Full list of similarities and differences is rather long. Let me summarize it.

Xvc has different commands; xvc file track is used instead of dvc add. Xvc doesn’t add files (like .dvc files) to your repository and keeps all metadata tracking under the .xvc directory. Checkout method is per-file, not configured globally, so you can keep track of your data directory with symlinks and your model directory as copies. Xvc uses BLAKE3 as the default hashing algorithm, and you can configure this to be BLAKE2, SHA-2, or SHA-3.

Pipelines are not defined using YAML files. You can write a shell script with xvc pipeline step ... or use Python xvc.pipeline().step().dependency(step_name="preprocess", param="hyperparams.yaml::batch_size") to define pipelines first. Then you can use xvc pipeline export and xvc pipeline import to modify the pipeline in YAML, JSON, or TOML.

There are more dependency options; e.g., a pipeline step may depend on a text file partially, by regex or by the line options. There is a generic dependency option; the output of a shell command can be used as a dependency to a step.

Remote storage options are rather limited for Xvc; local, ssh+rsync, and S3-compatible storages are supported for now. There is also a generic storage option where you can define upload and download commands for the tool you’re using, e.g., rclone or s5cmd, and Xvc can use it. I’ll add Azure and rclone as natively supported storage options eventually, but I don’t like the idea of keeping credentials, so there won’t be any OAuth-required storage options, e.g., Google Drive. (You’ll be able to use these through rclone, though.) All Xvc storages use environment variables for authentication.

Xvc doesn’t have experiment tracking yet. You can use --from-ref and --to-branch options to store artifacts from the pipeline to different branches. I’ll add features to run pipelines and commands quickly and compare these eventually (I need one too), but it may take some time.

Xvc doesn’t track anything about the user. It shouldn’t make any network connections if you don’t specifically ask it to do so. I’m planning to add binaries that only do file operations or pipeline operations, so if someone doesn’t need pipeline features, they will simply use xvc-file.

I saw an ad on Stack Overflow Jobs those days about employment by contributing to open source projects. I applied to that. A few weeks later, the CTO of [iterative.ai] got in touch and I started working on DVC documentation. Initially on a per-hour basis, and after May 2021, as a full-time employee.

Initially, I liked the tool we were building very much. The team was awesome. (Still, they are.) It was one of the best periods of my life, especially in my turbulent still-ongoing-divorce-period pressures. I know I will always miss them.

My job was learning DVC, documenting it, and helping newcomers grasp it easily. It was a fun job. Until then, I didn’t see myself as a technical writer. English is not my native tongue and I never have lived in an English-speaking country. Nevertheless, I think I wasn’t too bad at it.

When I was first learning the tool, I began to use it everywhere. I was an avid user of Git Annex once. DVC looked better. I don’t remember why I lost interest in Git Annex after many years, but it was probably related to symbolic links not working on Windows (or on Termux). DVC had multiple ways of connecting the cache and the files in the workspace, including hardlinks and copy, so it was a breath of fresh air for me.

I began to use it for my large collections. Keeping track of my binary files in Git was something I always desired. Git is the least sucking version control system among the ones I used previously (SVN, hg, darcs…) and I’d rather keep using it everywhere rather than learning new tools for binary files.

After some time I began to use the tool for my personal file collections. I noticed its performance became a burden. I was tracking maybe a few gigabytes of files with it and basic file operations became slower as I added more. I noticed I was becoming distracted after I wrote a dvc command. It took some time to confess that the tool I liked once and was earning my salary with was not a tool that I liked to use.

I don’t know what real professionals would do at this point. I never had a good LinkedIn profile. When I met a similar problem with the example repository that’s supposed to contain 70,000 small files, I brought the issue forward. I wrote a shell script that was basically doing the same thing as dvc add and it worked much faster than the actual command. The shell script was naïve and I thought DVC must have at least that level of speed. It didn’t. Simply calling md5sum on files and copying them to appropriate location in .dvc/cache was way faster. How could this be?

I had cursory observations on the codebase. I know some decisions (like a large central class that connects everything, separate .dvc files for each tracked file) that may lead to degradation. Although I don’t see it as the problem, Python was also not helpful. These are rough observations.

It was September 2021. I was also teaching myself Rust. I wrote an email to the CTO and CEO of the company to request a sabbatical to work on DVC. My plan was to rewrite certain portions (or commands) in Rust and wrap them with PyO3. It could fail. So to have skin in the game, I said I’ll work for free during this time and if I fail to make DVC faster for some reason, I’ll return to my writing position.

They didn’t accept. I didn’t try to persuade them. The decision was rational and although I’d say go ahead and see what happens if I were in their shoes just to make my employee happy, they aren’t crazy-managers as I once was. Probably there are many factors that I’m not aware of. I returned to my post and continued to write documentation for another 9 months. In the meantime I studied Rust and thought about how I could architect a similar tool. Where does DVC go wrong?

In April 2022, I informed the CTO that I’d like to take a sabbatical for my book. I have a political-SF book and after the Ukrainian war started with a (albeit minor) probability of nuclear attack on the other shore of Black Sea, I thought it’s not a time to work on something I stopped liking. My performance in the last quarter was also not something I was proud of. I didn’t feel good.

When I retired to sabbatical in July though, while writing the book, I thought writing the software that I wanted to see was also something on my mind before nuclear war. I had notes about the architecture I was planning. I wanted to see if I could apply an Entity-Component System to this basic problem, without any Object-Oriented conceptions. I believe it looks cool. I’m still simplifying and testing the idea, and it looks better to my mind than mixing data and functions for no reason.

After I made the repository public, I resigned from Iterative.

In a sense, Xvc owes its existence to DVC, and the name is a tribute to this. I hope they squash their bugs, and improve their user experience, and be a long-term player in the crowded market they are in. I don’t intend to be a “competitor”, because I prefer being developer/architect rather than a “technical steward to VC money”, and the license of Xvc is GPL-3 to signal this.

The Xvc command line interface, however, is as different as it can be from DVC. The command names are different; DVC has commands similar to Git (push, fetch, pull, commit), while Xvc tries to be different from Git to reduce the user’s mental load. For example, as a writer, I noticed that “Git remotes” and “DVC remotes” was confusing, so I called them “Xvc storages”. DVC calls the units of a pipeline stages; the same concept is called steps in Xvc, because stage in Git is something completely different.

Internally, the architecture is also very different. Xvc uses serialization (with serde) instead of YAML. It can export/import pipelines from YAML (or JSON), but YAML is not as central as in DVC. (I believe YAML is overused in our industry, and it’s an employment guarantee for another generation of developers but there is better work than keeping up a half-baked configuration format.) Xvc doesn’t keep its artifacts in the user’s workspace (except .xvcignore files). They are all stored in the .xvc/ directory. The DVC way of doing things makes merging .dvc files easier. To overcome the problems caused by merging large metadata files, Xvc keeps track of events and replays them to get the final state of the repository. All metadata storage and retrieval operations revolve around the XvcStore<T> struct in Xvc. Typically, if the user runs an xvc command, only the updated store events (added files, changed pipelines, etc.) are stored. There are optimizations in this front, but I profile first and optimize later.

Algorithms for data digests are configurable; by default Xvc uses Blake3, but it is configurable to use SHA2-256, SHA3-256, or Blake2s. It can be modified to use any 256-bit digest quickly. There are some features that are not found in DVC, and more will come. So, although I’m solving a similar problem, Xvc is not “DVC rewritten in Rust,” it’s a different tool completely.

Currently, it doesn’t have as much eye candy as DVC. In time, I plan to add Python, Julia, and R APIs, notebook integration, experiment tracking (without relying on Git internals), data labeling and filtering, and other MLOps features. I’m building with a goal to make these features available without making the rest of the software slower.

I’ve found the tool I was looking for to track my kids’ photos and Ottoman OCR datasets in a Git repository. I’m tracking more than 1TB of files in a single repository with Xvc and adding another 10TB looks feasible now.

Ah yeah, when you work for free, you can do so at any time you want, perhaps. You also don’t have team members, and that means when you sit in front of this, you can move it.

Umm, right. Let’s take a look at the outstanding PRs.

You have a documentation PR#93. You also have work that you’ve begun to integrate Git into. I think it’s better to focus on the latter today.

Right. Let’s think about the relationship between Git and Xvc. I believe we should identify a general relation to avoid ending up in a mess like DVC and Git.

Why do you think the DVC and Git relationship is a mess?

They don’t automate common Git operations like a commit after dvc add. There is only auto-stage, and that’s turned off by default. This makes it seem that DVC wants to intervene as little as possible with the user’s Git workflow. That’s understandable. I support this. But on the other hand, they use the .git/ directory itself to store and manage experiments in a custom way that creates custom stash objects for experiments. This is against the principle of minimum intervention.

So this makes it a mess?

The mess, in my opinion, is caused by the second factor. If DVC doesn’t perform any Git operations, that’s alright. It was intended to be VCS-agnostic. Then experiments came and used Git internals in a way that no other similar tool uses.

Git-LFS and Git-Annex seem to use some non-standard mechanisms as well.

Ok. Not no other tool uses, but in a way that no other tool has used.

You know, GitHub PRs are also stored in a similar way. They also use non-standard machinery.

Yeah, but these tools are all Git-specific tools. They accept the dominion of Git, and don’t try to bring any VCS-agnosticism.

And Xvc tries to have this agnosticism?

I believe the initial design of DVC, which aims to be VCS-agnostic or being able to run without a VCS, is valuable. I like the idea behind Git, but the interface and implementation show that it’s a gradual development. There is no library behind it.

Libgit?

Libgit2 is something different. Although it’s said to have some common code, it doesn’t support all features. Git is command-line software with a mix of scripts and compiled executables, and not all code seems to be written in a way that could be used by external tools.

Hmm. The comment you added to the issue says git stash push --staged is not available in libgit2. Can’t you mimic it like DVC does for branches?

I don’t want to depend on Git at that level.

So, you’ll be using the CLI and shell for Git?

Yes, I believe, at the moment, before any performance tests, that this doesn’t matter much. Running Git commands once in a while using the shell shouldn’t make much difference in overall performance.

Then you’ll use it like a command-line tool, like the user?

Yes, and I’ll make it run outside of the usual threads. All Git will be like a sandwich, wrapping around Xvc operations. If there are --git-ref instructions in an xvc command, it will be run before Xvc performs the command, and if there are any changes in Xvc metafiles, they will be committed to the current branch.

Like

graph LR

co["git checkout"] --> xvc
xvc --> cm["git commit"]

The first could be a branch as well. So we have:

graph LR

br["git branch"] --> xvc
co["git checkout"] --> xvc
xvc --> cm["git commit"]

Looks sensible. How will you reflect these in the command line?

With something like xvc --git-checkout my-branch file list

Hmm, and for a branch?

I think instead of different options for branch, checkout or tag, we can have a git-ref option that marks the option as a git reference. It will be checked out, or created as a branch from the current one if it doesn’t exist.

I think creating a branch is not a good idea. It should be explicit. You can just send the --git-ref value to git checkout and perform the Xvc operation. If the user wants to create a branch, I think they can do it themselves.

What about storing the results in a branch? After adding a bunch of files, they may want to store them in another branch, maybe?

That’s sensible. We can have another option, like --to-branch in certain operations.

Or in the xvc command as a general option. In that case, we can change the option names to --from-ref and --to-branch. It will be like:

graph LR

fr["git checkout $(--from-ref)"] --> xvc
xvc --> tb["git checkout --branch $(--to-branch)"]
tb --> co["git add .xvc && git commit -m 'xvc cmd'"]

If no such options are given, xvc will run without branching, right?

Yep. --from-ref and --to-branch options are just shortcuts for user behavior. Any other VCS tool could be used this way. We don’t need to integrate Git at the library level.

This brings up the question of portability, though. When you aim for the software to be portable, you can’t rely on the existence of Git on the host, right?

I think a git.command option in the configuration is a good idea. Xvc will issue a warning if it can’t run the commands.

Will you use the shell to run this command? Otherwise no $PATH configuration is possible, you know.

I believe that could be another option: git.use_shell. If git.command is set to an absolute path, Xvc may use it without the shell. Otherwise, it can use the shell. Running the process directly will make it faster and more secure.

There is also this option to run Xvc in another process. Because we may access Git in the shell that runs Xvc, and if we can access it, maybe we don’t need shell execution in the process.

That’s a cool idea. But I wouldn’t add that extra complexity. Instead, we can try to find the git executable if git.command is not an absolute path. If git.command = /usr/bin/git in the configuration, we use it as is. Otherwise, we can get $PATH or %PATH% from the environment and search for git.command in that to find the exact executable.

It looks like there is a crate called which that does exactly what we are looking for.

Ah, cool. Then we can just use that to find the executable and run it. We don’t need to drop to a shell.

Yep. Let’s go back to implementation now.