Full list of similarities and differences is rather long. Let me summarize it.

Xvc has different commands; xvc file track is used instead of dvc add. Xvc doesn’t add files (like .dvc files) to your repository and keeps all metadata tracking under the .xvc directory. Checkout method is per-file, not configured globally, so you can keep track of your data directory with symlinks and your model directory as copies. Xvc uses BLAKE3 as the default hashing algorithm, and you can configure this to be BLAKE2, SHA-2, or SHA-3.

Pipelines are not defined using YAML files. You can write a shell script with xvc pipeline step ... or use Python xvc.pipeline().step().dependency(step_name="preprocess", param="hyperparams.yaml::batch_size") to define pipelines first. Then you can use xvc pipeline export and xvc pipeline import to modify the pipeline in YAML, JSON, or TOML.

There are more dependency options; e.g., a pipeline step may depend on a text file partially, by regex or by the line options. There is a generic dependency option; the output of a shell command can be used as a dependency to a step.

Remote storage options are rather limited for Xvc; local, ssh+rsync, and S3-compatible storages are supported for now. There is also a generic storage option where you can define upload and download commands for the tool you’re using, e.g., rclone or s5cmd, and Xvc can use it. I’ll add Azure and rclone as natively supported storage options eventually, but I don’t like the idea of keeping credentials, so there won’t be any OAuth-required storage options, e.g., Google Drive. (You’ll be able to use these through rclone, though.) All Xvc storages use environment variables for authentication.

Xvc doesn’t have experiment tracking yet. You can use --from-ref and --to-branch options to store artifacts from the pipeline to different branches. I’ll add features to run pipelines and commands quickly and compare these eventually (I need one too), but it may take some time.

Xvc doesn’t track anything about the user. It shouldn’t make any network connections if you don’t specifically ask it to do so. I’m planning to add binaries that only do file operations or pipeline operations, so if someone doesn’t need pipeline features, they will simply use xvc-file.

I saw an ad on Stack Overflow Jobs those days about employment by contributing to open source projects. I applied to that. A few weeks later, the CTO of [iterative.ai] got in touch and I started working on DVC documentation. Initially on a per-hour basis, and after May 2021, as a full-time employee.

Initially, I liked the tool we were building very much. The team was awesome. (Still, they are.) It was one of the best periods of my life, especially in my turbulent still-ongoing-divorce-period pressures. I know I will always miss them.

My job was learning DVC, documenting it, and helping newcomers grasp it easily. It was a fun job. Until then, I didn’t see myself as a technical writer. English is not my native tongue and I never have lived in an English-speaking country. Nevertheless, I think I wasn’t too bad at it.

When I was first learning the tool, I began to use it everywhere. I was an avid user of Git Annex once. DVC looked better. I don’t remember why I lost interest in Git Annex after many years, but it was probably related to symbolic links not working on Windows (or on Termux). DVC had multiple ways of connecting the cache and the files in the workspace, including hardlinks and copy, so it was a breath of fresh air for me.

I began to use it for my large collections. Keeping track of my binary files in Git was something I always desired. Git is the least sucking version control system among the ones I used previously (SVN, hg, darcs…) and I’d rather keep using it everywhere rather than learning new tools for binary files.

After some time I began to use the tool for my personal file collections. I noticed its performance became a burden. I was tracking maybe a few gigabytes of files with it and basic file operations became slower as I added more. I noticed I was becoming distracted after I wrote a dvc command. It took some time to confess that the tool I liked once and was earning my salary with was not a tool that I liked to use.

I don’t know what real professionals would do at this point. I never had a good LinkedIn profile. When I met a similar problem with the example repository that’s supposed to contain 70,000 small files, I brought the issue forward. I wrote a shell script that was basically doing the same thing as dvc add and it worked much faster than the actual command. The shell script was naïve and I thought DVC must have at least that level of speed. It didn’t. Simply calling md5sum on files and copying them to appropriate location in .dvc/cache was way faster. How could this be?

I had cursory observations on the codebase. I know some decisions (like a large central class that connects everything, separate .dvc files for each tracked file) that may lead to degradation. Although I don’t see it as the problem, Python was also not helpful. These are rough observations.

It was September 2021. I was also teaching myself Rust. I wrote an email to the CTO and CEO of the company to request a sabbatical to work on DVC. My plan was to rewrite certain portions (or commands) in Rust and wrap them with PyO3. It could fail. So to have skin in the game, I said I’ll work for free during this time and if I fail to make DVC faster for some reason, I’ll return to my writing position.

They didn’t accept. I didn’t try to persuade them. The decision was rational and although I’d say go ahead and see what happens if I were in their shoes just to make my employee happy, they aren’t crazy-managers as I once was. Probably there are many factors that I’m not aware of. I returned to my post and continued to write documentation for another 9 months. In the meantime I studied Rust and thought about how I could architect a similar tool. Where does DVC go wrong?

In April 2022, I informed the CTO that I’d like to take a sabbatical for my book. I have a political-SF book and after the Ukrainian war started with a (albeit minor) probability of nuclear attack on the other shore of Black Sea, I thought it’s not a time to work on something I stopped liking. My performance in the last quarter was also not something I was proud of. I didn’t feel good.

When I retired to sabbatical in July though, while writing the book, I thought writing the software that I wanted to see was also something on my mind before nuclear war. I had notes about the architecture I was planning. I wanted to see if I could apply an Entity-Component System to this basic problem, without any Object-Oriented conceptions. I believe it looks cool. I’m still simplifying and testing the idea, and it looks better to my mind than mixing data and functions for no reason.

After I made the repository public, I resigned from Iterative.

In a sense, Xvc owes its existence to DVC, and the name is a tribute to this. I hope they squash their bugs, and improve their user experience, and be a long-term player in the crowded market they are in. I don’t intend to be a “competitor”, because I prefer being developer/architect rather than a “technical steward to VC money”, and the license of Xvc is GPL-3 to signal this.

The Xvc command line interface, however, is as different as it can be from DVC. The command names are different; DVC has commands similar to Git (push, fetch, pull, commit), while Xvc tries to be different from Git to reduce the user’s mental load. For example, as a writer, I noticed that “Git remotes” and “DVC remotes” was confusing, so I called them “Xvc storages”. DVC calls the units of a pipeline stages; the same concept is called steps in Xvc, because stage in Git is something completely different.

Internally, the architecture is also very different. Xvc uses serialization (with serde) instead of YAML. It can export/import pipelines from YAML (or JSON), but YAML is not as central as in DVC. (I believe YAML is overused in our industry, and it’s an employment guarantee for another generation of developers but there is better work than keeping up a half-baked configuration format.) Xvc doesn’t keep its artifacts in the user’s workspace (except .xvcignore files). They are all stored in the .xvc/ directory. The DVC way of doing things makes merging .dvc files easier. To overcome the problems caused by merging large metadata files, Xvc keeps track of events and replays them to get the final state of the repository. All metadata storage and retrieval operations revolve around the XvcStore<T> struct in Xvc. Typically, if the user runs an xvc command, only the updated store events (added files, changed pipelines, etc.) are stored. There are optimizations in this front, but I profile first and optimize later.

Algorithms for data digests are configurable; by default Xvc uses Blake3, but it is configurable to use SHA2-256, SHA3-256, or Blake2s. It can be modified to use any 256-bit digest quickly. There are some features that are not found in DVC, and more will come. So, although I’m solving a similar problem, Xvc is not “DVC rewritten in Rust,” it’s a different tool completely.

Currently, it doesn’t have as much eye candy as DVC. In time, I plan to add Python, Julia, and R APIs, notebook integration, experiment tracking (without relying on Git internals), data labeling and filtering, and other MLOps features. I’m building with a goal to make these features available without making the rest of the software slower.

I’ve found the tool I was looking for to track my kids’ photos and Ottoman OCR datasets in a Git repository. I’m tracking more than 1TB of files in a single repository with Xvc and adding another 10TB looks feasible now.

🔺 So, who is the typical user of Xvc? Who is the probable user?

🔹 I’m thinking of a user with large amounts of data who will be training machine learning models. They need to version this data.

🔺 What do you mean by data in this context?

🔹 The data in this context consists of files. Image files, audio files, text files—a whole bunch of files. Lots and lots of files. They may be organized into directories; they may be reserved for training, verification, and testing; and they may be stored in multiple locations.

🔺 So, not database files? Are you saying we can’t version the data in databases?

🔹 If the database you’re thinking of is something like SQLite, then it is possible to version it. It’s just another file. You can track the file with Xvc and move back and forth between versions to go back in time. But I doubt that’s the best way.

🔺 Why?

🔹 A database is an alternative to the file system for storing data. You can version the data by adding a timestamp or a version description and track individual records. In this case, you can go back and forth between versions by selecting a subset of the records. To my mind, it doesn’t make sense to replace the database file itself just to track the data version.

🔺 For tracking, yes, there may be other ways to version the data. Databases themselves often provide this functionality. Except for SQLite, replacing database files doesn’t work seamlessly. Databases maintain extra indices and auxiliary files for performance, and simply replacing the database files would likely cause them to fail. However, these databases can be sources of data in data pipelines. When you have, for example, a database table as a source, you should be able to define it as a dependency in the pipelines.

🔹 That sounds worthwhile, right.

🔺 So, the user could define a database table as a dependency. Suppose you have a table that contains a lot of data, and you want to update your ML model whenever this data changes. This must be a very common use case.

🔹 It seems so, yes. It’s more difficult than file system dependencies, though. At the very least, it must be customized per database engine. We need different connections for different databases.

🔺 This is something that can be abstracted. All databases have tables and queries. We can have a database connection layer to retrieve a table, check whether it has changed, and invalidate it as a dependency if it has. The user is responsible for getting the actual data and doing whatever they need to do with it anyway.

🔹 It looks like a nice model. We can also adapt it to SQL queries. When queries are run, they produce a result, and if this result has changed, we can consider the dependency invalidated. It would be useful to run a simple query to check whether a complex join has changed and update the models accordingly.

🔺 What would be the performance implications? A query would need to be run in each execution of the pipeline, I think.

🔹 Yep. If we define a database table dependency as xvc pipeline step dependency --database-table, then it should connect to the database during each xvc pipeline run and check whether the table has changed. A table change can be detected by selecting all records or via database internals. It’s not that big a deal if the table or query doesn’t produce too many results.

🔺 Maybe it’s possible to generalize this behavior to the command line as well. Xvc could run a command to check whether something has changed—like ls -R my-data—and if the result differs from the previous run, it can trigger a command. This could be the generic way to handle pipeline steps.

🔹 That sounds like a great idea. We can talk about this tomorrow.