🐇 I think we can start by improving xvc.py. I mean, releasing. Yesterday we finished our work with the Rust library.

🐢 Then we can start to look at the GUI, I believe. We can replace that 3-column view with a table and preview. It will be much easier that way.

🐇 Yep. Let’s finish and release the Python version first. Then we’ll go on to the GUI.

🐢 Let’s take a look at the PRs first.

ghpl 
36	Bump pyo3 from 0.22.2 to 0.23.3	dependabot/cargo/pyo3-0.23.3	OPEN	2024-12-04T03:52:03Z

gh pr close 36
✓ Closed pull request iesahin/xvc.py#36 (Bump pyo3 from 0.22.2 to 0.23.3)

🐢 We have already upgraded to pyo3 0.23. No need for this. Let’s create a PR for the current branch.

git push --set-upstream origin v0.6.13

ghpC --fill
https://github.com/iesahin/xvc.py/pull/37

🐇 We can also tag and push the tags.

🐢 I’d like to have some more coverage for certain parts of the code. Let’s add some tests.

🐇 It looks like we mainly lack the storage tests. They need configuration to add keys to GitHub, and we already skip some of these even in the Rust code.

🐢 Umm, I see. We also need a way to measure coverage. Could we do this with Codecov, I wonder?

🐇 I found an example here: https://github.com/codecov/example-python/blob/main/.github/workflows/ci.yml. It needs coverage and pytest-cov in the requirements.

🐢 Let’s try this then.

🐇 Added coverage.yml file. It’s simpler than the other GitHub action. Need to update the token now.

🐢 There is an issue installing the requirements.

🐇 I forgot to add sudo to apt-get. Will take care of it now.

🐢 Let’s check the run.

ghrl
in_progress		Release v0.6.13	coverage	v0.6.13	pull_request	12556590550	2m31s	2024-12-31T06:59:58Z
in_progress		Release v0.6.13	publish-to-pypi	v0.6.13	pull_request	12556590548	2m31s	2024-12-31T06:59:58Z
...

🐇 It takes a while and we still didn’t add the CODECOV_TOKEN.

🐢 Let’s add it, and after that, we need to take a look at this blink configuration. It adds letters after the selection.

🐇 Added the secret and configured xvc.py for coverage. There is an error with the build, though. Maybe the command we should be using is maturin develop instead of build to make Xvc available for the environment.

🐢 Let’s update and try it then.

🐇 maturin develop requires a virtual environment.

🐢 I checked the options for build, and I think there is an option, but let’s search first.

🐇 I searched, but it looks like we can just pass an --out directory or install xvc from the target/wheels/ directory. The second option requires less maintenance.

🐢 Okay. Let’s add a step to the action then.

🐇 Now, let’s wait for the run to finish with gh run watch.

🐢 It failed. Let’s take a look at the logs:

ghrl
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12556939836	3m26s	2024-12-31T07:38:54Z

gh run view 12556939836 --log-failed

🐇 It looks like some of the tests are failing. Let’s run the tests locally.

🐢 We should run with the --forked option, and it looks like we have a test to update with the xvc file list.

🐇 Updated the test, and I noticed we forgot to supply the new --show-directories option in the Python interface.

🐢 Yep. Passing these options as command-line options in strings is not robust. It’s very easy to forget things. I think we should start using CLI structs directly, but it’s not time yet.

🐇 I agree. It’s one of the goals for building a GUI, actually.

🐢 The tests failed again.

ghrl | rg failure
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12557031091	3m49s	2024-12-31T07:51:19Z

gh run view 12557031091 --log-failed

🐇 There is a Git error now. We need to add a Git user and email to the action.

🐢 Added those and watching the results again now.

🐇 Why do you think the publish action always works? It can only run with the main branch, I think. No need to run it with other pushes.

🐢 Yes, let’s configure it now.

ghrl | rg failure
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12557083393	3m21s	2024-12-31T07:59:18Z

gh run view 12557083393 --log-failed

🐇 It looks like we also need xvc-test-helper in the path. Let’s cargo install it and add it to the path.

🐢 Added .cargo/bin to the path like:

- name: Add cargo bin to PATH
  run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH

and installed the helper with cargo install xvc-test-helper.

🐇 By the way, GitHub Copilot is hallucinating about a method to update the path.

🐢 I searched and it may not be hallucinating. We can use the ::add-path:: command with echo, it looks like. This is new to me.

🐇 The tests failed again.

ghrl | rg failure
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12557208723	3m34s	2024-12-31T08:12:09Z

gh run view 12557208723 --log-failed

🐢 file().list() had a mistake, and we need to install rg for the tests.

🐇 Watching the test run. In the meantime, maybe we can review…

🐢 Failed again.

ghrl | rg failure
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12557294105	3m46s	2024-12-31T08:21:17Z

gh run view 12557294105 --log-failed

🐇 Added db.commit() to two places in the code. This should pass now.

🐢 Yeah!

ghrl | head -n 1
completed	success	Release v0.6.13	coverage	v0.6.13	pull_request	12557509836	3m46s	2024-12-31T08:44:33Z

gh run view 12557509836

✓ v0.6.13 coverage iesahin/xvc.py#37 · 12557509836
Triggered via pull_request about 4 minutes ago

JOBS
✓ linux in 3m36s (ID 35010253915)

ANNOTATIONS
! ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
linux: .github#1


For more information about the job, try: gh run view --job=35010253915
View this run on GitHub: https://github.com/iesahin/xvc.py/actions/runs/12557509836

🐇 I can’t see a coverage report on codecov.io, though.

🐢 It says no coverage report is generated. Let’s test to generate XML files locally.

🐇 The option in the docs seems incorrect. --cov-branch doesn’t produce anything.

🐢 Let’s wait for the run again.

🐇 Should we add a badge to the README?

🐢 It won’t show much, but yeah, let’s make it.

🐇 The results are in and it shows 100% coverage. This means it doesn’t actually test anything.

🐢 We need Rust coverage for this. Let’s search for it.

🐇 I found this: https://github.com/cjermain/rust-python-coverage. It runs cargo llvm-cov with the project and measures test coverage. But we don’t have any Rust tests.

🐢 It looks like we don’t need Rust tests. cargo llvm-cov can check coverage with the Python as well.

$ cargo llvm-cov show-env --export-prefix
export RUSTFLAGS=" -C instrument-coverage --cfg coverage --cfg trybuild_no_target"
export LLVM_PROFILE_FILE="/home/.../rust-python-coverage/target/rust-python-coverage-%m.profraw"
export CARGO_INCREMENTAL="0"
export CARGO_LLVM_COV_TARGET_DIR="/home/.../rust-python-coverage/target"

🐢 Let’s take a look at what remained for 0.6.13.

🐇 I think there is nothing left. Python must be published when we merged the PR.

🐢 Let’s take a look by searching xvc python.

🐇 The PyPI page is https://pypi.org/project/xvc/ and it still reports the version as 0.6.11. There must be something.

🐢 Now let’s take a look at

https://github.com/iesahin/xvc.py

🐇 The run seems to be OK though.

https://github.com/iesahin/xvc.py/actions/runs/12557780141/job/35010938462

🐢 Maybe the version in pyproject.toml is still 0.6.11 and we forgot to update it?

tmux new-window -c $HOME/github.com/iesahin/xvc.py/ nvim

🐇 There is no version string in ~/github.com/iesahin/xvc.py/pyproject.toml. It’s dynamic.

🐢 Then, let’s try to publish from local now.

🐇 The iex username requires email verification.

🐢 It looks like I publish xvc through the iesahin account, not iex. Maybe I can add both accounts to the project.

🐇 There seem to be no errors on the PyPI site.

🐢 Updating the token. Let’s add the token to pass to run maturin publish.

🐇 We’re receiving invalid or non-existent authentication information. Upgraded maturin to see if it fixes the issue.

🐢 We can also double-check the key.

🐇 Uploaded successfully from local. Let’s check the job again.

🐢 The release job was skipped because we didn’t tag after the merge. https://github.com/iesahin/xvc.py/actions/runs/12557780141/job/35011303045 That looks like the reason.

🐇 I should be more careful which is run and which is skipped.

🐢 Maybe we can relax the condition. We do this rarely. Maybe republishing is alright?

🐇 Yep, removed that. The jobs are running now. Let’s watch them to see what happens when we publish some of the packages.

🐢 In the meantime, let’s experiment with searching commands file with fzf-lua.

🐇 It requires more experimentation, but we can start from https://github.com/ibhagwan/fzf-lua/wiki/Advanced#interactive-shell-command.

🐢 The xvc publish jobs failed, btw.

ghrl
completed	failure	Remove if condition from release	publish-to-pypi	v0.6.13	push	12568863716	15m26s	2025-01-01T08:30:03Z

gh run view 12568863716 --log-failed
...
Release	Run actions/download-artifact@v4.1.7	        Please ensure that your artifact is not expired and the artifact was uploaded using a compatible version of toolkit/upload-artifact.
...

🐇 It was using an older version of upload-artifact.

🐢 Rerunning the job and it looks like it runs for both push to main and tag with v0.6.13. We can turn off push to main, I believe.

🐇 There was a missing upload-artifact again. Fixed and updated the tags.

ghrl completed failure update upload-artifacts publish-to-pypi main push 12569060915 14m39s 2025-01-01T08:57:22Z

ghrf 12569060915

🐢 There are conflicts with uploaded artifacts now. We may need to clean up the artifacts manually.

🐇 The conflicts were not about inter-workflow names. The names were conflicting because all files were named wheel. I added the platform and target to the names to avoid conflicts.

🐢 Let’s wait then. Maybe it will work this time. Could you search for a Lua console for Neovim?

🐇 Let’s try this one: return { “yarospace/lua-console.nvim”, lazy = true, keys = “`”, opts = {}, }

🐢 I couldn’t make it run but won’t spend much time ATM. How about the jobs?

ghrl
completed	failure	update upload artifact names	publish-to-pypi	v0.6.13	push	12569298414	13m57s	2025-01-01T09:27:00Z

ghrf 12569298414
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9180510Z ##[group]Run actions/download-artifact@v4
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9182098Z with:
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9182859Z   name: wheels
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9183843Z   merge-multiple: false
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9184817Z   repository: iesahin/xvc.py
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9185820Z   run-id: 12569298414
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9186903Z ##[endgroup]
Release	Run actions/download-artifact@v4	2025-01-01T09:40:53.1937617Z Downloading single artifact
Release	Run actions/download-artifact@v4	2025-01-01T09:40:53.4694769Z ##[error]Unable to download artifact(s): Artifact not found for name: wheels
Release	Run actions/download-artifact@v4	        Please ensure that your artifact is not expired and the artifact was uploaded using a compatible version of toolkit/upload-artifact.
Release	Run actions/download-artifact@v4	        For more information, visit the GitHub Artifacts FAQ: https://github.com/actions/toolkit/blob/main/packages/artifact/docs/faq.md

🐇 Now the download doesn’t work.

🐢 Update it to download using patterns.

🐇 Did so and let’s take a look at the jobs again.

ghrl
completed	failure	added pattern to download	.github/workflows/publish.yml	main	push	12569456859	0s	2025-01-01T09:49:19Z

ghrv 12569456859

X main .github/workflows/publish.yml · 12569456859
Triggered via push about 3 minutes ago

X This run likely failed because of a workflow file issue.

For more information, see: https://github.com/iesahin/xvc.py/actions/runs/12569456859

🐢 The line with the pattern was reported as broken.

🐇 Removed it and recommitted.

ghrl
✅ completed	success	remove pattern to download all	publish-to-pypi	v0.6.13	push	12569509435	14m41s	2025-01-01T09:58:49Z

🐢 And now this completes the v0.6.13 release.

🐇 We’ll see how it will work next time.

🐢 Yep. Let’s move on to the GUI for now. Is that okay with you?

I saw an ad on Stack Overflow Jobs those days about employment by contributing to open source projects. I applied to that. A few weeks later, the CTO of [iterative.ai] got in touch and I started working on DVC documentation. Initially on a per-hour basis, and after May 2021, as a full-time employee.

Initially, I liked the tool we were building very much. The team was awesome. (Still, they are.) It was one of the best periods of my life, especially in my turbulent still-ongoing-divorce-period pressures. I know I will always miss them.

My job was learning DVC, documenting it, and helping newcomers grasp it easily. It was a fun job. Until then, I didn’t see myself as a technical writer. English is not my native tongue and I never have lived in an English-speaking country. Nevertheless, I think I wasn’t too bad at it.

When I was first learning the tool, I began to use it everywhere. I was an avid user of Git Annex once. DVC looked better. I don’t remember why I lost interest in Git Annex after many years, but it was probably related to symbolic links not working on Windows (or on Termux). DVC had multiple ways of connecting the cache and the files in the workspace, including hardlinks and copy, so it was a breath of fresh air for me.

I began to use it for my large collections. Keeping track of my binary files in Git was something I always desired. Git is the least sucking version control system among the ones I used previously (SVN, hg, darcs…) and I’d rather keep using it everywhere rather than learning new tools for binary files.

After some time I began to use the tool for my personal file collections. I noticed its performance became a burden. I was tracking maybe a few gigabytes of files with it and basic file operations became slower as I added more. I noticed I was becoming distracted after I wrote a dvc command. It took some time to confess that the tool I liked once and was earning my salary with was not a tool that I liked to use.

I don’t know what real professionals would do at this point. I never had a good LinkedIn profile. When I met a similar problem with the example repository that’s supposed to contain 70,000 small files, I brought the issue forward. I wrote a shell script that was basically doing the same thing as dvc add and it worked much faster than the actual command. The shell script was naïve and I thought DVC must have at least that level of speed. It didn’t. Simply calling md5sum on files and copying them to appropriate location in .dvc/cache was way faster. How could this be?

I had cursory observations on the codebase. I know some decisions (like a large central class that connects everything, separate .dvc files for each tracked file) that may lead to degradation. Although I don’t see it as the problem, Python was also not helpful. These are rough observations.

It was September 2021. I was also teaching myself Rust. I wrote an email to the CTO and CEO of the company to request a sabbatical to work on DVC. My plan was to rewrite certain portions (or commands) in Rust and wrap them with PyO3. It could fail. So to have skin in the game, I said I’ll work for free during this time and if I fail to make DVC faster for some reason, I’ll return to my writing position.

They didn’t accept. I didn’t try to persuade them. The decision was rational and although I’d say go ahead and see what happens if I were in their shoes just to make my employee happy, they aren’t crazy-managers as I once was. Probably there are many factors that I’m not aware of. I returned to my post and continued to write documentation for another 9 months. In the meantime I studied Rust and thought about how I could architect a similar tool. Where does DVC go wrong?

In April 2022, I informed the CTO that I’d like to take a sabbatical for my book. I have a political-SF book and after the Ukrainian war started with a (albeit minor) probability of nuclear attack on the other shore of Black Sea, I thought it’s not a time to work on something I stopped liking. My performance in the last quarter was also not something I was proud of. I didn’t feel good.

When I retired to sabbatical in July though, while writing the book, I thought writing the software that I wanted to see was also something on my mind before nuclear war. I had notes about the architecture I was planning. I wanted to see if I could apply an Entity-Component System to this basic problem, without any Object-Oriented conceptions. I believe it looks cool. I’m still simplifying and testing the idea, and it looks better to my mind than mixing data and functions for no reason.

After I made the repository public, I resigned from Iterative.

In a sense, Xvc owes its existence to DVC, and the name is a tribute to this. I hope they squash their bugs, and improve their user experience, and be a long-term player in the crowded market they are in. I don’t intend to be a “competitor”, because I prefer being developer/architect rather than a “technical steward to VC money”, and the license of Xvc is GPL-3 to signal this.

The Xvc command line interface, however, is as different as it can be from DVC. The command names are different; DVC has commands similar to Git (push, fetch, pull, commit), while Xvc tries to be different from Git to reduce the user’s mental load. For example, as a writer, I noticed that “Git remotes” and “DVC remotes” was confusing, so I called them “Xvc storages”. DVC calls the units of a pipeline stages; the same concept is called steps in Xvc, because stage in Git is something completely different.

Internally, the architecture is also very different. Xvc uses serialization (with serde) instead of YAML. It can export/import pipelines from YAML (or JSON), but YAML is not as central as in DVC. (I believe YAML is overused in our industry, and it’s an employment guarantee for another generation of developers but there is better work than keeping up a half-baked configuration format.) Xvc doesn’t keep its artifacts in the user’s workspace (except .xvcignore files). They are all stored in the .xvc/ directory. The DVC way of doing things makes merging .dvc files easier. To overcome the problems caused by merging large metadata files, Xvc keeps track of events and replays them to get the final state of the repository. All metadata storage and retrieval operations revolve around the XvcStore<T> struct in Xvc. Typically, if the user runs an xvc command, only the updated store events (added files, changed pipelines, etc.) are stored. There are optimizations in this front, but I profile first and optimize later.

Algorithms for data digests are configurable; by default Xvc uses Blake3, but it is configurable to use SHA2-256, SHA3-256, or Blake2s. It can be modified to use any 256-bit digest quickly. There are some features that are not found in DVC, and more will come. So, although I’m solving a similar problem, Xvc is not “DVC rewritten in Rust,” it’s a different tool completely.

Currently, it doesn’t have as much eye candy as DVC. In time, I plan to add Python, Julia, and R APIs, notebook integration, experiment tracking (without relying on Git internals), data labeling and filtering, and other MLOps features. I’m building with a goal to make these features available without making the rest of the software slower.

I’ve found the tool I was looking for to track my kids’ photos and Ottoman OCR datasets in a Git repository. I’m tracking more than 1TB of files in a single repository with Xvc and adding another 10TB looks feasible now.