ghrl | get url | first
https://github.com/iesahin/xvc/actions/runs/13875177382

🐢 We forgot to update the doc tests. Let’s run them again to update storage remove and file untrack commands.

🐇 There are issues with elision.

ghpl

╭───┬──────────────────────┬────────────────────┬──────────────────────────────╮
│ # │     headRefName      │       title        │             url              │
├───┼──────────────────────┼────────────────────┼──────────────────────────────┤
│ 0 │ storage-remove-16674 │ xvc storage remove │ https://github.com/iesahin/x │
│   │                      │                    │ vc/pull/270                  │
╰───┴──────────────────────┴────────────────────┴──────────────────────────────╯

🐢 Tests are passing; we can merge the PR. But our commit hook to check the CHANGELOG doesn’t work. That’s weird; we don’t get any errors when the CHANGELOG is not in the push set.

tmux new-window -c ($env.HOME | path join github.com iesahin xvc.py)  nvim 

🐢 Now we can update the Python bindings as well.

🐇 We forgot to bump package versions. We need to create a checklist for releases.

• ✅ #🌻 CREATE a release checklist (2025-03-25 17:52)

🐢 Let’s check if the latest version is updated on PyPI.

pypi xvc

🐇 Yes, it is.

🐇 What are the options, though? xvc.py is just a wrapper around xvc, and that was the easiest way to get it working. We can list all options manually in the headers as documentation, but it will be harder to maintain.

🐢 Once we start doing it, we’ll find a good way to simplify and shorten it.

🐇 Let’s start to work on xvc file track, then.

🐢 Worked on it a bit, and I decided it’s not worth it at the moment. We need to supply default values for most of the options. Maintaining a separate list of default values may not be feasible; it will be error-prone in a different way.

🐇 It should only be needed to update the dependency versions in Cargo.toml, right?

🐢 Let’s start with that.

🐇 We have interface changes regarding aliases; let’s start using uv for building.

🦊 Added requirements to pyproject.toml by running:

uv add -r requirements.txt

🐢 There is a way if we start from the configuration, not just files. The default configuration is a TOML document. We can make it an XvcConfiguration struct and load and store it with confy.

🐇 We have a cascading set of configurations, but maybe we can start from the struct and serialize/deserialize it on demand.

🐲 I don’t think we need to update xvc-config at the moment. It’s working, and we don’t need to alter its inner workings in the near future.

🐢 I agree. We can use the config crate when we need to update and have enough time to work on this.

🐢 Yes, but what will T be? Although store structures have Serde implementations, they are not particularly useful for this.

🐇 We can have output types, named like XvcFileListOutput, that will be converted to strings with serialization.

🦊 We do something similar in xvc pipeline export and import commands. We use XvcPipelineSchema and XvcStepSchema just for the import and export commands. We’ll write similar structs for all JSON output and will use Serde to convert these to strings.

🐢 Unlike the import and export commands, we have optional fields in the output, though. I don’t want content digests to appear in JSON output if they are not required.

🐇 Let’s search for optional fields in Serde.

🦊 There is a crate for optional fields.

🐢 We don’t need another crate for this. Serde has the skip_serializing_if attribute for fields. We can add Option::is_none as a method to these to skip outputting None fields. All those fields, in this case, will be optional.

🐇 This is fine. We already use structs to format the xvc file list output. We can just use them to output JSON.

🐇 https://www.getzola.org/themes/pico/ is an option, but I don’t like its header.

🦊 Minimal Dark from the same author looks better: https://kuznetsov17.github.io/minimal-dark/notes/note1/

🐇 https://www.getzola.org/themes/no-style-please/ is also an option.

🦊 https://halve-z.netlify.app/posts/information/ looks interesting, but there is too much screen estate for the left bar.

🐢 Let’s start by running the site locally first.

🐇 There are errors in the configuration. That’s weird, but let’s fix these.

🐢 Fixed errors. These are probably related to a newer version. Can we take a look at netlify.toml to see if it downloads the same version?

🐇 There are breaking changes in Zola 0.19. Let’s update and push the Netlify config to see the results.

🐢 We need to add language support for Nushell to prevent warnings. Take a look at how to add a syntax file to Zola.

🐇 I added

extra_syntaxes_and_themes = ["syntaxes"]

to the [markdown] section and added a syntaxes/nushell.sublime-syntax file copied from https://github.com/kurokirasama/nushell_sublime_syntax. I’m getting:

Error: Reason: Error while compiling regex '\b(?x: 7z | ?
...
s-to-gdrive | usage | ver | verify | weather | wget-all | which-cd | wifi-info | wifi-pass | xls2csv | ydx | yt-api | ytcli | ytm | z | zi)\b'
Oniguruma error: target of repeat operator is not specified

🐢 The syntax highlighter may be a bit buggy. Let’s try to fix this if it’s a one-off.

🐇 Found the bug. There is a ? in the regex that causes it to fail. Now it compiles, and Nushell blocks are colored.

🐢 Cool. Let’s fix the other warnings now. shell and console are not recognized, it looks like.

🐇 There is only Bash listed in https://www.getzola.org/documentation/content/syntax-highlighting/.

🦁 What do you think about migrating to mdBook? We already maintain mdBook for XVC; what about just moving the site to mdBook?

🐢 I thought about this before, and the only downside is the lack of an RSS feed.

🦊 I found this: https://github.com/theowenyoung/mdbook-rss

🐢 Now, this changes everything. We can even move the nedriy.at site to mdBook in this case.

🐇 Then, let’s start working on this. The site will be a technical book site in this case.

🐢 Does mdBook support Nushell syntax?

🐇 Nushell is not in the listed languages. mdBook uses highlight.js, and in its listed languages, we don’t find Nu either.

🐢 We already added Nu support to Zola, and we can just change the theme. This is a blocker in my opinion.

🐇 I searched for Nushell highlight.js support, and nothing appears. I think we can just postpone until Nu has more support on this front.

🐢 Yes. Let’s first try this change in the non-technical blog, and we can come back to this issue. Now, we’ll update console and shell to bash, I think.

🐇 Replaced shell and console with bash.

🐢 There is a file for ggplot that has warnings from earlier incarnations. We also lack syntax highlighters for Vim and Tmux.

🐇 There is a sublime-syntax file for Tmux at https://raw.githubusercontent.com/gerardroche/sublime-tmux/refs/heads/master/Tmux.sublime-syntax, but do we need it for a single file?

🐢 Let’s set it to plain text.

🐇 Now we only have Mermaid warnings left.

🐢 There should be a diagram at https://emresahin.net/developing-a-gitignore-crate/, but it doesn’t show up. We need a shortcode to show these, like the YouTube shortcode. Now let’s get back to theme selection.

🐇 I tested Karzok, but it doesn’t have category and tags support.

🐢 And I tested https://github.com/micahkepe/radion, but the best so far is the apollo theme. I’m struggling to modify the index page, though. I forgot that I modified the theme’s index.html file. I have content in /content/_index.md and a modified index.html in /themes/anemone/templates/index.html to show the content, tags, categories, etc. It should be fixed now.

🐇 Ah, cool. Can we clean the recent duplicate pages now?

🐢 Yeah, let’s take a look.

🐇 What about adding at least command completions for Nushell?

🐢 Let’s read a bit about clap_complete_nushell.

🦊 There seems to be a nu-complete command. Let’s check its documentation.

🐇 Nothing was found, and Kagi doesn’t help much either.

🐢 There is a completions document for Nushell: https://www.nushell.sh/book/custom_completions.html

🐇 There is a tool called carapace to provide completions across shells.

🐢 Its documentation is thin, and I’m not sure if it supports dynamic completions out of the box. I believe instead of adding a carapace setup, we can just write a Nushell completion script that will use JSON output from the commands and add some (maybe hidden) utility commands to support it.

🐇 There are a set of example scripts in the Nushell repo: https://github.com/nushell/nu_scripts/tree/main/custom-completions

🐢 The reason I want to write custom completions for Nushell is that it will be an exercise for the scripting language. gh completions are not as scary as a Bash script.

🐇 git completions are a better example for XVC. They simply run git whenever necessary. We can start from a static completions command and update this with dynamic completions manually. It will teach a lot.

🐢 I forked the nu_scripts repo and will add XVC completions script there.

🐇 Then let’s begin by adding Nushell static completions. Shall we add a command for this?

🦊 Reviving the completion command we removed in 0.6.13?

🐢 We shouldn’t list it. We can make a _comp subcommand for the time being and generate and distribute completions in the repository. When clap_complete_nushell has the feature parity to provide dynamic completions, we can remove these commands.

🐇 What will we use this for other than generating completions?

🐢 Maybe dynamic completions can call this as well.

🐇 Added Nushell static completions to be output using xvc _comp generate-nushell. Let’s bump up the version to 0.6.15.

cargo set-version 0.6.15-alpha.1
   Upgrading xvc from 0.6.14 to 0.6.15-alpha.1
...

🐢 I noticed we forgot a line in the CLI command handler that asserts xvc_root_opt.is_some(), and this fails when we run xvc outside of repositories. We need to release this version quickly.

🐇 Oops, now, ok, let’s write a static Nushell generator and just release quickly.

🦊 Generating completions with

xvc comp generate-nushell

🐢 Completion command is run with comp instead of _comp. Should we rename it?

🐇 Renamed it to _comp. It’s not hidden, but at least we can be sure that it won’t be misunderstood as a common command.

🐢 Bumping the version again. Now let’s source the generated script and test it.

cargo set-version 0.6.15-alpha.2
   Upgrading xvc from 0.6.15-alpha.1 to 0.6.15-alpha.2
...

🦊 Yep, it works. We now have completions for Nushell.

🐢 Let’s update the completions documentation.

🐇 Done. Now, let’s take a look at CI and see what fails.

ghrl | first
╭──────────────┬─────────────────────────────────────────────────────────╮
│ conclusion   │ success                                                 │
│ displayTitle │ Add Nushell completions                                 │
│ headBranch   │ nushell-completions                                     │
│ url          │ https://github.com/iesahin/xvc/actions/runs/13070200765 │
╰──────────────┴─────────────────────────────────────────────────────────╯

🐢 It fails because of coverage, not the tests. Codecov says the new code isn’t tested.

🐇 The added xvc _comp command isn’t tested. We can add a test running those lines and testing if the command outputs a completion script.

🐢 We have a test for completions. We can add a test that runs the lines.

🐇 Added a test and bumping up the version.

cargo set-version 0.6.15-alpha.3
   Upgrading xvc from 0.6.15-alpha.2 to 0.6.15-alpha.3
...

🦊 We can add some more coverage while waiting for the tests.

🐇 XvcOutputLine implementation seems to have no tests. It’s weird because we use these everywhere.

🐢 I’m not sure we use this particular implementation; we just use XvcOutputLine::Info(s), not XvcOutputLine::info(s) anywhere. We can delete these methods actually.

🐇 We’ll add JSON output via this particular struct. Can we refactor these to use formatting for JSON, for example? Or use these to output JSON?

🦊 We can add a formatter to XvcOutputLine to output structures.

🐢 The enum is now defined as:

#[derive(Clone, Debug)]
pub enum XvcOutputLine {
    /// The output that we should be reporting to user
    Output(String),
    /// For informational messages
    Info(String),
    /// For debug output to show the internals of Xvc
    Debug(String),
    /// Warnings that are against some usual workflows
    Warn(String),
    /// Errors that interrupts a workflow but may be recoverable
    Error(String),
    /// Panics that interrupts the workflow and ends the program
    /// Note that this doesn't call panic! automatically
    Panic(String),
    /// Progress bar ticks.
    /// Self::Info is also used for Tick(1)
    Tick(usize),
}

Here, these fields can also have a formatter that will render the string in a particular format. For example, the output can be

XvcOutputLine::Output(XvcJsonFormatter, String)

🐇 I’m not sure this is a good idea. Output already specifies this string as output. We can have a wrapper instead, like,

struct XvcJsonOutput(Format<XvcStructuredOutput>, XvcStructuredOutput)

and we can use the supplied format to render XvcStructuredOutput to an output line with XvcOutputLine::Output. If we don’t provide output as structured, it will be too much error-prone work to convert the current outputs to structured.

🦊 The transition will also be gradual. We may not need structured output for most of the commands. We can start with xvc file list and convert others as we go.

🐢 This is sensible. By the way, coverage still didn’t increase. There may be something going on with Codecov or running the test.

ghrl | first
╭──────────────┬─────────────────────────────────────────────────────────╮
│ conclusion   │ success                                                 │
│ displayTitle │ Add Nushell completions                                 │
│ headBranch   │ nushell-completions                                     │
│ url          │ https://github.com/iesahin/xvc/actions/runs/13087431038 │
╰──────────────┴─────────────────────────────────────────────────────────╯

🐇 Let’s run the test:

cargo test -p xvc --test test_completions
...
test test_completions ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.80s

🐢 Can we make sure the output is a Nushell script and not an error message?

🐇 Let’s print it out.

🐢 It looks like when the COMPLETE environment variable is set, it never calls _comp subcommand and never calls those lines.

cargo set-version 0.6.15-alpha.4
   Upgrading xvc from 0.6.15-alpha.3 to 0.6.15-alpha.4
...

🐢 Let’s make a release for 0.6.15. Coverage is OK now.

cargo set-version 0.6.15
   Upgrading xvc from 0.6.15-alpha.4 to 0.6.15
...

gh pr merge --squash --body $"(open CHANGELOG.md | lines | skip 2 | take 5)" --subject "Add static nushell completions"

🐇 Merged the PR.

🐢 Releases should appear in a few minutes.

🐇 We need to tag the merge commit for this.

🐢 Oh, yep. AFAIK Lazygit doesn’t have something for git push --tags. Let’s push from the CLI.

git push --tags
You are on the main branch. Skipping CHANGELOG.md check.
To github.com:iesahin/xvc
 * [new tag]         v0.6.15 -> v0.6.15

🦊 These commands, especially tables, are not rendered correctly on the web. We need to change the theme, I think.

Currently, it only supports text. When I send a message, the bot adds it to a new file or appends it to a file modified within the last 30 minutes, which simplifies multi-paragraph inputs.

I need some more features to make it more useful.

I want it to save audio and image files to my inbox as well. I plan to run Xvc pipelines on these files to convert audio notes to text and make images searchable via OCR or tagging.

The first step to achieve this is to understand what kind of files Telegram can upload and download. I want to track these files with Xvc to avoid bloating the Git repository. I have a script that processes these files, but currently, it only commits changes to Git. I want to determine whether a file should be tracked by Xvc or Git.

I skimmed the Telegram API documentation and found that it supports a wide range of file types. This led me to change my approach: I will track only .txt and .md files with Git and use Xvc for everything else.

In the future, I plan to add a --binary-only option to xvc file track to track binary files only. This will help me to track only non-text files when using globs as targets. I might also extend this to a general predicate to decide which files to track based on their size, modification time, or other properties.

Rust has fearless concurrency for memory access, but for the file system, there seem to be no built-in locked access primitives.

I decided to write one. Parallel execution of file system operations is important for Xvc. The error messages are annoying; having identical files in a repository is common, and in those cases, these messages look like there is a problem. In theory, when files are identical, having only one of them written to the cache is not a problem, but there may be other issues preventing cache access. We could just swallow the error and get away with it, but that’s not ideal.

Two options came to mind. One is modifying the list of cache files before creating threads so that no two threads access the same cache file at the same time. This is hard to implement and brings extra complexity to thread creation. A one-in-a-thousand concern becomes an architectural burden.

The other solution is to lock paths while accessing them so two threads working on the same cache path wait for each other. This is easier and requires just dependency injection into the thread functions. It has the downside of making file system operations slightly slower, as each path operation will now require checking a mutex, but that seems of little concern for file system access, which is already much slower than memory operations.

First, I tried to implement this with a HashMap<PathBuf, Mutex<()>>, but this must also be passed to the function wrapped in Arc<Mutex<HashMap>>, which made the “ceremony” of acquiring the lock for a single file much longer.

The issue is that you don’t want the HashMap itself to be a bottleneck. Multiple threads shouldn’t wait for the HashMap to become available, as it’s not the HashMap we want to lock, but the values inside it.

The solution is to return the lock value from a method of a struct. Something like:

pub struct PathSync {
    locks: Arc<RwLock<HashMap<PathBuf, Arc<Mutex<()>>>>>,
}

Now the ceremony can be performed within the method, and threads working in different directories won’t need to wait for the hash map to become available.

However, after implementing this, I realized I’d probably forget to lock a path at some point. This is a general-purpose solution, and I should apply it to all path operations when multiple threads are working. In most cases, there are multiple paths to lock (cache_path, cache_dir, repository path), and if I forget to lock one of them, a future user, some time, somewhere, will probably see an error message.

So, I decided on a different approach: creating wrappers to run passed closures. This makes it much more obvious that paths Xvc works on must be locked before operations.

    pub fn with_sync_path(
        &self,
        path: &Path,
        mut f: impl FnMut(&Path) -> Result<()>,
    )

This works by passing the path and a closure that operates on that path. It first locks the path and then runs the closure. The locking mechanism allows threads with different paths to run in parallel, but if they try to operate on the same path, they will wait for each other.

The implementation is here.

Another issue is the restart script. The if in that script that restarts the server doesn’t work; it always restarts the notebook server.

I removed some println! statements from xvc.py. There doesn’t seem to be anything in the library related to outputs.

Let’s search for how to check if a jupyter-lab command with the port 7979 runs in the background.

It looks like the bug is in the condition; it’s not -s, it’s -z.

Oops, yeah.

Let’s do a bit of tidying and check if the script is fixed.

I have a clippy warning with a new function that says these usually don’t take self as a parameter. This is for the pipeline new command, and it receives a self as an XvcPipeline object. It seems best to turn off the clippy warning for this.

I allowed two clippy warnings, and the script seems to work fine.

Let’s go on to copying the content from the Xvc README to the notebook.

There is an issue with the run-after-commit script. When Xvc commits the changes, the command we give is run again.

The issue is that git initializes the directory in the test-data/ directory, while xvc works in the current directory. I think we can either git init in the current directory or xvc init in test-data.

We’re already deleting .git and .xvc directories in the start-readme script. I think it may be easier to update git init to just initialize in the current directory.

Yep, let’s do it that way.

I see there are still extra outputs from the commands. We need to deal with this first.

I removed them, but there are still pink outputs. These are from dbg! statements, it looks like, or we’re initializing the output thread incorrectly.

Fixed those as well. I wrote up the xvc file list command examples as well. Now we have issues with xvc storage new s3 not running, and not even showing any debug output. We’ll deal with it in the next devlog, though.

• The Disruptor is a concurrency framework created by LMAX to achieve very low latency and high throughput on their Java trading platform.
• LMAX found that using queues between system stages introduced latency, leading them to focus on optimizing this area.
• The Disruptor uses a lock-free “ring buffer” approach to avoid cache misses and locks at the CPU level, which are very costly.
• The Disruptor is intended as a general-purpose solution for concurrent programming, not just for financial applications.
• Applying the Disruptor pattern is not as simple as replacing all queues with the ring buffer; the user guide provides further guidance.
• Various blogs, articles, technical papers, and performance tests explain the Disruptor’s inner workings.
• A presentation and discussion group are also available for learning about the Disruptor from its creators.
• Martin Fowler has reviewed the Disruptor’s use at LMAX.
• The Disruptor is significantly faster than the ArrayBlockingQueue, as shown in the latency histogram.
• More performance results are available comparing the Disruptor to other approaches.

I saw an ad on Stack Overflow Jobs those days about employment by contributing to open source projects. I applied to that. A few weeks later, the CTO of [iterative.ai] got in touch and I started working on DVC documentation. Initially on a per-hour basis, and after May 2021, as a full-time employee.

Initially, I liked the tool we were building very much. The team was awesome. (Still, they are.) It was one of the best periods of my life, especially in my turbulent still-ongoing-divorce-period pressures. I know I will always miss them.

My job was learning DVC, documenting it, and helping newcomers grasp it easily. It was a fun job. Until then, I didn’t see myself as a technical writer. English is not my native tongue and I never have lived in an English-speaking country. Nevertheless, I think I wasn’t too bad at it.

When I was first learning the tool, I began to use it everywhere. I was an avid user of Git Annex once. DVC looked better. I don’t remember why I lost interest in Git Annex after many years, but it was probably related to symbolic links not working on Windows (or on Termux). DVC had multiple ways of connecting the cache and the files in the workspace, including hardlinks and copy, so it was a breath of fresh air for me.

I began to use it for my large collections. Keeping track of my binary files in Git was something I always desired. Git is the least sucking version control system among the ones I used previously (SVN, hg, darcs…) and I’d rather keep using it everywhere rather than learning new tools for binary files.

After some time I began to use the tool for my personal file collections. I noticed its performance became a burden. I was tracking maybe a few gigabytes of files with it and basic file operations became slower as I added more. I noticed I was becoming distracted after I wrote a dvc command. It took some time to confess that the tool I liked once and was earning my salary with was not a tool that I liked to use.

I don’t know what real professionals would do at this point. I never had a good LinkedIn profile. When I met a similar problem with the example repository that’s supposed to contain 70,000 small files, I brought the issue forward. I wrote a shell script that was basically doing the same thing as dvc add and it worked much faster than the actual command. The shell script was naïve and I thought DVC must have at least that level of speed. It didn’t. Simply calling md5sum on files and copying them to appropriate location in .dvc/cache was way faster. How could this be?

I had cursory observations on the codebase. I know some decisions (like a large central class that connects everything, separate .dvc files for each tracked file) that may lead to degradation. Although I don’t see it as the problem, Python was also not helpful. These are rough observations.

It was September 2021. I was also teaching myself Rust. I wrote an email to the CTO and CEO of the company to request a sabbatical to work on DVC. My plan was to rewrite certain portions (or commands) in Rust and wrap them with PyO3. It could fail. So to have skin in the game, I said I’ll work for free during this time and if I fail to make DVC faster for some reason, I’ll return to my writing position.

They didn’t accept. I didn’t try to persuade them. The decision was rational and although I’d say go ahead and see what happens if I were in their shoes just to make my employee happy, they aren’t crazy-managers as I once was. Probably there are many factors that I’m not aware of. I returned to my post and continued to write documentation for another 9 months. In the meantime I studied Rust and thought about how I could architect a similar tool. Where does DVC go wrong?

In April 2022, I informed the CTO that I’d like to take a sabbatical for my book. I have a political-SF book and after the Ukrainian war started with a (albeit minor) probability of nuclear attack on the other shore of Black Sea, I thought it’s not a time to work on something I stopped liking. My performance in the last quarter was also not something I was proud of. I didn’t feel good.

When I retired to sabbatical in July though, while writing the book, I thought writing the software that I wanted to see was also something on my mind before nuclear war. I had notes about the architecture I was planning. I wanted to see if I could apply an Entity-Component System to this basic problem, without any Object-Oriented conceptions. I believe it looks cool. I’m still simplifying and testing the idea, and it looks better to my mind than mixing data and functions for no reason.

After I made the repository public, I resigned from Iterative.

In a sense, Xvc owes its existence to DVC, and the name is a tribute to this. I hope they squash their bugs, and improve their user experience, and be a long-term player in the crowded market they are in. I don’t intend to be a “competitor”, because I prefer being developer/architect rather than a “technical steward to VC money”, and the license of Xvc is GPL-3 to signal this.

The Xvc command line interface, however, is as different as it can be from DVC. The command names are different; DVC has commands similar to Git (push, fetch, pull, commit), while Xvc tries to be different from Git to reduce the user’s mental load. For example, as a writer, I noticed that “Git remotes” and “DVC remotes” was confusing, so I called them “Xvc storages”. DVC calls the units of a pipeline stages; the same concept is called steps in Xvc, because stage in Git is something completely different.

Internally, the architecture is also very different. Xvc uses serialization (with serde) instead of YAML. It can export/import pipelines from YAML (or JSON), but YAML is not as central as in DVC. (I believe YAML is overused in our industry, and it’s an employment guarantee for another generation of developers but there is better work than keeping up a half-baked configuration format.) Xvc doesn’t keep its artifacts in the user’s workspace (except .xvcignore files). They are all stored in the .xvc/ directory. The DVC way of doing things makes merging .dvc files easier. To overcome the problems caused by merging large metadata files, Xvc keeps track of events and replays them to get the final state of the repository. All metadata storage and retrieval operations revolve around the XvcStore<T> struct in Xvc. Typically, if the user runs an xvc command, only the updated store events (added files, changed pipelines, etc.) are stored. There are optimizations in this front, but I profile first and optimize later.

Algorithms for data digests are configurable; by default Xvc uses Blake3, but it is configurable to use SHA2-256, SHA3-256, or Blake2s. It can be modified to use any 256-bit digest quickly. There are some features that are not found in DVC, and more will come. So, although I’m solving a similar problem, Xvc is not “DVC rewritten in Rust,” it’s a different tool completely.

Currently, it doesn’t have as much eye candy as DVC. In time, I plan to add Python, Julia, and R APIs, notebook integration, experiment tracking (without relying on Git internals), data labeling and filtering, and other MLOps features. I’m building with a goal to make these features available without making the rest of the software slower.

I’ve found the tool I was looking for to track my kids’ photos and Ottoman OCR datasets in a Git repository. I’m tracking more than 1TB of files in a single repository with Xvc and adding another 10TB looks feasible now.

• Although there is a common format for the manifest file, Outlook and Word add-ins seem to require separate files. I haven’t seen any add-in in AppSource that serves as both a Word and an Excel add-in.

• Microsoft provides a tool called Script Lab that allows you to run arbitrary scripts and test APIs directly within Office.

• For authentication, it is recommended to open a browser window to your web app using the Dialog API.

• It is usually not possible to use iframes to obtain authentication tokens from the task pane or sidebar itself. See this answer for more details.

• It is possible to add elements to the right-click menu in Word for add-in commands.

I first tried to use the rust-fsm library, but it became apparent that Xvc pipeline steps’ states are tied to XvcRoot. That is, if we are to check the presence of a file or the value of a parameter, we have to do it relative to the repository root. The root directory should be taken into consideration in every transition.

I checked the code and noticed that the FSM is actually very simple. I copied it, added &XvcRoot to the transition and output functions in the trait definition, and implemented it for XvcOutput, XvcDependency, and XvcStep.

These are the constituents of a pipeline. A pipeline is composed of an XvcStep that defines a command, and each step can have multiple XvcDependency and XvcOutput definitions. For each of these structs, I’ve added fields that represent their current state.

For example, an XvcOutput can be Missing, Found, Old, or Ok. An XvcStep that produces this XvcOutput doesn’t check the dependency content hash if an output is missing. However, if an XvcOutput is Found, the XvcStepStateMachine checks the XvcDependency states and their modification times to see if they have changed since the output was generated. An XvcStep is invalidated when an XvcOutput is Missing or an XvcDependency has changed after the last command run.

Unlike DVC, I added the ability for an XvcStep to depend on other XvcSteps. They communicate through outputs. I’ve added XvcDependency::Step(XvcStep) to the XvcDependency definition. I’m also planning XvcDependency::Pipeline(XvcPipeline) to allow steps to depend on other pipelines, so that pipelines can be run in order.

Currently, the following are included as XvcDependency:

• File: A (binary or text) file in the repository. If the metadata (size and modification time) or the content changes, the dependent step becomes invalidated.
• Directory: A directory that contains files. If a file is added to or removed from the directory, or any of the files are changed, the associated step becomes invalidated.
• Glob: A glob such as my-data/*.png. If the list of files changes or their content has changed, the associated step becomes invalidated.
• Parameter: Xvc can parse YAML, TOML, and JSON files and get the values of variables. It’s possible to define these (hyper)parameters as dependencies.
• URL: An HTTPS URL, which is checked first by metadata and then by content to see whether it has changed.
• Step: A previously defined step; if it’s invalidated, the depending step also becomes invalidated.

Additionally, I’m planning to add the following items to Xvc as dependencies:

• Lines {path, begin, end}: Lines in a text file. This can be used for general-purpose input tracking. If the given lines in a file are changed, the dependent step becomes invalidated.
• Regex {path, regex}: If the regular expression result on the file changes, the dependent stage becomes invalidated.
• Pipeline { name }: If any of the steps in a pipeline is invalidated, the pipeline is also invalidated, or the step that depends on this pipeline becomes invalidated.

Each of these dependencies is checked minimally; that is, when their size on disk is detected to have changed, they are considered changed without checking the content hash. It needs a very detailed state machine to track the changes without bugs.

I’ve noticed that if I can write such a state machine, most of the I/O operations can be done in parallel. If two steps do not depend on each other in the dependency graph, they can be run in parallel. The state machine’s granularity allows this.

🐢 I think I can implement Rsync today. Looking at ssh2-rs, though, I think we can implement file transfer without relying on rsync. It might be easier to implement everything within the code.

🐇 Then you should rename the issue to new ssh.

🐢 Fair. There is also the ssh_rs crate, but it doesn’t have full support for the protocol. Instead, we can have another command, like xvc storage new ssh, that uses libssh2 via the crate mentioned above. It has some limitations with OpenSSH on macOS.

🐇 From the crate’s README, it looks like you can enable the vendored-openssl feature to compile it statically.

🐢 Let’s go ahead then. It’s better to compile it behind a feature flag, though.

🐇 Yup. Rsync can be separate. I think for now you can implement rsync via Exec::cmd and make new ssh a new issue.

🐢 I’ll copy this conversation there.

🐇 Now, let’s start implementing rsync.

🐢 Do we really want to hide it behind a feature flag? It doesn’t bring any extra complexity to generic, for example—just using the commands and returning the errors.

🐇 I think so. If the user doesn’t have rsync on their system, they’ll just get errors. We don’t need to make the implementation optional, but the tests might be.

🐢 OK.

• ends_with in Path is different from ends_with in str: If you try to check whether a Path is a directory by its final character, you’ll find that path.ends_with("/") returns false.

• Path::strip_prefix also consumes the final slash: If you have a directory marker at the end, such as /Users/emre/mydir/, and pass it to strip_prefix("/Users/emre"), you’ll get mydir instead of mydir/.

In this post, I’ll update it. Some of the requirements have changed, and I need new features:

• walk_serial currently takes a Sender and uses channels to send the results. This is confusing, as it doesn’t use any parallelism to traverse the file system.

• There are long-running processes that modify the file system, which can affect behavior. We need to be notified about file system changes while using identical ignore rules.

For the first change, I’ll convert the signature of walk_serial to:

pub fn walk_serial(
    given: IgnoreRules,
    dir: &Path,
    walk_options: &WalkOptions,
    res_paths: &mut Vec<Result<PathMetadata>>,
) -> Result<IgnoreRules>

In the first version, instead of res_paths, we had a sender through which the results were sent.

Why don’t I simply return the paths instead of receiving a mut Vec parameter to fill? While it might be simpler, I don’t like the idea of creating a new Vec for each function call. The overhead of creating new vectors and merging them can become a burden for large directory trees. Instead, each call of the function works on the same vector and expands its capacity when necessary. Since the current Vec strategy is to double the capacity when needed, there are fewer memory allocations for complex and large directory hierarchies.

Other than this, instead of Sender::send, the function uses Vec::push to add elements to the vector. The rest of the implementation is similar.

One major difference in this version is that the function returns an IgnoreRules value instead of (). This contains the compiled ignore rules from all directories below the given directory. We can use this returned value to further check the paths we get from the notifier we’ll develop.

Adding a file system notifier

For the second change, we need to access the underlying OS machinery for file system notifications. My research led me to the notify crate. Although it seems to be undergoing a major overhaul as of this writing, it’s the only promising option other than interfacing directly with the OS.

Again, we begin by defining the enums and structs for this problem.

What do we want? We want to collect file system events to update the Vec<PathMetadata> that was built with walk_serial or walk_parallel. PathMetadata was defined as:

pub struct PathMetadata {
    pub path: PathBuf,
    pub metadata: Metadata,
}

We could build a HashMap<PathBuf, Metadata> or keep this as a vector, but for our purposes, we want to track new events. These events correspond to creating a new file, updating metadata (size, timestamp), or deleting a file. We don’t need, for example, permission changes, as we primarily want to track which file is newer or whose size has changed.

pub enum PathEvent {
    Create { path: PathBuf, metadata: Metadata },
    Update { path: PathBuf, metadata: Metadata },
    Delete { path: PathBuf },
}

notify provides an abstraction over different operating systems. It’s called RecommendedWatcher and is used to provide a consistent interface for events. A watcher is built by implementing the EventHandler trait.

Since we need to check whether paths are ignored before reporting them, the watcher should receive an IgnoreRules struct.

pub fn make_watcher(
    ignore_rules: IgnoreRules,
) -> Result<(RecommendedWatcher, Receiver<PathEvent>)> {

It returns a RecommendedWatcher and a (crossbeam) Receiver for the PathEvent we defined above. The implementation creates a channel, initializes the watcher, and starts watching.

    let (sender, receiver) = bounded(10000);
    let root = ignore_rules.root.clone();
    let mut watcher = notify::recommended_watcher(PathEventHandler {
        ignore_rules,
        sender,
    })?;
    watcher.watch(&root, RecursiveMode::Recursive)?;

PathEventHandler is the struct that implements the EventHandler trait. The implementation requires a single function to be implemented:

    fn handle_event(&mut self, event: notify::Result<Event>)

notify::Event is a hierarchical set of enums that describe the type of file system event. The event handler is expected to discriminate between these events and react accordingly.

pub struct Event {
    pub kind: EventKind,
    pub paths: Vec<PathBuf>,
    pub attrs: EventAttributes,
}

We’re interested in filtering events by EventKind, checking whether the paths are ignored, and sending the path and metadata through a channel. We have three kinds of events in PathEvent. PathEventHandler implements event filtering and calls handlers for these PathEvents.

The struct has functions that correspond to one or more PathEvent elements. These are create_event, write_event, remove_event, and rename_event. The first three correspond directly to members of PathEvent, and the latter sends both a PathEvent::Delete and a PathEvent::Create.

match event.kind {
    notify::EventKind::Create(_) => self.create_event(event.paths[0].clone()),
    notify::EventKind::Modify(mk) => match mk {
        notify::event::ModifyKind::Any => todo!(),
        notify::event::ModifyKind::Data(_) => self.write_event(event.paths[0].clone()),
        notify::event::ModifyKind::Metadata(_) => {
            self.write_event(event.paths[0].clone())
        }
        notify::event::ModifyKind::Name(rk) => match rk {
            notify::event::RenameMode::Any => {}
            notify::event::RenameMode::To => self.create_event(event.paths[0].clone()),
            notify::event::RenameMode::From => {
                self.remove_event(event.paths[0].clone())
            }
            notify::event::RenameMode::Both => {
                self.rename_event(event.paths[0].clone(), event.paths[1].clone())
            }
            notify::event::RenameMode::Other => {}
        },
        notify::event::ModifyKind::Other => {}
    },
    notify::EventKind::Remove(_) => self.remove_event(event.paths[0].clone()),
    notify::EventKind::Any => {}
    notify::EventKind::Access(_) => {}
    notify::EventKind::Other => {}
}

An example event handler should clarify how the others work:

fn write_event(&mut self, path: PathBuf) {
    match check_ignore(&self.ignore_rules, &path) {
        MatchResult::Whitelist | MatchResult::NoMatch => {
            self.sender
                .send(PathEvent::Create {
                    path: path.clone(),
                    metadata: path.metadata().map_err(Error::from).unwrap(),
                })
                .unwrap_or_else(|e| warn!("{}", e));
        }
        MatchResult::Ignore => {
            debug!("FS Notification Ignored: {}", path.to_string_lossy());
        }
    }
}

Other event handlers work similarly. They send PathEvent values that can be used to update a list of paths and their metadata.

To use this watcher, you first retrieve the list of files with walk_serial. It returns a snapshot of the directory and the IgnoreRules collected from the directories below it. Then, you create a watcher with:

let ignore_rules = walk_serial(...)?;
let (watcher, path_event_rec) = make_watcher(ignore_rules)?;

After this, you can create another thread to monitor file system changes:

crossbeam::scope(|s| {
    s.spawn(|_| {
        while let Ok(path_event) = path_event_rec.recv() {
            match path_event {
                PathEvent::Create { .. } => { /* ... */ }
                PathEvent::Delete { .. } => { /* ... */ }
                PathEvent::Update { .. } => { /* ... */ }
            }
        }
    });
});

Another point is that Rust, being a statically typed language, requires far fewer unit tests than dynamic languages. Traditionally, unit tests are used to validate “input consumption” (ensuring input is processed correctly) and “well-formed output” (verifying that the output meets requirements under specific conditions). In Rust, many of these input-output concerns are resolved at compile time through the use of expressive types like structs and enums.

For these reasons, I find myself more inclined toward integration tests that reflect real-world requirements, rather than focusing solely on unit tests in artificial settings.

The talk is insightful. He argues that Scrum is not agile and that it imposes a very strict set of rules that have nothing to do with true agility.

While I am convinced that Scrum isn’t agile, I’m not convinced that we don’t need processes or habits at all. Another talk of his is this:

Basically, what agile boils down to is self-managing teams composed of self-managing people working in close collaboration with the customer. This requires an endless supply of motivation for development. If we take the team is motivated as a given, then it’s certain that hindrances from rigid processes don’t make sense. For me, for Allen, and for many others, this may be true, but my experience shows me that those who are really motivated to develop even for free are in the minority. For most folks out there, development is a job like any other. Their supply of motivation is limited, and when we make them self-govern, they might not actually do the work; they might just read, doodle, spend their budget on useless things, get bored, and jump to another job.

In his second talk, he says, “We assume we’re grown-ups.” I’d like to assume that as well, and I certainly assume this for myself—I’d do development on my own time. Most of my development in the last 25 years has been essentially for free, without an explicit financial goal. However, when I’ve managed projects, I’ve seen that real life doesn’t always support such an assumption. Software development is a lucrative profession. If we start with mainly financial motivations, self-governing becomes a bit of a dream. If I’ll earn the same amount whether I work today or not, I might skip the work and engage in another joyful activity. In my case, that activity is another kind of development, but does it matter?

This doesn’t mean Scrum is good and should be endorsed. I don’t like it and haven’t used it much anyway. But the ideals Allen discusses can’t survive on their own. They need some way to supply motivation, some direction, some form of soft governing. Agile in its original sense looks like anarchy, but anarchy often gives birth to some form of governance at some point. In the past, there were people who had no hierarchical management, but when hierarchies were invented (or introduced by others), they became powerful.

We can continue to believe that developing like aboriginal tribes, in a people over processes fashion, leads to ultimate success. That may be true in certain scenarios, but in general, those tribes can’t maintain coherence after a certain size. Development teams, in my humble opinion, are similar.

I then came across the cached crate. It caches the results of functions for memoization. This is exactly what I need, I thought. Internally, it does what I was planning to do.

Later, I noticed the possible bugs that could arise. I’m thinking of using the function in a short-running process, so the metadata is not expected to change during the run. For some reason, suppose the runtime of the process began to get longer, or I decided to add a web server on top of it. At that time, probably many moons from now, I’ll have forgotten the decision I made about caches and my assumption that the metadata won’t change during the run. It will cause some weird bugs when file timestamp changes aren’t detected.

No one will notice that I’m fixing bugs if they never appear, but I believe this is the best kind of software engineering.

Commentary (2022-08-01)

• It looks like my assumption that RAM is significantly faster than disk access may also be wrong. SSDs are fast, and for parallel access, they may perform as fast as RAM.