🐇 It should only be needed to update the dependency versions in Cargo.toml, right?

🐢 Let’s start with that.

🐇 We have interface changes regarding aliases; let’s start using uv for building.

🦊 Added requirements to pyproject.toml by running:

uv add -r requirements.txt

🐢 There is a way if we start from the configuration, not just files. The default configuration is a TOML document. We can make it an XvcConfiguration struct and load and store it with confy.

🐇 We have a cascading set of configurations, but maybe we can start from the struct and serialize/deserialize it on demand.

🐲 I don’t think we need to update xvc-config at the moment. It’s working, and we don’t need to alter its inner workings in the near future.

🐢 I agree. We can use the config crate when we need to update and have enough time to work on this.

🐢 Yes, but what will T be? Although store structures have Serde implementations, they are not particularly useful for this.

🐇 We can have output types, named like XvcFileListOutput, that will be converted to strings with serialization.

🦊 We do something similar in xvc pipeline export and import commands. We use XvcPipelineSchema and XvcStepSchema just for the import and export commands. We’ll write similar structs for all JSON output and will use Serde to convert these to strings.

🐢 Unlike the import and export commands, we have optional fields in the output, though. I don’t want content digests to appear in JSON output if they are not required.

🐇 Let’s search for optional fields in Serde.

🦊 There is a crate for optional fields.

🐢 We don’t need another crate for this. Serde has the skip_serializing_if attribute for fields. We can add Option::is_none as a method to these to skip outputting None fields. All those fields, in this case, will be optional.

🐇 This is fine. We already use structs to format the xvc file list output. We can just use them to output JSON.

🐇 What about adding at least command completions for Nushell?

🐢 Let’s read a bit about clap_complete_nushell.

🦊 There seems to be a nu-complete command. Let’s check its documentation.

🐇 Nothing was found, and Kagi doesn’t help much either.

🐢 There is a completions document for Nushell: https://www.nushell.sh/book/custom_completions.html

🐇 There is a tool called carapace to provide completions across shells.

🐢 Its documentation is thin, and I’m not sure if it supports dynamic completions out of the box. I believe instead of adding a carapace setup, we can just write a Nushell completion script that will use JSON output from the commands and add some (maybe hidden) utility commands to support it.

🐇 There are a set of example scripts in the Nushell repo: https://github.com/nushell/nu_scripts/tree/main/custom-completions

🐢 The reason I want to write custom completions for Nushell is that it will be an exercise for the scripting language. gh completions are not as scary as a Bash script.

🐇 git completions are a better example for XVC. They simply run git whenever necessary. We can start from a static completions command and update this with dynamic completions manually. It will teach a lot.

🐢 I forked the nu_scripts repo and will add XVC completions script there.

🐇 Then let’s begin by adding Nushell static completions. Shall we add a command for this?

🦊 Reviving the completion command we removed in 0.6.13?

🐢 We shouldn’t list it. We can make a _comp subcommand for the time being and generate and distribute completions in the repository. When clap_complete_nushell has the feature parity to provide dynamic completions, we can remove these commands.

🐇 What will we use this for other than generating completions?

🐢 Maybe dynamic completions can call this as well.

🐇 Added Nushell static completions to be output using xvc _comp generate-nushell. Let’s bump up the version to 0.6.15.

cargo set-version 0.6.15-alpha.1
   Upgrading xvc from 0.6.14 to 0.6.15-alpha.1
...

🐢 I noticed we forgot a line in the CLI command handler that asserts xvc_root_opt.is_some(), and this fails when we run xvc outside of repositories. We need to release this version quickly.

🐇 Oops, now, ok, let’s write a static Nushell generator and just release quickly.

🦊 Generating completions with

xvc comp generate-nushell

🐢 Completion command is run with comp instead of _comp. Should we rename it?

🐇 Renamed it to _comp. It’s not hidden, but at least we can be sure that it won’t be misunderstood as a common command.

🐢 Bumping the version again. Now let’s source the generated script and test it.

cargo set-version 0.6.15-alpha.2
   Upgrading xvc from 0.6.15-alpha.1 to 0.6.15-alpha.2
...

🦊 Yep, it works. We now have completions for Nushell.

🐢 Let’s update the completions documentation.

🐇 Done. Now, let’s take a look at CI and see what fails.

ghrl | first
╭──────────────┬─────────────────────────────────────────────────────────╮
│ conclusion   │ success                                                 │
│ displayTitle │ Add Nushell completions                                 │
│ headBranch   │ nushell-completions                                     │
│ url          │ https://github.com/iesahin/xvc/actions/runs/13070200765 │
╰──────────────┴─────────────────────────────────────────────────────────╯

🐢 It fails because of coverage, not the tests. Codecov says the new code isn’t tested.

🐇 The added xvc _comp command isn’t tested. We can add a test running those lines and testing if the command outputs a completion script.

🐢 We have a test for completions. We can add a test that runs the lines.

🐇 Added a test and bumping up the version.

cargo set-version 0.6.15-alpha.3
   Upgrading xvc from 0.6.15-alpha.2 to 0.6.15-alpha.3
...

🦊 We can add some more coverage while waiting for the tests.

🐇 XvcOutputLine implementation seems to have no tests. It’s weird because we use these everywhere.

🐢 I’m not sure we use this particular implementation; we just use XvcOutputLine::Info(s), not XvcOutputLine::info(s) anywhere. We can delete these methods actually.

🐇 We’ll add JSON output via this particular struct. Can we refactor these to use formatting for JSON, for example? Or use these to output JSON?

🦊 We can add a formatter to XvcOutputLine to output structures.

🐢 The enum is now defined as:

#[derive(Clone, Debug)]
pub enum XvcOutputLine {
    /// The output that we should be reporting to user
    Output(String),
    /// For informational messages
    Info(String),
    /// For debug output to show the internals of Xvc
    Debug(String),
    /// Warnings that are against some usual workflows
    Warn(String),
    /// Errors that interrupts a workflow but may be recoverable
    Error(String),
    /// Panics that interrupts the workflow and ends the program
    /// Note that this doesn't call panic! automatically
    Panic(String),
    /// Progress bar ticks.
    /// Self::Info is also used for Tick(1)
    Tick(usize),
}

Here, these fields can also have a formatter that will render the string in a particular format. For example, the output can be

XvcOutputLine::Output(XvcJsonFormatter, String)

🐇 I’m not sure this is a good idea. Output already specifies this string as output. We can have a wrapper instead, like,

struct XvcJsonOutput(Format<XvcStructuredOutput>, XvcStructuredOutput)

and we can use the supplied format to render XvcStructuredOutput to an output line with XvcOutputLine::Output. If we don’t provide output as structured, it will be too much error-prone work to convert the current outputs to structured.

🦊 The transition will also be gradual. We may not need structured output for most of the commands. We can start with xvc file list and convert others as we go.

🐢 This is sensible. By the way, coverage still didn’t increase. There may be something going on with Codecov or running the test.

ghrl | first
╭──────────────┬─────────────────────────────────────────────────────────╮
│ conclusion   │ success                                                 │
│ displayTitle │ Add Nushell completions                                 │
│ headBranch   │ nushell-completions                                     │
│ url          │ https://github.com/iesahin/xvc/actions/runs/13087431038 │
╰──────────────┴─────────────────────────────────────────────────────────╯

🐇 Let’s run the test:

cargo test -p xvc --test test_completions
...
test test_completions ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.80s

🐢 Can we make sure the output is a Nushell script and not an error message?

🐇 Let’s print it out.

🐢 It looks like when the COMPLETE environment variable is set, it never calls _comp subcommand and never calls those lines.

cargo set-version 0.6.15-alpha.4
   Upgrading xvc from 0.6.15-alpha.3 to 0.6.15-alpha.4
...

🐢 Let’s make a release for 0.6.15. Coverage is OK now.

cargo set-version 0.6.15
   Upgrading xvc from 0.6.15-alpha.4 to 0.6.15
...

gh pr merge --squash --body $"(open CHANGELOG.md | lines | skip 2 | take 5)" --subject "Add static nushell completions"

🐇 Merged the PR.

🐢 Releases should appear in a few minutes.

🐇 We need to tag the merge commit for this.

🐢 Oh, yep. AFAIK Lazygit doesn’t have something for git push --tags. Let’s push from the CLI.

git push --tags
You are on the main branch. Skipping CHANGELOG.md check.
To github.com:iesahin/xvc
 * [new tag]         v0.6.15 -> v0.6.15

🦊 These commands, especially tables, are not rendered correctly on the web. We need to change the theme, I think.

🐇 It looks like you first need to turn off this ghost text from the blink output. It makes writing insufferable.

🐢 Yep, let’s do that first.

🐇 Now, let’s restart Neovim.

🐢 I don’t know why blink.cmp doesn’t prioritize the emojis I use. Maybe we can just use #tor and #rab for ourselves.

🐇 I think over time it will learn that the emojis I defined in the snippets file should have higher priority, but let’s skip this for now. What do we need to do to add cross-compilation?

🐢 Maybe we can just make the completion menu wait a bit longer. It shows up almost instantly, and I want it to wait for a few more milliseconds.

🐇 Okay, let’s look at the config.

🐢 The configuration file doesn’t seem to have a key for this. Let’s search: blink.cmp.

🐇 I think the culprit is typo resistance; it causes better options to be pushed down: https://cmp.saghen.dev/configuration/reference#fuzzy

🐢 Let’s turn that off.

🐇 There is an error in the configuration file. I don’t know why it’s failing.

🐢 I added emojis to Espanso and checked the error message. The configuration I copied from their docs seems to be broken. The message is:

...share/nvim/lazy/blink.cmp/lua/blink/cmp/config/utils.lua:14: fuzzy.max_items: unexpected field found in configuration

I’ll just delete that line.

🐇 It still doesn’t prioritize snippets, but we can look into this later. Espanso seems to be a better tool for this anyway.

🐢 Yep. Let’s look into cross-compilation support for Rust.

🐇 The well-known option is cross.rs: https://github.com/cross-rs/cross

🐢 We can start with that. Installation is from the Git repository:

$ cargo install cross --git https://github.com/cross-rs/cross
    Updating git repository `https://github.com/cross-rs/cross`
    Updating git submodule `https://github.com/cross-rs/cross-toolchains.git`
  Installing cross v0.2.5 (https://github.com/cross-rs/cross#4090beca)
...
   Installed package `cross v0.2.5 (https://github.com/cross-rs/cross#4090beca)` (executables `cross`, `cross-util`)

🐇 Now we have the cross and cross-util commands. Cross-compilation requires Podman on Linux or Docker on macOS. Do we have Docker?

🐢 It looks like we don’t. Maybe we can just set up a remote build using Podman or GitHub Actions. I saw a crate for that yesterday; I remember saving it somewhere but can’t find it now. Searching again seems easier, which says something about my archival and retrieval habits.

🐇 Maybe later you can add some vector search capabilities to your archive—semantic search.

🐢 Yep, sometime later.

🐇 Now let’s search for “adding rust cross compilation to github actions.”

🐢 I found a link to the action: https://github.com/marketplace/actions/build-rust-projects-with-cross

Let’s look at the example:

jobs:
  release:
    name: Release - ${{ matrix.platform.os-name }}
    strategy:
      matrix:
        platform:
          - os-name: FreeBSD-x86_64
            runs-on: ubuntu-20.04
            target: x86_64-unknown-freebsd
            skip_tests: true

          - os-name: Linux-x86_64
            runs-on: ubuntu-20.04
            target: x86_64-unknown-linux-musl

          - os-name: Linux-aarch64
            runs-on: ubuntu-20.04
            target: aarch64-unknown-linux-musl

          - os-name: Linux-riscv64
            runs-on: ubuntu-20.04
            target: riscv64gc-unknown-linux-gnu

          - os-name: Windows-x86_64
            runs-on: windows-latest
            target: x86_64-pc-windows-msvc

          - os-name: macOS-x86_64
            runs-on: macOS-latest
            target: x86_64-apple-darwin

          # more targets here ...

    runs-on: ${{ matrix.platform.runs-on }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build binary
        uses: houseabsolute/actions-rust-cross@v0
        with:
           command: ${{ matrix.platform.command }}
          target: ${{ matrix.platform.target }}
          args: "--locked --release"
          strip: true
      - name: Publish artifacts and release
        uses: houseabsolute/actions-rust-release@v0
        with:
          executable-name: ubi
          target: ${{ matrix.platform.target }}

🐇 We can just convert the current configuration to this and see if it works.

🐢 Let’s do that. I created .github/workflows/release.yml and will update it.

🐇 Let’s check the results with gh:

$ gh -R iesahin/xvc run list
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525070531	34s	2024-12-28T08:17:28Z

🐢 It says the release has completed.

$ gh -R iesahin/xvc run view 12525070531

X v0.6.13 Release iesahin/xvc#263 · 12525070531
Triggered via pull_request about 3 minutes ago

JOBS
X Release - FreeBSD-x86_64 in 12s (ID 34936257653)
  ✓ Set up job
  ✓ Checkout
  X Build binary
  - Publish artifacts and release
  ✓ Post Build binary
  ✓ Post Checkout
  ✓ Complete job
...

Now let’s look at the failure:

$ gh -R iesahin/xvc run view 12525070531 --log-failed

🐇 Change the order and see what happens. Maybe we should first succeed in a non-cross-compilation build.

$ gh -R iesahin/xvc run list
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525178664	35s	2024-12-28T08:34:18Z
...
$ gh -R iesahin/xvc run view   12525178664 --log-failed
...

🐢 It’s the same error. The usage example is actually much more sophisticated.

🐇 The issue is that we’re asking for command from the matrix, but the matrix doesn’t define it. That’s the second time today an example from the documentation has failed. I’ve added command for each platform. We can also add separate features this way to make platform-specific functionality work.

🐢 Agreed. Let’s look at it once more.

$ gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525253982	41s	2024-12-28T08:46:23Z

$ gh -R iesahin/xvc run view 12525253982 --log-failed
...
Release - macOS-x86_64	Build binary	2024-12-28T08:46:44.8464010Z  [1m [31merror [0m [1m: [0m the lock file /Users/runner/work/xvc/xvc/Cargo.lock needs to be updated but --locked was passed to prevent this

🐇 Ah, that’s a different error. Let’s remove the --locked flag and retry.

$ gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525289176	1m8s	2024-12-28T08:53:23Z

🐢 We’re finally starting to get some good news. Now we’re getting OpenSSL errors. We need feature flags for these platforms or to specify where OpenSSL is. Let’s add bundled-openssl to the failed ones. We also need bundled-sqlite for Windows binaries.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525354179	3m9s	2024-12-28T09:04:55Z

🐇 It looks like the Changes.md file is missing, and it can’t upload the binaries as a release because of this. I’ll set the changes file to CHANGELOG.md.

🐢 It’s weird to fail because of that. I think we should report these errors and possibly send a PR to make that file optional.

gh -R iesahin/xvc run list | rg Release | head -n 1
in_progress		v0.6.13	Release	v0.6.13	pull_request	12525449368	1m38s	2024-12-28T09:18:40Z
...
✓ Release - macOS-x86_64 in 2m39s (ID 34937017695)
✓ Release - macOS-aarch64 in 2m42s (ID 34937017787)
..
ARTIFACTS
xvc-macOS-x86_64.tar.gz
xvc-macOS-arm64.tar.gz

🐢 It looks like Linux-riscv64 has OpenSSL compilation errors. I think we can skip this platform for now. The goal was to add aarch64 for macOS, and that seems to have succeeded.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525504079	3m24s	2024-12-28T09:26:43Z
...
X Release - Linux-x86_64	Build binary	2024-12-28T09:29:16.2262518Z  [0m [1m [38;5;9merror[E0308] [0m [0m [1m: mismatched types [0m
...

🐢 It looks like Linux-x86_64 doesn’t support reflinks. Maybe we can make reflinks an optional feature and add it specifically to macOS and Windows targets.

🐇 I’ve removed reflink from the default features. This is a breaking change, but fortunately we don’t have many users who will be affected by it.

Let’s check the results once more.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525597866	3m39s	2024-12-28T09:42:44Z

🐢 Now turn off the NetBSD target as well.

🐇 FreeBSD and Linux-x86_64 targets are building. Let’s see why the ARM Linux targets are failing.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525643284	4m46s	2024-12-28T09:52:04Z

🐇 It looks like those targets don’t have libsqlite3 installed. Let’s add bundled-sqlite to these targets too.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525734325	4m11s	2024-12-28T10:07:36Z

🐢 Adding Android as a target didn’t work. Let’s remove it for now; it seems to require more work.

🐇 I think we’ll eventually move to using Xvc to distribute binaries. We could build the Android binary on Termux and link it on the releases page or push it as a release artifact.

🐢 I need to learn more about GitHub releases. If we can add artifacts to the release, maybe we can do some of this work locally.

🐇 We can start by looking at the capabilities of gh commands.

🐢 It looks like it’s possible to upload assets to releases. Let’s check how that works.

gh release upload --help

🐇 So basically, we can just upload files to tags. We can list releases and work with them like anything else.

gh -R iesahin/xvc release list

And we can delete releases:

for r in v0.4.2-alpha.8 v0.4.2-alpha.7 v0.4.2-alpha.6 v0.4.2-alpha.5 v0.4.2-alpha.0 v0.4.1-alpha.0; do
  gh -R iesahin/xvc release delete "${r}"
done

🐢 Now let’s list them again.

gh -R iesahin/xvc release list

🐇 The latest one has failed again.

X Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run

🐢 It looks like we’re coming to the end of this session. Under what conditions will we create a release?

🐇 I think it’s better to release only non-alpha tags.

🐢 Then the rule in the workflow will be something like:

on:
  workflow_dispatch:
  push:
    tags:
      - "v*.*.*"
      - "!v.*.*-alpha.*"

🐇 And now we have this result:

✓ v0.6.13 Release iesahin/xvc#263 · 12533498361
Triggered via pull_request about 21 minutes ago

JOBS
✓ Release - FreeBSD-x86_64 in 4m14s
✓ Release - Linux-x86_64 in 4m3s
✓ Release - Linux-aarch64 in 4m50s
✓ Release - Windows-x86_64 in 8m47s
✓ Release - Windows-aarch64 in 6m42s
✓ Release - macOS-x86_64 in 2m37s
✓ Release - macOS-aarch64 in 2m36s

🐢 Nice! Let’s check the release list.

gh -R iesahin/xvc release list

🐇 Why is there no “latest” release?

🐢 We have to tag it first.

🐇 Ah, right. Let’s tag it then.

🐢 I’ve pushed the changes and tagged them with v0.6.13-alpha.5.

🐇 I think it’s possible to make a release today.

🐢 It looks like it, yes.

🐇 We can merge the PR and tag it. Then everything should work.

🐢 We need some cleanup in the YAML files, though.

🐇 We can leave that for the next release.

gh -R iesahin/xvc run list
in_progress		v0.6.13	Rust-CI	v0.6.13	pull_request	12533690474	9m18s	2024-12-29T08:02:29Z
completed	success	Release	Release	v0.6.13-alpha.5	push	12533689099	9m11s	2024-12-29T08:02:19Z

🐢 Now we have another failure in the regular CI. Let’s look at it.

🐇 The issue seems to be in the doc tests:

- Total #: 8 Workspace Size:         276 Cached Size:          19
+ Total #: 8 Workspace Size:         278 Cached Size:          19

🐢 These tests are brittle, but they provide valuable information. Let’s fix it and push again.

🐇 Done. We can also remove some of the watches that produce so many logs.

🐢 I’m a bit ambivalent about them. I thought we could use these watches when debugging, but experience has shown that we need more granular watches during debugging and almost never use these otherwise. Let’s remove some of them.

🐇 We can increase the output for certain commands, but the tracing output doesn’t help much in regular runs. If Xvc gets popular enough that we can’t cope with bug reports, we can always add more watches.

🐢 Another option is to exclude the watch code from the release build, but that won’t change anything for our debug cycles.

🐇 I think removing them is a fair trial. We can always put them back when debugging.

🐢 Watches could also produce regular output instead of tracing. That way we won’t forget to remove them.

🐇 Ah, yep, that’s a good option too.

🐢 We can have a trace! macro similar to the current one for user consumption, and a watch! macro that sends output to stderr.

🐇 Good idea. Let’s do that in the next release.

🐢 Let’s check the tests before pushing this cleanup.

gh -R iesahin/xvc run list
completed	success	v0.6.13	Rust-CI	v0.6.13	pull_request	12533993607	13m49s	2024-12-29T08:47:50Z

🐇 CI succeeded, and the release didn’t run. Let’s do some more cleanup.

gh -R iesahin/xvc run list
in_progress		v0.6.13	Rust-CI	v0.6.13	pull_request	12534416152	43s	2024-12-29T09:57:02Z

🐇 It looks, from the coverage page, that our additions to HStore don’t have any tests. Maybe we can add some unit tests to the new joins.

🐢 I don’t find unit tests particularly useful, but let’s use GitHub Copilot to add some for us.

🐇 I added a unit test and a doc test for full_join. I think doc tests have more value; they provide documentation, and we can readily see how to use a function from its docs. It’s better to increase coverage with doc tests.

🐢 There are things I should keep in mind while writing doc tests. Imports must use the full path, not crate. Also, the tested struct doesn’t have implicit imports.

🐇 The ceremony for adding keys and values is a bit too much. It might be worthwhile to add an insert method for anything that implements Into<XvcEntity>.

🐢 That would certainly save time—no more typing .into() for each key! :)

🐇 Pushed to test again. Should we have some way to test coverage locally?

🐢 I don’t think we need to check coverage locally. It’s not worth our time right now.

🐇 Now, while waiting for the tests to complete, what else can we do?

gh -R iesahin/xvc run list
completed failure v0.6.13 Rust-CI v0.6.13 pull_request 12543831360 3m54s 2024-12-30T08:15:06Z
...

🐢 That didn’t take too long. Let’s view the results:

gh -R iesahin/xvc run view 12543831360
...
  X Run Current Dev Tests
...
To see what failed, try: 
View this run on GitHub: https://github.com/iesahin/xvc/actions/runs/12543831360

🐢 The current dev tests are failing for some reason. Let’s run them locally.

🐇 We’re missing llvm-tools-preview locally. How do I install this?

🐢 The command is rustup component add llvm-tools-preview:

info: component 'llvm-tools' for target 'aarch64-apple-darwin' is up to date

🐇 It’s already installed. We just need to set the environment variables.

🐢 Instead, we can just turn off dev tests for the time being. We don’t really need them; our local tests pass.

🐇 Yeah, okay. We don’t need to solve every single bit of these issues right now.

gh -R iesahin/xvc run list
...

🐇 Okay, let’s take a look at the run again.

gh -R iesahin/xvc run list
completed failure v0.6.13 Rust-CI v0.6.13 pull_request 12544020064 3m39s 2024-12-30T08:33:25Z

gh -R iesahin/xvc run view 12544020064
...
  X Test and Coverage
...

gh run view 12544020064 --log-failed
...
Test and Coverage (stable) Test and Coverage 2024-12-30T08:36:58.8911690Z Error: ProcessError { stdout: "", stderr: "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to e1.xvc.dev port 80 after 160 ms: Couldn't connect to server\n" }
...

🐢 We need to start Nginx on the server. We forgot to do that yesterday.

🐇 Ah, right. After that dufs installation. Okay.

🐢 We also need to add a reverse proxy for dufs somehow, but that’s for later.

🐇 For this use case, I don’t think it’s necessary. We can just adjust the port to a non-standard one if we need 443 for something else. Let’s look at the tests again.

🐢 Let’s add another doc test, this time to XvcStore.

cargo test -p xvc-ecs --doc
...
test result: ok. 8 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.87s

🐇 Sent the files again. Waiting for the tests to finish.

🐢 Let’s check the keymaps file in the meantime.

🐇 I tried reading the documentation but didn’t see an error. Maybe we should just set it to non-lazy and disallow remaps.

🐢 We’ve already spent too much time on this.

🐇 Yes, let’s check the tests again.

gh -R iesahin/xvc run list
completed success v0.6.13 Rust-CI v0.6.13 pull_request 12544291068 8m10s 2024-12-30T09:00:42Z
...

🐢 Oh, nice, the merge is ready.

🐇 Patch coverage is still behind the target, though.

🐢 Yeah, but let’s release this one and ensure the next one is better covered. Also, I’m not sure if the doc tests actually affected the coverage report.

🐇 If we look at the coverage page again, we can see.

🐢 It seems codecov.io doesn’t consider coverage for doc tests. That’s a bit weird, but let’s not spend more time on it.

🐇 Sure, let’s merge.

🐢 I think we forgot to bump the versions in the Cargo.toml files. We’ll have to do that in main.

🐇 Oh, yeah. Let’s bump them and tag the release as well.

🐢 Now we can wait for all the files to be produced. What’s next?

🐇 We can release the Python version too. It shouldn’t need any changes.

🐢 Right. Maybe we can add a few tests there as well.

🐇 Let’s bump the version first and see.

🐢 I’ve bumped the versions in Cargo.toml and run maturin develop.

🐇 It seems ready now.

🐢 The main branch is failing, though. The “Publish crates” action is looking for libsqlite3. Let’s investigate.

gh -R iesahin/xvc run list
completed failure Release v0.6.13 Publish Crates v0.6.13 push 12544753359 6m1s 2024-12-30T09:41:56Z

🐢 The issue is that the VM doesn’t have libsqlite3-dev. Let’s add it.

🐇 We need to restart the job manually. Let’s skip tagging this time.

🐢 Some of the packages were already published, and now they’re causing failures because crates.io says they already exist. Maybe we can check if a package is already published before trying.

🐇 Let’s see if we can make cargo publish more forgiving.

🐢 There doesn’t seem to be an easy option. Let’s search for “how to skip published packages in workspace to avoid errors with cargo publish.”

🐇 Claude is hallucinating again. Let’s try a manual approach: how to skip already published packages?

🐢 It might be easier to just add a check. How do we get that info?

🐇 Or we can just move on to the next package if one is already available.

🐢 Let’s push the missing packages manually this time.

🐢 We should have a key to open garden files quickly. What does Fzf-Lua files receive as arguments?

🐇 Let’s check the help page: fzf-lua.

🐢 Before that, maybe we can try to fix why selected lines are not searched in Visual-Line mode?

🐇 Yep, let’s fix our search first.

🐢 How do you set a key in visual line mode in Neovim Lua?

🐇 The abbreviation is V. Let’s try that.

🐢 Looks like we need to restart the session.

🐇 It’s still not working. Let’s check :map.

🐢 It seems it needs more care; I’ll check it later.

👨🏾‍🦲 🐢 Bence kendini biraz daha anlamlı bir işle uğraştırmalısın. 🐇 Ne gibi? 🐢 Belki biraz daha yayın yapmalısın, biraz daha işe başvurmalısın. 🐇 “Meli”, “malı” ile geçiyor ömrümüz.

⌚ 🐢 It looks like we can move most of the daily template to links or commands. We can call it ref/daily. My idea is to fill the page intentionally, without templates.

🐇 It might be better to start with a blank page, yeah. The template makes me a bit nervous. There are too many things to fill in, and most of them aren’t things I like being pushed into doing.

🐢 Let’s start by moving the daily templates to ref/daily. No more daily template.

🐇 Now we can delete the rest of this page. We’ll use the daily links page and maybe have reminders at the end of these sessions.

👨🏽‍⚕️

Currently, it only supports text. When I send a message, the bot adds it to a new file or appends it to a file modified within the last 30 minutes, which simplifies multi-paragraph inputs.

I need some more features to make it more useful.

I want it to save audio and image files to my inbox as well. I plan to run Xvc pipelines on these files to convert audio notes to text and make images searchable via OCR or tagging.

The first step to achieve this is to understand what kind of files Telegram can upload and download. I want to track these files with Xvc to avoid bloating the Git repository. I have a script that processes these files, but currently, it only commits changes to Git. I want to determine whether a file should be tracked by Xvc or Git.

I skimmed the Telegram API documentation and found that it supports a wide range of file types. This led me to change my approach: I will track only .txt and .md files with Git and use Xvc for everything else.

In the future, I plan to add a --binary-only option to xvc file track to track binary files only. This will help me to track only non-text files when using globs as targets. I might also extend this to a general predicate to decide which files to track based on their size, modification time, or other properties.

#!/bin/bash

# Git pre-push hook to check if CHANGELOG.md is included in the push and if the branch is develop

# Get the current branch name

current_branch=$(git rev-parse --abbrev-ref HEAD)

# Check if the branch is develop

if [ "$current_branch" == "main" ]; then
	echo "You are on the main branch. Skipping CHANGELOG.md check."
	exit 0
fi

# remote="$1"
# url="$2"

# Get the list of commits to be pushed
commits=$(git rev-list '@{u}..HEAD')

has_rust_files=$(false)

# TODO: We can iterate to get file list only once
# Get the list of files that are going to be pushed
for commit in $commits; do
	if git diff-tree --no-commit-id --name-only -r "${commit}" | grep -q "\\.rs$"; then
		has_rust_files=$(true)
	fi
done

if [[ ! $has_rust_files ]]; then
	echo "No .rs files in the push, no need to check CHANGELOG"
	exit 0
fi

# Check if CHANGELOG.md is among the files in the commits
for commit in $commits; do
	if git diff-tree --no-commit-id --name-only -r "${commit}" | grep -q "CHANGELOG.md"; then
		echo "CHANGELOG.md is included in the push."
		exit 0
	fi
done

echo "ERROR: CHANGELOG.md is not included in the push."
exit 1

Edit (2025-02-08): Added a check for when only code files are changed.

Full list of similarities and differences is rather long. Let me summarize it.

Xvc has different commands; xvc file track is used instead of dvc add. Xvc doesn’t add files (like .dvc files) to your repository and keeps all metadata tracking under the .xvc directory. Checkout method is per-file, not configured globally, so you can keep track of your data directory with symlinks and your model directory as copies. Xvc uses BLAKE3 as the default hashing algorithm, and you can configure this to be BLAKE2, SHA-2, or SHA-3.

Pipelines are not defined using YAML files. You can write a shell script with xvc pipeline step ... or use Python xvc.pipeline().step().dependency(step_name="preprocess", param="hyperparams.yaml::batch_size") to define pipelines first. Then you can use xvc pipeline export and xvc pipeline import to modify the pipeline in YAML, JSON, or TOML.

There are more dependency options; e.g., a pipeline step may depend on a text file partially, by regex or by the line options. There is a generic dependency option; the output of a shell command can be used as a dependency to a step.

Remote storage options are rather limited for Xvc; local, ssh+rsync, and S3-compatible storages are supported for now. There is also a generic storage option where you can define upload and download commands for the tool you’re using, e.g., rclone or s5cmd, and Xvc can use it. I’ll add Azure and rclone as natively supported storage options eventually, but I don’t like the idea of keeping credentials, so there won’t be any OAuth-required storage options, e.g., Google Drive. (You’ll be able to use these through rclone, though.) All Xvc storages use environment variables for authentication.

Xvc doesn’t have experiment tracking yet. You can use --from-ref and --to-branch options to store artifacts from the pipeline to different branches. I’ll add features to run pipelines and commands quickly and compare these eventually (I need one too), but it may take some time.

Xvc doesn’t track anything about the user. It shouldn’t make any network connections if you don’t specifically ask it to do so. I’m planning to add binaries that only do file operations or pipeline operations, so if someone doesn’t need pipeline features, they will simply use xvc-file.

I tried the same approach when testing Python bindings with Pytest:

def test_pipeline_step_dependency(empty_xvc_repo):
    dep_help = empty_xvc_repo.pipeline().step().dependency(help=True)
    expected = """
Usage: xvc pipeline step dependency [OPTIONS] --step-name <STEP_NAME>

Options:
  -s, --step-name <STEP_NAME>
          Name of the step to add the dependency to
"""
    assert dep_help == expected

This doesn’t work because the help text is generated by clap and skips the usual thread-based output handler. All command output and errors in Xvc are returned as strings from the command, except for the help text that’s generated by clap automatically.

There are probably workarounds for this, but I won’t pursue them further as I don’t test the actual functionality in the Python bindings anyway.

The single-loader allows only one instance of Xvc to be run in a single process. This is no problem for the shell, but it looks like it won’t be possible to use multiple Xvc instances in a single Python process.

It’s possible to overcome this with an elaborate multiprocessing setup in the wrapper, but I won’t bother with it for now.

I replaced the eprintln!s with println!, but it didn’t make any difference. Maybe we should remove those statements completely.

I can’t really find the place that kills the kernel. The last command to run is xvc file list:

print(xvc_test_data.file().list("test-data/dir-0002"))

and the output it produces is:

[src/output.rs:144:13] &output_str = "SS         131 2024-06-05 08:57:11 41e16be7          test-data/dir-0002/file-0003.bin\nSS         131 2024-06-05 08:57:11 27f0
efd0          test-data/dir-0002/file-0002.bin\nSS         131 2024-06-05 08:57:11 66de5084          test-data/dir-0002/file-0001.bin\nTotal #: 3 Workspace Size:
      393 Cached Size:        6006\n"

print may be causing the crash, but the more likely cause is the command that comes after this:

!ls -l test-data/dir-0001/

I replaced this with lsd, which also failed. Maybe it’s actually a Python crash or bug.

The way to understand is to create a notebook file with only that cell and try to run it.

The ls line runs fine with a new notebook. It even runs on the README file when run at the beginning. The line that makes the kernel crash is:

xvc_test_data.storage().new_s3(name="backup", bucket_name="xvc-test", region="eu-central-1", storage_prefix="xvc-storage")

We can start by removing the new_s3 part.

The storage() method runs fine. It returns an XvcStorage() object, as it should.

When I run storage().list(), it takes a very long time. The bug is likely related to storage().

It looks like the storage object was adding file instead of storage as a subcommand. I’ve fixed it now.

That was the bug. The README notebook now creates the S3 storage.

What was the reason behind this?

Parsing the CLI to the XvcCLI object was perhaps the culprit. Let’s look at it more clearly.

Let’s try xvc file new s3 as a command to see how it behaves.

It says unrecognized subcommand for new.

This is how it should be, but I wonder why it doesn’t work for the XvcCLI parser.

Anyway, it’s already 13:00, so let’s stop here for today.

To make debugging easier, we can add file deletions in the ./start-readme script to the notebook itself.

Instead, I’ll run my watcher run-after-commit.sh to ensure that the files are deleted after commits.

That may also work; you can use both as well.

I noticed cli-opts pass --no-system-config etc. by default. Let’s deal with this first.

I fixed that.

I’m testing whether we are in the directory that we should be in with xvc.root("--absolute"), but it looks like it’s not possible to pass a string argument to the command.

Let me take a look at this.

It looks like there have been some changes in optional parameter handling in PyO3. You may need to deal with keyword arguments with decorators.

Now, let’s try xvc.file().track() once more with dir-0001/.

It seems to be working now. With an existing directory, it doesn’t show an error.

What does xvc.file().list() show?

It shows a single string with \n in it. It looks like we need to handle this in the output thread.

I added a replace("\\n", "\n") to output_str at the end, but it didn’t make a difference. I tested in the notebook with list_result.split("\n"), and these \n characters are indeed CRLF. So Jupyter shows CRLF in strings with \n, and this is not something we should try to fix, I think.

You can search how to show \n characters in a Jupyter notebook with CRLF.

The output string should be fed into repr, as in:

# Define a string with CRLF characters
text_crlf = "Hello\r\nWorld\r\nThis is a test string."

# Use repr to show the \r\n characters explicitly
print(repr(text_crlf))

I tested this, and GPT misleads. repr is when you want to show \n, not vice versa. When I print(list_result), it prints the results properly.

It looks like we don’t need to make this a priority now. We can tell the user in the notebook that the commands are intentionally returning strings, and they can process or print them however they want.

The issue with having a mutable element is that we need to update XvcRoot from Arc<XvcRootInner> to Arc<RwLock<XvcRootInner>>, which will cause almost all machinery around XvcRoot to require locking the object first. This is too large a refactoring.

However, I can write an xvc_root! macro to replace xvc_root.read() or whatever is required to minimize the code changes. XvcRoot can also be a wrapper object, but that’s too much fuss, and the responsibility may be misplaced.

Let’s update the type and let the dispatcher update XvcRootInner only when necessary. Let’s see what will require updating.

We have a bug in the Python bindings where xvc.file.track() enters an infinite loop.

There can be multiple reasons, but are you sure that the xvc CLI works with the same command?

Let’s start testing by creating another notebook server.

The notebook creates an Xvc repo successfully and returns the root with xvc.root(), but xvc.file().track("test-data/dir-0001/") never finishes.

It’s likely that this is caused by something in the background threads.

The CLI command completes successfully, but we may be passing the command incorrectly. Let’s try dir-0001 only.

Even if an incorrect path is provided, it shouldn’t enter an infinite loop.

When I provide the correct path, dir-0001, it returns. There may be something going on with finding the root of the Xvc repo.

It returns, but it’s also giving an error that it cannot find a repository. I should take a closer look.

The Xvc struct wasn’t implementing Debug, so I added it.

Let’s add it to others, XvcFile, etc.

Added it. Recompiling to get more info when we run track with an incorrect path.

You can also make it run after a commit to avoid starting a new server if one is already running.

Added conditionals, and now it runs the server if none is running in the background.

PORT=7979
if [[ ! -s "$(ps ax | rg -v ps | rg jupyter-lab | rg $PORT)" ]] ; then
  jupyter lab --port=$PORT --notebook-dir=Readme/ &
  open http://localhost:${PORT}/lab/workspaces/auto-H/tree/Readme.ipynb
fi

I saw an ad on Stack Overflow Jobs those days about employment by contributing to open source projects. I applied to that. A few weeks later, the CTO of [iterative.ai] got in touch and I started working on DVC documentation. Initially on a per-hour basis, and after May 2021, as a full-time employee.

Initially, I liked the tool we were building very much. The team was awesome. (Still, they are.) It was one of the best periods of my life, especially in my turbulent still-ongoing-divorce-period pressures. I know I will always miss them.

My job was learning DVC, documenting it, and helping newcomers grasp it easily. It was a fun job. Until then, I didn’t see myself as a technical writer. English is not my native tongue and I never have lived in an English-speaking country. Nevertheless, I think I wasn’t too bad at it.

When I was first learning the tool, I began to use it everywhere. I was an avid user of Git Annex once. DVC looked better. I don’t remember why I lost interest in Git Annex after many years, but it was probably related to symbolic links not working on Windows (or on Termux). DVC had multiple ways of connecting the cache and the files in the workspace, including hardlinks and copy, so it was a breath of fresh air for me.

I began to use it for my large collections. Keeping track of my binary files in Git was something I always desired. Git is the least sucking version control system among the ones I used previously (SVN, hg, darcs…) and I’d rather keep using it everywhere rather than learning new tools for binary files.

After some time I began to use the tool for my personal file collections. I noticed its performance became a burden. I was tracking maybe a few gigabytes of files with it and basic file operations became slower as I added more. I noticed I was becoming distracted after I wrote a dvc command. It took some time to confess that the tool I liked once and was earning my salary with was not a tool that I liked to use.

I don’t know what real professionals would do at this point. I never had a good LinkedIn profile. When I met a similar problem with the example repository that’s supposed to contain 70,000 small files, I brought the issue forward. I wrote a shell script that was basically doing the same thing as dvc add and it worked much faster than the actual command. The shell script was naïve and I thought DVC must have at least that level of speed. It didn’t. Simply calling md5sum on files and copying them to appropriate location in .dvc/cache was way faster. How could this be?

I had cursory observations on the codebase. I know some decisions (like a large central class that connects everything, separate .dvc files for each tracked file) that may lead to degradation. Although I don’t see it as the problem, Python was also not helpful. These are rough observations.

It was September 2021. I was also teaching myself Rust. I wrote an email to the CTO and CEO of the company to request a sabbatical to work on DVC. My plan was to rewrite certain portions (or commands) in Rust and wrap them with PyO3. It could fail. So to have skin in the game, I said I’ll work for free during this time and if I fail to make DVC faster for some reason, I’ll return to my writing position.

They didn’t accept. I didn’t try to persuade them. The decision was rational and although I’d say go ahead and see what happens if I were in their shoes just to make my employee happy, they aren’t crazy-managers as I once was. Probably there are many factors that I’m not aware of. I returned to my post and continued to write documentation for another 9 months. In the meantime I studied Rust and thought about how I could architect a similar tool. Where does DVC go wrong?

In April 2022, I informed the CTO that I’d like to take a sabbatical for my book. I have a political-SF book and after the Ukrainian war started with a (albeit minor) probability of nuclear attack on the other shore of Black Sea, I thought it’s not a time to work on something I stopped liking. My performance in the last quarter was also not something I was proud of. I didn’t feel good.

When I retired to sabbatical in July though, while writing the book, I thought writing the software that I wanted to see was also something on my mind before nuclear war. I had notes about the architecture I was planning. I wanted to see if I could apply an Entity-Component System to this basic problem, without any Object-Oriented conceptions. I believe it looks cool. I’m still simplifying and testing the idea, and it looks better to my mind than mixing data and functions for no reason.

After I made the repository public, I resigned from Iterative.

In a sense, Xvc owes its existence to DVC, and the name is a tribute to this. I hope they squash their bugs, and improve their user experience, and be a long-term player in the crowded market they are in. I don’t intend to be a “competitor”, because I prefer being developer/architect rather than a “technical steward to VC money”, and the license of Xvc is GPL-3 to signal this.

The Xvc command line interface, however, is as different as it can be from DVC. The command names are different; DVC has commands similar to Git (push, fetch, pull, commit), while Xvc tries to be different from Git to reduce the user’s mental load. For example, as a writer, I noticed that “Git remotes” and “DVC remotes” was confusing, so I called them “Xvc storages”. DVC calls the units of a pipeline stages; the same concept is called steps in Xvc, because stage in Git is something completely different.

Internally, the architecture is also very different. Xvc uses serialization (with serde) instead of YAML. It can export/import pipelines from YAML (or JSON), but YAML is not as central as in DVC. (I believe YAML is overused in our industry, and it’s an employment guarantee for another generation of developers but there is better work than keeping up a half-baked configuration format.) Xvc doesn’t keep its artifacts in the user’s workspace (except .xvcignore files). They are all stored in the .xvc/ directory. The DVC way of doing things makes merging .dvc files easier. To overcome the problems caused by merging large metadata files, Xvc keeps track of events and replays them to get the final state of the repository. All metadata storage and retrieval operations revolve around the XvcStore<T> struct in Xvc. Typically, if the user runs an xvc command, only the updated store events (added files, changed pipelines, etc.) are stored. There are optimizations in this front, but I profile first and optimize later.

Algorithms for data digests are configurable; by default Xvc uses Blake3, but it is configurable to use SHA2-256, SHA3-256, or Blake2s. It can be modified to use any 256-bit digest quickly. There are some features that are not found in DVC, and more will come. So, although I’m solving a similar problem, Xvc is not “DVC rewritten in Rust,” it’s a different tool completely.

Currently, it doesn’t have as much eye candy as DVC. In time, I plan to add Python, Julia, and R APIs, notebook integration, experiment tracking (without relying on Git internals), data labeling and filtering, and other MLOps features. I’m building with a goal to make these features available without making the rest of the software slower.

I’ve found the tool I was looking for to track my kids’ photos and Ottoman OCR datasets in a Git repository. I’m tracking more than 1TB of files in a single repository with Xvc and adding another 10TB looks feasible now.

Can't locate IPC/Cmd.pm in @INC (@INC contains: /home/runner/work/xvc.py/xvc.py/target/x86_64-unknown-linux-gnu/release/build/openssl-sys-844a96d66ae533b1/out/openssl-build/build/src/util/perl /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /home/runner/work/xvc.py/xvc.py/target/x86_64-unknown-linux-gnu/release/build/openssl-sys-844a96d66ae533b1/out/openssl-build/build/src/external/perl/Text-Template-1.56/lib)

The issue seemed to be a missing Perl package. Building rust-openssl now appears to require the perl-core package.

I added apt-get update && apt-get install perl-core to the CI configuration, but that didn’t work.

However, as this GitHub issue comment suggests, Maturin with its manylinux support uses different kinds of Docker containers to build the packages. We must detect whether it is a CentOS or Debian-based container and install the missing packages accordingly.

The Building wheels step in the configuration should look similar to:

- name: Build wheels
  uses: PyO3/maturin-action@v1
  with:
    target: ${{ matrix.target }}
    manylinux: auto
    args: --release --out dist
    before-script-linux: |
      # If we're running on RHEL/CentOS, install needed packages.
      if command -v yum &> /dev/null; then
          yum update -y && yum install -y perl-core openssl openssl-devel pkgconfig libatomic

          # If we're running on i686, we need to symlink libatomic
          # in order to build openssl with the -latomic flag.
          if [[ ! -d "/usr/lib64" ]]; then
              ln -s /usr/lib/libatomic.so.1 /usr/lib/libatomic.so
          fi
      else
          # If we're running on a Debian-based system.
          apt update -y && apt-get install -y libssl-dev openssl pkg-config
      fi

This will run the specified script before executing the Maturin build command in the container, ensuring all missing packages are installed.

Note that there may be thousands of dependencies (e.g., files for a step), and creating these channels beforehand is not feasible.

The Crossbeam library has Select which allows waiting for an arbitrary number of channels. Now, I need to figure out how to structure the threads and update their states. A minor task! 😆

I first tried to use the rust-fsm library, but it became apparent that Xvc pipeline steps’ states are tied to XvcRoot. That is, if we are to check the presence of a file or the value of a parameter, we have to do it relative to the repository root. The root directory should be taken into consideration in every transition.

I checked the code and noticed that the FSM is actually very simple. I copied it, added &XvcRoot to the transition and output functions in the trait definition, and implemented it for XvcOutput, XvcDependency, and XvcStep.

These are the constituents of a pipeline. A pipeline is composed of an XvcStep that defines a command, and each step can have multiple XvcDependency and XvcOutput definitions. For each of these structs, I’ve added fields that represent their current state.

For example, an XvcOutput can be Missing, Found, Old, or Ok. An XvcStep that produces this XvcOutput doesn’t check the dependency content hash if an output is missing. However, if an XvcOutput is Found, the XvcStepStateMachine checks the XvcDependency states and their modification times to see if they have changed since the output was generated. An XvcStep is invalidated when an XvcOutput is Missing or an XvcDependency has changed after the last command run.

Unlike DVC, I added the ability for an XvcStep to depend on other XvcSteps. They communicate through outputs. I’ve added XvcDependency::Step(XvcStep) to the XvcDependency definition. I’m also planning XvcDependency::Pipeline(XvcPipeline) to allow steps to depend on other pipelines, so that pipelines can be run in order.

Currently, the following are included as XvcDependency:

• File: A (binary or text) file in the repository. If the metadata (size and modification time) or the content changes, the dependent step becomes invalidated.
• Directory: A directory that contains files. If a file is added to or removed from the directory, or any of the files are changed, the associated step becomes invalidated.
• Glob: A glob such as my-data/*.png. If the list of files changes or their content has changed, the associated step becomes invalidated.
• Parameter: Xvc can parse YAML, TOML, and JSON files and get the values of variables. It’s possible to define these (hyper)parameters as dependencies.
• URL: An HTTPS URL, which is checked first by metadata and then by content to see whether it has changed.
• Step: A previously defined step; if it’s invalidated, the depending step also becomes invalidated.

Additionally, I’m planning to add the following items to Xvc as dependencies:

• Lines {path, begin, end}: Lines in a text file. This can be used for general-purpose input tracking. If the given lines in a file are changed, the dependent step becomes invalidated.
• Regex {path, regex}: If the regular expression result on the file changes, the dependent stage becomes invalidated.
• Pipeline { name }: If any of the steps in a pipeline is invalidated, the pipeline is also invalidated, or the step that depends on this pipeline becomes invalidated.

Each of these dependencies is checked minimally; that is, when their size on disk is detected to have changed, they are considered changed without checking the content hash. It needs a very detailed state machine to track the changes without bugs.

I’ve noticed that if I can write such a state machine, most of the I/O operations can be done in parallel. If two steps do not depend on each other in the dependency graph, they can be run in parallel. The state machine’s granularity allows this.

🐢 I think I can implement Rsync today. Looking at ssh2-rs, though, I think we can implement file transfer without relying on rsync. It might be easier to implement everything within the code.

🐇 Then you should rename the issue to new ssh.

🐢 Fair. There is also the ssh_rs crate, but it doesn’t have full support for the protocol. Instead, we can have another command, like xvc storage new ssh, that uses libssh2 via the crate mentioned above. It has some limitations with OpenSSH on macOS.

🐇 From the crate’s README, it looks like you can enable the vendored-openssl feature to compile it statically.

🐢 Let’s go ahead then. It’s better to compile it behind a feature flag, though.

🐇 Yup. Rsync can be separate. I think for now you can implement rsync via Exec::cmd and make new ssh a new issue.

🐢 I’ll copy this conversation there.

🐇 Now, let’s start implementing rsync.

🐢 Do we really want to hide it behind a feature flag? It doesn’t bring any extra complexity to generic, for example—just using the commands and returning the errors.

🐇 I think so. If the user doesn’t have rsync on their system, they’ll just get errors. We don’t need to make the implementation optional, but the tests might be.

🐢 OK.