Devlog 1: XVC Root and Python Bindings Debugging

I should have a dispatch method that receives an XvcRootOpt and runs a command with it. The dispatcher can also update the XvcRoot from None to Some(XvcRoot) in some cases, so it should receive a mutable XvcRootOpt or return one after receiving ownership.

The issue with having a mutable element is that we need to update XvcRoot from Arc<XvcRootInner> to Arc<RwLock<XvcRootInner>>, which will cause almost all machinery around XvcRoot to require locking the object first. This is too large a refactoring.

However, I can write an xvc_root! macro to replace xvc_root.read() or whatever is required to minimize the code changes. XvcRoot can also be a wrapper object, but that’s too much fuss, and the responsibility may be misplaced.

Let’s update the type and let the dispatcher update XvcRootInner only when necessary. Let’s see what will require updating.

We have a bug in the Python bindings where xvc.file.track() enters an infinite loop.

There can be multiple reasons, but are you sure that the xvc CLI works with the same command?

Let’s start testing by creating another notebook server.

The notebook creates an Xvc repo successfully and returns the root with xvc.root(), but xvc.file().track("test-data/dir-0001/") never finishes.

It’s likely that this is caused by something in the background threads.

The CLI command completes successfully, but we may be passing the command incorrectly. Let’s try dir-0001 only.

Even if an incorrect path is provided, it shouldn’t enter an infinite loop.

When I provide the correct path, dir-0001, it returns. There may be something going on with finding the root of the Xvc repo.

It returns, but it’s also giving an error that it cannot find a repository. I should take a closer look.

The Xvc struct wasn’t implementing Debug, so I added it.

Let’s add it to others, XvcFile, etc.

Added it. Recompiling to get more info when we run track with an incorrect path.

You can also make it run after a commit to avoid starting a new server if one is already running.

Added conditionals, and now it runs the server if none is running in the background.

PORT=7979
if [[ ! -s "$(ps ax | rg -v ps | rg jupyter-lab | rg $PORT)" ]] ; then
  jupyter lab --port=$PORT --notebook-dir=Readme/ &
  open http://localhost:${PORT}/lab/workspaces/auto-H/tree/Readme.ipynb
fi

devlog 2

We were debugging Python bindings for Xvc. xvc.file.track enters an infinite loop when given a non-existent path.

To make debugging easier, we can add file deletions in the ./start-readme script to the notebook itself.

Instead, I’ll run my watcher run-after-commit.sh to ensure that the files are deleted after commits.

That may also work; you can use both as well.

I noticed cli-opts pass --no-system-config etc. by default. Let’s deal with this first.

I fixed that.

I’m testing whether we are in the directory that we should be in with xvc.root("--absolute"), but it looks like it’s not possible to pass a string argument to the command.

Let me take a look at this.

It looks like there have been some changes in optional parameter handling in PyO3. You may need to deal with keyword arguments with decorators.

Now, let’s try xvc.file().track() once more with dir-0001/.

It seems to be working now. With an existing directory, it doesn’t show an error.

What does xvc.file().list() show?

It shows a single string with \n in it. It looks like we need to handle this in the output thread.

I added a replace("\\n", "\n") to output_str at the end, but it didn’t make a difference. I tested in the notebook with list_result.split("\n"), and these \n characters are indeed CRLF. So Jupyter shows CRLF in strings with \n, and this is not something we should try to fix, I think.

You can search how to show \n characters in a Jupyter notebook with CRLF.

The output string should be fed into repr, as in:

# Define a string with CRLF characters
text_crlf = "Hello\r\nWorld\r\nThis is a test string."

# Use repr to show the \r\n characters explicitly
print(repr(text_crlf))

I tested this, and GPT misleads. repr is when you want to show \n, not vice versa. When I print(list_result), it prints the results properly.

It looks like we don’t need to make this a priority now. We can tell the user in the notebook that the commands are intentionally returning strings, and they can process or print them however they want.

devlog 3

Let’s begin this session by removing debug statements from both the Xvc library and the Python bindings.

Another issue is the restart script. The if in that script that restarts the server doesn’t work; it always restarts the notebook server.

I removed some println! statements from xvc.py. There doesn’t seem to be anything in the library related to outputs.

Let’s search for how to check if a jupyter-lab command with the port 7979 runs in the background.

It looks like the bug is in the condition; it’s not -s, it’s -z.

Oops, yeah.

Let’s do a bit of tidying and check if the script is fixed.

I have a clippy warning with a new function that says these usually don’t take self as a parameter. This is for the pipeline new command, and it receives a self as an XvcPipeline object. It seems best to turn off the clippy warning for this.

I allowed two clippy warnings, and the script seems to work fine.

Let’s go on to copying the content from the Xvc README to the notebook.

There is an issue with the run-after-commit script. When Xvc commits the changes, the command we give is run again.

The issue is that git initializes the directory in the test-data/ directory, while xvc works in the current directory. I think we can either git init in the current directory or xvc init in test-data.

We’re already deleting .git and .xvc directories in the start-readme script. I think it may be easier to update git init to just initialize in the current directory.

Yep, let’s do it that way.

I see there are still extra outputs from the commands. We need to deal with this first.

I removed them, but there are still pink outputs. These are from dbg! statements, it looks like, or we’re initializing the output thread incorrectly.

Fixed those as well. I wrote up the xvc file list command examples as well. Now we have issues with xvc storage new s3 not running, and not even showing any debug output. We’ll deal with it in the next devlog, though.

devlog 4

Let’s start by looking at the debug output issue. We can start by replacing the eprintln! macros with println!, perhaps.

I replaced the eprintln!s with println!, but it didn’t make any difference. Maybe we should remove those statements completely.

I can’t really find the place that kills the kernel. The last command to run is xvc file list:

print(xvc_test_data.file().list("test-data/dir-0002"))

and the output it produces is:

[src/output.rs:144:13] &output_str = "SS         131 2024-06-05 08:57:11 41e16be7          test-data/dir-0002/file-0003.bin\nSS         131 2024-06-05 08:57:11 27f0
efd0          test-data/dir-0002/file-0002.bin\nSS         131 2024-06-05 08:57:11 66de5084          test-data/dir-0002/file-0001.bin\nTotal #: 3 Workspace Size:
      393 Cached Size:        6006\n"

print may be causing the crash, but the more likely cause is the command that comes after this:

!ls -l test-data/dir-0001/

I replaced this with lsd, which also failed. Maybe it’s actually a Python crash or bug.

The way to understand is to create a notebook file with only that cell and try to run it.

The ls line runs fine with a new notebook. It even runs on the README file when run at the beginning. The line that makes the kernel crash is:

xvc_test_data.storage().new_s3(name="backup", bucket_name="xvc-test", region="eu-central-1", storage_prefix="xvc-storage")

We can start by removing the new_s3 part.

The storage() method runs fine. It returns an XvcStorage() object, as it should.

When I run storage().list(), it takes a very long time. The bug is likely related to storage().

It looks like the storage object was adding file instead of storage as a subcommand. I’ve fixed it now.

That was the bug. The README notebook now creates the S3 storage.

What was the reason behind this?

Parsing the CLI to the XvcCLI object was perhaps the culprit. Let’s look at it more clearly.

Let’s try xvc file new s3 as a command to see how it behaves.

It says unrecognized subcommand for new.

This is how it should be, but I wonder why it doesn’t work for the XvcCLI parser.

Anyway, it’s already 13:00, so let’s stop here for today.

devlog 5

While writing Xvc tests for Python, I hit an error caused by the ECS single-load protection.

The single-loader allows only one instance of Xvc to be run in a single process. This is no problem for the shell, but it looks like it won’t be possible to use multiple Xvc instances in a single Python process.

It’s possible to overcome this with an elaborate multiprocessing setup in the wrapper, but I won’t bother with it for now.

devlog 6

I have a habit of testing against the CLI’s help string output. It allows me to keep the documentation up to date and makes me aware of any undocumented options. When new features are added, the help text changes and the test fails, which prompts me to add those options to the documentation.

I tried the same approach when testing Python bindings with Pytest:

def test_pipeline_step_dependency(empty_xvc_repo):
    dep_help = empty_xvc_repo.pipeline().step().dependency(help=True)
    expected = """
Usage: xvc pipeline step dependency [OPTIONS] --step-name <STEP_NAME>

Options:
  -s, --step-name <STEP_NAME>
          Name of the step to add the dependency to
"""
    assert dep_help == expected

This doesn’t work because the help text is generated by clap and skips the usual thread-based output handler. All command output and errors in Xvc are returned as strings from the command, except for the help text that’s generated by clap automatically.

There are probably workarounds for this, but I won’t pursue them further as I don’t test the actual functionality in the Python bindings anyway.

devlog 7

I noticed that I often forget to update the Xvc CHANGELOG. To fix this, I added a pre-push hook that checks the files I’m pushing. If the CHANGELOG is not among them and there are changes to Rust files, it prevents the push. I hope this will remind me to update the logs more frequently.

#!/bin/bash

# Git pre-push hook to check if CHANGELOG.md is included in the push and if the branch is develop

# Get the current branch name

current_branch=$(git rev-parse --abbrev-ref HEAD)

# Check if the branch is develop

if [ "$current_branch" == "main" ]; then
	echo "You are on the main branch. Skipping CHANGELOG.md check."
	exit 0
fi

# remote="$1"
# url="$2"

# Get the list of commits to be pushed
commits=$(git rev-list '@{u}..HEAD')

has_rust_files=$(false)

# TODO: We can iterate to get file list only once
# Get the list of files that are going to be pushed
for commit in $commits; do
	if git diff-tree --no-commit-id --name-only -r "${commit}" | grep -q "\\.rs$"; then
		has_rust_files=$(true)
	fi
done

if [[ ! $has_rust_files ]]; then
	echo "No .rs files in the push, no need to check CHANGELOG"
	exit 0
fi

# Check if CHANGELOG.md is among the files in the commits
for commit in $commits; do
	if git diff-tree --no-commit-id --name-only -r "${commit}" | grep -q "CHANGELOG.md"; then
		echo "CHANGELOG.md is included in the push."
		exit 0
	fi
done

echo "ERROR: CHANGELOG.md is not included in the push."
exit 1

Edit (2025-02-08): Added a check for when only code files are changed.

devlog 8

🐢 The only failure was the patch coverage in Xvc. Let’s see what needs to be done.

🐇 It looks, from the coverage page, that our additions to HStore don’t have any tests. Maybe we can add some unit tests to the new joins.

🐢 I don’t find unit tests particularly useful, but let’s use GitHub Copilot to add some for us.

🐇 I added a unit test and a doc test for full_join. I think doc tests have more value; they provide documentation, and we can readily see how to use a function from its docs. It’s better to increase coverage with doc tests.

🐢 There are things I should keep in mind while writing doc tests. Imports must use the full path, not crate. Also, the tested struct doesn’t have implicit imports.

🐇 The ceremony for adding keys and values is a bit too much. It might be worthwhile to add an insert method for anything that implements Into<XvcEntity>.

🐢 That would certainly save time—no more typing .into() for each key! :)

🐇 Pushed to test again. Should we have some way to test coverage locally?

🐢 I don’t think we need to check coverage locally. It’s not worth our time right now.

🐇 Now, while waiting for the tests to complete, what else can we do?

gh -R iesahin/xvc run list
completed failure v0.6.13 Rust-CI v0.6.13 pull_request 12543831360 3m54s 2024-12-30T08:15:06Z
...

🐢 That didn’t take too long. Let’s view the results:

gh -R iesahin/xvc run view 12543831360
...
  X Run Current Dev Tests
...
To see what failed, try: 
View this run on GitHub: https://github.com/iesahin/xvc/actions/runs/12543831360

🐢 The current dev tests are failing for some reason. Let’s run them locally.

🐇 We’re missing llvm-tools-preview locally. How do I install this?

🐢 The command is rustup component add llvm-tools-preview:

info: component 'llvm-tools' for target 'aarch64-apple-darwin' is up to date

🐇 It’s already installed. We just need to set the environment variables.

🐢 Instead, we can just turn off dev tests for the time being. We don’t really need them; our local tests pass.

🐇 Yeah, okay. We don’t need to solve every single bit of these issues right now.

gh -R iesahin/xvc run list
...

🐇 Okay, let’s take a look at the run again.

gh -R iesahin/xvc run list
completed failure v0.6.13 Rust-CI v0.6.13 pull_request 12544020064 3m39s 2024-12-30T08:33:25Z

gh -R iesahin/xvc run view 12544020064
...
  X Test and Coverage
...

gh run view 12544020064 --log-failed
...
Test and Coverage (stable) Test and Coverage 2024-12-30T08:36:58.8911690Z Error: ProcessError { stdout: "", stderr: "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to e1.xvc.dev port 80 after 160 ms: Couldn't connect to server\n" }
...

🐢 We need to start Nginx on the server. We forgot to do that yesterday.

🐇 Ah, right. After that dufs installation. Okay.

🐢 We also need to add a reverse proxy for dufs somehow, but that’s for later.

🐇 For this use case, I don’t think it’s necessary. We can just adjust the port to a non-standard one if we need 443 for something else. Let’s look at the tests again.

🐢 Let’s add another doc test, this time to XvcStore.

cargo test -p xvc-ecs --doc
...
test result: ok. 8 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.87s

🐇 Sent the files again. Waiting for the tests to finish.

🐢 Let’s check the keymaps file in the meantime.

🐇 I tried reading the documentation but didn’t see an error. Maybe we should just set it to non-lazy and disallow remaps.

🐢 We’ve already spent too much time on this.

🐇 Yes, let’s check the tests again.

gh -R iesahin/xvc run list
completed success v0.6.13 Rust-CI v0.6.13 pull_request 12544291068 8m10s 2024-12-30T09:00:42Z
...

🐢 Oh, nice, the merge is ready.

🐇 Patch coverage is still behind the target, though.

🐢 Yeah, but let’s release this one and ensure the next one is better covered. Also, I’m not sure if the doc tests actually affected the coverage report.

🐇 If we look at the coverage page again, we can see.

🐢 It seems codecov.io doesn’t consider coverage for doc tests. That’s a bit weird, but let’s not spend more time on it.

🐇 Sure, let’s merge.

🐢 I think we forgot to bump the versions in the Cargo.toml files. We’ll have to do that in main.

🐇 Oh, yeah. Let’s bump them and tag the release as well.

🐢 Now we can wait for all the files to be produced. What’s next?

🐇 We can release the Python version too. It shouldn’t need any changes.

🐢 Right. Maybe we can add a few tests there as well.

🐇 Let’s bump the version first and see.

🐢 I’ve bumped the versions in Cargo.toml and run maturin develop.

🐇 It seems ready now.

🐢 The main branch is failing, though. The “Publish crates” action is looking for libsqlite3. Let’s investigate.

gh -R iesahin/xvc run list
completed failure Release v0.6.13 Publish Crates v0.6.13 push 12544753359 6m1s 2024-12-30T09:41:56Z

🐢 The issue is that the VM doesn’t have libsqlite3-dev. Let’s add it.

🐇 We need to restart the job manually. Let’s skip tagging this time.

🐢 Some of the packages were already published, and now they’re causing failures because crates.io says they already exist. Maybe we can check if a package is already published before trying.

🐇 Let’s see if we can make cargo publish more forgiving.

🐢 There doesn’t seem to be an easy option. Let’s search for “how to skip published packages in workspace to avoid errors with cargo publish.”

🐇 Claude is hallucinating again. Let’s try a manual approach: how to skip already published packages?

🐢 It might be easier to just add a check. How do we get that info?

🐇 Or we can just move on to the next package if one is already available.

🐢 Let’s push the missing packages manually this time.

🐢 We should have a key to open garden files quickly. What does Fzf-Lua files receive as arguments?

🐇 Let’s check the help page: fzf-lua.

🐢 Before that, maybe we can try to fix why selected lines are not searched in Visual-Line mode?

🐇 Yep, let’s fix our search first.

🐢 How do you set a key in visual line mode in Neovim Lua?

🐇 The abbreviation is V. Let’s try that.

🐢 Looks like we need to restart the session.

🐇 It’s still not working. Let’s check :map.

🐢 It seems it needs more care; I’ll check it later.

👨🏾‍🦲 🐢 Bence kendini biraz daha anlamlı bir işle uğraştırmalısın. 🐇 Ne gibi? 🐢 Belki biraz daha yayın yapmalısın, biraz daha işe başvurmalısın. 🐇 “Meli”, “malı” ile geçiyor ömrümüz.

⌚ 🐢 It looks like we can move most of the daily template to links or commands. We can call it ref/daily. My idea is to fill the page intentionally, without templates.

🐇 It might be better to start with a blank page, yeah. The template makes me a bit nervous. There are too many things to fill in, and most of them aren’t things I like being pushed into doing.

🐢 Let’s start by moving the daily templates to ref/daily. No more daily template.

🐇 Now we can delete the rest of this page. We’ll use the daily links page and maybe have reminders at the end of these sessions.

👨🏽‍⚕️

devlog 9

🐢 Today I’m planning to add cross-compilation to the Xvc 0.6.13 branch to provide more platform support.

🐇 It looks like you first need to turn off this ghost text from the blink output. It makes writing insufferable.

🐢 Yep, let’s do that first.

🐇 Now, let’s restart Neovim.

🐢 I don’t know why blink.cmp doesn’t prioritize the emojis I use. Maybe we can just use #tor and #rab for ourselves.

🐇 I think over time it will learn that the emojis I defined in the snippets file should have higher priority, but let’s skip this for now. What do we need to do to add cross-compilation?

🐢 Maybe we can just make the completion menu wait a bit longer. It shows up almost instantly, and I want it to wait for a few more milliseconds.

🐇 Okay, let’s look at the config.

🐢 The configuration file doesn’t seem to have a key for this. Let’s search: blink.cmp.

🐇 I think the culprit is typo resistance; it causes better options to be pushed down: https://cmp.saghen.dev/configuration/reference#fuzzy

🐢 Let’s turn that off.

🐇 There is an error in the configuration file. I don’t know why it’s failing.

🐢 I added emojis to Espanso and checked the error message. The configuration I copied from their docs seems to be broken. The message is:

...share/nvim/lazy/blink.cmp/lua/blink/cmp/config/utils.lua:14: fuzzy.max_items: unexpected field found in configuration

I’ll just delete that line.

🐇 It still doesn’t prioritize snippets, but we can look into this later. Espanso seems to be a better tool for this anyway.

🐢 Yep. Let’s look into cross-compilation support for Rust.

🐇 The well-known option is cross.rs: https://github.com/cross-rs/cross

🐢 We can start with that. Installation is from the Git repository:

$ cargo install cross --git https://github.com/cross-rs/cross
    Updating git repository `https://github.com/cross-rs/cross`
    Updating git submodule `https://github.com/cross-rs/cross-toolchains.git`
  Installing cross v0.2.5 (https://github.com/cross-rs/cross#4090beca)
...
   Installed package `cross v0.2.5 (https://github.com/cross-rs/cross#4090beca)` (executables `cross`, `cross-util`)

🐇 Now we have the cross and cross-util commands. Cross-compilation requires Podman on Linux or Docker on macOS. Do we have Docker?

🐢 It looks like we don’t. Maybe we can just set up a remote build using Podman or GitHub Actions. I saw a crate for that yesterday; I remember saving it somewhere but can’t find it now. Searching again seems easier, which says something about my archival and retrieval habits.

🐇 Maybe later you can add some vector search capabilities to your archive—semantic search.

🐢 Yep, sometime later.

🐇 Now let’s search for “adding rust cross compilation to github actions.”

🐢 I found a link to the action: https://github.com/marketplace/actions/build-rust-projects-with-cross

Let’s look at the example:

jobs:
  release:
    name: Release - ${{ matrix.platform.os-name }}
    strategy:
      matrix:
        platform:
          - os-name: FreeBSD-x86_64
            runs-on: ubuntu-20.04
            target: x86_64-unknown-freebsd
            skip_tests: true

          - os-name: Linux-x86_64
            runs-on: ubuntu-20.04
            target: x86_64-unknown-linux-musl

          - os-name: Linux-aarch64
            runs-on: ubuntu-20.04
            target: aarch64-unknown-linux-musl

          - os-name: Linux-riscv64
            runs-on: ubuntu-20.04
            target: riscv64gc-unknown-linux-gnu

          - os-name: Windows-x86_64
            runs-on: windows-latest
            target: x86_64-pc-windows-msvc

          - os-name: macOS-x86_64
            runs-on: macOS-latest
            target: x86_64-apple-darwin

          # more targets here ...

    runs-on: ${{ matrix.platform.runs-on }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build binary
        uses: houseabsolute/actions-rust-cross@v0
        with:
           command: ${{ matrix.platform.command }}
          target: ${{ matrix.platform.target }}
          args: "--locked --release"
          strip: true
      - name: Publish artifacts and release
        uses: houseabsolute/actions-rust-release@v0
        with:
          executable-name: ubi
          target: ${{ matrix.platform.target }}

🐇 We can just convert the current configuration to this and see if it works.

🐢 Let’s do that. I created .github/workflows/release.yml and will update it.

🐇 Let’s check the results with gh:

$ gh -R iesahin/xvc run list
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525070531	34s	2024-12-28T08:17:28Z

🐢 It says the release has completed.

$ gh -R iesahin/xvc run view 12525070531

X v0.6.13 Release iesahin/xvc#263 · 12525070531
Triggered via pull_request about 3 minutes ago

JOBS
X Release - FreeBSD-x86_64 in 12s (ID 34936257653)
  ✓ Set up job
  ✓ Checkout
  X Build binary
  - Publish artifacts and release
  ✓ Post Build binary
  ✓ Post Checkout
  ✓ Complete job
...

Now let’s look at the failure:

$ gh -R iesahin/xvc run view 12525070531 --log-failed

🐇 Change the order and see what happens. Maybe we should first succeed in a non-cross-compilation build.

$ gh -R iesahin/xvc run list
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525178664	35s	2024-12-28T08:34:18Z
...
$ gh -R iesahin/xvc run view   12525178664 --log-failed
...

🐢 It’s the same error. The usage example is actually much more sophisticated.

🐇 The issue is that we’re asking for command from the matrix, but the matrix doesn’t define it. That’s the second time today an example from the documentation has failed. I’ve added command for each platform. We can also add separate features this way to make platform-specific functionality work.

🐢 Agreed. Let’s look at it once more.

$ gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525253982	41s	2024-12-28T08:46:23Z

$ gh -R iesahin/xvc run view 12525253982 --log-failed
...
Release - macOS-x86_64	Build binary	2024-12-28T08:46:44.8464010Z  [1m [31merror [0m [1m: [0m the lock file /Users/runner/work/xvc/xvc/Cargo.lock needs to be updated but --locked was passed to prevent this

🐇 Ah, that’s a different error. Let’s remove the --locked flag and retry.

$ gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525289176	1m8s	2024-12-28T08:53:23Z

🐢 We’re finally starting to get some good news. Now we’re getting OpenSSL errors. We need feature flags for these platforms or to specify where OpenSSL is. Let’s add bundled-openssl to the failed ones. We also need bundled-sqlite for Windows binaries.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525354179	3m9s	2024-12-28T09:04:55Z

🐇 It looks like the Changes.md file is missing, and it can’t upload the binaries as a release because of this. I’ll set the changes file to CHANGELOG.md.

🐢 It’s weird to fail because of that. I think we should report these errors and possibly send a PR to make that file optional.

gh -R iesahin/xvc run list | rg Release | head -n 1
in_progress		v0.6.13	Release	v0.6.13	pull_request	12525449368	1m38s	2024-12-28T09:18:40Z
...
✓ Release - macOS-x86_64 in 2m39s (ID 34937017695)
✓ Release - macOS-aarch64 in 2m42s (ID 34937017787)
..
ARTIFACTS
xvc-macOS-x86_64.tar.gz
xvc-macOS-arm64.tar.gz

🐢 It looks like Linux-riscv64 has OpenSSL compilation errors. I think we can skip this platform for now. The goal was to add aarch64 for macOS, and that seems to have succeeded.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525504079	3m24s	2024-12-28T09:26:43Z
...
X Release - Linux-x86_64	Build binary	2024-12-28T09:29:16.2262518Z  [0m [1m [38;5;9merror[E0308] [0m [0m [1m: mismatched types [0m
...

🐢 It looks like Linux-x86_64 doesn’t support reflinks. Maybe we can make reflinks an optional feature and add it specifically to macOS and Windows targets.

🐇 I’ve removed reflink from the default features. This is a breaking change, but fortunately we don’t have many users who will be affected by it.

Let’s check the results once more.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525597866	3m39s	2024-12-28T09:42:44Z

🐢 Now turn off the NetBSD target as well.

🐇 FreeBSD and Linux-x86_64 targets are building. Let’s see why the ARM Linux targets are failing.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525643284	4m46s	2024-12-28T09:52:04Z

🐇 It looks like those targets don’t have libsqlite3 installed. Let’s add bundled-sqlite to these targets too.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525734325	4m11s	2024-12-28T10:07:36Z

🐢 Adding Android as a target didn’t work. Let’s remove it for now; it seems to require more work.

🐇 I think we’ll eventually move to using Xvc to distribute binaries. We could build the Android binary on Termux and link it on the releases page or push it as a release artifact.

🐢 I need to learn more about GitHub releases. If we can add artifacts to the release, maybe we can do some of this work locally.

🐇 We can start by looking at the capabilities of gh commands.

🐢 It looks like it’s possible to upload assets to releases. Let’s check how that works.

gh release upload --help

🐇 So basically, we can just upload files to tags. We can list releases and work with them like anything else.

gh -R iesahin/xvc release list

And we can delete releases:

for r in v0.4.2-alpha.8 v0.4.2-alpha.7 v0.4.2-alpha.6 v0.4.2-alpha.5 v0.4.2-alpha.0 v0.4.1-alpha.0; do
  gh -R iesahin/xvc release delete "${r}"
done

🐢 Now let’s list them again.

gh -R iesahin/xvc release list

🐇 The latest one has failed again.

X Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run

🐢 It looks like we’re coming to the end of this session. Under what conditions will we create a release?

🐇 I think it’s better to release only non-alpha tags.

🐢 Then the rule in the workflow will be something like:

on:
  workflow_dispatch:
  push:
    tags:
      - "v*.*.*"
      - "!v.*.*-alpha.*"

🐇 And now we have this result:

✓ v0.6.13 Release iesahin/xvc#263 · 12533498361
Triggered via pull_request about 21 minutes ago

JOBS
✓ Release - FreeBSD-x86_64 in 4m14s
✓ Release - Linux-x86_64 in 4m3s
✓ Release - Linux-aarch64 in 4m50s
✓ Release - Windows-x86_64 in 8m47s
✓ Release - Windows-aarch64 in 6m42s
✓ Release - macOS-x86_64 in 2m37s
✓ Release - macOS-aarch64 in 2m36s

🐢 Nice! Let’s check the release list.

gh -R iesahin/xvc release list

🐇 Why is there no “latest” release?

🐢 We have to tag it first.

🐇 Ah, right. Let’s tag it then.

🐢 I’ve pushed the changes and tagged them with v0.6.13-alpha.5.

🐇 I think it’s possible to make a release today.

🐢 It looks like it, yes.

🐇 We can merge the PR and tag it. Then everything should work.

🐢 We need some cleanup in the YAML files, though.

🐇 We can leave that for the next release.

gh -R iesahin/xvc run list
in_progress		v0.6.13	Rust-CI	v0.6.13	pull_request	12533690474	9m18s	2024-12-29T08:02:29Z
completed	success	Release	Release	v0.6.13-alpha.5	push	12533689099	9m11s	2024-12-29T08:02:19Z

🐢 Now we have another failure in the regular CI. Let’s look at it.

🐇 The issue seems to be in the doc tests:

- Total #: 8 Workspace Size:         276 Cached Size:          19
+ Total #: 8 Workspace Size:         278 Cached Size:          19

🐢 These tests are brittle, but they provide valuable information. Let’s fix it and push again.

🐇 Done. We can also remove some of the watches that produce so many logs.

🐢 I’m a bit ambivalent about them. I thought we could use these watches when debugging, but experience has shown that we need more granular watches during debugging and almost never use these otherwise. Let’s remove some of them.

🐇 We can increase the output for certain commands, but the tracing output doesn’t help much in regular runs. If Xvc gets popular enough that we can’t cope with bug reports, we can always add more watches.

🐢 Another option is to exclude the watch code from the release build, but that won’t change anything for our debug cycles.

🐇 I think removing them is a fair trial. We can always put them back when debugging.

🐢 Watches could also produce regular output instead of tracing. That way we won’t forget to remove them.

🐇 Ah, yep, that’s a good option too.

🐢 We can have a trace! macro similar to the current one for user consumption, and a watch! macro that sends output to stderr.

🐇 Good idea. Let’s do that in the next release.

🐢 Let’s check the tests before pushing this cleanup.

gh -R iesahin/xvc run list
completed	success	v0.6.13	Rust-CI	v0.6.13	pull_request	12533993607	13m49s	2024-12-29T08:47:50Z

🐇 CI succeeded, and the release didn’t run. Let’s do some more cleanup.

gh -R iesahin/xvc run list
in_progress		v0.6.13	Rust-CI	v0.6.13	pull_request	12534416152	43s	2024-12-29T09:57:02Z

Devlog 10: Codecov, Doctests, and Cargo Publish Struggles

🐢 Now we’re going into the real deep work. The only failure was the Codecov patch in Xvc. Let’s see what needs to be done.

🐇 It looks, from the coverage page, that our additions to HStore don’t have any tests. We can add some unit tests to new joins, maybe.

🐢 I don’t find unit tests particularly useful, but let’s use GitHub Copilot to add tests for us.

🐇 Added a unit test and a doc test for full_join, and I think doc tests have more value. They provide documentation, and we can readily see how to use a function from its docs. Better to increase coverage with doc tests.

🐢 There are points that I should learn while writing doc tests. The imports must use the full path, not crate::. The tested struct also doesn’t have implicit imports.

🐇 The ceremony of adding keys and values is a bit too much. It may be worthwhile to add insert for any Into<XvcEntity>.

🐢 It will certainly save time if we don’t have to type .into() for each key :)

🐇 Pushed to test again. Should we have some means to test coverage locally?

🐢 I don’t think we need to consider coverage locally. It’s not worth our time.

🐇 Now while waiting for tests to be completed, what can we do?

gh -R iesahin/xvc run list
completed	failure	v0.6.13	Rust-CI	v0.6.13	pull_request	12543831360	3m54s	2024-12-30T08:15:06Z
...

🐢 I don’t think it took too much time. Let’s view the results:

gh -R iesahin/xvc run view 12543831360
...
  X Run Current Dev Tests
...
To see what failed, try: 
View this run on GitHub: https://github.com/iesahin/xvc/actions/runs/12543831360

🐢 The current dev tests fail for some reason. Let’s run these locally.

🐇 We’re missing llvm-tools-preview locally. How do we install this?

🐢 The command is:

rustup component add llvm-tools-preview
info: component 'llvm-tools' for target 'aarch64-apple-darwin' is up to date

🐇 It’s already installed. We need to set the environment variables.

🐢 Instead, we can just turn off dev tests for the time being. We don’t need them. Our local tests pass.

🐇 Yeah, ok, we don’t need to solve each and every bit of these issues.

gh -R iesahin/xvc run list

… 🐇 Ok. Let’s take a look at the run again.

gh -R iesahin/xvc run list
completed	failure	v0.6.13	Rust-CI	v0.6.13	pull_request	12544020064	3m39s	2024-12-30T08:33:25Z

gh -R iesahin/xvc run view 12544020064
...
  X Test and Coverage
...

gh run view 12544020064 --log-failed
...
Test and Coverage (stable)	Test and Coverage	2024-12-30T08:36:58.8911690Z Error: ProcessError { stdout: "", stderr: "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to e1.xvc.dev port 80 after 160 ms: Couldn't connect to server\n" }
...

🐢 We need to start nginx on the server. We forgot it yesterday.

🐇 Ah, yeah. After adding that dufs installation. Ok.

🐢 We also need to add a reverse proxy to dufs somehow, but this is for later.

🐇 For the use case, I don’t think it’s necessary. We can just adjust the port to a non-standard one if we need to use 443 for another thing, but let’s take a look at the tests again.

🐢 Let’s add another doc test. This time to XvcStore.

cargo test -p xvc-ecs --doc
...
test result: ok. 8 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.87s

🐇 Sent the files again. Waiting for tests to finish.

🐢 Let’s check the keymaps file in the meantime.

🐇 I tried to read some documentation but didn’t see an error. Maybe we should just set it to non-lazy and avoid allowing remaps.

🐢 We already spent too much time with this.

🐇 Yes, let’s take a look at the tests again.

gh -R iesahin/xvc run list
completed	success	v0.6.13	Rust-CI	v0.6.13	pull_request	12544291068	8m10s	2024-12-30T09:00:42Z
...

🐢 Oh, yeah, the merge is ready.

🐇 Patch coverage is still behind the target, though.

🐢 Yeah, but let’s release this one and make the next better covered. Also, I’m not sure if the doc tests had any effect on coverage.

🐇 If we look at the coverage page again, we can see if the doc tests had any effect.

🐢 It seems codecov.io doesn’t consider coverage for doc tests. This is a bit weird, but let’s not spend more time on this.

🐇 Sure, let’s merge.

🐢 I think we forgot to bump the version in Cargo.toml. We’ll have to do that in main.

🐇 Oh, yeah. Let’s bump it and tag as well.

🐢 Now, we can wait for all files to be produced. What will we do next?

🐇 We can just release the Python version as well. It shouldn’t need any changes.

🐢 Umm, right. Maybe we can add a few tests as well.

🐇 Let’s bump the version first and see.

🐢 Bumped versions in Cargo.toml and ran maturin develop.

🐇 It seems ready now.

🐢 The main fails, though. The “Publish Crates” action looks for libsqlite3. Let’s take a look at it.

gh -R iesahin/xvc run list
completed	failure	Release v0.6.13	Publish Crates	v0.6.13	push	12544753359	6m1s	2024-12-30T09:41:56Z

🐢 The issue is that the VM doesn’t have libsqlite3-dev. Let’s add it.

🐇 We need to start the job manually again. Let’s not tag this time.

🐢 Some of the packages were already published. Now they break. crates.io says they’re already published. Maybe we can check if a package is already published.

🐇 Let’s check if we can make cargo publish more forgiving.

🐢 There doesn’t seem to be an option. Let’s search “how to skip published packages in the workspace to avoid errors with cargo publish”.

🐇 Claude is bullshitting again. Let’s try a manual approach: how to skip already published packages.

🐢 It may be easier to just add a check if the package is published. How do we get the info?

🐇 Or we can just go on to the next package if the package is already available.

🐢 Let’s do this manually this time.

Devlog 11: Homebrew Taps and Automation

🐢 Let’s move slowly. Let’s create a brew project first.

🦊 There is a brew create command, but could we really copy it from someone else?

🐢 I read the espanso example, and I’ll start by adding a repository. Welcome https://github.com/iesahin/homebrew-xvc

🦊 We can search to automate this for Rust stuff first. Maybe it’s easier to automate than to do it manually with examples.

🐢 Let’s search “how to automate homebrew tap formula updates with github actions”

🐲 There is a shell script that updates the brew repo with a shell script. It fires the action from the main repository with a command like:

gh workflow run release.yml -f version=${{ env.VERSION }} -R itspriddle/homebrew-slack-notify

🐢 It looks a little brittle, though.

🐲 There is also a GitHub action: https://github.com/marketplace/actions/homebrew-tap but it doesn’t look very popular. There is another one https://github.com/marketplace/actions/bump-homebrew-formula but this is for formulas, or the default settings are those.

🐢 There is an example https://github.com/marketplace/actions/bump-homebrew-formula#examples for taps as well.

🐲 This one is simpler, and probably we can just add this first: https://github.com/marketplace/actions/homebrew-bump-formula Most of the fields are optional.

🐇 I think this is much easier than the other. Let’s start with this.

🐢 Now let’s add a token to the Xvc repo. ✅

🦊 Add a branch to Xvc and add the release action file.

🐢 Added the release action file ~/github.com/iesahin/xvc/.github/workflows/homebrew.yml, but we still don’t have a tap; maybe we can just add one.

🦊 Can you check espanso’s for example?

🐢 It has a cask and is in the core. We need a tap example.

🐲 I believe we need to clone homebrew-xvc and run gh workflow run.

🐢 Yes, let’s clone the repo and try to run it manually.

🐇 We don’t have a formula yet.

🦊 What’s the directory structure of a Homebrew tap project?

🐢 It’s something like this, but we don’t need tests for this, I believe.

.
├── Formula
│   └── <formula_name>.rb
├── LICENSE
├── README.md
└── .github
    └── workflows
        └── tests.yml

🐇 What goes into the Formula?

🦊 Let’s take a look at ripgrep’s example:

class Ripgrep < Formula
  desc "Search tool like grep and The Silver Searcher"
  homepage "https://github.com/BurntSushi/ripgrep"
  url "https://github.com/BurntSushi/ripgrep/archive/refs/tags/14.1.1.tar.gz"
  sha256 "4dad02a2f9c8c3c8d89434e47337aa654cb0e2aa50e806589132f186bf5c2b66"
  license "Unlicense"
  head "https://github.com/BurntSushi/ripgrep.git", branch: "master"

  livecheck do
    url :stable
    strategy :github_latest
  end

  bottle do
    sha256 cellar: :any,                 arm64_sequoia:  "b8bf5e73c9c9b441de067ec86ac167b071ecc2078dcb1d89d2cebbb151feab35"
    sha256 cellar: :any,                 arm64_sonoma:   "47b9c3515c866b147f0e98735cab165d6471b9f28fab1ba2c57e59c43da5c10b"
    sha256 cellar: :any,                 arm64_ventura:  "e14a94e84c028ff53c1be3b106fdeb5aca4d7c893a819e7fb967e0719b946a28"
    sha256 cellar: :any,                 arm64_monterey: "ad8dc4ab475c84e2a1e60f5b3107f52dd59e33f84a08284b19681d8b98508fd7"
    sha256 cellar: :any,                 sonoma:         "71d434eeabc2af220285b037f7264563ce9bc77a41af35eabe2213276a37ec2b"
    sha256 cellar: :any,                 ventura:        "0cdb547c696992d08c6613c40934218964f4a061b5413c4b2f013c3f0c3ed253"
    sha256 cellar: :any,                 monterey:       "2ce54302e4524ad28389aca5a16333d4193128e911de2881e6b0e953559d89cd"
    sha256 cellar: :any_skip_relocation, x86_64_linux:   "97d7cbd33b4d0ed09551e3dbc07f830d3df018c2aefbb2222a12ccfb829aae30"
  end

  depends_on "asciidoctor" => :build
  depends_on "pkgconf" => :build
  depends_on "rust" => :build
  depends_on "pcre2"

  def install
    system "cargo", "install", "--features", "pcre2", *std_cargo_args

    generate_completions_from_executable(bin/"rg", "--generate", shell_parameter_format: "complete-")
    (man1/"rg.1").write Utils.safe_popen_read(bin/"rg", "--generate", "man")
  end

  test do
    (testpath/"Hello.txt").write("Hello World!")
    system bin/"rg", "Hello World!", testpath
  end
end

🐇 What’s in that tar file?

wget https://github.com/BurntSushi/ripgrep/archive/refs/tags/14.1.1.tar.gz 
--2025-01-03 06:28:07--  https://github.com/BurntSushi/ripgrep/archive/refs/tags/14.1.1.tar.gz
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/BurntSushi/ripgrep/tar.gz/refs/tags/14.1.1 [following]
--2025-01-03 06:28:08--  https://codeload.github.com/BurntSushi/ripgrep/tar.gz/refs/tags/14.1.1
Resolving codeload.github.com (codeload.github.com)... 140.82.121.10
Connecting to codeload.github.com (codeload.github.com)|140.82.121.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘14.1.1.tar.gz’

     0K .......... .......... .......... .......... ..........  355K
    50K .......... .......... .......... .......... .......... 1.05M
   100K .......... .......... .......... .......... .......... 1.35M
   150K .......... .......... .......... .......... .......... 3.61M
   200K .......... .......... .......... .......... ..........  876K
   250K .......... .......... .......... .......... .......... 6.50M
   300K .......... .......... .......... .......... .......... 1.15M
   350K .......... .......... .......... .......... .......... 1.65M
   400K .......... .......... .......... .......... .......... 4.88M
   450K .......... .......... .......... .......... .......... 1.22M
   500K .......... .......... .......... .......... .......... 1.56M
   550K .......... .......                                     3.62M=0.5s

2025-01-03 06:28:09 (1.21 MB/s) - ‘14.1.1.tar.gz’ saved [581402]

mv 14.1.1.tar.gz $HOME/Downloads/ripgrep-14.1.1.tar.gz

tar xvzf $HOME/Downloads/ripgrep-14.1.1.tar.gz 

🐢 It’s a source distribution. It doesn’t contain any binaries.

🦊 We can use the same URL format, it looks. Let’s try this:

wget https://github.com/iesahin/xvc/archive/refs/tags/0.6.13.tar.gz 
--2025-01-03 06:34:00--  https://github.com/iesahin/xvc/archive/refs/tags/0.6.13.tar.gz
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/iesahin/xvc/tar.gz/refs/tags/0.6.13 [following]
--2025-01-03 06:34:01--  https://codeload.github.com/iesahin/xvc/tar.gz/refs/tags/0.6.13
Resolving codeload.github.com (codeload.github.com)... 140.82.121.9
Connecting to codeload.github.com (codeload.github.com)|140.82.121.9|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2025-01-03 06:34:01 ERROR 404: Not Found.

🐢 Our source should be in another location.

🐇 We had an unpublished release. Didn’t we release the binaries a few days ago?

🐢 Umm, there must be something about the gh command where we set draft=false, but seemingly it didn’t work out.

🐇 Ok, the URL is something like:

wget https://github.com/iesahin/xvc/archive/refs/tags/v0.6.13.tar.gz 

🐢 Ok, this works. We’ll use this one for the URL.

🐇 What should we put for SHA256?

🦊 Let’s check ripgrep’s again.

🐢 Downloading and using xvc file hash (alias xvcfh):

wget https://github.com/BurntSushi/ripgrep/archive/refs/tags/14.1.1.tar.gz 
...

xvc file hash -a sha2 14.1.1.tar.gz
4dad02a2f9c8c3c8d89434e47337aa654cb0e2aa50e806589132f186bf5c2b66	14.1.1.tar.gz

🐢 Yep, this matches the source.

🦊 Then we can just use the same.

🐇 I want to learn how to download to $TMPDIR with wget.

🐢 The -P option is used for this.

wget -P $TMPDIR https://github.com/iesahin/xvc/archive/refs/tags/v0.6.13.tar.gz 
--2025-01-03 06:55:02--  https://github.com/iesahin/xvc/archive/refs/tags/v0.6.13.tar.gz
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/iesahin/xvc/tar.gz/refs/tags/v0.6.13 [following]
--2025-01-03 06:55:03--  https://codeload.github.com/iesahin/xvc/tar.gz/refs/tags/v0.6.13
Resolving codeload.github.com (codeload.github.com)... 140.82.121.9
Connecting to codeload.github.com (codeload.github.com)|140.82.121.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘/var/folders/gf/18gghtz15_zf0cr8v6dymglw0000gn/T/v0.6.13.tar.gz’
...
2025-01-03 06:55:07 (2.59 MB/s) - ‘/var/folders/gf/18gghtz15_zf0cr8v6dymglw0000gn/T/v0.6.13.tar.gz’ saved [8785449]

xvcfh -a sha2 $TMPDIR/v0.6.13.tar.gz
01bee5d840eefec7be1f52cc75546e1ffd7e332dfac83d875655f84e61e3a9f6	/var/folders/gf/18gghtz15_zf0cr8v6dymglw0000gn/T//v0.6.13.tar.gz

🐇 According to documentation, bottles are produced by Brew itself, and the documentation around bottling taps is limited.

🦊 We can just start with the source distribution. We can test it now.

🐢 The source distribution seems to work, but it requires Rust, and Brew downloads everything related to it.

🐇 I think that may be enough for the time being. With 0.6.14, we can work on providing bottles.

🐢 Umm, yes. I’m already bored with this stuff.

devlog 12

🐢 What are today’s plans?

🐇 I think we can start by improving xvc.py. I mean, releasing. Yesterday we finished our work with the Rust library.

🐢 Then we can start to look at the GUI, I believe. We can replace that 3-column view with a table and preview. It will be much easier that way.

🐇 Yep. Let’s finish and release the Python version first. Then we’ll go on to the GUI.

🐢 Let’s take a look at the PRs first.

ghpl 
36	Bump pyo3 from 0.22.2 to 0.23.3	dependabot/cargo/pyo3-0.23.3	OPEN	2024-12-04T03:52:03Z

gh pr close 36
✓ Closed pull request iesahin/xvc.py#36 (Bump pyo3 from 0.22.2 to 0.23.3)

🐢 We have already upgraded to pyo3 0.23. No need for this. Let’s create a PR for the current branch.

git push --set-upstream origin v0.6.13

ghpC --fill
https://github.com/iesahin/xvc.py/pull/37

🐇 We can also tag and push the tags.

🐢 I’d like to have some more coverage for certain parts of the code. Let’s add some tests.

🐇 It looks like we mainly lack the storage tests. They need configuration to add keys to GitHub, and we already skip some of these even in the Rust code.

🐢 Umm, I see. We also need a way to measure coverage. Could we do this with Codecov, I wonder?

🐇 I found an example here: https://github.com/codecov/example-python/blob/main/.github/workflows/ci.yml. It needs coverage and pytest-cov in the requirements.

🐢 Let’s try this then.

🐇 Added coverage.yml file. It’s simpler than the other GitHub action. Need to update the token now.

🐢 There is an issue installing the requirements.

🐇 I forgot to add sudo to apt-get. Will take care of it now.

🐢 Let’s check the run.

ghrl
in_progress		Release v0.6.13	coverage	v0.6.13	pull_request	12556590550	2m31s	2024-12-31T06:59:58Z
in_progress		Release v0.6.13	publish-to-pypi	v0.6.13	pull_request	12556590548	2m31s	2024-12-31T06:59:58Z
...

🐇 It takes a while and we still didn’t add the CODECOV_TOKEN.

🐢 Let’s add it, and after that, we need to take a look at this blink configuration. It adds letters after the selection.

🐇 Added the secret and configured xvc.py for coverage. There is an error with the build, though. Maybe the command we should be using is maturin develop instead of build to make Xvc available for the environment.

🐢 Let’s update and try it then.

🐇 maturin develop requires a virtual environment.

🐢 I checked the options for build, and I think there is an option, but let’s search first.

🐇 I searched, but it looks like we can just pass an --out directory or install xvc from the target/wheels/ directory. The second option requires less maintenance.

🐢 Okay. Let’s add a step to the action then.

🐇 Now, let’s wait for the run to finish with gh run watch.

🐢 It failed. Let’s take a look at the logs:

ghrl
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12556939836	3m26s	2024-12-31T07:38:54Z

gh run view 12556939836 --log-failed

🐇 It looks like some of the tests are failing. Let’s run the tests locally.

🐢 We should run with the --forked option, and it looks like we have a test to update with the xvc file list.

🐇 Updated the test, and I noticed we forgot to supply the new --show-directories option in the Python interface.

🐢 Yep. Passing these options as command-line options in strings is not robust. It’s very easy to forget things. I think we should start using CLI structs directly, but it’s not time yet.

🐇 I agree. It’s one of the goals for building a GUI, actually.

🐢 The tests failed again.

ghrl | rg failure
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12557031091	3m49s	2024-12-31T07:51:19Z

gh run view 12557031091 --log-failed

🐇 There is a Git error now. We need to add a Git user and email to the action.

🐢 Added those and watching the results again now.

🐇 Why do you think the publish action always works? It can only run with the main branch, I think. No need to run it with other pushes.

🐢 Yes, let’s configure it now.

ghrl | rg failure
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12557083393	3m21s	2024-12-31T07:59:18Z

gh run view 12557083393 --log-failed

🐇 It looks like we also need xvc-test-helper in the path. Let’s cargo install it and add it to the path.

🐢 Added .cargo/bin to the path like:

- name: Add cargo bin to PATH
  run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH

and installed the helper with cargo install xvc-test-helper.

🐇 By the way, GitHub Copilot is hallucinating about a method to update the path.

🐢 I searched and it may not be hallucinating. We can use the ::add-path:: command with echo, it looks like. This is new to me.

🐇 The tests failed again.

ghrl | rg failure
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12557208723	3m34s	2024-12-31T08:12:09Z

gh run view 12557208723 --log-failed

🐢 file().list() had a mistake, and we need to install rg for the tests.

🐇 Watching the test run. In the meantime, maybe we can review…

🐢 Failed again.

ghrl | rg failure
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12557294105	3m46s	2024-12-31T08:21:17Z

gh run view 12557294105 --log-failed

🐇 Added db.commit() to two places in the code. This should pass now.

🐢 Yeah!

ghrl | head -n 1
completed	success	Release v0.6.13	coverage	v0.6.13	pull_request	12557509836	3m46s	2024-12-31T08:44:33Z

gh run view 12557509836

✓ v0.6.13 coverage iesahin/xvc.py#37 · 12557509836
Triggered via pull_request about 4 minutes ago

JOBS
✓ linux in 3m36s (ID 35010253915)

ANNOTATIONS
! ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
linux: .github#1


For more information about the job, try: gh run view --job=35010253915
View this run on GitHub: https://github.com/iesahin/xvc.py/actions/runs/12557509836

🐇 I can’t see a coverage report on codecov.io, though.

🐢 It says no coverage report is generated. Let’s test to generate XML files locally.

🐇 The option in the docs seems incorrect. --cov-branch doesn’t produce anything.

🐢 Let’s wait for the run again.

🐇 Should we add a badge to the README?

🐢 It won’t show much, but yeah, let’s make it.

🐇 The results are in and it shows 100% coverage. This means it doesn’t actually test anything.

🐢 We need Rust coverage for this. Let’s search for it.

🐇 I found this: https://github.com/cjermain/rust-python-coverage. It runs cargo llvm-cov with the project and measures test coverage. But we don’t have any Rust tests.

🐢 It looks like we don’t need Rust tests. cargo llvm-cov can check coverage with the Python as well.

$ cargo llvm-cov show-env --export-prefix
export RUSTFLAGS=" -C instrument-coverage --cfg coverage --cfg trybuild_no_target"
export LLVM_PROFILE_FILE="/home/.../rust-python-coverage/target/rust-python-coverage-%m.profraw"
export CARGO_INCREMENTAL="0"
export CARGO_LLVM_COV_TARGET_DIR="/home/.../rust-python-coverage/target"

🐢 Let’s take a look at what remained for 0.6.13.

🐇 I think there is nothing left. Python must be published when we merged the PR.

🐢 Let’s take a look by searching xvc python.

🐇 The PyPI page is https://pypi.org/project/xvc/ and it still reports the version as 0.6.11. There must be something.

🐢 Now let’s take a look at

https://github.com/iesahin/xvc.py

🐇 The run seems to be OK though.

https://github.com/iesahin/xvc.py/actions/runs/12557780141/job/35010938462

🐢 Maybe the version in pyproject.toml is still 0.6.11 and we forgot to update it?

tmux new-window -c $HOME/github.com/iesahin/xvc.py/ nvim

🐇 There is no version string in ~/github.com/iesahin/xvc.py/pyproject.toml. It’s dynamic.

🐢 Then, let’s try to publish from local now.

🐇 The iex username requires email verification.

🐢 It looks like I publish xvc through the iesahin account, not iex. Maybe I can add both accounts to the project.

🐇 There seem to be no errors on the PyPI site.

🐢 Updating the token. Let’s add the token to pass to run maturin publish.

🐇 We’re receiving invalid or non-existent authentication information. Upgraded maturin to see if it fixes the issue.

🐢 We can also double-check the key.

🐇 Uploaded successfully from local. Let’s check the job again.

🐢 The release job was skipped because we didn’t tag after the merge. https://github.com/iesahin/xvc.py/actions/runs/12557780141/job/35011303045 That looks like the reason.

🐇 I should be more careful which is run and which is skipped.

🐢 Maybe we can relax the condition. We do this rarely. Maybe republishing is alright?

🐇 Yep, removed that. The jobs are running now. Let’s watch them to see what happens when we publish some of the packages.

🐢 In the meantime, let’s experiment with searching commands file with fzf-lua.

🐇 It requires more experimentation, but we can start from https://github.com/ibhagwan/fzf-lua/wiki/Advanced#interactive-shell-command.

🐢 The xvc publish jobs failed, btw.

ghrl
completed	failure	Remove if condition from release	publish-to-pypi	v0.6.13	push	12568863716	15m26s	2025-01-01T08:30:03Z

gh run view 12568863716 --log-failed
...
Release	Run actions/download-artifact@v4.1.7	        Please ensure that your artifact is not expired and the artifact was uploaded using a compatible version of toolkit/upload-artifact.
...

🐇 It was using an older version of upload-artifact.

🐢 Rerunning the job and it looks like it runs for both push to main and tag with v0.6.13. We can turn off push to main, I believe.

🐇 There was a missing upload-artifact again. Fixed and updated the tags.

ghrl completed failure update upload-artifacts publish-to-pypi main push 12569060915 14m39s 2025-01-01T08:57:22Z

ghrf 12569060915

🐢 There are conflicts with uploaded artifacts now. We may need to clean up the artifacts manually.

🐇 The conflicts were not about inter-workflow names. The names were conflicting because all files were named wheel. I added the platform and target to the names to avoid conflicts.

🐢 Let’s wait then. Maybe it will work this time. Could you search for a Lua console for Neovim?

🐇 Let’s try this one: return { “yarospace/lua-console.nvim”, lazy = true, keys = “`”, opts = {}, }

🐢 I couldn’t make it run but won’t spend much time ATM. How about the jobs?

ghrl
completed	failure	update upload artifact names	publish-to-pypi	v0.6.13	push	12569298414	13m57s	2025-01-01T09:27:00Z

ghrf 12569298414
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9180510Z ##[group]Run actions/download-artifact@v4
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9182098Z with:
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9182859Z   name: wheels
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9183843Z   merge-multiple: false
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9184817Z   repository: iesahin/xvc.py
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9185820Z   run-id: 12569298414
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9186903Z ##[endgroup]
Release	Run actions/download-artifact@v4	2025-01-01T09:40:53.1937617Z Downloading single artifact
Release	Run actions/download-artifact@v4	2025-01-01T09:40:53.4694769Z ##[error]Unable to download artifact(s): Artifact not found for name: wheels
Release	Run actions/download-artifact@v4	        Please ensure that your artifact is not expired and the artifact was uploaded using a compatible version of toolkit/upload-artifact.
Release	Run actions/download-artifact@v4	        For more information, visit the GitHub Artifacts FAQ: https://github.com/actions/toolkit/blob/main/packages/artifact/docs/faq.md

🐇 Now the download doesn’t work.

🐢 Update it to download using patterns.

🐇 Did so and let’s take a look at the jobs again.

ghrl
completed	failure	added pattern to download	.github/workflows/publish.yml	main	push	12569456859	0s	2025-01-01T09:49:19Z

ghrv 12569456859

X main .github/workflows/publish.yml · 12569456859
Triggered via push about 3 minutes ago

X This run likely failed because of a workflow file issue.

For more information, see: https://github.com/iesahin/xvc.py/actions/runs/12569456859

🐢 The line with the pattern was reported as broken.

🐇 Removed it and recommitted.

ghrl
✅ completed	success	remove pattern to download all	publish-to-pypi	v0.6.13	push	12569509435	14m41s	2025-01-01T09:58:49Z

🐢 And now this completes the v0.6.13 release.

🐇 We’ll see how it will work next time.

🐢 Yep. Let’s move on to the GUI for now. Is that okay with you?

devlog 13

🐇 So, what’s next?

🐲 We can work on completions or the GUI.

🐢 Completions are similar to the Homebrew work. It’s in another ecosystem and feels boring.

🐲 We can try to make it in Rust: “Writing shell completions in Rust.”

🐢 There are two solutions: One is clap_complete (https://docs.rs/clap_complete/latest/clap_complete/) and the other is shell_completion (https://github.com/JoshMcguigan/shell_completion). The latter is very new but may be what we need: writing completions only with Rust.

🐲 The shell_completion crate is very bare-bones. It doesn’t have anything, actually. https://github.com/JoshMcguigan/shell_completion/issues/1

🐇 The feature that we need is dynamic completion. We’ll write something similar to ArgValueCompleter for this.

🦊 I think we can just start a new branch.

🐲 We have a branch for Homebrew; should we merge it?

🐢 I think, yes, we can merge it. We’ll fix it if it breaks. It’s a separate workflow file anyway.

🦊 Created the PR.

ghpl
264	add brew tap	add-brew-tap	OPEN	2025-01-03T05:14:45Z

🐢 Merged it.

🐇 Now let’s create a new branch and add clap-complete to Cargo.toml.

🐲 There are some packages that we need to take a look at. thiserror now has v2.

🐢 Upgraded packages and compiled. It works.

🐲 Let’s update the version.

🐢 Done. Now we can add clap-complete.

🦊 Added with the unstable-dynamic command. Can we build it again?

🐢 Built it. Now we can add a completions subcommand to Xvc.

🐲 Is this the right name for this?

🐢 It will output a shell script that we can source in the shell.

🐲 Okay, let’s add the subcommand now.

🐢 The example doesn’t work with the clap builder.

🐇 Let’s search for it.

🐢 I think we’re extending ourselves a bit when trying to add all dynamic features of completion at once. We can just start by adding completion to xvc-test-helper.

🐇 Yep, that’s a better idea. We can have many more examples, and it’s certainly more straightforward.

🐢 One thing I noticed when working yesterday is that using auto-save was actually preventing many of these swap errors.

🐲 It was a bit slow. Can we take a look at a few others?

🐢 https://github.com/okuuva/auto-save.nvim seems a bit more polished.

🐇 Installed it, but haven’t seen an effect yet.

🐢 We may need to restart this.

🐲 Now, we can get into the test-helper completions.

🐢 Done. Added these quickly as you thought. Now adding this to main seems much easier.

🦊 That was a nice approach. Let’s dive into adding this to main.

🐇 Should we go with a separate command or just an option?

🐲 Our commands have distinct initial letters, allowing them to be used with just those letters. Adding another top-level command for this will make c useless.

🐢 I don’t think that’s a valid concern. We can create aliases for all commands and completions shouldn’t have to have an alias. But I agree that completions should not be a top-level command. It will be run once for installation at most.

🦊 I agree. Let’s call it --completions. It will be run as xvc --completions zsh and will print out the completions.

🐢 Okay. Let’s do this.

🦊 No errors left. Let’s install this version.

🐢 We get:

error: 'xvc' requires a subcommand but one was not provided
  [subcommands: file, init, pipeline, storage, root, check-ignore, aliases, help]

Usage: xvc [OPTIONS] <COMMAND>

For more information, try '--help'.

🦊 Umm, okay. I think we don’t have an option to create a command like that. Then we’ll have to print completions with a subcommand.

🐲 The above discussion is now moot. ❌

🐢 We can try to push, but probably it’s not worth it.

🐲 Let’s not lose time on this. I think we can remove the aliases command and replace it with completions, and add single-letter aliases to subcommands.

🐢 We can have an option in the completions command to print aliases. That will work.

🦊 I don’t think people will use the aliases command if we have single-letter command aliases.

🐢 You may be right. No need to try to maintain that at this time.

🐢 Good morning, and I’m fed up with these blink.cmp errors, you know.

🐇 Reinstalling blink works. It’s interesting to rely on such unreliable software.

🐢 Oh, yeah. It’s interesting. Where were we yesterday?

🐲 We decided to rename the aliases command to completions.

🐇 And waiting for rust-analyzer to complete its analysis.

🐢 Uh, yeah. I see.

🦊 The code itself is very short, actually.

    if let Some(shell) = cli_opts.completions {
        let mut cmd = XvcCLI::command();
        generate(shell, &mut cmd, "xvc", &mut io::stdout());
        return Ok(None);
    }

🐲 We’ll use the output! macro instead of writing to io::stdout(), right?

🐢 Yes, we can rely on the usual output system. It will be slower to create an output thread but shouldn’t matter for outputting a shell script.

🐇 It takes a while for rust-analyzer to scan all the directories, it looks like. When I close the project, it just cannot reload it immediately.

🐢 We can use cargo clean from time to time.

🦊 rust-analyzer finished scanning; let’s try to rename AliasesCLI.

🐲 Let’s keep these here; maybe we’ll need them in our scripts:

# Standard Xvc command aliases for longer commands.
alias xls='xvc file list'
alias pvc='xvc pipeline'
alias fvc='xvc file'
alias xvcf='xvc file'
alias xvcft='xvc file track'
alias xvcfl='xvc file list'
alias xvcfs='xvc file send'
alias xvcfb='xvc file bring'
alias xvcfh='xvc file hash'
alias xvcfco='xvc file checkout'
alias xvcfr='xvc file recheck'
alias xvcp='xvc pipeline'
alias xvcpr='xvc pipeline run'
alias xvcps='xvc pipeline step'
alias xvcpsn='xvc pipeline step new'
alias xvcpsd='xvc pipeline step dependency'
alias xvcpso='xvc pipeline step output'
alias xvcpi='xvc pipeline import'
alias xvcpe='xvc pipeline export'
alias xvcpl='xvc pipeline list'
alias xvcpn='xvc pipeline new'
alias xvcpu='xvc pipeline update'
alias xvcpd='xvc pipeline dag'
alias xvcs='xvc storage'
alias xvcsn='xvc storage new'
alias xvcsl='xvc storage list'
alias xvcsr='xvc storage remove'

🦊 We can use them when adding single-letter aliases.

🐇 There is a from_env method for Shell; will we support it?

🦊 I think we can support it. It’s much easier to use if we omit the shell.

🐲 We had aliases in xvc-core, but we need to access XvcCLI from completions. The completions module must be moved to xvc.

🦊 We added the clap_complete dependency to the xvc-pipeline and xvc-file crates, but I don’t think they are necessary. Let’s remove them now.

🐢 Now only the test-helper and xvc crates have the clap_complete dependency.

🐇 Are we ready to test?

🐢 Completions are working. 🎉

🐲 Now we need to update the docs and doc tests, I believe.

🐢 There are also tests to update.

🐇 Tests are running now. In the meantime, can we take a look at the blink configuration?

🐲 When we removed xvc aliases, we also removed pvc, xls, and other aliases. Maybe we can add these to the docs.

🐢 We can put them in the xvc completions reference for now.

🐢 It takes a while for rust-analyzer to finish analyzing the codebase.

🦊 Added aliases for xvc pipeline commands. Do you think we need to repeat root-level flags in xvc-pipeline?

🐢 No need to divert attention, I believe. Also, I still think there is an easier way to do that.

🐇 Okay. Do you think we should add easier subcommands to step? Like, step new becoming xvc p s n instead of xvc p s n? (Wait, that’s the same). I mean, more concise.

🐢 I think we can extend these even to the top-level, but shouldn’t make them visible. It will pollute the help text. We can add xvc fl for file list to avoid the space, but hide these from the help text.

🐲 That’s a good idea, but it will require including sub-crate level modules at the top level. We must test it first.

🐢 Let’s finish up the current changes and release them first.

🐇 By the way, we didn’t add two-letter abbreviations to the xvc storage new subcommands. Do you think we need them?

🐢 When we think about the frequency of these commands, no, I don’t think we need them. Users won’t add a new storage every day.

🐇 By that logic, we shouldn’t need an n for xvc storage new.

🐢 Actually, yeah. Maybe we should remove even s.

🦊 storage list may be useful.

🐢 s is a very common letter, though. We can have other uses for that letter.

🐇 Updating xvc p s dependency options, but it looks like making dependency options subcommands is a better way.

🦊 We didn’t do it because currently we can supply multiple options with a single command. If we go the subcommand route, we’ll have to write all dependencies one by one. Adding multiple subcommands with clap is a bit tricky.

🐢 Umm. Yeah, I see.

🐇 Maybe we can provide a separate command to add multiple dependencies. Like, xvc p s d add 'lines=myfile.csv::10-20; file=myimage.jpg; glob=dir/image-10*'

🐲 That looks like the start of a language. We need a parser for those strings. They will be freeform.

🦊 Another option is to keep the current options and add commands with the same names.

🐢 That will be confusing. The user will have both --param and param, and they will work differently.

🐇 We can have an add command that accepts the current options. Parsing will be done by clap just like now, but for a subcommand of dependency.

🐢 The full command will be something like xvc pipeline step --step-name preprocessing dependency add --params 'params.json::batch_size'

🐇 With shorter commands, it’s like xvc p s -s preprocessing d a --params 'params.json::batch_size', and this doesn’t look like ffmpeg monstrosities.

🐲 In any case, this is a backward-incompatible change. This should wait for v0.7 along with ECS changes.

🐢 ECS changes are not user-visible, but these are. Certainly needs a minor version update.

🐇 Then, okay, let’s keep the current ones for this version and think about updating them in a future version.

🐢 Okay. Let’s finish up and we’ll create a separate invisible subcommand to ask questions to the Xvc repo for completions.

🐇 We completed completions for the common command structure. Now, we need a way to show certain info after certain commands. A tab after xvc p r -p should show pipeline names, for example.

🐢 Umm, yeah. And these should be as quick as possible. They shouldn’t check Git or any other things.

🐇 I looked here and there, and the only working example I found is here: https://github.com/clap-rs/clap/blob/master/clap_complete/tests/testsuite/zsh.rs#L248 in clap_complete tests.

🐢 Let’s try to dive in. We can start by cloning the repo, I believe.

devlog 14

🐢 Now, let’s return to clap and its dynamic completions. Last time you said:

🐇 The examples are only found in the tests: https://github.com/clap-rs/clap/blob/master/clap_complete/tests/testsuite/engine.rs#L604

and we can try to create those random strings this way.

🦊 The issue is that we may not be able to use the derive method to add these custom completers. All examples are using the builder API.

🐢 We should be able to obtain the end result from derives and modify them to use these custom completers, but let’s try to use add = ArgValueCompleter first.

🐇 It looks like we have another problem. We need to add a rust-toolchain.toml file to the project, you know, to keep it on the stable channel.

🐢 Ah, yeah, we now use multiple channels. Let’s do that.

🐇 I found an example, actually:

#[derive(Debug, Parser)]
struct Cli {
    #[arg(long, add = ArgValueCompleter::new(custom_completer))]
    custom: Option<String>,
}

🐢 I’m trying to add this to the --from-ref option of Xvc. It’s not running anything. The custom completer should return a random string, but it doesn’t.

🦊 Maybe test with a constant first.

🐢 It didn’t work that way either.

🐇 Let’s do a cargo clean.

🐢 Nothing has changed.

🐇 Let’s take a look at the output of xvc completions:

xvc completions
...
'(--skip-git)--from-ref=[Checkout the given Git reference (branch, tag, commit etc.) before performing the Xvc operation. This runs \`git checkout <given-value>\` before running the command]:FROM_REF:_default' \
...

🐢 This is the only place --from-ref is mentioned and, as far as I can see, there is nothing that calls a dynamic command here.

🐇 Umm, right. There’s something weird here. Maybe we lack the ArgExt trait or something.

🐢 It should be turned on by clap-complete with its unstable-dynamic feature, but who knows.

🐇 It didn’t work.

🐢 ArgExt is available, but it isn’t being used.

🐇 I think I found the answer in https://jj-vcs.github.io/jj/latest/install-and-setup/#command-line-completion:

source <(COMPLETE=zsh xvc)

is the command we should use.

🐢 It doesn’t produce a completion script, though.

🐇 We have this in jj/cli/src/cli_util.rs:

        if env::var_os("COMPLETE").is_some() {
            return handle_shell_completion(ui, &self.app, &config, &cwd);
        }

🐢 I think the whole handle_shell_completion set is written by them. I’d like to see what jj with COMPLETE=zsh outputs.

🐇 Building it to see now.

🐢 As expected, it calls a function:

❯ COMPLETE=zsh target/debug/jj
#compdef jj
function _clap_dynamic_completer_jj() {
    local _CLAP_COMPLETE_INDEX=$(expr $CURRENT - 1)
    local _CLAP_IFS=$'\n'

    local completions=("${(@f)$( \
        _CLAP_IFS="$_CLAP_IFS" \
        _CLAP_COMPLETE_INDEX="$_CLAP_COMPLETE_INDEX" \
        COMPLETE="zsh" \
        /Users/iex/github.com/etc/jj/target/debug/jj -- ${words} 2>/dev/null \
    )}")

    if [[ -n $completions ]]; then
        _describe 'values' completions
    fi
}

compdef _clap_dynamic_completer_jj jj

🐇 Now we should make it the same; the shell must call xvc to get the completions. That’s how all these will work.

🐢 Our goal now is to make a random string output from the --from-ref completion.

🐇 The plan is to make completions work as quickly as possible.

🐢 We have done it. 🎉🥳

🐇 Cool. Now we need to fill up all those completion methods.

🦊 Yeah. We can start with basic ones and move from there.

🐢 There will probably be architectural changes as well. We cannot just run xvc functions directly. We need to run internal functions, but not through commands. At that point, when the user hits tab, we don’t know which command to run.

🐇 Maybe it’s time to move to a Git library. “What is the best Git library for Rust?”

🐲 Should we use a Git library?

🐢 It looks like Gitoxide is the default Rust way to interact with Git repositories. We can keep our way of using the Git binary and use this as an experimental way to learn.

🦊 Yep. That’s a good idea. We can include it in the lib for the time being and start to use it in completions.

devlog 15

🐢 Should we go into completions directly, or do we have anything to update in the working scripts?

🐇 Starting with completions is better. One question I have is whether the completions work for subcommands as usual, like in the static completions.

🐢 Yes, they work. We need to add:

source <(COMPLETE=zsh xvc)

to .zshrc, though.

🐇 It looks like we can drop the xvc completions command. We’re just checking the COMPLETE environment variable, and there’s no need for a separate command in this case.

🐢 Yes, let’s remove that.

🐇 Maybe we can employ a completions module for all completion-related functionality. It’s a separate thing, you know.

🐢 Which completions do we need?

🐇 We can just mark them with TODOs now.

🐢 Yes, let’s check where we need completions and what.

🐇 I noticed we’re repeating options in XvcFileCLI, and some of the options are missing, e.g., --from-ref and --to-branch.

🐢 Because it can compile into a different binary, and global options don’t work that way. Maybe we can move all these binaries into different files under lib. We can have feature flags to turn off certain features and compile different binaries.

🦊 Let’s search for it; I haven’t seen it before: building different binaries with feature flags using cargo.

🐢 There is no clear-cut solution, but it looks like we can move all binaries to the root with the required-features field.

🐲 We can postpone this to another release.

🐇 Yes, let’s not spend time on this now.

🐢 Should we try making pipeline_name a global option? It’s repeating everywhere.

🐇 That will make it easier to maintain.

🐢 We’ll have to pass pipeline_name to subcommands, though.

🐇 I think we can have a set of global options that we can pass. For the time being, that’s only the pipeline_name.

🐢 Okay. Let’s do this.

🐇 A similar option is the step name for pipeline steps.

🐢 Dependencies will need a revamp in the next version anyway. So let’s keep it for now.

🐲 Also, the semantics of step-name are different for these commands. step new interprets it as a new name, while step dependency interprets it as an existing name. The first can be renamed to --name, and the second can be --to, as one of its aliases suggests.

🐢 Yes, let’s keep it for now, and we’ll continue to work on others.

🦊 I want to post a comment in the clap discussions.

🐢 Wrote it. Now let’s build it after changing the pipeline_name.

🐇 It compiles. Should we test it?

🐢 I think we’ll test after all this completion work is done.

🐇 We did most of the strum-related completion.

🐢 Didn’t test them yet, though.

🐇 This is Rust. It will work if it compiles, and it compiles.

devlog 16

🐢 Good morning. Yesterday we wrote a proxy server for a job application. It was a pleasing experience, but I wasn’t able to write a dialogue.

🐇 It was one of those nice days. I have that feeling in my head that I used it to its full potential.

🐢 So we can do some light work today.

🐇 Like completions?

🐢 Yep, let’s fill in some missing pieces there.

🐇 Let’s see what those missing pieces are.

🐢 We need to clean up the code comments we copied from jj.

🦊 Cleaned up quite a bit and moved everything to main. It’s a single function call.

🐢 I think we completed all strum-related completions. Now we need to discuss store completions a bit.

🐇 They can’t all have the same function like we used for strum-based enums. They will look up different parts of components. For XvcPath, it will be the inner RelativePathBuf; for StorageIdentifier, that will be the names of all recorded storage names and GUIDs. The field we’re going to complete will change from case to case.

🦊 They can depend on the same machinery. It will detect .xvc, load a store, and return a field from the component. The first two are the same for all.

🐢 Yep. Let’s write a few TODOs in the code, then.

🦊 One thing to discuss is whether we should load all config and root. It may be a bit time-consuming. Just loading the stores should be enough.

🐢 I think so. ECS doesn’t depend on XvcRoot, and we can load stores without loading XvcRoot. We have convenience functions in XvcRoot, but they are for convenience. There is nothing that prevents us from loading stores without loading the root.

🦊 Let’s take a look at XvcRoot’s loading of these.

🐢 Tomorrow, hopefully.

devlog 17

🐢 We are progressing towards adding all store completions. In the meantime, I switched to nushell from zsh. Completions for nushell require a different crate and it doesn’t have dynamic completions yet.

🐇 We can just complete the completions for this version and add JSON output for all commands to use in nushell. In the meantime, dynamic completions support for nushell is likely to be added.

🐢 Umm, yep. Then we’ll continue to run tests with zsh for the current version.

🐇 run-tests is a zsh file, and we will continue to use it. We can switch to nushell eventually as we add more compatibility with it, but let’s not do this now.

🐢 Where were we the last time for completions?

🐇 We added xvc_path_completers. Let’s check the TODO list now.

🐢 We still need tracked_targets completions in a few places.

🐇 Added those.

🐢 Now we have some strum completers. Let’s finish these up as well.

🐇 Ok. They are done as well.

🐢 Now, let’s start adding a storage_identifier completer. It should read all XvcStorage records and list their names.

🐇 Added a storage_identifier completer. Let’s add it to all places where we use storage_identifiers.

🐢 Now, we need completers for pipeline and step names. These are straightforward.

🐇 Added these. Let’s fill up where they are needed.

🐢 I want to skip some of the fine-grained completions in this version. We can have a specific completer for --params options for files and params inside, for example. Or a special completer for directories.

🐇 Let’s not forget these. Reading YAML files and extracting hyperparameter keys for the prompt would be a really good feature for the user.

🐢 I agree. These are good features in general, but we need to ship the current version as soon as possible.

🐇 Let’s check the TODO comments once more.

$ rg 'TODO:' 
...
- ✅ pipeline/src/pipeline/api/update.rs:    /// TODO: Add a repository_dirs completer (11:08)
- ✅ file/src/remove/mod.rs:    /// TODO: Add a storage_identifier completer (11:08)
- ✅ file/src/bring/mod.rs:    /// TODO: Add a storage_identifier completer (11:08)
...

🦊 We can use xvc_path_completer for the tracked directory completer for the time being. For pipeline update, we can use a ValueHint instead. It’s not necessary to use a custom completion.

🐢 For the xvc file copy destination, we can have a file or a directory that we track and is not available, or we don’t track and is available. It’s a similar situation to xvc_path_completer, but we also need to check the local paths. It requires some more care.

🐇 Now we can begin to run the tests.

🐢 Is this building?

🐇 It should.

🐢 We also have one bug. xvc shouldn’t print help text when the COMPLETE environment variable is set.

🐇 Umm, yeah, that’s a blocker.

🐢 Fixed it. We can bump the version and push the changes.

devlog 18

🐇 There is no xvc completions command anymore. Let’s remove the test.

🐢 Instead, we can make it test with environment variables for each of these shells.

🐇 It looks like the test fails with wait_status.

cargo test -p xvc --test test_completions
...
failures:

---- test_completions stdout ----
...
ExitStatus(unix_wait_status(512))

thread 'test_completions' panicked at lib/tests/common/mod.rs:46:5:
Command failed: Command { cmd: "/Users/iex/github.com/iesahin/xvc/target/debug/xvc", stdin: None, timeout: None }
...

🐢 The failure is in the line:

    let mut cmd = Command::cargo_bin("xvc").unwrap();

So it probably waits for user input to complete, but it doesn’t have any and cannot complete it, so it returns an error.

🦊 Let’s comment the earlier test out and check if other tests work.

🐢 Yes, they fail the same. When there are no commands to run, xvc returns an error. Maybe we can change this behavior or handle the error in the Command::cargo_bin line above.

🐇 I checked the code and we don’t handle the “no arguments” case anywhere. It looks like the behavior to return an error code is inherited from clap.

🐢 Updated the error handling code to report a more descriptive message from the source.

cargo test -p xvc --test test_completions
...
failures:

---- test_completions stdout ----
Output { status: ExitStatus(unix_wait_status(25856)), stdout: "", stderr: "\nthread 'main' panicked at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/debug_asserts.rs:341:13:\nCommand pipeline: command `export` alias `l` is duplicated\nstack backtrace:\n   0: rust_begin_unwind\n             at /rustc/b1a7dfb91106018f47ed9dc9b27aee1977682868/library/std/src/panicking.rs:692:5\n   1: core::panicking::panic_fmt\n             at /rustc/b1a7dfb91106018f47ed9dc9b27aee1977682868/library/core/src/panicking.rs:75:14\n   2: clap_builder::builder::debug_asserts::assert_app\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/debug_asserts.rs:341:13\n   3: clap_builder::builder::command::Command::_build_self\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4173:13\n   4: clap_builder::builder::command::Command::_build_recursive\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4076:9\n   5: clap_builder::builder::command::Command::_build_recursive\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4078:13\n   6: clap_builder::builder::command::Command::build\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4071:9\n   7: clap_complete::env::CompleteEnv<F>::try_complete_\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_complete-4.5.40/src/env/mod.rs:229:9\n   8: clap_complete::env::CompleteEnv<F>::try_complete\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_complete-4.5.40/src/env/mod.rs:210:9\n   9: xvc::main\n             at ./src/main.rs:18:30\n  10: core::ops::function::FnOnce::call_once\n             at... [truncated]
ExitStatus(unix_wait_status(25856))

...

error: test failed, to rerun pass `-p xvc --test test_completions`

🐢 The real issue is when we assert the successful run, at this line:

            assert!(output.status.success(), "Command failed: {:?}", prepared);

🐇 If you look at the error message carefully, you’ll see that this is a different error. We added an alias to xvc pipeline export with l, which is a duplicate.

🐢 Oops, yeah.

cargo test -p xvc --test test_completions
...
test test_completions ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.81s

🐇 There are some warnings in compilation. Let’s fix these.

cargo build
   Compiling xvc-storage v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/storage)
   Compiling xvc-file v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/file)
   Compiling xvc-pipeline v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/pipeline)
   Compiling xvc v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/lib)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.28s

🐢 Finished the build without warnings. Now we can run the whole test suite again.

🐇 Tests are running fine. Updated some error messages and types, and doc tests also pass. There is an issue with the Rsync storage ref.

🐢 It looks like we try to elide output with [...], while the proper format is [..].

🐇 Replaced them with ... that will elide multiple lines.

🐢 Pushed changes to the server.

devlog 19

🐇 Let’s start by checking the GitHub Actions results.

$ ghrl | first
╭──────────────┬─────────────────────────────────────────────────────────╮
│ conclusion   │ failure                                                 │
│ displayTitle │ Add CLI completions                                     │
│ headBranch   │ clap-complete-16608                                     │
│ url          │ https://github.com/iesahin/xvc/actions/runs/13007688178 │
╰──────────────┴─────────────────────────────────────────────────────────╯

🐢 Although I turned off most of the watches, logs are still so large that it’s not possible to view them from the interface. Downloaded the log archive.

🦊 We can have different GitHub Actions steps for each test. Claude can help write such a repeating set of steps.

🐇 We can at least separate z_test_docs to see if integration tests or that fails.

🐢 We’ll have to add caching for test artifacts to upload them for coverage. I’m not sure we really need to add that complexity to the process just to avoid downloading the logs.

🐇 We can actually move all to Xvc. We need a GitHub Action to run an Xvc pipeline.

🐢 Eventually yes, we should move our testing to Xvc itself. Now, the logs show that there are differences in z_test_docs actually.

🐇 When I run the command below, it passes. We may have a different config in GitHub Actions.

$ XVC_TRYCMD_TESTS=storage,file,pipeline,core,start TRYCMD=overwrite rws cargo test --features test-ci -p xvc --test z_test_docs
test z_doc_tests ... ok
...
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 15.34s

🐢 We don’t have an XVC_TRYCMD_TESTS=storage,file,pipeline,core,start definition in GitHub Actions; let’s add it, bump the version, and try again.

$ cargo set-version "0.6.14-alpha.10"
   Upgrading xvc from 0.6.14-alpha.9 to 0.6.14-alpha.10
...

🦊 We could also update run-tests.zsh to get a quick response.

🐢 Yep, let’s do that as well.

🐇 Dev tests pass, but there are differences in the documents still. For some reason, the local run-tests.zsh doesn’t update xvc/book/src/ref/xvc-storage.md. It only has the help text as a reference, but it’s not updated with the aliases, and it breaks the CI. This is weird but a small issue. Fixed it manually.

$ ghrl | first 2
╭───┬────────────┬───────────────────┬────────────────────┬────────────────────╮
│ # │ conclusion │   displayTitle    │     headBranch     │        url         │
├───┼────────────┼───────────────────┼────────────────────┼────────────────────┤
│ 0 │            │ Add CLI           │ clap-complete-1660 │ https://github.com │
│   │            │ completions       │ 8                  │ /iesahin/xvc/actio │
│   │            │                   │                    │ ns/runs/1302770108 │
│   │            │                   │                    │ 2                  │
│ 1 │ success    │ Add CLI           │ clap-complete-1660 │ https://github.com │
│   │            │ completions       │ 8                  │ /iesahin/xvc/actio │
│   │            │                   │                    │ ns/runs/1302755462 │
│   │            │                   │                    │ 2                  │
╰───┴────────────┴───────────────────┴────────────────────┴────────────────────╯

🐢 The earlier one has passed. It looks like we’re ready to merge. Let’s update the CHANGELOG.md for release.

🐇 Setting the release version:

cargo set-version "0.6.14"
   Upgrading xvc from 0.6.14-alpha.10 to 0.6.14
...

🐢 Let’s check the CI

$ ghrl | first 2

╭───┬────────────┬───────────────────┬────────────────────┬────────────────────╮
│ # │ conclusion │   displayTitle    │     headBranch     │        url         │
├───┼────────────┼───────────────────┼────────────────────┼────────────────────┤
│ 0 │ success    │ Add CLI           │ clap-complete-1660 │ https://github.com │
│   │            │ completions       │ 8                  │ /iesahin/xvc/actio │
│   │            │                   │                    │ ns/runs/1302770108 │
│   │            │                   │                    │ 2                  │
│ 1 │ success    │ Add CLI           │ clap-complete-1660 │ https://github.com │
│   │            │ completions       │ 8                  │ /iesahin/xvc/actio │
│   │            │                   │                    │ ns/runs/1302755462 │
│   │            │                   │                    │ 2                  │
╰───┴────────────┴───────────────────┴────────────────────┴────────────────────╯

🐇 And merge:

$ ghpM --body $"(open CHANGELOG.md | lines | skip 2 | take 7)" --subject "Add completions" --squash

🐢 Tagged main and pushed. Packages should be built in a few minutes.

devlog 20

🐢 We have an error in the publish action. Let’s fix it and rerun it.

🐇 There are two errors. One resulted from forgetting sudo when installing dependencies. The other was the incorrect name for the OpenSSL library. It should be libssl-dev instead of openssl-dev. Maybe we can link the command list in the docs.

🐢 Oops, yes, we should at least keep that in mind.

🐇 I’m checking the most popular crates. getrandom retrieves a random number from the system. It looks much lighter than the rand crate.

🦊 We can use once_cell to initialize XvcEntityCounter. We currently use Once for this purpose.

🐇 I don’t think it will provide any better features.

🦊 For that case, yes, no better features. But the interface is something like:

impl<T> OnceCell<T> {
    const fn new() -> OnceCell<T> { ... }
    fn set(&self, value: T) -> Result<(), T> { ... }
    fn get(&self) -> Option<&T> { ... }
}

And this makes, for example, working with XvcRoot much easier. We are passing Arc<RwLock<XvcRootInner>>> everywhere. This is a heavy price when we only use it in a read-only manner. We can prevent most of these, when we use a read lock, by using OnceCell.

🐇 XvcConfig can benefit from this as well. We don’t update the config during runs.

🐢 Why do we want to assign it, though? We currently have a config field in XvcRootInner, and we get a reference to it with the config() method.

🐇 Ok. Let’s skip this for now. No need to worry before measuring the performance impact.

devlog 21

🐢 Now, the next version will have a --json output for xvc file list. We can start working on it or update the Readme file?

🐇 What about adding at least command completions for Nushell?

🐢 Let’s read a bit about clap_complete_nushell.

🦊 There seems to be a nu-complete command. Let’s check its documentation.

🐇 Nothing was found, and Kagi doesn’t help much either.

🐢 There is a completions document for Nushell: https://www.nushell.sh/book/custom_completions.html

🐇 There is a tool called carapace to provide completions across shells.

🐢 Its documentation is thin, and I’m not sure if it supports dynamic completions out of the box. I believe instead of adding a carapace setup, we can just write a Nushell completion script that will use JSON output from the commands and add some (maybe hidden) utility commands to support it.

🐇 There are a set of example scripts in the Nushell repo: https://github.com/nushell/nu_scripts/tree/main/custom-completions

🐢 The reason I want to write custom completions for Nushell is that it will be an exercise for the scripting language. gh completions are not as scary as a Bash script.

🐇 git completions are a better example for XVC. They simply run git whenever necessary. We can start from a static completions command and update this with dynamic completions manually. It will teach a lot.

🐢 I forked the nu_scripts repo and will add XVC completions script there.

🐇 Then let’s begin by adding Nushell static completions. Shall we add a command for this?

🦊 Reviving the completion command we removed in 0.6.13?

🐢 We shouldn’t list it. We can make a _comp subcommand for the time being and generate and distribute completions in the repository. When clap_complete_nushell has the feature parity to provide dynamic completions, we can remove these commands.

🐇 What will we use this for other than generating completions?

🐢 Maybe dynamic completions can call this as well.

🐇 Added Nushell static completions to be output using xvc _comp generate-nushell. Let’s bump up the version to 0.6.15.

cargo set-version 0.6.15-alpha.1
   Upgrading xvc from 0.6.14 to 0.6.15-alpha.1
...

🐢 I noticed we forgot a line in the CLI command handler that asserts xvc_root_opt.is_some(), and this fails when we run xvc outside of repositories. We need to release this version quickly.

🐇 Oops, now, ok, let’s write a static Nushell generator and just release quickly.

🦊 Generating completions with

xvc comp generate-nushell

🐢 Completion command is run with comp instead of _comp. Should we rename it?

🐇 Renamed it to _comp. It’s not hidden, but at least we can be sure that it won’t be misunderstood as a common command.

🐢 Bumping the version again. Now let’s source the generated script and test it.

cargo set-version 0.6.15-alpha.2
   Upgrading xvc from 0.6.15-alpha.1 to 0.6.15-alpha.2
...

🦊 Yep, it works. We now have completions for Nushell.

🐢 Let’s update the completions documentation.

🐇 Done. Now, let’s take a look at CI and see what fails.

ghrl | first
╭──────────────┬─────────────────────────────────────────────────────────╮
│ conclusion   │ success                                                 │
│ displayTitle │ Add Nushell completions                                 │
│ headBranch   │ nushell-completions                                     │
│ url          │ https://github.com/iesahin/xvc/actions/runs/13070200765 │
╰──────────────┴─────────────────────────────────────────────────────────╯

🐢 It fails because of coverage, not the tests. Codecov says the new code isn’t tested.

🐇 The added xvc _comp command isn’t tested. We can add a test running those lines and testing if the command outputs a completion script.

🐢 We have a test for completions. We can add a test that runs the lines.

🐇 Added a test and bumping up the version.

cargo set-version 0.6.15-alpha.3
   Upgrading xvc from 0.6.15-alpha.2 to 0.6.15-alpha.3
...

🦊 We can add some more coverage while waiting for the tests.

🐇 XvcOutputLine implementation seems to have no tests. It’s weird because we use these everywhere.

🐢 I’m not sure we use this particular implementation; we just use XvcOutputLine::Info(s), not XvcOutputLine::info(s) anywhere. We can delete these methods actually.

🐇 We’ll add JSON output via this particular struct. Can we refactor these to use formatting for JSON, for example? Or use these to output JSON?

🦊 We can add a formatter to XvcOutputLine to output structures.

🐢 The enum is now defined as:

#[derive(Clone, Debug)]
pub enum XvcOutputLine {
    /// The output that we should be reporting to user
    Output(String),
    /// For informational messages
    Info(String),
    /// For debug output to show the internals of Xvc
    Debug(String),
    /// Warnings that are against some usual workflows
    Warn(String),
    /// Errors that interrupts a workflow but may be recoverable
    Error(String),
    /// Panics that interrupts the workflow and ends the program
    /// Note that this doesn't call panic! automatically
    Panic(String),
    /// Progress bar ticks.
    /// Self::Info is also used for Tick(1)
    Tick(usize),
}

Here, these fields can also have a formatter that will render the string in a particular format. For example, the output can be

XvcOutputLine::Output(XvcJsonFormatter, String)

🐇 I’m not sure this is a good idea. Output already specifies this string as output. We can have a wrapper instead, like,

struct XvcJsonOutput(Format<XvcStructuredOutput>, XvcStructuredOutput)

and we can use the supplied format to render XvcStructuredOutput to an output line with XvcOutputLine::Output. If we don’t provide output as structured, it will be too much error-prone work to convert the current outputs to structured.

🦊 The transition will also be gradual. We may not need structured output for most of the commands. We can start with xvc file list and convert others as we go.

🐢 This is sensible. By the way, coverage still didn’t increase. There may be something going on with Codecov or running the test.

ghrl | first
╭──────────────┬─────────────────────────────────────────────────────────╮
│ conclusion   │ success                                                 │
│ displayTitle │ Add Nushell completions                                 │
│ headBranch   │ nushell-completions                                     │
│ url          │ https://github.com/iesahin/xvc/actions/runs/13087431038 │
╰──────────────┴─────────────────────────────────────────────────────────╯

🐇 Let’s run the test:

cargo test -p xvc --test test_completions
...
test test_completions ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.80s

🐢 Can we make sure the output is a Nushell script and not an error message?

🐇 Let’s print it out.

🐢 It looks like when the COMPLETE environment variable is set, it never calls _comp subcommand and never calls those lines.

cargo set-version 0.6.15-alpha.4
   Upgrading xvc from 0.6.15-alpha.3 to 0.6.15-alpha.4
...

🐢 Let’s make a release for 0.6.15. Coverage is OK now.

cargo set-version 0.6.15
   Upgrading xvc from 0.6.15-alpha.4 to 0.6.15
...

gh pr merge --squash --body $"(open CHANGELOG.md | lines | skip 2 | take 5)" --subject "Add static nushell completions"

🐇 Merged the PR.

🐢 Releases should appear in a few minutes.

🐇 We need to tag the merge commit for this.

🐢 Oh, yep. AFAIK Lazygit doesn’t have something for git push --tags. Let’s push from the CLI.

git push --tags
You are on the main branch. Skipping CHANGELOG.md check.
To github.com:iesahin/xvc
 * [new tag]         v0.6.15 -> v0.6.15

🦊 These commands, especially tables, are not rendered correctly on the web. We need to change the theme, I think.

devlog 22

🐢 We need to find a theme for the blog. The current one breaks Nushell output tables because the code blocks are too narrow.

🐇 https://www.getzola.org/themes/pico/ is an option, but I don’t like its header.

🦊 Minimal Dark from the same author looks better: https://kuznetsov17.github.io/minimal-dark/notes/note1/

🐇 https://www.getzola.org/themes/no-style-please/ is also an option.

🦊 https://halve-z.netlify.app/posts/information/ looks interesting, but there is too much screen estate for the left bar.

🐢 Let’s start by running the site locally first.

🐇 There are errors in the configuration. That’s weird, but let’s fix these.

🐢 Fixed errors. These are probably related to a newer version. Can we take a look at netlify.toml to see if it downloads the same version?

🐇 There are breaking changes in Zola 0.19. Let’s update and push the Netlify config to see the results.

🐢 We need to add language support for Nushell to prevent warnings. Take a look at how to add a syntax file to Zola.

🐇 I added

extra_syntaxes_and_themes = ["syntaxes"]

to the [markdown] section and added a syntaxes/nushell.sublime-syntax file copied from https://github.com/kurokirasama/nushell_sublime_syntax. I’m getting:

Error: Reason: Error while compiling regex '\b(?x: 7z | ?
...
s-to-gdrive | usage | ver | verify | weather | wget-all | which-cd | wifi-info | wifi-pass | xls2csv | ydx | yt-api | ytcli | ytm | z | zi)\b'
Oniguruma error: target of repeat operator is not specified

🐢 The syntax highlighter may be a bit buggy. Let’s try to fix this if it’s a one-off.

🐇 Found the bug. There is a ? in the regex that causes it to fail. Now it compiles, and Nushell blocks are colored.

🐢 Cool. Let’s fix the other warnings now. shell and console are not recognized, it looks like.

🐇 There is only Bash listed in https://www.getzola.org/documentation/content/syntax-highlighting/.

🦁 What do you think about migrating to mdBook? We already maintain mdBook for XVC; what about just moving the site to mdBook?

🐢 I thought about this before, and the only downside is the lack of an RSS feed.

🦊 I found this: https://github.com/theowenyoung/mdbook-rss

🐢 Now, this changes everything. We can even move the nedriy.at site to mdBook in this case.

🐇 Then, let’s start working on this. The site will be a technical book site in this case.

🐢 Does mdBook support Nushell syntax?

🐇 Nushell is not in the listed languages. mdBook uses highlight.js, and in its listed languages, we don’t find Nu either.

🐢 We already added Nu support to Zola, and we can just change the theme. This is a blocker in my opinion.

🐇 I searched for Nushell highlight.js support, and nothing appears. I think we can just postpone until Nu has more support on this front.

🐢 Yes. Let’s first try this change in the non-technical blog, and we can come back to this issue. Now, we’ll update console and shell to bash, I think.

🐇 Replaced shell and console with bash.

🐢 There is a file for ggplot that has warnings from earlier incarnations. We also lack syntax highlighters for Vim and Tmux.

🐇 There is a sublime-syntax file for Tmux at https://raw.githubusercontent.com/gerardroche/sublime-tmux/refs/heads/master/Tmux.sublime-syntax, but do we need it for a single file?

🐢 Let’s set it to plain text.

🐇 Now we only have Mermaid warnings left.

🐢 There should be a diagram at https://emresahin.net/developing-a-gitignore-crate/, but it doesn’t show up. We need a shortcode to show these, like the YouTube shortcode. Now let’s get back to theme selection.

🐇 I tested Karzok, but it doesn’t have category and tags support.

🐢 And I tested https://github.com/micahkepe/radion, but the best so far is the apollo theme. I’m struggling to modify the index page, though. I forgot that I modified the theme’s index.html file. I have content in /content/_index.md and a modified index.html in /themes/anemone/templates/index.html to show the content, tags, categories, etc. It should be fixed now.

🐇 Ah, cool. Can we clean the recent duplicate pages now?

🐢 Yeah, let’s take a look.

devlog 23

🐇 Let’s turn to discussing the JSON output changes. We can add another option to XvcOutputLine, like XvcOutputLine::Json(T: Serialize), that will output the type using serde_json. It won’t introduce any other type.

🐢 Yes, but what will T be? Although store structures have Serde implementations, they are not particularly useful for this.

🐇 We can have output types, named like XvcFileListOutput, that will be converted to strings with serialization.

🦊 We do something similar in xvc pipeline export and import commands. We use XvcPipelineSchema and XvcStepSchema just for the import and export commands. We’ll write similar structs for all JSON output and will use Serde to convert these to strings.

🐢 Unlike the import and export commands, we have optional fields in the output, though. I don’t want content digests to appear in JSON output if they are not required.

🐇 Let’s search for optional fields in Serde.

🦊 There is a crate for optional fields.

🐢 We don’t need another crate for this. Serde has the skip_serializing_if attribute for fields. We can add Option::is_none as a method to these to skip outputting None fields. All those fields, in this case, will be optional.

🐇 This is fine. We already use structs to format the xvc file list output. We can just use them to output JSON.

devlog 24

🐇 Is there a way to implement Default for CLI structs?

🐢 There is a way if we start from the configuration, not just files. The default configuration is a TOML document. We can make it an XvcConfiguration struct and load and store it with confy.

🐇 We have a cascading set of configurations, but maybe we can start from the struct and serialize/deserialize it on demand.

🐲 I don’t think we need to update xvc-config at the moment. It’s working, and we don’t need to alter its inner workings in the near future.

🐢 I agree. We can use the config crate when we need to update and have enough time to work on this.

devlog 25

🐢 I want to update xvc.py to the latest version.

🐇 It should only be needed to update the dependency versions in Cargo.toml, right?

🐢 Let’s start with that.

🐇 We have interface changes regarding aliases; let’s start using uv for building.

🦊 Added requirements to pyproject.toml by running:

uv add -r requirements.txt

devlog 26

🐢 We can add type checking to xvc.py’s command-line handler. Currently, it builds a command line manually from the given options and parses it with clap. It’s error-prone.

🐇 What are the options, though? xvc.py is just a wrapper around xvc, and that was the easiest way to get it working. We can list all options manually in the headers as documentation, but it will be harder to maintain.

🐢 Once we start doing it, we’ll find a good way to simplify and shorten it.

🐇 Let’s start to work on xvc file track, then.

🐢 Worked on it a bit, and I decided it’s not worth it at the moment. We need to supply default values for most of the options. Maintaining a separate list of default values may not be feasible; it will be error-prone in a different way.

devlog 27

🐇 Tests are failing again:

ghrl | get url | first
https://github.com/iesahin/xvc/actions/runs/13875177382

🐢 We forgot to update the doc tests. Let’s run them again to update storage remove and file untrack commands.

🐇 There are issues with elision.

ghpl

╭───┬──────────────────────┬────────────────────┬──────────────────────────────╮
│ # │     headRefName      │       title        │             url              │
├───┼──────────────────────┼────────────────────┼──────────────────────────────┤
│ 0 │ storage-remove-16674 │ xvc storage remove │ https://github.com/iesahin/x │
│   │                      │                    │ vc/pull/270                  │
╰───┴──────────────────────┴────────────────────┴──────────────────────────────╯

🐢 Tests are passing; we can merge the PR. But our commit hook to check the CHANGELOG doesn’t work. That’s weird; we don’t get any errors when the CHANGELOG is not in the push set.

tmux new-window -c ($env.HOME | path join github.com iesahin xvc.py)  nvim 

🐢 Now we can update the Python bindings as well.

🐇 We forgot to bump package versions. We need to create a checklist for releases.

• ✅ #🌻 CREATE a release checklist (2025-03-25 17:52)

🐢 Let’s check if the latest version is updated on PyPI.

pypi xvc

🐇 Yes, it is.

devlog 28

🐢 We can start adding an rclone remote as well.

🐢 Created the PR, waiting for the tests.

🦊 Let’s search if there is a crate to manage rclone. Maybe it will be easier that way.

🐇 It will add another dependency though.

🦊 We can always add a feature flag for this.

🐢 There is a librclone crate that can be used to call rclone commands like https://rclone.org/rc/#supported-commands

🦊 We can test it from the command line perhaps.

rclone rc 
2025/03/15 17:41:09 NOTICE: Failed to rc: failed to list: connection failed: Post "http://localhost:5572/rc/list": dial tcp [::1]:5572: connect: connection refused

🐇 It requires the backend to be running in the background.

🦊 There may be examples in the repository.

🐢 There are none. We can search GH for this crate though.

🦊 It’s also possible to search for dependents in crates.io.

🐢 I think Xvc will be the first dependent of this crate: https://crates.io/crates/librclone/reverse_dependencies

🐇 The following two projects depend on librclone:

• https://github.com/Sh3mm/WarpDrive/tree/master
• https://github.com/hwittenborn/celeste

🐢 Let’s clone Celeste. It uses librclone and looks like it’s a user interface for rclone written in Rust.

🐇 The examples are in celeste/src/rclone.rs.

🐢 Cool. Let’s take a look at how commands are run:

    /// Common function for some of the below command.
    fn common(command: &str, remote_name: &str, path: &str) -> Result<(), RcloneError> {
        let resp = run(
            command,
            &json!({
                "fs": get_remote_name(remote_name),
                "remote": util::strip_slashes(path),
            })
            .to_string(),
        );

        match resp {
            Ok(_) => Ok(()),
            Err(json_str) => Err(serde_json::from_str(&json_str).unwrap()),
        }
    }

All commands are run like librclone::rpc(method, input)) and the commands are like:

    /// make a directory on the remote.
    pub fn mkdir(remote_name: &str, path: &str) -> Result<(), RcloneError> {
        common("operations/mkdir", remote_name, path)
    }

🐇 We have all commands in this file that are relevant to Xvc. Let’s list them here:

• make directory: common("operations/mkdir", remote_name, path)
• delete file: common("operations/delete", remote_name, path)
• remove a dir and all of its contents: common("operations/purge", remote_name, path)
• copy file:

run( "operations/copyfile",
            &json!({
                "srcFs": src_fs,
                "srcRemote": util::strip_slashes(src_remote),
                "dstFs": dst_fs,
                "dstRemote": util::strip_slashes(dst_remote)
            })

and

    /// Copy a file from the local machine to the remote.
    pub fn copy_to_remote(
        local_file: &str,
        remote_name: &str,
        remote_destination: &str,
    ) -> Result<(), RcloneError> {
        copy(
            "/",
            local_file,
            &get_remote_name(remote_name),
            remote_destination,
        )
    }

    /// Copy a file from the remote to the local machine.
    pub fn copy_to_local(
        local_destination: &str,
        remote_name: &str,
        remote_file: &str,
    ) -> Result<(), RcloneError> {
        copy(
            &get_remote_name(remote_name),
            remote_file,
            "/",
            local_destination,
        )
    }

🐇 It looks like that’s all we need. We can organize the commands differently, but these examples are enough to use librclone. It seems rather straightforward.

devlog 29

🐢 As the new version is updated, we can go back to the project. What’s the next step?

🐇 We can continue working on rclone.

🐢 Umm, ok. Let’s try to focus on adding another storage type.

• ✅ #🌻 ADD rclone storage type (2025-04-24 06:27)

🐢 I think the first option is to run the commands from the command line. We can just use a modified generic storage type without trying to make it fast.

🐇 Let’s make it run first, you say?

🐢 Yes, let’s make it run first and then we can think about making it run fast.

🐇 You’re right!

• ✅ #🌻 ADD generic rclone tests (2025-04-24 06:27)

It’s possible to use the alias remote with a local path.

devlog 30

🐢 Let’s discuss how to move Xvc forward—maybe we can write a post to Reddit and the Rust forum in the meantime. 🐇 I think the next step is rclone remotes. It will allow us to use all remote storages supported by Rclone, which is a nice feature. 🐢 Don’t you think we need to publish the current version to Reddit and the forum? 🐇 We can do that as well. 🦊 Adding rclone remote must be a straightforward task. 🐢 We need to understand rclone paths, but overall, yes. We’ll just need to get the remote name, like drive://, and a path, like my-xvc-storage, and build paths with these. 🐇 What are the commands? 🐢 We need to learn how to upload files from local to remote and how to download these files. We can also list the files and get files as well. 🐲 How about adding a paths.txt to folders in remotes to show which paths the files in 0.jpg belong to? This will change the remote cache structure a bit. We will have a reverse index of files and they will be findable. 🐢 What’s the reason for this? 🐲 When I upload a file to Drive with only the content hash, I lose track of the actual path. This is not desirable. We can add a file to the directory, called paths.txt, to get the paths for a file. 🐢 This may prove to be a feat, though; adding these XvcPaths to a file requires a lookup. 🐇 Maybe a JSON file? It might be possible to look up a path with a JSON file, and it will be easier to parse. 🐲 I don’t think the issue is about parsing, though. We can just have a plain text file that lists the paths. It’s a text file, which is the most compatible across all storages. 🐢 Storages, you mean. 🐲 Ugh, yeah. If I have a file called Alan Watts but I only have the content, this file will be immensely useful. 🐢 This makes XvcCachePath and XvcPath coupled. Architecture-wise, it may not be a good thing, though. 🐇 Also, there may be common storages for multiple repositories. 🐲 Umm, that’s a good point. I don’t think the architecture will be much compromised, though. We already keep the file paths and their cache paths somewhere. 🐢 Cache paths are generated from the content, but any number of paths can point to a single path in the cache. If I have 1 million copies of the same file, will I add all these files to the paths.txt you mentioned? 🐲 That’s a good point too. We can have a limit, like 1,000 or something, not to make these files too big. 🐢 Instead of this, we can store the output of xvc file list at the storage root and allow looking up the files that way. 🐲 It has the same problem, though; if we have a million files, their list will be too large. 🐢 There can be a manual command, like xvc file index --to storage, that will show content hashes and paths of each file. We can also add URLs to files if possible. 🐲 No one will use it when it’s manual, though. 🐢 We can add functionality to update this index when we send a file, though.
🐲 So, after each send, we’ll update the index for the repository on that storage. Is that correct? 🐢 Not after each send. After each send session, maybe. 🐇 We can have an incremental way of updating the index, like we do in ECS? 🐢 It will be overkill for this functionality and add too much noise to the storage. 🐲 Let’s keep this discussion here, but I also want to have an index merge or index cleanup mechanism for the entity generator and the ECS. 🐢 We can have a “merge indices” functionality in ECS. That will remove all older entity-generator files and merge all store files. 🐇 Removing older entity files is easy, but what about merging the store files? 🐢 It’s easy too. We’ll just load all event logs from the directory, remove all other files, and save the event log to a file. 🐇 Will this be manual or automatic? 🐢 I think the first version can be manual, something like xvc fsck merge-store-files or something like that. We can notify the user if the number of files is > 10,000 or something like that. I don’t think we need to make it automatic unless we measure the impact of these files. There is no point in trying to do it at every command. 🐇 Then we’ll have two new commands for the next version? 🐢 I think we can just add rclone remote now and release it, then make changes in the ECS for this new xvc fsck command. 🐇 Can the name be doctor or something? Or util? Or can we add a top-level merge indices command? 🐢 xvc doctor seems like a better alternative. We can have a diagnose subcommand as well to check for possible inconsistencies. xvc doctor merge-store-files is a better command. 🐲 Will we use d for this command? 🐢 No need to add a single-letter command for this, I believe. It shouldn’t be required to run frequently. 🐇 Hmm, ok. What do we need to know for rclone remote? 🐲 I noticed we don’t have the xvc storage remove command implemented yet. Maybe we can start from that. 🐢 Hmm, yeap. Let’s start by implementing that first. We can add the rclone command next. 🐇 Will we use a feature flag for rclone? It will run the command only with an external binary. 🐢 It’s better to have a feature flag. I think we can add a feature flag for rsync remote as well. 🐇 We can use the generic one to update the feature flag. 🐢 I think the only two items of information we need for rclone are the remote name and the remote directory. Will we make these required?

devlog 31

🐢 I think sending Alt keys in Ghostty needs some love. Let’s start with it as an easy task for the day.

🐇 While checking macos-option-as-alt as a potential source of the problem, I spent time adjusting the Ghostty icon. I changed the app icon. This is how I spend my time being extremely productive.

🐢 Weird. The Option as alt key setting is fine.

🐇 Actually, the Alt key works; the reason we cannot make panes larger is that Alt-Right and Alt-Left are not bound to this.

🐢 You may be right, yes. Then, what’s the key to enlarge a pane?

🦊 List all the keys. Check the keys. You should have done this already.

$ tmux list-keys | rg resize

bind-key    -T prefix       >                      display-menu -T "#[align=centre]#{pane_index} (#{pane_id})" -x P -y P "#{?#{m/r:(copy|view)-mode,#{pane_mode}},Go To Top,}" < { send-keys -X history-top } "#{?#{m/r:(copy|view)-mode,#{pane_mode}},Go To Bottom,}" > { send-keys -X history-bottom } '' "#{?mouse_word,Search For #[underscore]#{=/9/...:mouse_word},}" C-r { if-shell -F "#{?#{m/r:(copy|view)-mode,#{pane_mode}},0,1}" "copy-mode -t=" ; send-keys -X -t = search-backward "#{q:mouse_word}" } "#{?mouse_word,Type #[underscore]#{=/9/...:mouse_word},}" C-y { copy-mode -q ; send-keys -l "#{q:mouse_word}" } "#{?mouse_word,Copy #[underscore]#{=/9/...:mouse_word},}" c { copy-mode -q ; set-buffer "#{q:mouse_word}" } "#{?mouse_line,Copy Line,}" l { copy-mode -q ; set-buffer "#{q:mouse_line}" } '' "#{?mouse_hyperlink,Type #[underscore]#{=/9/...:mouse_hyperlink},}" C-h { copy-mode -q ; send-keys -l "#{q:mouse_hyperlink}" } "#{?mouse_hyperlink,Copy #[underscore]#{=/9/...:mouse_hyperlink},}" h { copy-mode -q ; set-buffer "#{q:mouse_hyperlink}" } '' "Horizontal Split" h { split-window -h } "Vertical Split" v { split-window -v } '' "#{?#{>:#{window_panes},1},,-}Swap Up" u { swap-pane -U } "#{?#{>:#{window_panes},1},,-}Swap Down" d { swap-pane -D } "#{?pane_marked_set,,-}Swap Marked" s { swap-pane } '' Kill X { kill-pane } Respawn R { respawn-pane -k } "#{?pane_marked,Unmark,Mark}" m { select-pane -m } "#{?#{>:#{window_panes},1},,-}#{?window_zoomed_flag,Unzoom,Zoom}" z { resize-pane -Z }
bind-key    -T prefix       z                      resize-pane -Z
bind-key -r -T prefix       M-Up                   resize-pane -U 5
bind-key -r -T prefix       M-Down                 resize-pane -D 5
bind-key -r -T prefix       M-Left                 resize-pane -L 5
bind-key -r -T prefix       C-Up                   resize-pane -U
bind-key -r -T prefix       C-Down                 resize-pane -D
bind-key -r -T prefix       C-Left                 resize-pane -L
bind-key -r -T prefix       C-Right                resize-pane -R
bind-key    -T root         MouseDown3Pane         if-shell -F -t = "#{||:#{mouse_any_flag},#{&&:#{pane_in_mode},#{?#{m/r:(copy|view)-mode,#{pane_mode}},0,1}}}" { select-pane -t = ; send-keys -M } { display-menu -T "#[align=centre]#{pane_index} (#{pane_id})" -t = -x M -y M "#{?#{m/r:(copy|view)-mode,#{pane_mode}},Go To Top,}" < { send-keys -X history-top } "#{?#{m/r:(copy|view)-mode,#{pane_mode}},Go To Bottom,}" > { send-keys -X history-bottom } '' "#{?mouse_word,Search For #[underscore]#{=/9/...:mouse_word},}" C-r { if-shell -F "#{?#{m/r:(copy|view)-mode,#{pane_mode}},0,1}" "copy-mode -t=" ; send-keys -X -t = search-backward "#{q:mouse_word}" } "#{?mouse_word,Type #[underscore]#{=/9/...:mouse_word},}" C-y { copy-mode -q ; send-keys -l "#{q:mouse_word}" } "#{?mouse_word,Copy #[underscore]#{=/9/...:mouse_word},}" c { copy-mode -q ; set-buffer "#{q:mouse_word}" } "#{?mouse_line,Copy Line,}" l { copy-mode -q ; set-buffer "#{q:mouse_line}" } '' "#{?mouse_hyperlink,Type #[underscore]#{=/9/...:mouse_hyperlink},}" C-h { copy-mode -q ; send-keys -l "#{q:mouse_hyperlink}" } "#{?mouse_hyperlink,Copy #[underscore]#{=/9/...:mouse_hyperlink},}" h { copy-mode -q ; set-buffer "#{q:mouse_hyperlink}" } '' "Horizontal Split" h { split-window -h } "Vertical Split" v { split-window -v } '' "#{?#{>:#{window_panes},1},,-}Swap Up" u { swap-pane -U } "#{?#{>:#{window_panes},1},,-}Swap Down" d { swap-pane -D } "#{?pane_marked_set,,-}Swap Marked" s { swap-pane } '' Kill X { kill-pane } Respawn R { respawn-pane -k } "#{?pane_marked,Unmark,Mark}" m { select-pane -m } "#{?#{>:#{window_panes},1},,-}#{?window_zoomed_flag,Unzoom,Zoom}" z { resize-pane -Z } }
bind-key    -T root         MouseDrag1Border       resize-pane -M
bind-key    -T root         M-MouseDown3Pane       display-menu -T "#[align=centre]#{pane_index} (#{pane_id})" -t = -x M -y M "#{?#{m/r:(copy|view)-mode,#{pane_mode}},Go To Top,}" < { send-keys -X history-top } "#{?#{m/r:(copy|view)-mode,#{pane_mode}},Go To Bottom,}" > { send-keys -X history-bottom } '' "#{?mouse_word,Search For #[underscore]#{=/9/...:mouse_word},}" C-r { if-shell -F "#{?#{m/r:(copy|view)-mode,#{pane_mode}},0,1}" "copy-mode -t=" ; send-keys -X -t = search-backward "#{q:mouse_word}" } "#{?mouse_word,Type #[underscore]#{=/9/...:mouse_word},}" C-y { copy-mode -q ; send-keys -l "#{q:mouse_word}" } "#{?mouse_word,Copy #[underscore]#{=/9/...:mouse_word},}" c { copy-mode -q ; set-buffer "#{q:mouse_word}" } "#{?mouse_line,Copy Line,}" l { copy-mode -q ; set-buffer "#{q:mouse_line}" } '' "#{?mouse_hyperlink,Type #[underscore]#{=/9/...:mouse_hyperlink},}" C-h { copy-mode -q ; send-keys -l "#{q:mouse_hyperlink}" } "#{?mouse_hyperlink,Copy #[underscore]#{=/9/...:mouse_hyperlink},}" h { copy-mode -q ; set-buffer "#{q:mouse_hyperlink}" } '' "Horizontal Split" h { split-window -h } "Vertical Split" v { split-window -v } '' "#{?#{>:#{window_panes},1},,-}Swap Up" u { swap-pane -U } "#{?#{>:#{window_panes},1},,-}Swap Down" d { swap-pane -D } "#{?pane_marked_set,,-}Swap Marked" s { swap-pane } '' Kill X { kill-pane } Respawn R { respawn-pane -k } "#{?pane_marked,Unmark,Mark}" m { select-pane -m } "#{?#{>:#{window_panes},1},,-}#{?window_zoomed_flag,Unzoom,Zoom}" z { resize-pane -Z }

🐇 resize-pane has keys, but they conflict with C-Right, etc., which are macOS display selection keys. I defined M-S-Right, etc., to resize and M-C-Right, etc., to swap the panes. Defining new keys with M-S…

bind-key -n "M-S-Right" resize-pane -R 10
bind-key -n "M-S-Left" resize-pane -L 10
bind-key -n "M-S-Up" resize-pane -U 10
bind-key -n "M-S-Down" resize-pane -D 10

🐢 We can set keys to swap windows too, maybe to M-C-PageDown, etc. M-PageDown now moves between windows.

🐇 Let’s take a look at the command options:

     swap-window [-d] [-s src-window] [-t dst-window]
                   (alias: swapw)
             This is similar to link-window, except the source and destination windows are swapped.  It is an error if no window exists at src-window.  If -d is given, the new window
             does not become the current window.

             If -s is omitted and a marked pane is present (see select-pane -m), the window containing the marked pane is used rather than the current window.

🐢 The use case will be limited, though.

🐇 Umm, right. Let’s stop procrastinating here.

🐢 It was productive procrastination, though. Now I can resize my tmux panes.

devlog 32

This dialog is about converting ColQwen2 to the ONNX format.

🐢 Now, I have two classes. This one is a copy of the ONNX Patcher that’s used to convert ColQwen2 to ONNX format. But this one is valid only for the underlying model, that is Qwen2VLForConditionalGeneration, and ColQwen2 also has some modifications to call it.

🐇 What are these modifications?

🐢 The forward method looks something like this:

ColQwen2

    def forward(self, *args, **kwargs) -> torch.Tensor:
        kwargs.pop("output_hidden_states", None)

        # Handle the custom "pixel_values" input obtained with `ColQwen2Processor` through unpadding
        if "pixel_values" in kwargs:
            offsets = kwargs["image_grid_thw"][:, 1] * kwargs["image_grid_thw"][:, 2]  # (batch_size,)
            kwargs["pixel_values"] = torch.cat(
                [pixel_sequence[:offset] for pixel_sequence, offset in zip(kwargs["pixel_values"], offsets)],
                dim=0,
            )

        position_ids, rope_deltas = self.get_rope_index(
            input_ids=kwargs["input_ids"],
            image_grid_thw=kwargs.get("image_grid_thw", None),
            video_grid_thw=None,
            attention_mask=kwargs.get("attention_mask", None),
        )
        last_hidden_states = self.inner_forward(
            *args, **kwargs, position_ids=position_ids, use_cache=False, output_hidden_states=True
        )  # (batch_size, sequence_length, hidden_size)

        proj = self.custom_text_proj(last_hidden_states)  # (batch_size, sequence_length, dim)

        # L2 normalization
        proj = proj / proj.norm(dim=-1, keepdim=True)  # (batch_size, sequence_length, dim)
        proj = proj * kwargs["attention_mask"].unsqueeze(-1)  # (batch_size, sequence_length, dim)

        if "pixel_values" in kwargs and self.mask_non_image_embeddings:
            # Pools only the image embeddings
            image_mask = (kwargs["input_ids"] == self.config.image_token_id).unsqueeze(-1)
            proj = proj * image_mask
        return proj

🐇 How about the inner_forward method? Does it just directly call Qwen2VLForConditionalGeneration’s forward?

🐢 No, it has some conditionals:

        if inputs_embeds is None:
            inputs_embeds = self.model.embed_tokens(input_ids)
            if pixel_values is not None:
                pixel_values = pixel_values.type(self.visual.get_dtype())
                image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw)
                image_mask = (input_ids == self.config.image_token_id).unsqueeze(-1).expand_as(inputs_embeds)
                image_embeds = image_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
                inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)

            if pixel_values_videos is not None:
                pixel_values_videos = pixel_values_videos.type(self.visual.get_dtype())
                video_embeds = self.visual(pixel_values_videos, grid_thw=video_grid_thw)
                video_mask = (input_ids == self.config.video_token_id).unsqueeze(-1).expand_as(inputs_embeds)
                video_embeds = video_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
                inputs_embeds = inputs_embeds.masked_scatter(video_mask, video_embeds)

            if attention_mask is not None:
                attention_mask = attention_mask.to(inputs_embeds.device)

        outputs = self.model(
            input_ids=None,
            position_ids=position_ids,
            attention_mask=attention_mask,
            past_key_values=past_key_values,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_states = outputs[0]
        return hidden_states

🦊 The patcher only modifies the past_key_values_args and moves them to a DynamicCache. There are no other changes for patching.

🐢 But the return types are also different. Qwen2VLForConditionalGeneration.forward returns a Union[Tuple, Qwen2VLCausalLMOutputWithPast], but ColQwen2.forward returns a torch.Tensor.

🐇 How is that torch.Tensor calculated from the return type of Qwen2VLForConditionalGeneration? ❓

🦊 There are multiple return types in Qwen2VLForConditionalGeneration. It’s determined by the return_dict argument.

        if not return_dict:
            output = (logits,) + outputs[1:]
            return (loss,) + output if loss is not None else output

        return Qwen2VLCausalLMOutputWithPast(
            loss=loss,
            logits=logits,
            past_key_values=outputs.past_key_values,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
            rope_deltas=self.rope_deltas,
        )

🐇 What’s that argument in ColQwen2?

🐢 It’s passed from the caller; it’s a kwarg.

🐇 Is there any modification to this in the ONNX patcher?

🐢 Nope.

🐇 Then we can consider it as the default. What’s the default?

🐢 The default is None. Hence it returns (logits,) + outputs[1:].

🐇 Then, logits are the first parameter by default.

🐢 Yes, we can assume this.

🐇 What does ColQwen2 do with these logits?

🦊 By the way, it calls inner_forward with use_cache=False and output_hidden_states=True. What do these change in Qwen2VLForConditionalGeneration?

🐢 These hidden states are actually output. In ColQwen2.forward, the inner_forward call is actually:

        last_hidden_states = self.inner_forward(
            *args, **kwargs, position_ids=position_ids, use_cache=False, output_hidden_states=True
        )  # (batch_size, sequence_length, hidden_size)

and the output is the hidden states.

🦊 It then runs:

        proj = self.custom_text_proj(last_hidden_states)  # (batch_size, sequence_length, dim)

and calculates a set of projections.

🐢 custom_set_proj is defined as:

        self.custom_text_proj = nn.Linear(self.model.config.hidden_size, self.dim)

hence these projections are fully connected layer calculations.

🦊 It returns these after L2 normalization and checks if there are pixel_values to consider.

🐢 In this case, the output from ColQwen2 is this single tensor.

🐇 Ok. Then it’s actually simpler than wrapping up the whole Qwen2VLForConditionalGeneration.

🐢 It looks so, yes. But the example for ColQwen2 also has a post-processing step:

scores = processor.score_multi_vector(query_embeddings, image_embeddings)

🐇 I think we can leave that post-processing for the time being. We only need multi-vector embeddings for FastEmbed.

🐢 Maybe it’s convertible to a forward method that we can use with ONNX.

🐇 Let’s see how this scoring works, then.

🐢 score_multi_vector receives two tensors or tensor lists and compares query vectors with passage vectors. It’s a rather straightforward implementation that requires Torch, but not Transformers.

🐇 In theory, we can also convert this to an ONNX model.

🦊 We can also write a custom model for this.

🐢 It has two for loops for comparisons. Can we convert all of these?

🦊 The loop indices can be seen as dynamic_axes, and it’s possible to convert the whole thing as a Torch model, then use torch.onnx.export just as we do for the model itself.

🐢 I see, but I don’t think that’s what we must do now.

🐇 Yes, let’s skip that for the time being and convert the model itself. We’ll have multivectors for patches and queries at the end.

🐢 So, we’ll keep the processing part, send BatchFeature objects that are output from the processor, and send this to two models: one for images and one for text.

🐇 Yep, that’s the plan. In the end, we’ll have two ONNX models that require BatchFeatures.

🐢 Then, we’ll modify past_key_values in this argument to use DynamicCache.

🐇 Yes, that’s alright. We can start by moving past_key_values_converter to a method in PatchedColQwen2.

🐢 past_key_values wasn’t used much, so I completely removed it. I also began to use processor outputs as dummy input in the exporter. However, when using BatchFeature, we get:

RuntimeError: Only tuples, lists and Variables are supported as JIT inputs/outputs. Dictionaries and strings are also accepted, but their usage is not recommended. Here, received an input of unsupported type: BatchFeature

🐇 So, we can try Dynamo first, I think. If that doesn’t work, we can just collect items as tensors and build up a BatchFeature inside the patcher.

🐢 Let’s try dynamo=True for this first.

🦊 This time, it’s about BatchFeature again, but the error is different: KeyError: 'Indexing with integers is not available when using Python based feature extractors'

🐇 In this case, we can just create BatchFeature inside the patcher.

PatchedColQwen2

class PatchedColQwen2(ColQwen2):
    def forward(self, *args):
        (
            input_ids,
            inputs_embeds,
            attention_mask,
            position_ids,
            *past_key_values_args,
        ) = args
        # Convert past_key_values list to DynamicCache
        if len(past_key_values_args) == 0:
            past_key_values = None
        else:
            past_key_values = DynamicCache()
            for i in range(self.config.num_hidden_layers):
                key = past_key_values_args.pop(0)
                value = past_key_values_args.pop(0)
                past_key_values.update(key_states=key, value_states=value, layer_idx=i)

        breakpoint()
        o = super().forward(
            input_ids=input_ids,
            inputs_embeds=inputs_embeds,
            attention_mask=attention_mask,
            # position_ids=position_ids,
            past_key_values=past_key_values,
        )

        flattened_past_key_values_outputs = {
            "logits": o.logits,
        }
        output_past_key_values: DynamicCache = o.past_key_values
        for i, (key, value) in enumerate(
            zip(output_past_key_values.key_cache, output_past_key_values.value_cache)
        ):
            flattened_past_key_values_outputs[f"present.{i}.key"] = key
            flattened_past_key_values_outputs[f"present.{i}.value"] = value

        return flattened_past_key_values_outputs