🐇 What are the options, though? xvc.py is just a wrapper around xvc, and that was the easiest way to get it working. We can list all options manually in the headers as documentation, but it will be harder to maintain.

🐢 Once we start doing it, we’ll find a good way to simplify and shorten it.

🐇 Let’s start to work on xvc file track, then.

🐢 Worked on it a bit, and I decided it’s not worth it at the moment. We need to supply default values for most of the options. Maintaining a separate list of default values may not be feasible; it will be error-prone in a different way.

🐢 Instead, we can make it test with environment variables for each of these shells.

🐇 It looks like the test fails with wait_status.

cargo test -p xvc --test test_completions
...
failures:

---- test_completions stdout ----
...
ExitStatus(unix_wait_status(512))

thread 'test_completions' panicked at lib/tests/common/mod.rs:46:5:
Command failed: Command { cmd: "/Users/iex/github.com/iesahin/xvc/target/debug/xvc", stdin: None, timeout: None }
...

🐢 The failure is in the line:

    let mut cmd = Command::cargo_bin("xvc").unwrap();

So it probably waits for user input to complete, but it doesn’t have any and cannot complete it, so it returns an error.

🦊 Let’s comment the earlier test out and check if other tests work.

🐢 Yes, they fail the same. When there are no commands to run, xvc returns an error. Maybe we can change this behavior or handle the error in the Command::cargo_bin line above.

🐇 I checked the code and we don’t handle the “no arguments” case anywhere. It looks like the behavior to return an error code is inherited from clap.

🐢 Updated the error handling code to report a more descriptive message from the source.

cargo test -p xvc --test test_completions
...
failures:

---- test_completions stdout ----
Output { status: ExitStatus(unix_wait_status(25856)), stdout: "", stderr: "\nthread 'main' panicked at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/debug_asserts.rs:341:13:\nCommand pipeline: command `export` alias `l` is duplicated\nstack backtrace:\n   0: rust_begin_unwind\n             at /rustc/b1a7dfb91106018f47ed9dc9b27aee1977682868/library/std/src/panicking.rs:692:5\n   1: core::panicking::panic_fmt\n             at /rustc/b1a7dfb91106018f47ed9dc9b27aee1977682868/library/core/src/panicking.rs:75:14\n   2: clap_builder::builder::debug_asserts::assert_app\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/debug_asserts.rs:341:13\n   3: clap_builder::builder::command::Command::_build_self\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4173:13\n   4: clap_builder::builder::command::Command::_build_recursive\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4076:9\n   5: clap_builder::builder::command::Command::_build_recursive\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4078:13\n   6: clap_builder::builder::command::Command::build\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4071:9\n   7: clap_complete::env::CompleteEnv<F>::try_complete_\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_complete-4.5.40/src/env/mod.rs:229:9\n   8: clap_complete::env::CompleteEnv<F>::try_complete\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_complete-4.5.40/src/env/mod.rs:210:9\n   9: xvc::main\n             at ./src/main.rs:18:30\n  10: core::ops::function::FnOnce::call_once\n             at... [truncated]
ExitStatus(unix_wait_status(25856))

...

error: test failed, to rerun pass `-p xvc --test test_completions`

🐢 The real issue is when we assert the successful run, at this line:

            assert!(output.status.success(), "Command failed: {:?}", prepared);

🐇 If you look at the error message carefully, you’ll see that this is a different error. We added an alias to xvc pipeline export with l, which is a duplicate.

🐢 Oops, yeah.

cargo test -p xvc --test test_completions
...
test test_completions ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.81s

🐇 There are some warnings in compilation. Let’s fix these.

cargo build
   Compiling xvc-storage v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/storage)
   Compiling xvc-file v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/file)
   Compiling xvc-pipeline v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/pipeline)
   Compiling xvc v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/lib)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.28s

🐢 Finished the build without warnings. Now we can run the whole test suite again.

🐇 Tests are running fine. Updated some error messages and types, and doc tests also pass. There is an issue with the Rsync storage ref.

🐢 It looks like we try to elide output with [...], while the proper format is [..].

🐇 Replaced them with ... that will elide multiple lines.

🐢 Pushed changes to the server.

🐇 We can just complete the completions for this version and add JSON output for all commands to use in nushell. In the meantime, dynamic completions support for nushell is likely to be added.

🐢 Umm, yep. Then we’ll continue to run tests with zsh for the current version.

🐇 run-tests is a zsh file, and we will continue to use it. We can switch to nushell eventually as we add more compatibility with it, but let’s not do this now.

🐢 Where were we the last time for completions?

🐇 We added xvc_path_completers. Let’s check the TODO list now.

🐢 We still need tracked_targets completions in a few places.

🐇 Added those.

🐢 Now we have some strum completers. Let’s finish these up as well.

🐇 Ok. They are done as well.

🐢 Now, let’s start adding a storage_identifier completer. It should read all XvcStorage records and list their names.

🐇 Added a storage_identifier completer. Let’s add it to all places where we use storage_identifiers.

🐢 Now, we need completers for pipeline and step names. These are straightforward.

🐇 Added these. Let’s fill up where they are needed.

🐢 I want to skip some of the fine-grained completions in this version. We can have a specific completer for --params options for files and params inside, for example. Or a special completer for directories.

🐇 Let’s not forget these. Reading YAML files and extracting hyperparameter keys for the prompt would be a really good feature for the user.

🐢 I agree. These are good features in general, but we need to ship the current version as soon as possible.

🐇 Let’s check the TODO comments once more.

$ rg 'TODO:' 
...
- ✅ pipeline/src/pipeline/api/update.rs:    /// TODO: Add a repository_dirs completer (11:08)
- ✅ file/src/remove/mod.rs:    /// TODO: Add a storage_identifier completer (11:08)
- ✅ file/src/bring/mod.rs:    /// TODO: Add a storage_identifier completer (11:08)
...

🦊 We can use xvc_path_completer for the tracked directory completer for the time being. For pipeline update, we can use a ValueHint instead. It’s not necessary to use a custom completion.

🐢 For the xvc file copy destination, we can have a file or a directory that we track and is not available, or we don’t track and is available. It’s a similar situation to xvc_path_completer, but we also need to check the local paths. It requires some more care.

🐇 Now we can begin to run the tests.

🐢 Is this building?

🐇 It should.

🐢 We also have one bug. xvc shouldn’t print help text when the COMPLETE environment variable is set.

🐇 Umm, yeah, that’s a blocker.

🐢 Fixed it. We can bump the version and push the changes.

🐇 Starting with completions is better. One question I have is whether the completions work for subcommands as usual, like in the static completions.

🐢 Yes, they work. We need to add:

source <(COMPLETE=zsh xvc)

to .zshrc, though.

🐇 It looks like we can drop the xvc completions command. We’re just checking the COMPLETE environment variable, and there’s no need for a separate command in this case.

🐢 Yes, let’s remove that.

🐇 Maybe we can employ a completions module for all completion-related functionality. It’s a separate thing, you know.

🐢 Which completions do we need?

🐇 We can just mark them with TODOs now.

🐢 Yes, let’s check where we need completions and what.

🐇 I noticed we’re repeating options in XvcFileCLI, and some of the options are missing, e.g., --from-ref and --to-branch.

🐢 Because it can compile into a different binary, and global options don’t work that way. Maybe we can move all these binaries into different files under lib. We can have feature flags to turn off certain features and compile different binaries.

🦊 Let’s search for it; I haven’t seen it before: building different binaries with feature flags using cargo.

🐢 There is no clear-cut solution, but it looks like we can move all binaries to the root with the required-features field.

🐲 We can postpone this to another release.

🐇 Yes, let’s not spend time on this now.

🐢 Should we try making pipeline_name a global option? It’s repeating everywhere.

🐇 That will make it easier to maintain.

🐢 We’ll have to pass pipeline_name to subcommands, though.

🐇 I think we can have a set of global options that we can pass. For the time being, that’s only the pipeline_name.

🐢 Okay. Let’s do this.

🐇 A similar option is the step name for pipeline steps.

🐢 Dependencies will need a revamp in the next version anyway. So let’s keep it for now.

🐲 Also, the semantics of step-name are different for these commands. step new interprets it as a new name, while step dependency interprets it as an existing name. The first can be renamed to --name, and the second can be --to, as one of its aliases suggests.

🐢 Yes, let’s keep it for now, and we’ll continue to work on others.

🦊 I want to post a comment in the clap discussions.

🐢 Wrote it. Now let’s build it after changing the pipeline_name.

🐇 It compiles. Should we test it?

🐢 I think we’ll test after all this completion work is done.

🐇 We did most of the strum-related completion.

🐢 Didn’t test them yet, though.

🐇 This is Rust. It will work if it compiles, and it compiles.

🐇 The examples are only found in the tests: https://github.com/clap-rs/clap/blob/master/clap_complete/tests/testsuite/engine.rs#L604

and we can try to create those random strings this way.

🦊 The issue is that we may not be able to use the derive method to add these custom completers. All examples are using the builder API.

🐢 We should be able to obtain the end result from derives and modify them to use these custom completers, but let’s try to use add = ArgValueCompleter first.

🐇 It looks like we have another problem. We need to add a rust-toolchain.toml file to the project, you know, to keep it on the stable channel.

🐢 Ah, yeah, we now use multiple channels. Let’s do that.

🐇 I found an example, actually:

#[derive(Debug, Parser)]
struct Cli {
    #[arg(long, add = ArgValueCompleter::new(custom_completer))]
    custom: Option<String>,
}

🐢 I’m trying to add this to the --from-ref option of Xvc. It’s not running anything. The custom completer should return a random string, but it doesn’t.

🦊 Maybe test with a constant first.

🐢 It didn’t work that way either.

🐇 Let’s do a cargo clean.

🐢 Nothing has changed.

🐇 Let’s take a look at the output of xvc completions:

xvc completions
...
'(--skip-git)--from-ref=[Checkout the given Git reference (branch, tag, commit etc.) before performing the Xvc operation. This runs \`git checkout <given-value>\` before running the command]:FROM_REF:_default' \
...

🐢 This is the only place --from-ref is mentioned and, as far as I can see, there is nothing that calls a dynamic command here.

🐇 Umm, right. There’s something weird here. Maybe we lack the ArgExt trait or something.

🐢 It should be turned on by clap-complete with its unstable-dynamic feature, but who knows.

🐇 It didn’t work.

🐢 ArgExt is available, but it isn’t being used.

🐇 I think I found the answer in https://jj-vcs.github.io/jj/latest/install-and-setup/#command-line-completion:

source <(COMPLETE=zsh xvc)

is the command we should use.

🐢 It doesn’t produce a completion script, though.

🐇 We have this in jj/cli/src/cli_util.rs:

        if env::var_os("COMPLETE").is_some() {
            return handle_shell_completion(ui, &self.app, &config, &cwd);
        }

🐢 I think the whole handle_shell_completion set is written by them. I’d like to see what jj with COMPLETE=zsh outputs.

🐇 Building it to see now.

🐢 As expected, it calls a function:

❯ COMPLETE=zsh target/debug/jj
#compdef jj
function _clap_dynamic_completer_jj() {
    local _CLAP_COMPLETE_INDEX=$(expr $CURRENT - 1)
    local _CLAP_IFS=$'\n'

    local completions=("${(@f)$( \
        _CLAP_IFS="$_CLAP_IFS" \
        _CLAP_COMPLETE_INDEX="$_CLAP_COMPLETE_INDEX" \
        COMPLETE="zsh" \
        /Users/iex/github.com/etc/jj/target/debug/jj -- ${words} 2>/dev/null \
    )}")

    if [[ -n $completions ]]; then
        _describe 'values' completions
    fi
}

compdef _clap_dynamic_completer_jj jj

🐇 Now we should make it the same; the shell must call xvc to get the completions. That’s how all these will work.

🐢 Our goal now is to make a random string output from the --from-ref completion.

🐇 The plan is to make completions work as quickly as possible.

🐢 We have done it. 🎉🥳

🐇 Cool. Now we need to fill up all those completion methods.

🦊 Yeah. We can start with basic ones and move from there.

🐢 There will probably be architectural changes as well. We cannot just run xvc functions directly. We need to run internal functions, but not through commands. At that point, when the user hits tab, we don’t know which command to run.

🐇 Maybe it’s time to move to a Git library. “What is the best Git library for Rust?”

🐲 Should we use a Git library?

🐢 It looks like Gitoxide is the default Rust way to interact with Git repositories. We can keep our way of using the Git binary and use this as an experimental way to learn.

🦊 Yep. That’s a good idea. We can include it in the lib for the time being and start to use it in completions.

I tried the same approach when testing Python bindings with Pytest:

def test_pipeline_step_dependency(empty_xvc_repo):
    dep_help = empty_xvc_repo.pipeline().step().dependency(help=True)
    expected = """
Usage: xvc pipeline step dependency [OPTIONS] --step-name <STEP_NAME>

Options:
  -s, --step-name <STEP_NAME>
          Name of the step to add the dependency to
"""
    assert dep_help == expected

This doesn’t work because the help text is generated by clap and skips the usual thread-based output handler. All command output and errors in Xvc are returned as strings from the command, except for the help text that’s generated by clap automatically.

There are probably workarounds for this, but I won’t pursue them further as I don’t test the actual functionality in the Python bindings anyway.

I replaced the eprintln!s with println!, but it didn’t make any difference. Maybe we should remove those statements completely.

I can’t really find the place that kills the kernel. The last command to run is xvc file list:

print(xvc_test_data.file().list("test-data/dir-0002"))

and the output it produces is:

[src/output.rs:144:13] &output_str = "SS         131 2024-06-05 08:57:11 41e16be7          test-data/dir-0002/file-0003.bin\nSS         131 2024-06-05 08:57:11 27f0
efd0          test-data/dir-0002/file-0002.bin\nSS         131 2024-06-05 08:57:11 66de5084          test-data/dir-0002/file-0001.bin\nTotal #: 3 Workspace Size:
      393 Cached Size:        6006\n"

print may be causing the crash, but the more likely cause is the command that comes after this:

!ls -l test-data/dir-0001/

I replaced this with lsd, which also failed. Maybe it’s actually a Python crash or bug.

The way to understand is to create a notebook file with only that cell and try to run it.

The ls line runs fine with a new notebook. It even runs on the README file when run at the beginning. The line that makes the kernel crash is:

xvc_test_data.storage().new_s3(name="backup", bucket_name="xvc-test", region="eu-central-1", storage_prefix="xvc-storage")

We can start by removing the new_s3 part.

The storage() method runs fine. It returns an XvcStorage() object, as it should.

When I run storage().list(), it takes a very long time. The bug is likely related to storage().

It looks like the storage object was adding file instead of storage as a subcommand. I’ve fixed it now.

That was the bug. The README notebook now creates the S3 storage.

What was the reason behind this?

Parsing the CLI to the XvcCLI object was perhaps the culprit. Let’s look at it more clearly.

Let’s try xvc file new s3 as a command to see how it behaves.

It says unrecognized subcommand for new.

This is how it should be, but I wonder why it doesn’t work for the XvcCLI parser.

Anyway, it’s already 13:00, so let’s stop here for today.