🐢 Instead, we can make it test with environment variables for each of these shells.

🐇 It looks like the test fails with wait_status.

cargo test -p xvc --test test_completions
...
failures:

---- test_completions stdout ----
...
ExitStatus(unix_wait_status(512))

thread 'test_completions' panicked at lib/tests/common/mod.rs:46:5:
Command failed: Command { cmd: "/Users/iex/github.com/iesahin/xvc/target/debug/xvc", stdin: None, timeout: None }
...

🐢 The failure is in the line:

    let mut cmd = Command::cargo_bin("xvc").unwrap();

So it probably waits for user input to complete, but it doesn’t have any and cannot complete it, so it returns an error.

🦊 Let’s comment the earlier test out and check if other tests work.

🐢 Yes, they fail the same. When there are no commands to run, xvc returns an error. Maybe we can change this behavior or handle the error in the Command::cargo_bin line above.

🐇 I checked the code and we don’t handle the “no arguments” case anywhere. It looks like the behavior to return an error code is inherited from clap.

🐢 Updated the error handling code to report a more descriptive message from the source.

cargo test -p xvc --test test_completions
...
failures:

---- test_completions stdout ----
Output { status: ExitStatus(unix_wait_status(25856)), stdout: "", stderr: "\nthread 'main' panicked at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/debug_asserts.rs:341:13:\nCommand pipeline: command `export` alias `l` is duplicated\nstack backtrace:\n   0: rust_begin_unwind\n             at /rustc/b1a7dfb91106018f47ed9dc9b27aee1977682868/library/std/src/panicking.rs:692:5\n   1: core::panicking::panic_fmt\n             at /rustc/b1a7dfb91106018f47ed9dc9b27aee1977682868/library/core/src/panicking.rs:75:14\n   2: clap_builder::builder::debug_asserts::assert_app\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/debug_asserts.rs:341:13\n   3: clap_builder::builder::command::Command::_build_self\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4173:13\n   4: clap_builder::builder::command::Command::_build_recursive\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4076:9\n   5: clap_builder::builder::command::Command::_build_recursive\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4078:13\n   6: clap_builder::builder::command::Command::build\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4071:9\n   7: clap_complete::env::CompleteEnv<F>::try_complete_\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_complete-4.5.40/src/env/mod.rs:229:9\n   8: clap_complete::env::CompleteEnv<F>::try_complete\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_complete-4.5.40/src/env/mod.rs:210:9\n   9: xvc::main\n             at ./src/main.rs:18:30\n  10: core::ops::function::FnOnce::call_once\n             at... [truncated]
ExitStatus(unix_wait_status(25856))

...

error: test failed, to rerun pass `-p xvc --test test_completions`

🐢 The real issue is when we assert the successful run, at this line:

            assert!(output.status.success(), "Command failed: {:?}", prepared);

🐇 If you look at the error message carefully, you’ll see that this is a different error. We added an alias to xvc pipeline export with l, which is a duplicate.

🐢 Oops, yeah.

cargo test -p xvc --test test_completions
...
test test_completions ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.81s

🐇 There are some warnings in compilation. Let’s fix these.

cargo build
   Compiling xvc-storage v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/storage)
   Compiling xvc-file v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/file)
   Compiling xvc-pipeline v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/pipeline)
   Compiling xvc v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/lib)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.28s

🐢 Finished the build without warnings. Now we can run the whole test suite again.

🐇 Tests are running fine. Updated some error messages and types, and doc tests also pass. There is an issue with the Rsync storage ref.

🐢 It looks like we try to elide output with [...], while the proper format is [..].

🐇 Replaced them with ... that will elide multiple lines.

🐢 Pushed changes to the server.

🐇 We can just complete the completions for this version and add JSON output for all commands to use in nushell. In the meantime, dynamic completions support for nushell is likely to be added.

🐢 Umm, yep. Then we’ll continue to run tests with zsh for the current version.

🐇 run-tests is a zsh file, and we will continue to use it. We can switch to nushell eventually as we add more compatibility with it, but let’s not do this now.

🐢 Where were we the last time for completions?

🐇 We added xvc_path_completers. Let’s check the TODO list now.

🐢 We still need tracked_targets completions in a few places.

🐇 Added those.

🐢 Now we have some strum completers. Let’s finish these up as well.

🐇 Ok. They are done as well.

🐢 Now, let’s start adding a storage_identifier completer. It should read all XvcStorage records and list their names.

🐇 Added a storage_identifier completer. Let’s add it to all places where we use storage_identifiers.

🐢 Now, we need completers for pipeline and step names. These are straightforward.

🐇 Added these. Let’s fill up where they are needed.

🐢 I want to skip some of the fine-grained completions in this version. We can have a specific completer for --params options for files and params inside, for example. Or a special completer for directories.

🐇 Let’s not forget these. Reading YAML files and extracting hyperparameter keys for the prompt would be a really good feature for the user.

🐢 I agree. These are good features in general, but we need to ship the current version as soon as possible.

🐇 Let’s check the TODO comments once more.

$ rg 'TODO:' 
...
- ✅ pipeline/src/pipeline/api/update.rs:    /// TODO: Add a repository_dirs completer (11:08)
- ✅ file/src/remove/mod.rs:    /// TODO: Add a storage_identifier completer (11:08)
- ✅ file/src/bring/mod.rs:    /// TODO: Add a storage_identifier completer (11:08)
...

🦊 We can use xvc_path_completer for the tracked directory completer for the time being. For pipeline update, we can use a ValueHint instead. It’s not necessary to use a custom completion.

🐢 For the xvc file copy destination, we can have a file or a directory that we track and is not available, or we don’t track and is available. It’s a similar situation to xvc_path_completer, but we also need to check the local paths. It requires some more care.

🐇 Now we can begin to run the tests.

🐢 Is this building?

🐇 It should.

🐢 We also have one bug. xvc shouldn’t print help text when the COMPLETE environment variable is set.

🐇 Umm, yeah, that’s a blocker.

🐢 Fixed it. We can bump the version and push the changes.

🐇 Starting with completions is better. One question I have is whether the completions work for subcommands as usual, like in the static completions.

🐢 Yes, they work. We need to add:

source <(COMPLETE=zsh xvc)

to .zshrc, though.

🐇 It looks like we can drop the xvc completions command. We’re just checking the COMPLETE environment variable, and there’s no need for a separate command in this case.

🐢 Yes, let’s remove that.

🐇 Maybe we can employ a completions module for all completion-related functionality. It’s a separate thing, you know.

🐢 Which completions do we need?

🐇 We can just mark them with TODOs now.

🐢 Yes, let’s check where we need completions and what.

🐇 I noticed we’re repeating options in XvcFileCLI, and some of the options are missing, e.g., --from-ref and --to-branch.

🐢 Because it can compile into a different binary, and global options don’t work that way. Maybe we can move all these binaries into different files under lib. We can have feature flags to turn off certain features and compile different binaries.

🦊 Let’s search for it; I haven’t seen it before: building different binaries with feature flags using cargo.

🐢 There is no clear-cut solution, but it looks like we can move all binaries to the root with the required-features field.

🐲 We can postpone this to another release.

🐇 Yes, let’s not spend time on this now.

🐢 Should we try making pipeline_name a global option? It’s repeating everywhere.

🐇 That will make it easier to maintain.

🐢 We’ll have to pass pipeline_name to subcommands, though.

🐇 I think we can have a set of global options that we can pass. For the time being, that’s only the pipeline_name.

🐢 Okay. Let’s do this.

🐇 A similar option is the step name for pipeline steps.

🐢 Dependencies will need a revamp in the next version anyway. So let’s keep it for now.

🐲 Also, the semantics of step-name are different for these commands. step new interprets it as a new name, while step dependency interprets it as an existing name. The first can be renamed to --name, and the second can be --to, as one of its aliases suggests.

🐢 Yes, let’s keep it for now, and we’ll continue to work on others.

🦊 I want to post a comment in the clap discussions.

🐢 Wrote it. Now let’s build it after changing the pipeline_name.

🐇 It compiles. Should we test it?

🐢 I think we’ll test after all this completion work is done.

🐇 We did most of the strum-related completion.

🐢 Didn’t test them yet, though.

🐇 This is Rust. It will work if it compiles, and it compiles.

🐇 The examples are only found in the tests: https://github.com/clap-rs/clap/blob/master/clap_complete/tests/testsuite/engine.rs#L604

and we can try to create those random strings this way.

🦊 The issue is that we may not be able to use the derive method to add these custom completers. All examples are using the builder API.

🐢 We should be able to obtain the end result from derives and modify them to use these custom completers, but let’s try to use add = ArgValueCompleter first.

🐇 It looks like we have another problem. We need to add a rust-toolchain.toml file to the project, you know, to keep it on the stable channel.

🐢 Ah, yeah, we now use multiple channels. Let’s do that.

🐇 I found an example, actually:

#[derive(Debug, Parser)]
struct Cli {
    #[arg(long, add = ArgValueCompleter::new(custom_completer))]
    custom: Option<String>,
}

🐢 I’m trying to add this to the --from-ref option of Xvc. It’s not running anything. The custom completer should return a random string, but it doesn’t.

🦊 Maybe test with a constant first.

🐢 It didn’t work that way either.

🐇 Let’s do a cargo clean.

🐢 Nothing has changed.

🐇 Let’s take a look at the output of xvc completions:

xvc completions
...
'(--skip-git)--from-ref=[Checkout the given Git reference (branch, tag, commit etc.) before performing the Xvc operation. This runs \`git checkout <given-value>\` before running the command]:FROM_REF:_default' \
...

🐢 This is the only place --from-ref is mentioned and, as far as I can see, there is nothing that calls a dynamic command here.

🐇 Umm, right. There’s something weird here. Maybe we lack the ArgExt trait or something.

🐢 It should be turned on by clap-complete with its unstable-dynamic feature, but who knows.

🐇 It didn’t work.

🐢 ArgExt is available, but it isn’t being used.

🐇 I think I found the answer in https://jj-vcs.github.io/jj/latest/install-and-setup/#command-line-completion:

source <(COMPLETE=zsh xvc)

is the command we should use.

🐢 It doesn’t produce a completion script, though.

🐇 We have this in jj/cli/src/cli_util.rs:

        if env::var_os("COMPLETE").is_some() {
            return handle_shell_completion(ui, &self.app, &config, &cwd);
        }

🐢 I think the whole handle_shell_completion set is written by them. I’d like to see what jj with COMPLETE=zsh outputs.

🐇 Building it to see now.

🐢 As expected, it calls a function:

❯ COMPLETE=zsh target/debug/jj
#compdef jj
function _clap_dynamic_completer_jj() {
    local _CLAP_COMPLETE_INDEX=$(expr $CURRENT - 1)
    local _CLAP_IFS=$'\n'

    local completions=("${(@f)$( \
        _CLAP_IFS="$_CLAP_IFS" \
        _CLAP_COMPLETE_INDEX="$_CLAP_COMPLETE_INDEX" \
        COMPLETE="zsh" \
        /Users/iex/github.com/etc/jj/target/debug/jj -- ${words} 2>/dev/null \
    )}")

    if [[ -n $completions ]]; then
        _describe 'values' completions
    fi
}

compdef _clap_dynamic_completer_jj jj

🐇 Now we should make it the same; the shell must call xvc to get the completions. That’s how all these will work.

🐢 Our goal now is to make a random string output from the --from-ref completion.

🐇 The plan is to make completions work as quickly as possible.

🐢 We have done it. 🎉🥳

🐇 Cool. Now we need to fill up all those completion methods.

🦊 Yeah. We can start with basic ones and move from there.

🐢 There will probably be architectural changes as well. We cannot just run xvc functions directly. We need to run internal functions, but not through commands. At that point, when the user hits tab, we don’t know which command to run.

🐇 Maybe it’s time to move to a Git library. “What is the best Git library for Rust?”

🐲 Should we use a Git library?

🐢 It looks like Gitoxide is the default Rust way to interact with Git repositories. We can keep our way of using the Git binary and use this as an experimental way to learn.

🦊 Yep. That’s a good idea. We can include it in the lib for the time being and start to use it in completions.

🐲 We can work on completions or the GUI.

🐢 Completions are similar to the Homebrew work. It’s in another ecosystem and feels boring.

🐲 We can try to make it in Rust: “Writing shell completions in Rust.”

🐢 There are two solutions: One is clap_complete (https://docs.rs/clap_complete/latest/clap_complete/) and the other is shell_completion (https://github.com/JoshMcguigan/shell_completion). The latter is very new but may be what we need: writing completions only with Rust.

🐲 The shell_completion crate is very bare-bones. It doesn’t have anything, actually. https://github.com/JoshMcguigan/shell_completion/issues/1

🐇 The feature that we need is dynamic completion. We’ll write something similar to ArgValueCompleter for this.

🦊 I think we can just start a new branch.

🐲 We have a branch for Homebrew; should we merge it?

🐢 I think, yes, we can merge it. We’ll fix it if it breaks. It’s a separate workflow file anyway.

🦊 Created the PR.

ghpl
264	add brew tap	add-brew-tap	OPEN	2025-01-03T05:14:45Z

🐢 Merged it.

🐇 Now let’s create a new branch and add clap-complete to Cargo.toml.

🐲 There are some packages that we need to take a look at. thiserror now has v2.

🐢 Upgraded packages and compiled. It works.

🐲 Let’s update the version.

🐢 Done. Now we can add clap-complete.

🦊 Added with the unstable-dynamic command. Can we build it again?

🐢 Built it. Now we can add a completions subcommand to Xvc.

🐲 Is this the right name for this?

🐢 It will output a shell script that we can source in the shell.

🐲 Okay, let’s add the subcommand now.

🐢 The example doesn’t work with the clap builder.

🐇 Let’s search for it.

🐢 I think we’re extending ourselves a bit when trying to add all dynamic features of completion at once. We can just start by adding completion to xvc-test-helper.

🐇 Yep, that’s a better idea. We can have many more examples, and it’s certainly more straightforward.

🐢 One thing I noticed when working yesterday is that using auto-save was actually preventing many of these swap errors.

🐲 It was a bit slow. Can we take a look at a few others?

🐢 https://github.com/okuuva/auto-save.nvim seems a bit more polished.

🐇 Installed it, but haven’t seen an effect yet.

🐢 We may need to restart this.

🐲 Now, we can get into the test-helper completions.

🐢 Done. Added these quickly as you thought. Now adding this to main seems much easier.

🦊 That was a nice approach. Let’s dive into adding this to main.

🐇 Should we go with a separate command or just an option?

🐲 Our commands have distinct initial letters, allowing them to be used with just those letters. Adding another top-level command for this will make c useless.

🐢 I don’t think that’s a valid concern. We can create aliases for all commands and completions shouldn’t have to have an alias. But I agree that completions should not be a top-level command. It will be run once for installation at most.

🦊 I agree. Let’s call it --completions. It will be run as xvc --completions zsh and will print out the completions.

🐢 Okay. Let’s do this.

🦊 No errors left. Let’s install this version.

🐢 We get:

error: 'xvc' requires a subcommand but one was not provided
  [subcommands: file, init, pipeline, storage, root, check-ignore, aliases, help]

Usage: xvc [OPTIONS] <COMMAND>

For more information, try '--help'.

🦊 Umm, okay. I think we don’t have an option to create a command like that. Then we’ll have to print completions with a subcommand.

🐲 The above discussion is now moot. ❌

🐢 We can try to push, but probably it’s not worth it.

🐲 Let’s not lose time on this. I think we can remove the aliases command and replace it with completions, and add single-letter aliases to subcommands.

🐢 We can have an option in the completions command to print aliases. That will work.

🦊 I don’t think people will use the aliases command if we have single-letter command aliases.

🐢 You may be right. No need to try to maintain that at this time.

🐢 Good morning, and I’m fed up with these blink.cmp errors, you know.

🐇 Reinstalling blink works. It’s interesting to rely on such unreliable software.

🐢 Oh, yeah. It’s interesting. Where were we yesterday?

🐲 We decided to rename the aliases command to completions.

🐇 And waiting for rust-analyzer to complete its analysis.

🐢 Uh, yeah. I see.

🦊 The code itself is very short, actually.

    if let Some(shell) = cli_opts.completions {
        let mut cmd = XvcCLI::command();
        generate(shell, &mut cmd, "xvc", &mut io::stdout());
        return Ok(None);
    }

🐲 We’ll use the output! macro instead of writing to io::stdout(), right?

🐢 Yes, we can rely on the usual output system. It will be slower to create an output thread but shouldn’t matter for outputting a shell script.

🐇 It takes a while for rust-analyzer to scan all the directories, it looks like. When I close the project, it just cannot reload it immediately.

🐢 We can use cargo clean from time to time.

🦊 rust-analyzer finished scanning; let’s try to rename AliasesCLI.

🐲 Let’s keep these here; maybe we’ll need them in our scripts:

# Standard Xvc command aliases for longer commands.
alias xls='xvc file list'
alias pvc='xvc pipeline'
alias fvc='xvc file'
alias xvcf='xvc file'
alias xvcft='xvc file track'
alias xvcfl='xvc file list'
alias xvcfs='xvc file send'
alias xvcfb='xvc file bring'
alias xvcfh='xvc file hash'
alias xvcfco='xvc file checkout'
alias xvcfr='xvc file recheck'
alias xvcp='xvc pipeline'
alias xvcpr='xvc pipeline run'
alias xvcps='xvc pipeline step'
alias xvcpsn='xvc pipeline step new'
alias xvcpsd='xvc pipeline step dependency'
alias xvcpso='xvc pipeline step output'
alias xvcpi='xvc pipeline import'
alias xvcpe='xvc pipeline export'
alias xvcpl='xvc pipeline list'
alias xvcpn='xvc pipeline new'
alias xvcpu='xvc pipeline update'
alias xvcpd='xvc pipeline dag'
alias xvcs='xvc storage'
alias xvcsn='xvc storage new'
alias xvcsl='xvc storage list'
alias xvcsr='xvc storage remove'

🦊 We can use them when adding single-letter aliases.

🐇 There is a from_env method for Shell; will we support it?

🦊 I think we can support it. It’s much easier to use if we omit the shell.

🐲 We had aliases in xvc-core, but we need to access XvcCLI from completions. The completions module must be moved to xvc.

🦊 We added the clap_complete dependency to the xvc-pipeline and xvc-file crates, but I don’t think they are necessary. Let’s remove them now.

🐢 Now only the test-helper and xvc crates have the clap_complete dependency.

🐇 Are we ready to test?

🐢 Completions are working. 🎉

🐲 Now we need to update the docs and doc tests, I believe.

🐢 There are also tests to update.

🐇 Tests are running now. In the meantime, can we take a look at the blink configuration?

🐲 When we removed xvc aliases, we also removed pvc, xls, and other aliases. Maybe we can add these to the docs.

🐢 We can put them in the xvc completions reference for now.

🐢 It takes a while for rust-analyzer to finish analyzing the codebase.

🦊 Added aliases for xvc pipeline commands. Do you think we need to repeat root-level flags in xvc-pipeline?

🐢 No need to divert attention, I believe. Also, I still think there is an easier way to do that.

🐇 Okay. Do you think we should add easier subcommands to step? Like, step new becoming xvc p s n instead of xvc p s n? (Wait, that’s the same). I mean, more concise.

🐢 I think we can extend these even to the top-level, but shouldn’t make them visible. It will pollute the help text. We can add xvc fl for file list to avoid the space, but hide these from the help text.

🐲 That’s a good idea, but it will require including sub-crate level modules at the top level. We must test it first.

🐢 Let’s finish up the current changes and release them first.

🐇 By the way, we didn’t add two-letter abbreviations to the xvc storage new subcommands. Do you think we need them?

🐢 When we think about the frequency of these commands, no, I don’t think we need them. Users won’t add a new storage every day.

🐇 By that logic, we shouldn’t need an n for xvc storage new.

🐢 Actually, yeah. Maybe we should remove even s.

🦊 storage list may be useful.

🐢 s is a very common letter, though. We can have other uses for that letter.

🐇 Updating xvc p s dependency options, but it looks like making dependency options subcommands is a better way.

🦊 We didn’t do it because currently we can supply multiple options with a single command. If we go the subcommand route, we’ll have to write all dependencies one by one. Adding multiple subcommands with clap is a bit tricky.

🐢 Umm. Yeah, I see.

🐇 Maybe we can provide a separate command to add multiple dependencies. Like, xvc p s d add 'lines=myfile.csv::10-20; file=myimage.jpg; glob=dir/image-10*'

🐲 That looks like the start of a language. We need a parser for those strings. They will be freeform.

🦊 Another option is to keep the current options and add commands with the same names.

🐢 That will be confusing. The user will have both --param and param, and they will work differently.

🐇 We can have an add command that accepts the current options. Parsing will be done by clap just like now, but for a subcommand of dependency.

🐢 The full command will be something like xvc pipeline step --step-name preprocessing dependency add --params 'params.json::batch_size'

🐇 With shorter commands, it’s like xvc p s -s preprocessing d a --params 'params.json::batch_size', and this doesn’t look like ffmpeg monstrosities.

🐲 In any case, this is a backward-incompatible change. This should wait for v0.7 along with ECS changes.

🐢 ECS changes are not user-visible, but these are. Certainly needs a minor version update.

🐇 Then, okay, let’s keep the current ones for this version and think about updating them in a future version.

🐢 Okay. Let’s finish up and we’ll create a separate invisible subcommand to ask questions to the Xvc repo for completions.

🐇 We completed completions for the common command structure. Now, we need a way to show certain info after certain commands. A tab after xvc p r -p should show pipeline names, for example.

🐢 Umm, yeah. And these should be as quick as possible. They shouldn’t check Git or any other things.

🐇 I looked here and there, and the only working example I found is here: https://github.com/clap-rs/clap/blob/master/clap_complete/tests/testsuite/zsh.rs#L248 in clap_complete tests.

🐢 Let’s try to dive in. We can start by cloning the repo, I believe.