🐇 The examples are only found in the tests: https://github.com/clap-rs/clap/blob/master/clap_complete/tests/testsuite/engine.rs#L604

and we can try to create those random strings this way.

🦊 The issue is that we may not be able to use the derive method to add these custom completers. All examples are using the builder API.

🐢 We should be able to obtain the end result from derives and modify them to use these custom completers, but let’s try to use add = ArgValueCompleter first.

🐇 It looks like we have another problem. We need to add a rust-toolchain.toml file to the project, you know, to keep it on the stable channel.

🐢 Ah, yeah, we now use multiple channels. Let’s do that.

🐇 I found an example, actually:

#[derive(Debug, Parser)]
struct Cli {
    #[arg(long, add = ArgValueCompleter::new(custom_completer))]
    custom: Option<String>,
}

🐢 I’m trying to add this to the --from-ref option of Xvc. It’s not running anything. The custom completer should return a random string, but it doesn’t.

🦊 Maybe test with a constant first.

🐢 It didn’t work that way either.

🐇 Let’s do a cargo clean.

🐢 Nothing has changed.

🐇 Let’s take a look at the output of xvc completions:

xvc completions
...
'(--skip-git)--from-ref=[Checkout the given Git reference (branch, tag, commit etc.) before performing the Xvc operation. This runs \`git checkout <given-value>\` before running the command]:FROM_REF:_default' \
...

🐢 This is the only place --from-ref is mentioned and, as far as I can see, there is nothing that calls a dynamic command here.

🐇 Umm, right. There’s something weird here. Maybe we lack the ArgExt trait or something.

🐢 It should be turned on by clap-complete with its unstable-dynamic feature, but who knows.

🐇 It didn’t work.

🐢 ArgExt is available, but it isn’t being used.

🐇 I think I found the answer in https://jj-vcs.github.io/jj/latest/install-and-setup/#command-line-completion:

source <(COMPLETE=zsh xvc)

is the command we should use.

🐢 It doesn’t produce a completion script, though.

🐇 We have this in jj/cli/src/cli_util.rs:

        if env::var_os("COMPLETE").is_some() {
            return handle_shell_completion(ui, &self.app, &config, &cwd);
        }

🐢 I think the whole handle_shell_completion set is written by them. I’d like to see what jj with COMPLETE=zsh outputs.

🐇 Building it to see now.

🐢 As expected, it calls a function:

❯ COMPLETE=zsh target/debug/jj
#compdef jj
function _clap_dynamic_completer_jj() {
    local _CLAP_COMPLETE_INDEX=$(expr $CURRENT - 1)
    local _CLAP_IFS=$'\n'

    local completions=("${(@f)$( \
        _CLAP_IFS="$_CLAP_IFS" \
        _CLAP_COMPLETE_INDEX="$_CLAP_COMPLETE_INDEX" \
        COMPLETE="zsh" \
        /Users/iex/github.com/etc/jj/target/debug/jj -- ${words} 2>/dev/null \
    )}")

    if [[ -n $completions ]]; then
        _describe 'values' completions
    fi
}

compdef _clap_dynamic_completer_jj jj

🐇 Now we should make it the same; the shell must call xvc to get the completions. That’s how all these will work.

🐢 Our goal now is to make a random string output from the --from-ref completion.

🐇 The plan is to make completions work as quickly as possible.

🐢 We have done it. 🎉🥳

🐇 Cool. Now we need to fill up all those completion methods.

🦊 Yeah. We can start with basic ones and move from there.

🐢 There will probably be architectural changes as well. We cannot just run xvc functions directly. We need to run internal functions, but not through commands. At that point, when the user hits tab, we don’t know which command to run.

🐇 Maybe it’s time to move to a Git library. “What is the best Git library for Rust?”

🐲 Should we use a Git library?

🐢 It looks like Gitoxide is the default Rust way to interact with Git repositories. We can keep our way of using the Git binary and use this as an experimental way to learn.

🦊 Yep. That’s a good idea. We can include it in the lib for the time being and start to use it in completions.

vim.keymap.set({ "n", "i" }, "<F2>", '<esc><esc>yy:r!<c-r>"')

I type a command, like ls, on a line and hit <F2>. That line is copied, executed, and the output is read into the buffer just below it.

I use it for all sorts of tasks, from listing open PRs with gh to fetching website text with w3m. It has significantly improved my workflow, allowing me to use the shell effectively within textual notebooks.

The issue with having a mutable element is that we need to update XvcRoot from Arc<XvcRootInner> to Arc<RwLock<XvcRootInner>>, which will cause almost all machinery around XvcRoot to require locking the object first. This is too large a refactoring.

However, I can write an xvc_root! macro to replace xvc_root.read() or whatever is required to minimize the code changes. XvcRoot can also be a wrapper object, but that’s too much fuss, and the responsibility may be misplaced.

Let’s update the type and let the dispatcher update XvcRootInner only when necessary. Let’s see what will require updating.

We have a bug in the Python bindings where xvc.file.track() enters an infinite loop.

There can be multiple reasons, but are you sure that the xvc CLI works with the same command?

Let’s start testing by creating another notebook server.

The notebook creates an Xvc repo successfully and returns the root with xvc.root(), but xvc.file().track("test-data/dir-0001/") never finishes.

It’s likely that this is caused by something in the background threads.

The CLI command completes successfully, but we may be passing the command incorrectly. Let’s try dir-0001 only.

Even if an incorrect path is provided, it shouldn’t enter an infinite loop.

When I provide the correct path, dir-0001, it returns. There may be something going on with finding the root of the Xvc repo.

It returns, but it’s also giving an error that it cannot find a repository. I should take a closer look.

The Xvc struct wasn’t implementing Debug, so I added it.

Let’s add it to others, XvcFile, etc.

Added it. Recompiling to get more info when we run track with an incorrect path.

You can also make it run after a commit to avoid starting a new server if one is already running.

Added conditionals, and now it runs the server if none is running in the background.

PORT=7979
if [[ ! -s "$(ps ax | rg -v ps | rg jupyter-lab | rg $PORT)" ]] ; then
  jupyter lab --port=$PORT --notebook-dir=Readme/ &
  open http://localhost:${PORT}/lab/workspaces/auto-H/tree/Readme.ipynb
fi

I needed a way to quickly open URLs in the Kagi Summarizer from my terminal. I wrote a small shell function to do that:

ksum() {
  local urlenc=$(jq -rn --arg x "$1" '$x|@uri')
  open "https://kagi.com/summarizer/index.html?target_language=&summary=takeaway&url=${urlenc}"
}

🐢 It looks like we don’t have many architectural problems.

🐇 You’re forgetting how Exec::cmd works, though. You have those kinds of problems.

🐢 Yeah, I’m figuring that out. Exec::shell requires a string to run on the shell, while Exec::cmd just needs a command name. You have to supply args with another function. I’m writing a closure now to handle this.

🐢 It started working, but it revealed a much bigger problem: .xvc/ is not added to .gitignore in xvc init.

🐇 Wow, that’s a showstopper.

🐢 Yup. I think I should fix it as well.

🐇 On closer glance, I think the problem is not xvc init not modifying .gitignore; it modifies it incorrectly. Maybe putting certain files on a whitelist is a better idea than trying to blacklist everything.

🐢 Yeah, we should define a set of Git-tracked files and directories, whitelist them, and let all other files be ignored.

🐇 Go ahead, then.

🐢 I think I’ve fixed it. There were two problems: the initial gitignore content was wrong, and it was placed in the root of the repository instead of .xvc.

🐇 Maybe putting it directly in the root is better than hiding it in .xvc. What do you think?

🐢 There seems to be an assumption in file track to have a .gitignore file somewhere. I think we should also handle changes in .gitignore files throughout the repository.

🐇 Umm, yes. We should handle .gitignore files as well. But not all .gitignore files are changed by Xvc. How can we make sure they were modified by Xvc?

🐢 I think there are ways to do that, like tracking them somewhere. But I believe we shouldn’t try. We should just get a list of .gitignore files, git add them, and include them in the commit.

🐇 Ok. Let’s write a test for this as well. No .gitignore should appear in git status -s.

🐢 I wrote the test. It fails now. Do you think we should check the output of Git status to determine which files to add?

🐇 That may be a good idea. Git status should already know which files need to be added. We can use that information.

🐢 It looks like pathspec is enough to modify git add behavior. We should be able to write *.gitignore and let all gitignore files be added.

🐇 Let’s try this manually.

🐢 Yep, it works. git add '*.gitignore' adds all .gitignore files in the subdirectories, too.

🐇 Congrats. 👏🥳

🐢 I’m writing the missing documentation for the crates. There are proc_macro not expanded errors all over the code.

🐇 I think you should update rust-analyzer and everything. There must be a command for this.

🐢 I reinstalled RA with :LspInstall and restarted the LSP. It seems to work correctly now.

Ah yeah, when you work for free, you can do so at any time you want, perhaps. You also don’t have team members, and that means when you sit in front of this, you can move it.

Umm, right. Let’s take a look at the outstanding PRs.

You have a documentation PR#93. You also have work that you’ve begun to integrate Git into. I think it’s better to focus on the latter today.

Right. Let’s think about the relationship between Git and Xvc. I believe we should identify a general relation to avoid ending up in a mess like DVC and Git.

Why do you think the DVC and Git relationship is a mess?

They don’t automate common Git operations like a commit after dvc add. There is only auto-stage, and that’s turned off by default. This makes it seem that DVC wants to intervene as little as possible with the user’s Git workflow. That’s understandable. I support this. But on the other hand, they use the .git/ directory itself to store and manage experiments in a custom way that creates custom stash objects for experiments. This is against the principle of minimum intervention.

So this makes it a mess?

The mess, in my opinion, is caused by the second factor. If DVC doesn’t perform any Git operations, that’s alright. It was intended to be VCS-agnostic. Then experiments came and used Git internals in a way that no other similar tool uses.

Git-LFS and Git-Annex seem to use some non-standard mechanisms as well.

Ok. Not no other tool uses, but in a way that no other tool has used.

You know, GitHub PRs are also stored in a similar way. They also use non-standard machinery.

Yeah, but these tools are all Git-specific tools. They accept the dominion of Git, and don’t try to bring any VCS-agnosticism.

And Xvc tries to have this agnosticism?

I believe the initial design of DVC, which aims to be VCS-agnostic or being able to run without a VCS, is valuable. I like the idea behind Git, but the interface and implementation show that it’s a gradual development. There is no library behind it.

Libgit?

Libgit2 is something different. Although it’s said to have some common code, it doesn’t support all features. Git is command-line software with a mix of scripts and compiled executables, and not all code seems to be written in a way that could be used by external tools.

Hmm. The comment you added to the issue says git stash push --staged is not available in libgit2. Can’t you mimic it like DVC does for branches?

I don’t want to depend on Git at that level.

So, you’ll be using the CLI and shell for Git?

Yes, I believe, at the moment, before any performance tests, that this doesn’t matter much. Running Git commands once in a while using the shell shouldn’t make much difference in overall performance.

Then you’ll use it like a command-line tool, like the user?

Yes, and I’ll make it run outside of the usual threads. All Git will be like a sandwich, wrapping around Xvc operations. If there are --git-ref instructions in an xvc command, it will be run before Xvc performs the command, and if there are any changes in Xvc metafiles, they will be committed to the current branch.

Like

graph LR

co["git checkout"] --> xvc
xvc --> cm["git commit"]

The first could be a branch as well. So we have:

graph LR

br["git branch"] --> xvc
co["git checkout"] --> xvc
xvc --> cm["git commit"]

Looks sensible. How will you reflect these in the command line?

With something like xvc --git-checkout my-branch file list

Hmm, and for a branch?

I think instead of different options for branch, checkout or tag, we can have a git-ref option that marks the option as a git reference. It will be checked out, or created as a branch from the current one if it doesn’t exist.

I think creating a branch is not a good idea. It should be explicit. You can just send the --git-ref value to git checkout and perform the Xvc operation. If the user wants to create a branch, I think they can do it themselves.

What about storing the results in a branch? After adding a bunch of files, they may want to store them in another branch, maybe?

That’s sensible. We can have another option, like --to-branch in certain operations.

Or in the xvc command as a general option. In that case, we can change the option names to --from-ref and --to-branch. It will be like:

graph LR

fr["git checkout $(--from-ref)"] --> xvc
xvc --> tb["git checkout --branch $(--to-branch)"]
tb --> co["git add .xvc && git commit -m 'xvc cmd'"]

If no such options are given, xvc will run without branching, right?

Yep. --from-ref and --to-branch options are just shortcuts for user behavior. Any other VCS tool could be used this way. We don’t need to integrate Git at the library level.

This brings up the question of portability, though. When you aim for the software to be portable, you can’t rely on the existence of Git on the host, right?

I think a git.command option in the configuration is a good idea. Xvc will issue a warning if it can’t run the commands.

Will you use the shell to run this command? Otherwise no $PATH configuration is possible, you know.

I believe that could be another option: git.use_shell. If git.command is set to an absolute path, Xvc may use it without the shell. Otherwise, it can use the shell. Running the process directly will make it faster and more secure.

There is also this option to run Xvc in another process. Because we may access Git in the shell that runs Xvc, and if we can access it, maybe we don’t need shell execution in the process.

That’s a cool idea. But I wouldn’t add that extra complexity. Instead, we can try to find the git executable if git.command is not an absolute path. If git.command = /usr/bin/git in the configuration, we use it as is. Otherwise, we can get $PATH or %PATH% from the environment and search for git.command in that to find the exact executable.

It looks like there is a crate called which that does exactly what we are looking for.

Ah, cool. Then we can just use that to find the executable and run it. We don’t need to drop to a shell.

Yep. Let’s go back to implementation now.

🐢 Yep. It was a nice October. I would like to finish by increasing the coverage.

🐇 How about setting up a runner on your machine to test them locally?

🐢 Not now. I should fix the tests as always. Later I can take a look at it. I don’t want to distract myself with it.

🐇 You’ll need to create Apple Silicon binaries, though.

🐢 I think I can cross-compile for aarch.

🐇 You were doing this in the past, trying to build all binaries on a single Ubuntu VM. It didn’t work, as far as I remember.

🐢 I hadn’t even read this section then, and there is a dedicated project for it. The proper way is to use that. I’ll return to this after fixing the tests. I think we can release Xvc on all platforms supported by Rust.

🐇 Using cross-rs on GitHub Actions may be challenging. It seems Apple Silicon doesn’t need that much work either. You can set up cross-compilation with rustup targets.

🐢 Yeah. It looks so. Let’s take a look at the previous devlog.

🐇 The page seems missing. Would you like to fix your site first?

🐢 Ah, yeah. I should, maybe. Now it’s time to fix those PRs. I’ll take care of the site later. I should change the theme anyway.

🐇 Ok. Let’s take a look at outstanding PRs.

🐢 I want to fix the failed tests locally first. Yesterday I wasn’t able to redirect the output. It looks like the correct way of doing it is cargo test --all-features --no-fail-fast > $TMPDIR/xvc-test.log 2>&1. It’s required to put 2>&1 at the end, not between the redirection and output.

🐇 There is also the &> operator that you may want to use. cargo test --all-features --no-fail-fast &> $TMPDIR/xvc-test.log should be equivalent to this.

🐢 I’ll try it when the tests finish.

🐇 In the meantime, you can work to fix the documentation. In another repository, perhaps.

🐢 Good idea. Let me clone that.

🐇 Looks like the tests have finished and the only error is Minio. It couldn’t find the env variables you defined.

🐢 Restarted the shell and the tests. Returned to docs now.

🐢 Added a few function docs. Storage tests seem to pass. Probably they will fail in GA because of rsync tests using one.emresult.com login in my name.

🐇 If that’s expected, you can push and begin to fix it.

🐢 Yeah, let’s push the tests.

🐢 It looks like connecting to localhost also poses a challenge. Instead, I can create a user for xvc on the server and limit its usage.

🐇 Hmm. Good idea for now.

🐢 I created a new user xvc-test@one.emresult.com and its SSH keys. I’ll update the action to write the secret to the keyfile.

🐇 Looks like you forgot to install mc for Minio connection.

🐢 Yeah, I must convert Minio tests to use s3cmd.

🐇 And your region in s3 configuration seems to be wrong.

🐢 I’ll need to check this.

🐇 There is this line in the tests let region = env::var("AWS_DEFAULT_REGION").unwrap_or("us-east-1".to_string()); that’s probably causing that error. You should define the region properly.

🐢 I set this to eu-central-1 directly and will check the Minio error in the next session.

🐇 👏

🐢 It’s working except for the rsync tests now.

🐇 I think you can just use localhost to run the tests. You may need to install openssh-server, but it should work with localhost without configuration.

🐢 There may be a step missing in the configuration. I’ll try to make it run.

🐢 I’m trying to use .ssh/config in GA to allow connections to the server. It doesn’t work for some reason.

🐇 You may try to log in to the server outside of the tests and check if it’s running.

🐢 Yesterday’s last attempt was successful, and now we have working remote storage tests.

🐇 👏👏👏🥳

🐢 I’m merging the PR.

i3-msg workspace "n"

zenity --info --text="Merge complete. Updated 3 of 10 files."

Bir dizindeki tüm dosyalar: ls *

Bulunduğumuz dizin ve tüm alt dizinlerdeki dosyalar: ls **/*

Son iki haftada değiştirilmiş dosyalar: ls **/*(.mw-2)

Boyutu 100 MB’tan büyük olan dosyalar: ls **/*(.Lm+100)

Sadece okunabilir olan dosyalar: ls **/*(.R)

/etc dizininde dünya tarafından yazılabilir olan dosyalar: ls /etc/**/*(.W)

/etc dizininde son bir haftada değiştirilmiş ve dünya tarafından yazılabilir olan dosyalar: ls /etc/**/*(.Wmw-1)

zsh’in sağladığı bu gibi seçeneklerin yanında dosya adını parçalamak da kolay. Örneğin dosyanın uzantıdan önceki kısmını almak için *(:r) yazıyoruz. Dizindeki jpg dosyalarını ImageMagick’in convert programıyla aynı isimli png dosyalarına çevirmek şöyle mümkün:

for f in **/*.jpg(.) ; do
   convert ${f:r}.jpg ${f:r}.png
done

Burada anlatılanları find yoluyla yapmak da mümkün ancak çok daha uzun sürebiliyor. Bilhassa dosya adlarını parçalamak, dönüştürmek gibi işlemleri yaparken zsh’in nimetleri çok. Tek satırlık bir operasyonla çok güçlü bazı dönüşümler yapabilmek mümkün.

I have used ~ several times in scripts, but it may not always expand as expected depending on the environment. Use $HOME to refer to the home directory; it is much more reliable and always works.