🐢 I think I can implement Rsync today. Looking at ssh2-rs, though, I think we can implement file transfer without relying on rsync. It might be easier to implement everything within the code.

🐇 Then you should rename the issue to new ssh.

🐢 Fair. There is also the ssh_rs crate, but it doesn’t have full support for the protocol. Instead, we can have another command, like xvc storage new ssh, that uses libssh2 via the crate mentioned above. It has some limitations with OpenSSH on macOS.

🐇 From the crate’s README, it looks like you can enable the vendored-openssl feature to compile it statically.

🐢 Let’s go ahead then. It’s better to compile it behind a feature flag, though.

🐇 Yup. Rsync can be separate. I think for now you can implement rsync via Exec::cmd and make new ssh a new issue.

🐢 I’ll copy this conversation there.

🐇 Now, let’s start implementing rsync.

🐢 Do we really want to hide it behind a feature flag? It doesn’t bring any extra complexity to generic, for example—just using the commands and returning the errors.

🐇 I think so. If the user doesn’t have rsync on their system, they’ll just get errors. We don’t need to make the implementation optional, but the tests might be.

🐢 OK.

🐢 The tests for yesterday’s Git integration PR failed. I’ll begin by checking the logs.

🐇 Maybe run the tests locally to see. They might fail on your machine as well. It might be a simple thing.

🐢 Probably, yes. I’ve started the tests now. I shouldn’t forget to run them before testing the PR; it may save some time.

🐇 Ah, yeah, maybe. Yesterday you were already at the end of the workday, so it didn’t matter much. But you can shorten the testing time by reducing the number of files, etc. I think a separate benchmark suite might be good to have. Tests should be shorter; benchmarks should only run for tags.

🐢 Yup. I should shorten the tests. I should make them use a shorter list of files, maybe.

🐇 Local tests have passed. You’ll have to check the logs.

🐢 It seems git diff --name-only --cached returns files not yet added. That’s causing an error in new repositories with stash. Stashing shouldn’t run at all when there are no staged files.

🐇 Git behavior between the local version and the remote version seems different.

🐢 I think the trace should also show the Git version string.

🐇 You should also require a minimum Git version for the commands. You’re using some newer options, and not all users may have them.

🐢 Yep. Let’s see the version on the CI now.

🐢 Ah, I see—the problem is not about versions. It’s that the CI doesn’t configure git config --global user.email and user.name. Commits don’t work without these.

🐇 You should remove the Git version report, then. It’s one more process call for no reason.

🐢 I think it should be in trace!; I can check the verbosity level and call it only if it’s trace.

🐢 I completed the xvc-config docs as well. I think I can merge it before the tests finish, as it’s just a documentation update.

🐇 You seem to be rushing for the release.

🐢 Yeah, I want to really start testing this on the servers.

🐇 Then go ahead; let’s release a new version.

🐇 It looks like you’re back for an evening session. I think it’s time to start using Xvc on your torrent server.

🐢 Ah, yeah. Let’s see how it goes.

🐇 Create a repository for torrents. Then you can add files to it with cache-type = symlink or cache-type = hardlink.

🐢 I think creating a local storage may also help. I can use it to store files to retrieve them later.

🐇 Umm. We don’t have a “garbage collection” facility yet, you know. There are no file deletions at the moment. It may be better to have a repository-to-repository transfer feature, using SSH.

🐢 Yeah, we don’t have SSH storage either. I think this highlights a lack of features.

🐇 It’s possible to mimic Rsync storage with xvc storage new generic, but it doesn’t feel quite the same.

🐢 Then I’m adding these three tickets.

🐇 I think for 0.3.4, the command we add might be xvc file delete. You can work on this and add rsync support as well.

🐢 There is a librsync bindings library for Rust. But it looks like it doesn’t allow transferring file contents between hosts. There is also rusync, which is similar to rsync and implemented in Rust. There is also fast_rsync in pure Rust. It uses MD4 to calculate deltas, though, I think.

🐇 None of these seem to have network capability, though.

🐢 I think it’s better just to use the process for now. We are trying to come up with the simplest solution for now. We are just trying to be more general.

🐇 Yes, I think we can just use the process for the time being.

🐢 Then let’s start by adding it.

🐇 We should also begin to create release notes. GitHub can generate them from PRs. Automatically generated release notes

🐢 Created an issue for that. Now we’re starting xvc storage new rsync #111.

🐇 Ok. Let’s do this.

🐢 It looks like we don’t have many architectural problems.

🐇 You’re forgetting how Exec::cmd works, though. You have those kinds of problems.

🐢 Yeah, I’m figuring that out. Exec::shell requires a string to run on the shell, while Exec::cmd just needs a command name. You have to supply args with another function. I’m writing a closure now to handle this.

🐢 It started working, but it revealed a much bigger problem: .xvc/ is not added to .gitignore in xvc init.

🐇 Wow, that’s a showstopper.

🐢 Yup. I think I should fix it as well.

🐇 On closer glance, I think the problem is not xvc init not modifying .gitignore; it modifies it incorrectly. Maybe putting certain files on a whitelist is a better idea than trying to blacklist everything.

🐢 Yeah, we should define a set of Git-tracked files and directories, whitelist them, and let all other files be ignored.

🐇 Go ahead, then.

🐢 I think I’ve fixed it. There were two problems: the initial gitignore content was wrong, and it was placed in the root of the repository instead of .xvc.

🐇 Maybe putting it directly in the root is better than hiding it in .xvc. What do you think?

🐢 There seems to be an assumption in file track to have a .gitignore file somewhere. I think we should also handle changes in .gitignore files throughout the repository.

🐇 Umm, yes. We should handle .gitignore files as well. But not all .gitignore files are changed by Xvc. How can we make sure they were modified by Xvc?

🐢 I think there are ways to do that, like tracking them somewhere. But I believe we shouldn’t try. We should just get a list of .gitignore files, git add them, and include them in the commit.

🐇 Ok. Let’s write a test for this as well. No .gitignore should appear in git status -s.

🐢 I wrote the test. It fails now. Do you think we should check the output of Git status to determine which files to add?

🐇 That may be a good idea. Git status should already know which files need to be added. We can use that information.

🐢 It looks like pathspec is enough to modify git add behavior. We should be able to write *.gitignore and let all gitignore files be added.

🐇 Let’s try this manually.

🐢 Yep, it works. git add '*.gitignore' adds all .gitignore files in the subdirectories, too.

🐇 Congrats. 👏🥳

🐢 I’m writing the missing documentation for the crates. There are proc_macro not expanded errors all over the code.

🐇 I think you should update rust-analyzer and everything. There must be a command for this.

🐢 I reinstalled RA with :LspInstall and restarted the LSP. It seems to work correctly now.

Ah yeah, when you work for free, you can do so at any time you want, perhaps. You also don’t have team members, and that means when you sit in front of this, you can move it.

Umm, right. Let’s take a look at the outstanding PRs.

You have a documentation PR#93. You also have work that you’ve begun to integrate Git into. I think it’s better to focus on the latter today.

Right. Let’s think about the relationship between Git and Xvc. I believe we should identify a general relation to avoid ending up in a mess like DVC and Git.

Why do you think the DVC and Git relationship is a mess?

They don’t automate common Git operations like a commit after dvc add. There is only auto-stage, and that’s turned off by default. This makes it seem that DVC wants to intervene as little as possible with the user’s Git workflow. That’s understandable. I support this. But on the other hand, they use the .git/ directory itself to store and manage experiments in a custom way that creates custom stash objects for experiments. This is against the principle of minimum intervention.

So this makes it a mess?

The mess, in my opinion, is caused by the second factor. If DVC doesn’t perform any Git operations, that’s alright. It was intended to be VCS-agnostic. Then experiments came and used Git internals in a way that no other similar tool uses.

Git-LFS and Git-Annex seem to use some non-standard mechanisms as well.

Ok. Not no other tool uses, but in a way that no other tool has used.

You know, GitHub PRs are also stored in a similar way. They also use non-standard machinery.

Yeah, but these tools are all Git-specific tools. They accept the dominion of Git, and don’t try to bring any VCS-agnosticism.

And Xvc tries to have this agnosticism?

I believe the initial design of DVC, which aims to be VCS-agnostic or being able to run without a VCS, is valuable. I like the idea behind Git, but the interface and implementation show that it’s a gradual development. There is no library behind it.

Libgit?

Libgit2 is something different. Although it’s said to have some common code, it doesn’t support all features. Git is command-line software with a mix of scripts and compiled executables, and not all code seems to be written in a way that could be used by external tools.

Hmm. The comment you added to the issue says git stash push --staged is not available in libgit2. Can’t you mimic it like DVC does for branches?

I don’t want to depend on Git at that level.

So, you’ll be using the CLI and shell for Git?

Yes, I believe, at the moment, before any performance tests, that this doesn’t matter much. Running Git commands once in a while using the shell shouldn’t make much difference in overall performance.

Then you’ll use it like a command-line tool, like the user?

Yes, and I’ll make it run outside of the usual threads. All Git will be like a sandwich, wrapping around Xvc operations. If there are --git-ref instructions in an xvc command, it will be run before Xvc performs the command, and if there are any changes in Xvc metafiles, they will be committed to the current branch.

Like

graph LR

co["git checkout"] --> xvc
xvc --> cm["git commit"]

The first could be a branch as well. So we have:

graph LR

br["git branch"] --> xvc
co["git checkout"] --> xvc
xvc --> cm["git commit"]

Looks sensible. How will you reflect these in the command line?

With something like xvc --git-checkout my-branch file list

Hmm, and for a branch?

I think instead of different options for branch, checkout or tag, we can have a git-ref option that marks the option as a git reference. It will be checked out, or created as a branch from the current one if it doesn’t exist.

I think creating a branch is not a good idea. It should be explicit. You can just send the --git-ref value to git checkout and perform the Xvc operation. If the user wants to create a branch, I think they can do it themselves.

What about storing the results in a branch? After adding a bunch of files, they may want to store them in another branch, maybe?

That’s sensible. We can have another option, like --to-branch in certain operations.

Or in the xvc command as a general option. In that case, we can change the option names to --from-ref and --to-branch. It will be like:

graph LR

fr["git checkout $(--from-ref)"] --> xvc
xvc --> tb["git checkout --branch $(--to-branch)"]
tb --> co["git add .xvc && git commit -m 'xvc cmd'"]

If no such options are given, xvc will run without branching, right?

Yep. --from-ref and --to-branch options are just shortcuts for user behavior. Any other VCS tool could be used this way. We don’t need to integrate Git at the library level.

This brings up the question of portability, though. When you aim for the software to be portable, you can’t rely on the existence of Git on the host, right?

I think a git.command option in the configuration is a good idea. Xvc will issue a warning if it can’t run the commands.

Will you use the shell to run this command? Otherwise no $PATH configuration is possible, you know.

I believe that could be another option: git.use_shell. If git.command is set to an absolute path, Xvc may use it without the shell. Otherwise, it can use the shell. Running the process directly will make it faster and more secure.

There is also this option to run Xvc in another process. Because we may access Git in the shell that runs Xvc, and if we can access it, maybe we don’t need shell execution in the process.

That’s a cool idea. But I wouldn’t add that extra complexity. Instead, we can try to find the git executable if git.command is not an absolute path. If git.command = /usr/bin/git in the configuration, we use it as is. Otherwise, we can get $PATH or %PATH% from the environment and search for git.command in that to find the exact executable.

It looks like there is a crate called which that does exactly what we are looking for.

Ah, cool. Then we can just use that to find the executable and run it. We don’t need to drop to a shell.

Yep. Let’s go back to implementation now.

🐢 Yep. It was a nice October. I would like to finish by increasing the coverage.

🐇 How about setting up a runner on your machine to test them locally?

🐢 Not now. I should fix the tests as always. Later I can take a look at it. I don’t want to distract myself with it.

🐇 You’ll need to create Apple Silicon binaries, though.

🐢 I think I can cross-compile for aarch.

🐇 You were doing this in the past, trying to build all binaries on a single Ubuntu VM. It didn’t work, as far as I remember.

🐢 I hadn’t even read this section then, and there is a dedicated project for it. The proper way is to use that. I’ll return to this after fixing the tests. I think we can release Xvc on all platforms supported by Rust.

🐇 Using cross-rs on GitHub Actions may be challenging. It seems Apple Silicon doesn’t need that much work either. You can set up cross-compilation with rustup targets.

🐢 Yeah. It looks so. Let’s take a look at the previous devlog.

🐇 The page seems missing. Would you like to fix your site first?

🐢 Ah, yeah. I should, maybe. Now it’s time to fix those PRs. I’ll take care of the site later. I should change the theme anyway.

🐇 Ok. Let’s take a look at outstanding PRs.

🐢 I want to fix the failed tests locally first. Yesterday I wasn’t able to redirect the output. It looks like the correct way of doing it is cargo test --all-features --no-fail-fast > $TMPDIR/xvc-test.log 2>&1. It’s required to put 2>&1 at the end, not between the redirection and output.

🐇 There is also the &> operator that you may want to use. cargo test --all-features --no-fail-fast &> $TMPDIR/xvc-test.log should be equivalent to this.

🐢 I’ll try it when the tests finish.

🐇 In the meantime, you can work to fix the documentation. In another repository, perhaps.

🐢 Good idea. Let me clone that.

🐇 Looks like the tests have finished and the only error is Minio. It couldn’t find the env variables you defined.

🐢 Restarted the shell and the tests. Returned to docs now.

🐢 Added a few function docs. Storage tests seem to pass. Probably they will fail in GA because of rsync tests using one.emresult.com login in my name.

🐇 If that’s expected, you can push and begin to fix it.

🐢 Yeah, let’s push the tests.

🐢 It looks like connecting to localhost also poses a challenge. Instead, I can create a user for xvc on the server and limit its usage.

🐇 Hmm. Good idea for now.

🐢 I created a new user xvc-test@one.emresult.com and its SSH keys. I’ll update the action to write the secret to the keyfile.

🐇 Looks like you forgot to install mc for Minio connection.

🐢 Yeah, I must convert Minio tests to use s3cmd.

🐇 And your region in s3 configuration seems to be wrong.

🐢 I’ll need to check this.

🐇 There is this line in the tests let region = env::var("AWS_DEFAULT_REGION").unwrap_or("us-east-1".to_string()); that’s probably causing that error. You should define the region properly.

🐢 I set this to eu-central-1 directly and will check the Minio error in the next session.

🐇 👏

🐢 It’s working except for the rsync tests now.

🐇 I think you can just use localhost to run the tests. You may need to install openssh-server, but it should work with localhost without configuration.

🐢 There may be a step missing in the configuration. I’ll try to make it run.

🐢 I’m trying to use .ssh/config in GA to allow connections to the server. It doesn’t work for some reason.

🐇 You may try to log in to the server outside of the tests and check if it’s running.

🐢 Yesterday’s last attempt was successful, and now we have working remote storage tests.

🐇 👏👏👏🥳

🐢 I’m merging the PR.

🐢 I’ve merged PR#66. Let’s list the outstanding issues now.

65	OPEN	Add Mermaid Support to Netlify Installation
64	OPEN	Fix `xvc-storage` compilation warnings	bug
54	OPEN	Fix all references to `xvc data` in the documentation
51	OPEN	Add a benchmark script to compare Xvc with other tools
50	OPEN	Consider changing `xvc file push` to `xvc file send` and `xvc file pull` to `xvc file retrieve`
49	OPEN	Add `--generic-command` as a dependency type to `xvc pipeline dependency`
48	OPEN	Write rsync example for `xvc storage generic`	documentation
47	OPEN	Write rclone example for `xvc storage generic`	documentation
46	OPEN	Allow to skip init in remotes with `--skip-init` option
45	OPEN	Update `VStore::to_store` to use some internal mechanism to avoid `XvcStore::insert`
43	OPEN	Update to `clap 4.0`
42	OPEN	Clean up xvc crate dependencies
36	OPEN	Add version to released file binaries
33	OPEN	Update `arch/remotes.md` for new naming	documentation
32	OPEN	Remove arc42 sections from the documentation	documentation
29	OPEN	Add documentation for `xvc storage new gcs`	documentation
28	OPEN	`xvc storage new yandex`	enhancement
24	OPEN	Create a logo for Xvc	documentation
23	OPEN	Add a Github action to upload new versions to crates.io	automation
20	OPEN	ref: Add rsync example for `xvc storage new generic`
18	OPEN	Add a new workflow for remote tests	automation
12	OPEN	Create a website for Xvc
5	OPEN	Add storage tests to Github Actions	automation
4	OPEN	fix clippy warnings	bug
1	OPEN	`xvc storage new` for all S3 compatible cloud services supported by `rust-s3`

🐇 As far as I know, you removed the arc42 stubs from the docs yesterday. You can close issue #32.

🐢 I closed it via gh issue close.

🐇 You should add a comment to that.

🐢 Added the comment.

🐇 It’s October 30th. Looking at the logs, you’ve closed some more issues since we last talked.

🐢 I’ve been productive over the last few days. Now, let’s take a look at the open issues.

🐇 I think you have a more pressing problem. Your tests are failing. You need to fix them first before moving to another task.

🐢 Created a new issue. The logs say:

81Z error: 7 targets failed:
2022-10-29T18:18:36.8969467Z ##[error]    `-p xvc-workflow-tests --test test_storage_new_digital_ocean`
2022-10-29T18:18:36.8970697Z     `-p xvc-workflow-tests --test test_storage_new_gcp`
2022-10-29T18:18:36.8971279Z     `-p xvc-workflow-tests --test test_storage_new_generic_rsync`
2022-10-29T18:18:36.8971840Z     `-p xvc-workflow-tests --test test_storage_new_minio`
2022-10-29T18:18:36.8972387Z     `-p xvc-workflow-tests --test test_storage_new_r2`
2022-10-29T18:18:36.8973035Z     `-p xvc-workflow-tests --test test_storage_new_s3`
2022-10-29T18:18:36.8973577Z     `-p xvc-workflow-tests --test test_storage_new_wasabi`
2022-10-29T18:18:36.9368079Z ##[error]The process '/home/runner/.cargo/bin/cargo' failed with exit code 101

🐇 Looking at the logs, you should clean up those older branches.

🐢 Let’s do it now.

🐢 Done. I’ve deleted all branches except main.

🐇 Good. I think the best way is to delete them as soon as you merge them.

🐢 I’ve activated that setting. These were older branches.

🐇 Cool. Now, will you be checking the failing tests?

🐢 I think remote tests should never run if there are other errors. We can run coverage and remote tests as a second step after the first succeed.

🐇 That seems like a neat idea. You want to split the current job into two: one for compiling and testing the non-remote parts, and the second for coverage and storage tests. It won’t waste CI minutes that way?

🐢 On second thought, I think at the moment, that’s not a pressing issue. We should start fixing these tests at once. We can split them later.

🐇 Ok. It looks from the logs like secrets are not being made available to your jobs.

2022-10-29T18:18:17.5039311Z test test_storage_new_digital_ocean ... FAILED
2022-10-29T18:18:17.5039550Z
2022-10-29T18:18:17.5039652Z failures:
2022-10-29T18:18:17.5039784Z
2022-10-29T18:18:17.5040197Z ---- test_storage_new_digital_ocean stdout ----
2022-10-29T18:18:17.5040558Z Error: VarError { source: NotPresent }
2022-10-29T18:18:17.5040758Z
2022-10-29T18:18:17.5040765Z

🐢 I added them now. Let’s wait until the job ends to get a new set of logs.

🐇 You can write some documentation in the meantime.

🐢 I think #82 is a good candidate for this. There must not be too many missing docs in the ECS crate.

🐇 It was about walker and you fixed the ecs. You’re the most absentminded developer here, I believe.

🐢 Ooops, you’re right. I’ll add them together in a single PR.

🐇 In the meantime, the storage testing job has ended with 🔴.

🐢 Checking the raw logs. It looks like we didn’t update the tests to match the current options:

2022-10-30T14:45:50.1058348Z error: Found argument '--storage-prefix' which wasn't expected, or isn't valid in this context
2022-10-30T14:45:50.1059809Z ##[error]Found argument '--storage-prefix' which wasn't expected, or isn't valid in this context
2022-10-30T14:45:50.1061740Z 	If you tried to supply `--storage-prefix` as a value rather than a flag, use `-- --storage-prefix`
2022-10-30T14:45:50.1063220Z
2022-10-30T14:45:50.1063394Z USAGE:
2022-10-30T14:45:50.1064302Z     xvc storage new digital-ocean --name <NAME> --bucket-name <BUCKET_NAME> --region <REGION>

🐇 Is it --storage-prefix or --remote-prefix? Which one is clearer?

🐢 I think updating the tests to conform to the current options is better for now. We can update the options later if desired.

🐇 Another failure is rg. You assume it exists on the testing system.

🐢 ripgrep is available in Ubuntu 20.04, so we can just update the initial package list.

🐇 The same --storage-prefix failure appears in the minio tests.

🐢 Ok. Fixing it.

🐇 S3 tests want to run the new-s3 subcommand. I think we now see why we need these tests in the first place.

🐢 Yeah. Fixing the prefix option, too.

🐇 s3cmd is also required. You should add it, too.

🐢 Ok. I think we’ve added all missing dependencies. I’ll have to convert the mc tests for Minio to use s3cmd.

🐇 Then we can try again. Now, we can get back to documentation.

🐢 Added some more documentation to PR#93. I think it’s better to return to the storage test errors now.

🐇 It looks like you have missing DigitalOcean credentials.

🐢 Let’s take a look.

🐢 I’ve updated the tests to use a config file instead of command-line arguments that would be visible in the logs.

🐇 Good practice. You also need to remove previous logs.

🐢 Maybe there is an option for that.

🐇 It looks like there isn’t. There are masking options; when we add them to secrets, they are masked. But you shouldn’t use them in calls anyway.