emre şahin's digital garden 🍃 - storage

devlog 30

Emre Şahin — 2025-05-11T18:17:52+00:00

🐢 Let’s discuss how to move Xvc forward—maybe we can write a post to Reddit and the Rust forum in the meantime. 🐇 I think the next step is rclone remotes. It will allow us to use all remote storages supported by Rclone, which is a nice feature. 🐢 Don’t you think we need to publish the current version to Reddit and the forum? 🐇 We can do that as well. 🦊 Adding rclone remote must be a straightforward task. 🐢 We need to understand rclone paths, but overall, yes. We’ll just need to get the remote name, like drive://, and a path, like my-xvc-storage, and build paths with these. 🐇 What are the commands? 🐢 We need to learn how to upload files from local to remote and how to download these files. We can also list the files and get files as well. 🐲 How about adding a paths.txt to folders in remotes to show which paths the files in 0.jpg belong to? This will change the remote cache structure a bit. We will have a reverse index of files and they will be findable. 🐢 What’s the reason for this? 🐲 When I upload a file to Drive with only the content hash, I lose track of the actual path. This is not desirable. We can add a file to the directory, called paths.txt, to get the paths for a file. 🐢 This may prove to be a feat, though; adding these XvcPaths to a file requires a lookup. 🐇 Maybe a JSON file? It might be possible to look up a path with a JSON file, and it will be easier to parse. 🐲 I don’t think the issue is about parsing, though. We can just have a plain text file that lists the paths. It’s a text file, which is the most compatible across all storages. 🐢 Storages, you mean. 🐲 Ugh, yeah. If I have a file called Alan Watts but I only have the content, this file will be immensely useful. 🐢 This makes XvcCachePath and XvcPath coupled. Architecture-wise, it may not be a good thing, though. 🐇 Also, there may be common storages for multiple repositories. 🐲 Umm, that’s a good point. I don’t think the architecture will be much compromised, though. We already keep the file paths and their cache paths somewhere. 🐢 Cache paths are generated from the content, but any number of paths can point to a single path in the cache. If I have 1 million copies of the same file, will I add all these files to the paths.txt you mentioned? 🐲 That’s a good point too. We can have a limit, like 1,000 or something, not to make these files too big. 🐢 Instead of this, we can store the output of xvc file list at the storage root and allow looking up the files that way. 🐲 It has the same problem, though; if we have a million files, their list will be too large. 🐢 There can be a manual command, like xvc file index --to storage, that will show content hashes and paths of each file. We can also add URLs to files if possible. 🐲 No one will use it when it’s manual, though. 🐢 We can add functionality to update this index when we send a file, though.
🐲 So, after each send, we’ll update the index for the repository on that storage. Is that correct? 🐢 Not after each send. After each send session, maybe. 🐇 We can have an incremental way of updating the index, like we do in ECS? 🐢 It will be overkill for this functionality and add too much noise to the storage. 🐲 Let’s keep this discussion here, but I also want to have an index merge or index cleanup mechanism for the entity generator and the ECS. 🐢 We can have a “merge indices” functionality in ECS. That will remove all older entity-generator files and merge all store files. 🐇 Removing older entity files is easy, but what about merging the store files? 🐢 It’s easy too. We’ll just load all event logs from the directory, remove all other files, and save the event log to a file. 🐇 Will this be manual or automatic? 🐢 I think the first version can be manual, something like xvc fsck merge-store-files or something like that. We can notify the user if the number of files is > 10,000 or something like that. I don’t think we need to make it automatic unless we measure the impact of these files. There is no point in trying to do it at every command. 🐇 Then we’ll have two new commands for the next version? 🐢 I think we can just add rclone remote now and release it, then make changes in the ECS for this new xvc fsck command. 🐇 Can the name be doctor or something? Or util? Or can we add a top-level merge indices command? 🐢 xvc doctor seems like a better alternative. We can have a diagnose subcommand as well to check for possible inconsistencies. xvc doctor merge-store-files is a better command. 🐲 Will we use d for this command? 🐢 No need to add a single-letter command for this, I believe. It shouldn’t be required to run frequently. 🐇 Hmm, ok. What do we need to know for rclone remote? 🐲 I noticed we don’t have the xvc storage remove command implemented yet. Maybe we can start from that. 🐢 Hmm, yeap. Let’s start by implementing that first. We can add the rclone command next. 🐇 Will we use a feature flag for rclone? It will run the command only with an external binary. 🐢 It’s better to have a feature flag. I think we can add a feature flag for rsync remote as well. 🐇 We can use the generic one to update the feature flag. 🐢 I think the only two items of information we need for rclone are the remote name and the remote directory. Will we make these required?

Xvc Devlog - 221109

Emre Şahin — 2022-11-10T09:26:00+00:00

🐇 How do you want to proceed from here, Mr. 🐢?

🐢 I think I can implement Rsync today. Looking at ssh2-rs, though, I think we can implement file transfer without relying on rsync. It might be easier to implement everything within the code.

🐇 Then you should rename the issue to new ssh.

🐢 Fair. There is also the ssh_rs crate, but it doesn’t have full support for the protocol. Instead, we can have another command, like xvc storage new ssh, that uses libssh2 via the crate mentioned above. It has some limitations with OpenSSH on macOS.

🐇 From the crate’s README, it looks like you can enable the vendored-openssl feature to compile it statically.

🐢 Let’s go ahead then. It’s better to compile it behind a feature flag, though.

🐇 Yup. Rsync can be separate. I think for now you can implement rsync via Exec::cmd and make new ssh a new issue.

🐢 I’ll copy this conversation there.

🐇 Now, let’s start implementing rsync.

🐢 Do we really want to hide it behind a feature flag? It doesn’t bring any extra complexity to generic, for example—just using the commands and returning the errors.

🐇 I think so. If the user doesn’t have rsync on their system, they’ll just get errors. We don’t need to make the implementation optional, but the tests might be.

🐢 OK.

Xvc Devlog - 221108

Emre Şahin — 2022-11-09T10:42:00+00:00

🐇 I think we can begin by checking GitHub PRs. What do you have today?

🐢 The tests for yesterday’s Git integration PR failed. I’ll begin by checking the logs.

🐇 Maybe run the tests locally to see. They might fail on your machine as well. It might be a simple thing.

🐢 Probably, yes. I’ve started the tests now. I shouldn’t forget to run them before testing the PR; it may save some time.

🐇 Ah, yeah, maybe. Yesterday you were already at the end of the workday, so it didn’t matter much. But you can shorten the testing time by reducing the number of files, etc. I think a separate benchmark suite might be good to have. Tests should be shorter; benchmarks should only run for tags.

🐢 Yup. I should shorten the tests. I should make them use a shorter list of files, maybe.

🐇 Local tests have passed. You’ll have to check the logs.

🐢 It seems git diff --name-only --cached returns files not yet added. That’s causing an error in new repositories with stash. Stashing shouldn’t run at all when there are no staged files.

🐇 Git behavior between the local version and the remote version seems different.

🐢 I think the trace should also show the Git version string.

🐇 You should also require a minimum Git version for the commands. You’re using some newer options, and not all users may have them.

🐢 Yep. Let’s see the version on the CI now.

🐢 Ah, I see—the problem is not about versions. It’s that the CI doesn’t configure git config --global user.email and user.name. Commits don’t work without these.

🐇 You should remove the Git version report, then. It’s one more process call for no reason.

🐢 I think it should be in trace!; I can check the verbosity level and call it only if it’s trace.

🐢 I completed the xvc-config docs as well. I think I can merge it before the tests finish, as it’s just a documentation update.

🐇 You seem to be rushing for the release.

🐢 Yeah, I want to really start testing this on the servers.

🐇 Then go ahead; let’s release a new version.

🐇 It looks like you’re back for an evening session. I think it’s time to start using Xvc on your torrent server.

🐢 Ah, yeah. Let’s see how it goes.

🐇 Create a repository for torrents. Then you can add files to it with cache-type = symlink or cache-type = hardlink.

🐢 I think creating a local storage may also help. I can use it to store files to retrieve them later.

🐇 Umm. We don’t have a “garbage collection” facility yet, you know. There are no file deletions at the moment. It may be better to have a repository-to-repository transfer feature, using SSH.

🐢 Yeah, we don’t have SSH storage either. I think this highlights a lack of features.

🐇 It’s possible to mimic Rsync storage with xvc storage new generic, but it doesn’t feel quite the same.

🐢 Then I’m adding these three tickets.

🐇 I think for 0.3.4, the command we add might be xvc file delete. You can work on this and add rsync support as well.

🐢 There is a librsync bindings library for Rust. But it looks like it doesn’t allow transferring file contents between hosts. There is also rusync, which is similar to rsync and implemented in Rust. There is also fast_rsync in pure Rust. It uses MD4 to calculate deltas, though, I think.

🐇 None of these seem to have network capability, though.

🐢 I think it’s better just to use the process for now. We are trying to come up with the simplest solution for now. We are just trying to be more general.

🐇 Yes, I think we can just use the process for the time being.

🐢 Then let’s start by adding it.

🐇 We should also begin to create release notes. GitHub can generate them from PRs. Automatically generated release notes

🐢 Created an issue for that. Now we’re starting xvc storage new rsync #111.

🐇 Ok. Let’s do this.