emre şahin's digital garden 🍃 - ecs

devlog 30

Emre Şahin — 2025-05-11T18:17:52+00:00

🐢 Let’s discuss how to move Xvc forward—maybe we can write a post to Reddit and the Rust forum in the meantime. 🐇 I think the next step is rclone remotes. It will allow us to use all remote storages supported by Rclone, which is a nice feature. 🐢 Don’t you think we need to publish the current version to Reddit and the forum? 🐇 We can do that as well. 🦊 Adding rclone remote must be a straightforward task. 🐢 We need to understand rclone paths, but overall, yes. We’ll just need to get the remote name, like drive://, and a path, like my-xvc-storage, and build paths with these. 🐇 What are the commands? 🐢 We need to learn how to upload files from local to remote and how to download these files. We can also list the files and get files as well. 🐲 How about adding a paths.txt to folders in remotes to show which paths the files in 0.jpg belong to? This will change the remote cache structure a bit. We will have a reverse index of files and they will be findable. 🐢 What’s the reason for this? 🐲 When I upload a file to Drive with only the content hash, I lose track of the actual path. This is not desirable. We can add a file to the directory, called paths.txt, to get the paths for a file. 🐢 This may prove to be a feat, though; adding these XvcPaths to a file requires a lookup. 🐇 Maybe a JSON file? It might be possible to look up a path with a JSON file, and it will be easier to parse. 🐲 I don’t think the issue is about parsing, though. We can just have a plain text file that lists the paths. It’s a text file, which is the most compatible across all storages. 🐢 Storages, you mean. 🐲 Ugh, yeah. If I have a file called Alan Watts but I only have the content, this file will be immensely useful. 🐢 This makes XvcCachePath and XvcPath coupled. Architecture-wise, it may not be a good thing, though. 🐇 Also, there may be common storages for multiple repositories. 🐲 Umm, that’s a good point. I don’t think the architecture will be much compromised, though. We already keep the file paths and their cache paths somewhere. 🐢 Cache paths are generated from the content, but any number of paths can point to a single path in the cache. If I have 1 million copies of the same file, will I add all these files to the paths.txt you mentioned? 🐲 That’s a good point too. We can have a limit, like 1,000 or something, not to make these files too big. 🐢 Instead of this, we can store the output of xvc file list at the storage root and allow looking up the files that way. 🐲 It has the same problem, though; if we have a million files, their list will be too large. 🐢 There can be a manual command, like xvc file index --to storage, that will show content hashes and paths of each file. We can also add URLs to files if possible. 🐲 No one will use it when it’s manual, though. 🐢 We can add functionality to update this index when we send a file, though.
🐲 So, after each send, we’ll update the index for the repository on that storage. Is that correct? 🐢 Not after each send. After each send session, maybe. 🐇 We can have an incremental way of updating the index, like we do in ECS? 🐢 It will be overkill for this functionality and add too much noise to the storage. 🐲 Let’s keep this discussion here, but I also want to have an index merge or index cleanup mechanism for the entity generator and the ECS. 🐢 We can have a “merge indices” functionality in ECS. That will remove all older entity-generator files and merge all store files. 🐇 Removing older entity files is easy, but what about merging the store files? 🐢 It’s easy too. We’ll just load all event logs from the directory, remove all other files, and save the event log to a file. 🐇 Will this be manual or automatic? 🐢 I think the first version can be manual, something like xvc fsck merge-store-files or something like that. We can notify the user if the number of files is > 10,000 or something like that. I don’t think we need to make it automatic unless we measure the impact of these files. There is no point in trying to do it at every command. 🐇 Then we’ll have two new commands for the next version? 🐢 I think we can just add rclone remote now and release it, then make changes in the ECS for this new xvc fsck command. 🐇 Can the name be doctor or something? Or util? Or can we add a top-level merge indices command? 🐢 xvc doctor seems like a better alternative. We can have a diagnose subcommand as well to check for possible inconsistencies. xvc doctor merge-store-files is a better command. 🐲 Will we use d for this command? 🐢 No need to add a single-letter command for this, I believe. It shouldn’t be required to run frequently. 🐇 Hmm, ok. What do we need to know for rclone remote? 🐲 I noticed we don’t have the xvc storage remove command implemented yet. Maybe we can start from that. 🐢 Hmm, yeap. Let’s start by implementing that first. We can add the rclone command next. 🐇 Will we use a feature flag for rclone? It will run the command only with an external binary. 🐢 It’s better to have a feature flag. I think we can add a feature flag for rsync remote as well. 🐇 We can use the generic one to update the feature flag. 🐢 I think the only two items of information we need for rclone are the remote name and the remote directory. Will we make these required?

devlog 16

Emre Şahin — 2025-01-20T12:58:54+00:00

🐢 Good morning. Yesterday we wrote a proxy server for a job application. It was a pleasing experience, but I wasn’t able to write a dialogue.

🐇 It was one of those nice days. I have that feeling in my head that I used it to its full potential.

🐢 So we can do some light work today.

🐇 Like completions?

🐢 Yep, let’s fill in some missing pieces there.

🐇 Let’s see what those missing pieces are.

🐢 We need to clean up the code comments we copied from jj.

🦊 Cleaned up quite a bit and moved everything to main. It’s a single function call.

🐢 I think we completed all strum-related completions. Now we need to discuss store completions a bit.

🐇 They can’t all have the same function like we used for strum-based enums. They will look up different parts of components. For XvcPath, it will be the inner RelativePathBuf; for StorageIdentifier, that will be the names of all recorded storage names and GUIDs. The field we’re going to complete will change from case to case.

🦊 They can depend on the same machinery. It will detect .xvc, load a store, and return a field from the component. The first two are the same for all.

🐢 Yep. Let’s write a few TODOs in the code, then.

🦊 One thing to discuss is whether we should load all config and root. It may be a bit time-consuming. Just loading the stores should be enough.

🐢 I think so. ECS doesn’t depend on XvcRoot, and we can load stores without loading XvcRoot. We have convenience functions in XvcRoot, but they are for convenience. There is nothing that prevents us from loading stores without loading the root.

🦊 Let’s take a look at XvcRoot’s loading of these.

🐢 Tomorrow, hopefully.

Devlog 10: Codecov, Doctests, and Cargo Publish Struggles

Emre Şahin — 2025-01-19T09:30:49+00:00

🐢 Now we’re going into the real deep work. The only failure was the Codecov patch in Xvc. Let’s see what needs to be done.

🐇 It looks, from the coverage page, that our additions to HStore don’t have any tests. We can add some unit tests to new joins, maybe.

🐢 I don’t find unit tests particularly useful, but let’s use GitHub Copilot to add tests for us.

🐇 Added a unit test and a doc test for full_join, and I think doc tests have more value. They provide documentation, and we can readily see how to use a function from its docs. Better to increase coverage with doc tests.

🐢 There are points that I should learn while writing doc tests. The imports must use the full path, not crate::. The tested struct also doesn’t have implicit imports.

🐇 The ceremony of adding keys and values is a bit too much. It may be worthwhile to add insert for any Into.

🐢 It will certainly save time if we don’t have to type .into() for each key :)

🐇 Pushed to test again. Should we have some means to test coverage locally?

🐢 I don’t think we need to consider coverage locally. It’s not worth our time.

🐇 Now while waiting for tests to be completed, what can we do?

gh -R iesahin/xvc run list
completed	failure	v0.6.13	Rust-CI	v0.6.13	pull_request	12543831360	3m54s	2024-12-30T08:15:06Z
...

🐢 I don’t think it took too much time. Let’s view the results:

gh -R iesahin/xvc run view 12543831360
...
  X Run Current Dev Tests
...
To see what failed, try: 
View this run on GitHub: https://github.com/iesahin/xvc/actions/runs/12543831360

🐢 The current dev tests fail for some reason. Let’s run these locally.

🐇 We’re missing llvm-tools-preview locally. How do we install this?

🐢 The command is:

rustup component add llvm-tools-preview
info: component 'llvm-tools' for target 'aarch64-apple-darwin' is up to date

🐇 It’s already installed. We need to set the environment variables.

🐢 Instead, we can just turn off dev tests for the time being. We don’t need them. Our local tests pass.

🐇 Yeah, ok, we don’t need to solve each and every bit of these issues.

gh -R iesahin/xvc run list

… 🐇 Ok. Let’s take a look at the run again.

gh -R iesahin/xvc run list
completed	failure	v0.6.13	Rust-CI	v0.6.13	pull_request	12544020064	3m39s	2024-12-30T08:33:25Z

gh -R iesahin/xvc run view 12544020064
...
  X Test and Coverage
...

gh run view 12544020064 --log-failed
...
Test and Coverage (stable)	Test and Coverage	2024-12-30T08:36:58.8911690Z Error: ProcessError { stdout: "", stderr: "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to e1.xvc.dev port 80 after 160 ms: Couldn't connect to server\n" }
...

🐢 We need to start nginx on the server. We forgot it yesterday.

🐇 Ah, yeah. After adding that dufs installation. Ok.

🐢 We also need to add a reverse proxy to dufs somehow, but this is for later.

🐇 For the use case, I don’t think it’s necessary. We can just adjust the port to a non-standard one if we need to use 443 for another thing, but let’s take a look at the tests again.

🐢 Let’s add another doc test. This time to XvcStore.

cargo test -p xvc-ecs --doc
...
test result: ok. 8 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.87s

🐇 Sent the files again. Waiting for tests to finish.

🐢 Let’s check the keymaps file in the meantime.

🐇 I tried to read some documentation but didn’t see an error. Maybe we should just set it to non-lazy and avoid allowing remaps.

🐢 We already spent too much time with this.

🐇 Yes, let’s take a look at the tests again.

gh -R iesahin/xvc run list
completed	success	v0.6.13	Rust-CI	v0.6.13	pull_request	12544291068	8m10s	2024-12-30T09:00:42Z
...

🐢 Oh, yeah, the merge is ready.

🐇 Patch coverage is still behind the target, though.

🐢 Yeah, but let’s release this one and make the next better covered. Also, I’m not sure if the doc tests had any effect on coverage.

🐇 If we look at the coverage page again, we can see if the doc tests had any effect.

🐢 It seems codecov.io doesn’t consider coverage for doc tests. This is a bit weird, but let’s not spend more time on this.

🐇 Sure, let’s merge.

🐢 I think we forgot to bump the version in Cargo.toml. We’ll have to do that in main.

🐇 Oh, yeah. Let’s bump it and tag as well.

🐢 Now, we can wait for all files to be produced. What will we do next?

🐇 We can just release the Python version as well. It shouldn’t need any changes.

🐢 Umm, right. Maybe we can add a few tests as well.

🐇 Let’s bump the version first and see.

🐢 Bumped versions in Cargo.toml and ran maturin develop.

🐇 It seems ready now.

🐢 The main fails, though. The “Publish Crates” action looks for libsqlite3. Let’s take a look at it.

gh -R iesahin/xvc run list
completed	failure	Release v0.6.13	Publish Crates	v0.6.13	push	12544753359	6m1s	2024-12-30T09:41:56Z

🐢 The issue is that the VM doesn’t have libsqlite3-dev. Let’s add it.

🐇 We need to start the job manually again. Let’s not tag this time.

🐢 Some of the packages were already published. Now they break. crates.io says they’re already published. Maybe we can check if a package is already published.

🐇 Let’s check if we can make cargo publish more forgiving.

🐢 There doesn’t seem to be an option. Let’s search “how to skip published packages in the workspace to avoid errors with cargo publish”.

🐇 Claude is bullshitting again. Let’s try a manual approach: how to skip already published packages.

🐢 It may be easier to just add a check if the package is published. How do we get the info?

🐇 Or we can just go on to the next package if the package is already available.

🐢 Let’s do this manually this time.

devlog 5

Emre Şahin — 2024-07-08T20:35:24+00:00

While writing Xvc tests for Python, I hit an error caused by the ECS single-load protection.

The single-loader allows only one instance of Xvc to be run in a single process. This is no problem for the shell, but it looks like it won’t be possible to use multiple Xvc instances in a single Python process.

It’s possible to overcome this with an elaborate multiprocessing setup in the wrapper, but I won’t bother with it for now.