devlog 30

🐢 Let’s discuss how to move Xvc forward—maybe we can write a post to Reddit and the Rust forum in the meantime. 🐇 I think the next step is rclone remotes. It will allow us to use all remote storages supported by Rclone, which is a nice feature. 🐢 Don’t you think we need to publish the current version to Reddit and the forum? 🐇 We can do that as well. 🦊 Adding rclone remote must be a straightforward task. 🐢 We need to understand rclone paths, but overall, yes. We’ll just need to get the remote name, like drive://, and a path, like my-xvc-storage, and build paths with these. 🐇 What are the commands? 🐢 We need to learn how to upload files from local to remote and how to download these files. We can also list the files and get files as well. 🐲 How about adding a paths.txt to folders in remotes to show which paths the files in 0.jpg belong to? This will change the remote cache structure a bit. We will have a reverse index of files and they will be findable. 🐢 What’s the reason for this? 🐲 When I upload a file to Drive with only the content hash, I lose track of the actual path. This is not desirable. We can add a file to the directory, called paths.txt, to get the paths for a file. 🐢 This may prove to be a feat, though; adding these XvcPaths to a file requires a lookup. 🐇 Maybe a JSON file? It might be possible to look up a path with a JSON file, and it will be easier to parse. 🐲 I don’t think the issue is about parsing, though. We can just have a plain text file that lists the paths. It’s a text file, which is the most compatible across all storages. 🐢 Storages, you mean. 🐲 Ugh, yeah. If I have a file called Alan Watts but I only have the content, this file will be immensely useful. 🐢 This makes XvcCachePath and XvcPath coupled. Architecture-wise, it may not be a good thing, though. 🐇 Also, there may be common storages for multiple repositories. 🐲 Umm, that’s a good point. I don’t think the architecture will be much compromised, though. We already keep the file paths and their cache paths somewhere. 🐢 Cache paths are generated from the content, but any number of paths can point to a single path in the cache. If I have 1 million copies of the same file, will I add all these files to the paths.txt you mentioned? 🐲 That’s a good point too. We can have a limit, like 1,000 or something, not to make these files too big. 🐢 Instead of this, we can store the output of xvc file list at the storage root and allow looking up the files that way. 🐲 It has the same problem, though; if we have a million files, their list will be too large. 🐢 There can be a manual command, like xvc file index --to storage, that will show content hashes and paths of each file. We can also add URLs to files if possible. 🐲 No one will use it when it’s manual, though. 🐢 We can add functionality to update this index when we send a file, though.
🐲 So, after each send, we’ll update the index for the repository on that storage. Is that correct? 🐢 Not after each send. After each send session, maybe. 🐇 We can have an incremental way of updating the index, like we do in ECS? 🐢 It will be overkill for this functionality and add too much noise to the storage. 🐲 Let’s keep this discussion here, but I also want to have an index merge or index cleanup mechanism for the entity generator and the ECS. 🐢 We can have a “merge indices” functionality in ECS. That will remove all older entity-generator files and merge all store files. 🐇 Removing older entity files is easy, but what about merging the store files? 🐢 It’s easy too. We’ll just load all event logs from the directory, remove all other files, and save the event log to a file. 🐇 Will this be manual or automatic? 🐢 I think the first version can be manual, something like xvc fsck merge-store-files or something like that. We can notify the user if the number of files is > 10,000 or something like that. I don’t think we need to make it automatic unless we measure the impact of these files. There is no point in trying to do it at every command. 🐇 Then we’ll have two new commands for the next version? 🐢 I think we can just add rclone remote now and release it, then make changes in the ECS for this new xvc fsck command. 🐇 Can the name be doctor or something? Or util? Or can we add a top-level merge indices command? 🐢 xvc doctor seems like a better alternative. We can have a diagnose subcommand as well to check for possible inconsistencies. xvc doctor merge-store-files is a better command. 🐲 Will we use d for this command? 🐢 No need to add a single-letter command for this, I believe. It shouldn’t be required to run frequently. 🐇 Hmm, ok. What do we need to know for rclone remote? 🐲 I noticed we don’t have the xvc storage remove command implemented yet. Maybe we can start from that. 🐢 Hmm, yeap. Let’s start by implementing that first. We can add the rclone command next. 🐇 Will we use a feature flag for rclone? It will run the command only with an external binary. 🐢 It’s better to have a feature flag. I think we can add a feature flag for rsync remote as well. 🐇 We can use the generic one to update the feature flag. 🐢 I think the only two items of information we need for rclone are the remote name and the remote directory. Will we make these required?