🐢 Let’s discuss how to move Xvc forward – maybe we can write a post to Reddit and the Rust forum in the meantime. 🐇 I think the next step is rclone remotes. It will allow to use all remote storages supported by Rclone and that’s a nice feature 🐢 Don’t you think we need to publish the current version to reddits and the forum 🐇 We can do that as well 🦊 Adding rclone remote must be a straightforward task 🐢 We need to understand rclone paths but overall yes. We’ll just to get remote name, like drive:// and a path, like my-xvc-storage and build paths with these. 🐇 What are the commands? 🐢 We need to learn how to upload files from local to remote and how to download these files. We can also list the files and get files as well 🐲 How about adding a paths.txt to folders in remotes to show the paths files in 0.jpg belongs. This will change the remote cache structure a bit. We will have a reverse index of files and they will be findable 🐢 What’s the reason for this? 🐲 When I upload a file to drive with only the content hash, I lose track of the actual path. This is not desirable. We can add a file to the directory, called paths.txt to get the paths for a file. 🐢 This may prove to be a feat, though, adding these XvcPaths to a file requires a lookup 🐇 Maybe a JSON file? It can be possible to lookup a path with a JSON file and it will be easier to parse 🐲 I don’t think the issue is about parsing though. We can just have a plain text file that lists the paths. It’s a text file that’s the most compatible across all files. 🐢 Storages, you mean 🐲 Ugh, yeah. If I have a file called Alan Watts but I only have the content, this file will immensely be useful. 🐢 This makes XvcCachePath and XvcPath to be coupled. Architecture-wise it may not be a good thing, though. 🐇 Also, there may common storages for multiple repositories. 🐲 Umm, that’s a good point. I don’t think architecture will be much offended though. We already keep the file paths and their cache paths somewhere. 🐢 Cache paths are generated from the content but any number of paths can point to a single path in the cache. If I have 1 million copies of the same file, will I add all these files to the paths.txt you mention? 🐲 That’s a good point too. We can have a limit, like 1000 or something not to make these files too big. 🐢 Instead of this, we can store the output of xvc file list at the storage root and allow to look up the files that way. 🐲 It has the same problem though, if we have a million files, their list will be too big 🐢 There can be a manual command, like xvc file index –to storage that will show content hashes and paths of each file. We can also add URLs to files if possible. 🐲 No one will use it when it’s manual, though. 🐢 We can add to update this index when we send a file though.
🐲 So, after each send we’ll update the index for the repository on that storage. Is that correct? 🐢 Not after each send. After each send session, maybe. 🐇 We can have an incremental way of updating the index, like we do in ECS? 🐢 It will be an overkill for this functionality and add too much noise to the storage 🐲 Let’s keep this discussion here but I also want to have an index merge or index cleanup mechanism for entity generator and the ECS. 🐢 We can have a merge indices functionality in ECS. That will remove all older entity-generator files and merge all store files. 🐇 Removing older entity files is easy but what about merging the store files? 🐢 It’s easy too. We’ll just load all event logs from the dir, remove all other files and save the event log to a file. 🐇 Will this be manual or automatic? 🐢 I think the first version can be manual, something like xvc fsck merge-store-files or something like that. We can notify the user if the number of files are > 10000 or something like that. I don’t think we need to make it automatic unless we measure the impact of these files. There is no point to try to do it at every command. 🐇 Then we’ll have two new commands for the next version? 🐢 I think we can just add rclone remote now and release it, then make changes in the ECS for this new xvc fsck command. 🐇 Can the name be doctor or something? Or util? Or can we add a top level merge indices command? 🐢 xvc doctor seems to be a better alternative. We can have a diagnose subcommand as well to check possible inconsistencies. xvc doctor merge-store-files is a better command 🐲 Will we use d for this command? 🐢 No need to add a single letter command for this, I believe. It shouldn’t be required to run frequently. 🐇 Hmm, ok. What do we need to know for rclone remote? 🐲 I noticed we don’t have xvc storage remove command implemented yet. Maybe we can start from that. 🐢 Hmm, yeap. Let’s start to implement that first. We can add rclone command next. 🐇 Will we use a feature flag for rclone? It will run the command with only an external binary 🐢 It’s better to have feature flag. I think we can add a feature flag for rsync remote as well. A 🐇 We can use the generic one to update the feature flag 🐢 I think the only two items of information we need for rclone is remote name and the remote directory. Will be make these required?