🐇 We can continue working on rclone.

🐢 Umm, ok. Let’s try to focus on adding another storage type.

• ✅ #🌻 ADD rclone storage type (2025-04-24 06:27)

🐢 I think the first option is to run the commands from the command line. We can just use a modified generic storage type without trying to make it fast.

🐇 Let’s make it run first, you say?

🐢 Yes, let’s make it run first and then we can think about making it run fast.

🐇 You’re right!

• ✅ #🌻 ADD generic rclone tests (2025-04-24 06:27)

It’s possible to use the alias remote with a local path.

🐢 Created the PR, waiting for the tests.

🦊 Let’s search if there is a crate to manage rclone. Maybe it will be easier that way.

🐇 It will add another dependency though.

🦊 We can always add a feature flag for this.

🐢 There is a librclone crate that can be used to call rclone commands like https://rclone.org/rc/#supported-commands

🦊 We can test it from the command line perhaps.

rclone rc 
2025/03/15 17:41:09 NOTICE: Failed to rc: failed to list: connection failed: Post "http://localhost:5572/rc/list": dial tcp [::1]:5572: connect: connection refused

🐇 It requires the backend to be running in the background.

🦊 There may be examples in the repository.

🐢 There are none. We can search GH for this crate though.

🦊 It’s also possible to search for dependents in crates.io.

🐢 I think Xvc will be the first dependent of this crate: https://crates.io/crates/librclone/reverse_dependencies

🐇 The following two projects depend on librclone:

• https://github.com/Sh3mm/WarpDrive/tree/master
• https://github.com/hwittenborn/celeste

🐢 Let’s clone Celeste. It uses librclone and looks like it’s a user interface for rclone written in Rust.

🐇 The examples are in celeste/src/rclone.rs.

🐢 Cool. Let’s take a look at how commands are run:

    /// Common function for some of the below command.
    fn common(command: &str, remote_name: &str, path: &str) -> Result<(), RcloneError> {
        let resp = run(
            command,
            &json!({
                "fs": get_remote_name(remote_name),
                "remote": util::strip_slashes(path),
            })
            .to_string(),
        );

        match resp {
            Ok(_) => Ok(()),
            Err(json_str) => Err(serde_json::from_str(&json_str).unwrap()),
        }
    }

All commands are run like librclone::rpc(method, input)) and the commands are like:

    /// make a directory on the remote.
    pub fn mkdir(remote_name: &str, path: &str) -> Result<(), RcloneError> {
        common("operations/mkdir", remote_name, path)
    }

🐇 We have all commands in this file that are relevant to Xvc. Let’s list them here:

• make directory: common("operations/mkdir", remote_name, path)
• delete file: common("operations/delete", remote_name, path)
• remove a dir and all of its contents: common("operations/purge", remote_name, path)
• copy file:

run( "operations/copyfile",
            &json!({
                "srcFs": src_fs,
                "srcRemote": util::strip_slashes(src_remote),
                "dstFs": dst_fs,
                "dstRemote": util::strip_slashes(dst_remote)
            })

and

    /// Copy a file from the local machine to the remote.
    pub fn copy_to_remote(
        local_file: &str,
        remote_name: &str,
        remote_destination: &str,
    ) -> Result<(), RcloneError> {
        copy(
            "/",
            local_file,
            &get_remote_name(remote_name),
            remote_destination,
        )
    }

    /// Copy a file from the remote to the local machine.
    pub fn copy_to_local(
        local_destination: &str,
        remote_name: &str,
        remote_file: &str,
    ) -> Result<(), RcloneError> {
        copy(
            &get_remote_name(remote_name),
            remote_file,
            "/",
            local_destination,
        )
    }

🐇 It looks like that’s all we need. We can organize the commands differently, but these examples are enough to use librclone. It seems rather straightforward.

Full list of similarities and differences is rather long. Let me summarize it.

Xvc has different commands; xvc file track is used instead of dvc add. Xvc doesn’t add files (like .dvc files) to your repository and keeps all metadata tracking under the .xvc directory. Checkout method is per-file, not configured globally, so you can keep track of your data directory with symlinks and your model directory as copies. Xvc uses BLAKE3 as the default hashing algorithm, and you can configure this to be BLAKE2, SHA-2, or SHA-3.

Pipelines are not defined using YAML files. You can write a shell script with xvc pipeline step ... or use Python xvc.pipeline().step().dependency(step_name="preprocess", param="hyperparams.yaml::batch_size") to define pipelines first. Then you can use xvc pipeline export and xvc pipeline import to modify the pipeline in YAML, JSON, or TOML.

There are more dependency options; e.g., a pipeline step may depend on a text file partially, by regex or by the line options. There is a generic dependency option; the output of a shell command can be used as a dependency to a step.

Remote storage options are rather limited for Xvc; local, ssh+rsync, and S3-compatible storages are supported for now. There is also a generic storage option where you can define upload and download commands for the tool you’re using, e.g., rclone or s5cmd, and Xvc can use it. I’ll add Azure and rclone as natively supported storage options eventually, but I don’t like the idea of keeping credentials, so there won’t be any OAuth-required storage options, e.g., Google Drive. (You’ll be able to use these through rclone, though.) All Xvc storages use environment variables for authentication.

Xvc doesn’t have experiment tracking yet. You can use --from-ref and --to-branch options to store artifacts from the pipeline to different branches. I’ll add features to run pipelines and commands quickly and compare these eventually (I need one too), but it may take some time.

Xvc doesn’t track anything about the user. It shouldn’t make any network connections if you don’t specifically ask it to do so. I’m planning to add binaries that only do file operations or pipeline operations, so if someone doesn’t need pipeline features, they will simply use xvc-file.

Nowadays, I’m using rclone. It has become my primary way of using Drive on Linux after that half-baked sync application. I discovered that rclone supports server-side MD5 hashes for files. I decided to write a script to delete the duplicate files in Drive.

First, I obtain the MD5 hashes of all files in Drive using:

$ rclone md5hash drive:/ > $HOME/Google-Drive-md5-$(date +%F).txt

This may take some time depending on the number of files, but it finished more quickly than I expected.

The file contents look like this:

39044094333de4a47d7478227cfa22a9  Facebin/vgg-clean/clea_duvall/00000501.jpg
398d44f0b32d1b4855e838754c2c49fc  Facebin/vgg-clean/Bob_Barker/00000445.jpg
c2090f9412b93d71fed884bb45b26518  Facebin/vgg-clean/Adam_Goldberg/00000432.jpg
cd6971f809a534ab02ee1b03eb6c1183  CHECK Uploads/Google Photos/2017/06/IMG_1461.JPG
1fb80cc9ef622410557cb42c4abf26a8  Facebin/vgg-clean/Angell_Conwell/00000510.jpg
160793fd8f24fcf27efb9a2b3698a9c8  Facebin/makimface-artifacts/dataset-images/user-test-v4/00056--/img-5c1fdc5d4fd82c1e12a7d49937d7f47b.png
bcddf6ebbc1eed83950d908c2ccbac4a  Facebin/vgg-clean/danny_pino/00000102.jpg
eb2746a9559e7a93fd0002f4bfd90517  Facebin/makimface-artifacts/dataset-images/user-test/00009--/ds-767cd16bfdf4e31860d597708a586979.png
9605d69aaf5fb83d309f10f9e2544630  Facebin/vgg-clean/Adam_Beach/00000953.jpg
68e6d3ec3cca8a728a78fb60c9a1ddba  Facebin/vgg-clean/corey_stoll/00000137.jpg

The first 32 characters are the MD5 hash of the file, and the rest of the line is the file path.

You’ll probably see some blanks for the MD5 hashes of certain files. It’s important to remove these from the list, as these are your Google Docs, spreadsheets, etc.

$ grep '[^0-9a-f]' $HOME/Google-Drive-md5-$(date +%F).txt > $HOME/Google-Drive-md5-cleaned.txt

Next, we sort the file using:

$ sort $HOME/Google-Drive-md5-cleaned.txt > $HOME/Google-Drive-md5-sorted.txt

We keep all intermediate files because you might want to review the differences after the cleanup.

Then, we write the following script and run it:

#!/bin/zsh

PREV_MD5=""
PREV_PATH=""
CURRENT_MD5=""
CURRENT_PATH=""
MD5_FILE=$HOME/Google-Drive-md5-sorted.txt
cat $MD5_FILE | while read current_line ; do
    # echo $current_line
    CURRENT_MD5=$(echo "$current_line" | cut -c -32)
    CURRENT_PATH=$(echo "$current_line" | cut -c 35-)
    # echo $CURRENT_MD5
    # echo "$CURRENT_PATH"
    if [[ "$CURRENT_MD5" == "$PREV_MD5" ]] ; then
        echo "EQUAL: $CURRENT_MD5 $PREV_MD5"
        echo "DELETE: drive:/$CURRENT_PATH"
        rclone -v delete "drive:/$CURRENT_PATH"
    else
        PREV_MD5=$CURRENT_MD5
        PREV_PATH=$CURRENT_PATH
    fi
done

You can save this script as google-drive-delete-duplicates.sh and run:

$ chmod +x google-drive-delete-duplicates.sh
$ ./google-drive-delete-duplicates.sh

The script checks each line one by one; if two duplicate MD5 hashes are found consecutively, the second file is deleted. It keeps only one of the duplicates, even if there are more than two copies.

I have gained about 200GB by running this script.