While writing a HOWTO post for the documentation, I found a bug where multiple carry-in commands were causing file system failures. When multiple threads were accessing the same cache directory, if one of them tried to set up the cache directory while another was still working on it, it caused a permissions error.

Rust has fearless concurrency for memory access, but for the file system, there seem to be no built-in locked access primitives.

I decided to write one. Parallel execution of file system operations is important for Xvc. The error messages are annoying; having identical files in a repository is common, and in those cases, these messages look like there is a problem. In theory, when files are identical, having only one of them written to the cache is not a problem, but there may be other issues preventing cache access. We could just swallow the error and get away with it, but that’s not ideal.

Two options came to mind. One is modifying the list of cache files before creating threads so that no two threads access the same cache file at the same time. This is hard to implement and brings extra complexity to thread creation. A one-in-a-thousand concern becomes an architectural burden.

The other solution is to lock paths while accessing them so two threads working on the same cache path wait for each other. This is easier and requires just dependency injection into the thread functions. It has the downside of making file system operations slightly slower, as each path operation will now require checking a mutex, but that seems of little concern for file system access, which is already much slower than memory operations.

First, I tried to implement this with a HashMap<PathBuf, Mutex<()>>, but this must also be passed to the function wrapped in Arc<Mutex<HashMap>>, which made the “ceremony” of acquiring the lock for a single file much longer.

The issue is that you don’t want the HashMap itself to be a bottleneck. Multiple threads shouldn’t wait for the HashMap to become available, as it’s not the HashMap we want to lock, but the values inside it.

The solution is to return the lock value from a method of a struct. Something like:

pub struct PathSync {
    locks: Arc<RwLock<HashMap<PathBuf, Arc<Mutex<()>>>>>,
}

Now the ceremony can be performed within the method, and threads working in different directories won’t need to wait for the hash map to become available.

However, after implementing this, I realized I’d probably forget to lock a path at some point. This is a general-purpose solution, and I should apply it to all path operations when multiple threads are working. In most cases, there are multiple paths to lock (cache_path, cache_dir, repository path), and if I forget to lock one of them, a future user, some time, somewhere, will probably see an error message.

So, I decided on a different approach: creating wrappers to run passed closures. This makes it much more obvious that paths Xvc works on must be locked before operations.

    pub fn with_sync_path(
        &self,
        path: &Path,
        mut f: impl FnMut(&Path) -> Result<()>,
    )

This works by passing the path and a closure that operates on that path. It first locks the path and then runs the closure. The locking mechanism allows threads with different paths to run in parallel, but if they try to operate on the same path, they will wait for each other.

The implementation is here.