Rust has fearless concurrency for memory access, but for the file system, there seem to be no built-in locked access primitives.

I decided to write one. Parallel execution of file system operations is important for Xvc. The error messages are annoying; having identical files in a repository is common, and in those cases, these messages look like there is a problem. In theory, when files are identical, having only one of them written to the cache is not a problem, but there may be other issues preventing cache access. We could just swallow the error and get away with it, but that’s not ideal.

Two options came to mind. One is modifying the list of cache files before creating threads so that no two threads access the same cache file at the same time. This is hard to implement and brings extra complexity to thread creation. A one-in-a-thousand concern becomes an architectural burden.

The other solution is to lock paths while accessing them so two threads working on the same cache path wait for each other. This is easier and requires just dependency injection into the thread functions. It has the downside of making file system operations slightly slower, as each path operation will now require checking a mutex, but that seems of little concern for file system access, which is already much slower than memory operations.

First, I tried to implement this with a HashMap<PathBuf, Mutex<()>>, but this must also be passed to the function wrapped in Arc<Mutex<HashMap>>, which made the “ceremony” of acquiring the lock for a single file much longer.

The issue is that you don’t want the HashMap itself to be a bottleneck. Multiple threads shouldn’t wait for the HashMap to become available, as it’s not the HashMap we want to lock, but the values inside it.

The solution is to return the lock value from a method of a struct. Something like:

pub struct PathSync {
    locks: Arc<RwLock<HashMap<PathBuf, Arc<Mutex<()>>>>>,
}

Now the ceremony can be performed within the method, and threads working in different directories won’t need to wait for the hash map to become available.

However, after implementing this, I realized I’d probably forget to lock a path at some point. This is a general-purpose solution, and I should apply it to all path operations when multiple threads are working. In most cases, there are multiple paths to lock (cache_path, cache_dir, repository path), and if I forget to lock one of them, a future user, some time, somewhere, will probably see an error message.

So, I decided on a different approach: creating wrappers to run passed closures. This makes it much more obvious that paths Xvc works on must be locked before operations.

    pub fn with_sync_path(
        &self,
        path: &Path,
        mut f: impl FnMut(&Path) -> Result<()>,
    )

This works by passing the path and a closure that operates on that path. It first locks the path and then runs the closure. The locking mechanism allows threads with different paths to run in parallel, but if they try to operate on the same path, they will wait for each other.

The implementation is here.

My current project on word spotting in historical documents is fairly complete in functionality, but I decided that searching for word images on page images concurrently would be better for speeding it up. I’m already using Boost for much of the functionality, and instead of creating a dependency on the not-yet-mature C++11 support in various compilers, I decided to use boost::thread.

Suppose we have a functor such as:

class Search_t
{
   public:
       Search_t(Document d) { ... };
       SearchResult operator()(SearchItem i) { ... };
};

And we want to use this functor in multiple threads. We can’t simply do the following:

std::vector<SearchResult> results;
Search_t search(document);
// search_items is a vector<SearchItem> and si is an iterator over it.
for (si = search_items.begin(); si != search_items.end(); ++si)
{
   boost::thread task(boost::bind(search, *si));
   results.push_back(task); // ERROR!
}

This is because task does not return a SearchResult.

Instead, we need to store results within the object and retrieve them after they are generated.

I didn’t want to change the interface of Search_t because multithreading should be optional, and other parts of the program may depend on this interface. Instead, a wrapper class that runs these threads with a similar interface seemed like a better solution.

class SearchMT_t
{
   boost::shared_ptr<std::vector<boost::unique_future<SearchResult> > > futures_;

   public:
   SearchMT_t(Document d) :
   /* The most important assumption here is that Search_t does not alter
      the Document object's state in any way. Otherwise, we need to ensure that 
      document_ is accessed by only a single thread at a time using mutexes. */
   document_(d),
   futures_(new std::vector<boost::unique_future<SearchResult> >)
   {};

   void operator()(SearchItem si)
   {
       /* If you are sure that there won't be any race conditions
       between Search_t threads during search, you can move the following line
       to the constructor and use a single object for all searches. */
       Search_t search(document_);

       boost::packaged_task<SearchResult> search_task(std::bind(search, si));
       futures_->push_back(search_task.get_future());
       boost::thread task(boost::move(search_task));
   };

   std::vector<SearchResult> results()
   {
      std::vector<SearchResult> results;

      /* Wait for all threads to complete their work. */
      boost::wait_for_all(futures_->begin(), futures_->end());

      for(int i = 0; i < futures_->size(); ++i)
      {
          results.push_back((*futures_)[i].get());
      }

      return results;
   }
};

This way, it becomes much more straightforward to use multithreading in a loop:

std::vector<SearchResult> results;
SearchMT_t search(document);
// search_items is a vector<SearchItem> and si is an iterator over it.
for (si = search_items.begin(); si != search_items.end(); ++si)
{
    search(*si);
}

results = search.results();

We kept the Search_t class intact and used a much simpler approach in the loop.