🐇 It looks, from the coverage page, that our additions to HStore don’t have any tests. We can add some unit tests to new joins, maybe.

🐢 I don’t find unit tests particularly useful, but let’s use GitHub Copilot to add tests for us.

🐇 Added a unit test and a doc test for full_join, and I think doc tests have more value. They provide documentation, and we can readily see how to use a function from its docs. Better to increase coverage with doc tests.

🐢 There are points that I should learn while writing doc tests. The imports must use the full path, not crate::. The tested struct also doesn’t have implicit imports.

🐇 The ceremony of adding keys and values is a bit too much. It may be worthwhile to add insert for any Into<XvcEntity>.

🐢 It will certainly save time if we don’t have to type .into() for each key :)

🐇 Pushed to test again. Should we have some means to test coverage locally?

🐢 I don’t think we need to consider coverage locally. It’s not worth our time.

🐇 Now while waiting for tests to be completed, what can we do?

gh -R iesahin/xvc run list
completed	failure	v0.6.13	Rust-CI	v0.6.13	pull_request	12543831360	3m54s	2024-12-30T08:15:06Z
...

🐢 I don’t think it took too much time. Let’s view the results:

gh -R iesahin/xvc run view 12543831360
...
  X Run Current Dev Tests
...
To see what failed, try: 
View this run on GitHub: https://github.com/iesahin/xvc/actions/runs/12543831360

🐢 The current dev tests fail for some reason. Let’s run these locally.

🐇 We’re missing llvm-tools-preview locally. How do we install this?

🐢 The command is:

rustup component add llvm-tools-preview
info: component 'llvm-tools' for target 'aarch64-apple-darwin' is up to date

🐇 It’s already installed. We need to set the environment variables.

🐢 Instead, we can just turn off dev tests for the time being. We don’t need them. Our local tests pass.

🐇 Yeah, ok, we don’t need to solve each and every bit of these issues.

gh -R iesahin/xvc run list

… 🐇 Ok. Let’s take a look at the run again.

gh -R iesahin/xvc run list
completed	failure	v0.6.13	Rust-CI	v0.6.13	pull_request	12544020064	3m39s	2024-12-30T08:33:25Z

gh -R iesahin/xvc run view 12544020064
...
  X Test and Coverage
...

gh run view 12544020064 --log-failed
...
Test and Coverage (stable)	Test and Coverage	2024-12-30T08:36:58.8911690Z Error: ProcessError { stdout: "", stderr: "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to e1.xvc.dev port 80 after 160 ms: Couldn't connect to server\n" }
...

🐢 We need to start nginx on the server. We forgot it yesterday.

🐇 Ah, yeah. After adding that dufs installation. Ok.

🐢 We also need to add a reverse proxy to dufs somehow, but this is for later.

🐇 For the use case, I don’t think it’s necessary. We can just adjust the port to a non-standard one if we need to use 443 for another thing, but let’s take a look at the tests again.

🐢 Let’s add another doc test. This time to XvcStore.

cargo test -p xvc-ecs --doc
...
test result: ok. 8 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.87s

🐇 Sent the files again. Waiting for tests to finish.

🐢 Let’s check the keymaps file in the meantime.

🐇 I tried to read some documentation but didn’t see an error. Maybe we should just set it to non-lazy and avoid allowing remaps.

🐢 We already spent too much time with this.

🐇 Yes, let’s take a look at the tests again.

gh -R iesahin/xvc run list
completed	success	v0.6.13	Rust-CI	v0.6.13	pull_request	12544291068	8m10s	2024-12-30T09:00:42Z
...

🐢 Oh, yeah, the merge is ready.

🐇 Patch coverage is still behind the target, though.

🐢 Yeah, but let’s release this one and make the next better covered. Also, I’m not sure if the doc tests had any effect on coverage.

🐇 If we look at the coverage page again, we can see if the doc tests had any effect.

🐢 It seems codecov.io doesn’t consider coverage for doc tests. This is a bit weird, but let’s not spend more time on this.

🐇 Sure, let’s merge.

🐢 I think we forgot to bump the version in Cargo.toml. We’ll have to do that in main.

🐇 Oh, yeah. Let’s bump it and tag as well.

🐢 Now, we can wait for all files to be produced. What will we do next?

🐇 We can just release the Python version as well. It shouldn’t need any changes.

🐢 Umm, right. Maybe we can add a few tests as well.

🐇 Let’s bump the version first and see.

🐢 Bumped versions in Cargo.toml and ran maturin develop.

🐇 It seems ready now.

🐢 The main fails, though. The “Publish Crates” action looks for libsqlite3. Let’s take a look at it.

gh -R iesahin/xvc run list
completed	failure	Release v0.6.13	Publish Crates	v0.6.13	push	12544753359	6m1s	2024-12-30T09:41:56Z

🐢 The issue is that the VM doesn’t have libsqlite3-dev. Let’s add it.

🐇 We need to start the job manually again. Let’s not tag this time.

🐢 Some of the packages were already published. Now they break. crates.io says they’re already published. Maybe we can check if a package is already published.

🐇 Let’s check if we can make cargo publish more forgiving.

🐢 There doesn’t seem to be an option. Let’s search “how to skip published packages in the workspace to avoid errors with cargo publish”.

🐇 Claude is bullshitting again. Let’s try a manual approach: how to skip already published packages.

🐢 It may be easier to just add a check if the package is published. How do we get the info?

🐇 Or we can just go on to the next package if the package is already available.

🐢 Let’s do this manually this time.

Can't locate IPC/Cmd.pm in @INC (@INC contains: /home/runner/work/xvc.py/xvc.py/target/x86_64-unknown-linux-gnu/release/build/openssl-sys-844a96d66ae533b1/out/openssl-build/build/src/util/perl /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /home/runner/work/xvc.py/xvc.py/target/x86_64-unknown-linux-gnu/release/build/openssl-sys-844a96d66ae533b1/out/openssl-build/build/src/external/perl/Text-Template-1.56/lib)

The issue seemed to be a missing Perl package. Building rust-openssl now appears to require the perl-core package.

I added apt-get update && apt-get install perl-core to the CI configuration, but that didn’t work.

However, as this GitHub issue comment suggests, Maturin with its manylinux support uses different kinds of Docker containers to build the packages. We must detect whether it is a CentOS or Debian-based container and install the missing packages accordingly.

The Building wheels step in the configuration should look similar to:

- name: Build wheels
  uses: PyO3/maturin-action@v1
  with:
    target: ${{ matrix.target }}
    manylinux: auto
    args: --release --out dist
    before-script-linux: |
      # If we're running on RHEL/CentOS, install needed packages.
      if command -v yum &> /dev/null; then
          yum update -y && yum install -y perl-core openssl openssl-devel pkgconfig libatomic

          # If we're running on i686, we need to symlink libatomic
          # in order to build openssl with the -latomic flag.
          if [[ ! -d "/usr/lib64" ]]; then
              ln -s /usr/lib/libatomic.so.1 /usr/lib/libatomic.so
          fi
      else
          # If we're running on a Debian-based system.
          apt update -y && apt-get install -y libssl-dev openssl pkg-config
      fi

This will run the specified script before executing the Maturin build command in the container, ensuring all missing packages are installed.

The Cargo.toml file can have a [features] section. In this section, features can be defined, and they can enable other features.

[features]
my-feature = []
another-feature = ["my-feature"]

These markers are used in the code as follows:

#[cfg(feature = "my-feature")]
pub mod my_module;

The default features are listed under the default key of the [features] section. If it’s not defined, it’s considered empty, so cargo build supplies no features to the build system by default.

If you don’t want default features added automatically, you can use the --no-default-features option with cargo.

Starting from Rust 1.60, dependencies can be tied to features. When you define a dependency in the [dependencies] section, add optional = true to its options. Then, in [features], use the dep:package syntax.

[dependencies]

pack = { version = "1.0", optional = true }

[features]

my-feature = ["dep:pack"]

This way, Cargo doesn’t compile pack if my-feature is not enabled.

A caveat to using features is that they should be additive. There shouldn’t be a no-std feature to remove the standard library dependency; instead, you should define a std feature and enable it by default. The reason behind this is that Cargo ensures there is a single copy of a package with all the requested features enabled for the entire dependency graph. If you have a no-x feature that removes functionality, it breaks the additive logic of Cargo’s feature resolution.

• ends_with in Path is different from ends_with in str: If you try to check whether a Path is a directory by its final character, you’ll find that path.ends_with("/") returns false.

• Path::strip_prefix also consumes the final slash: If you have a directory marker at the end, such as /Users/emre/mydir/, and pass it to strip_prefix("/Users/emre"), you’ll get mydir instead of mydir/.

In this post, I’ll update it. Some of the requirements have changed, and I need new features:

• walk_serial currently takes a Sender and uses channels to send the results. This is confusing, as it doesn’t use any parallelism to traverse the file system.

• There are long-running processes that modify the file system, which can affect behavior. We need to be notified about file system changes while using identical ignore rules.

For the first change, I’ll convert the signature of walk_serial to:

pub fn walk_serial(
    given: IgnoreRules,
    dir: &Path,
    walk_options: &WalkOptions,
    res_paths: &mut Vec<Result<PathMetadata>>,
) -> Result<IgnoreRules>

In the first version, instead of res_paths, we had a sender through which the results were sent.

Why don’t I simply return the paths instead of receiving a mut Vec parameter to fill? While it might be simpler, I don’t like the idea of creating a new Vec for each function call. The overhead of creating new vectors and merging them can become a burden for large directory trees. Instead, each call of the function works on the same vector and expands its capacity when necessary. Since the current Vec strategy is to double the capacity when needed, there are fewer memory allocations for complex and large directory hierarchies.

Other than this, instead of Sender::send, the function uses Vec::push to add elements to the vector. The rest of the implementation is similar.

One major difference in this version is that the function returns an IgnoreRules value instead of (). This contains the compiled ignore rules from all directories below the given directory. We can use this returned value to further check the paths we get from the notifier we’ll develop.

Adding a file system notifier

For the second change, we need to access the underlying OS machinery for file system notifications. My research led me to the notify crate. Although it seems to be undergoing a major overhaul as of this writing, it’s the only promising option other than interfacing directly with the OS.

Again, we begin by defining the enums and structs for this problem.

What do we want? We want to collect file system events to update the Vec<PathMetadata> that was built with walk_serial or walk_parallel. PathMetadata was defined as:

pub struct PathMetadata {
    pub path: PathBuf,
    pub metadata: Metadata,
}

We could build a HashMap<PathBuf, Metadata> or keep this as a vector, but for our purposes, we want to track new events. These events correspond to creating a new file, updating metadata (size, timestamp), or deleting a file. We don’t need, for example, permission changes, as we primarily want to track which file is newer or whose size has changed.

pub enum PathEvent {
    Create { path: PathBuf, metadata: Metadata },
    Update { path: PathBuf, metadata: Metadata },
    Delete { path: PathBuf },
}

notify provides an abstraction over different operating systems. It’s called RecommendedWatcher and is used to provide a consistent interface for events. A watcher is built by implementing the EventHandler trait.

Since we need to check whether paths are ignored before reporting them, the watcher should receive an IgnoreRules struct.

pub fn make_watcher(
    ignore_rules: IgnoreRules,
) -> Result<(RecommendedWatcher, Receiver<PathEvent>)> {

It returns a RecommendedWatcher and a (crossbeam) Receiver for the PathEvent we defined above. The implementation creates a channel, initializes the watcher, and starts watching.

    let (sender, receiver) = bounded(10000);
    let root = ignore_rules.root.clone();
    let mut watcher = notify::recommended_watcher(PathEventHandler {
        ignore_rules,
        sender,
    })?;
    watcher.watch(&root, RecursiveMode::Recursive)?;

PathEventHandler is the struct that implements the EventHandler trait. The implementation requires a single function to be implemented:

    fn handle_event(&mut self, event: notify::Result<Event>)

notify::Event is a hierarchical set of enums that describe the type of file system event. The event handler is expected to discriminate between these events and react accordingly.

pub struct Event {
    pub kind: EventKind,
    pub paths: Vec<PathBuf>,
    pub attrs: EventAttributes,
}

We’re interested in filtering events by EventKind, checking whether the paths are ignored, and sending the path and metadata through a channel. We have three kinds of events in PathEvent. PathEventHandler implements event filtering and calls handlers for these PathEvents.

The struct has functions that correspond to one or more PathEvent elements. These are create_event, write_event, remove_event, and rename_event. The first three correspond directly to members of PathEvent, and the latter sends both a PathEvent::Delete and a PathEvent::Create.

match event.kind {
    notify::EventKind::Create(_) => self.create_event(event.paths[0].clone()),
    notify::EventKind::Modify(mk) => match mk {
        notify::event::ModifyKind::Any => todo!(),
        notify::event::ModifyKind::Data(_) => self.write_event(event.paths[0].clone()),
        notify::event::ModifyKind::Metadata(_) => {
            self.write_event(event.paths[0].clone())
        }
        notify::event::ModifyKind::Name(rk) => match rk {
            notify::event::RenameMode::Any => {}
            notify::event::RenameMode::To => self.create_event(event.paths[0].clone()),
            notify::event::RenameMode::From => {
                self.remove_event(event.paths[0].clone())
            }
            notify::event::RenameMode::Both => {
                self.rename_event(event.paths[0].clone(), event.paths[1].clone())
            }
            notify::event::RenameMode::Other => {}
        },
        notify::event::ModifyKind::Other => {}
    },
    notify::EventKind::Remove(_) => self.remove_event(event.paths[0].clone()),
    notify::EventKind::Any => {}
    notify::EventKind::Access(_) => {}
    notify::EventKind::Other => {}
}

An example event handler should clarify how the others work:

fn write_event(&mut self, path: PathBuf) {
    match check_ignore(&self.ignore_rules, &path) {
        MatchResult::Whitelist | MatchResult::NoMatch => {
            self.sender
                .send(PathEvent::Create {
                    path: path.clone(),
                    metadata: path.metadata().map_err(Error::from).unwrap(),
                })
                .unwrap_or_else(|e| warn!("{}", e));
        }
        MatchResult::Ignore => {
            debug!("FS Notification Ignored: {}", path.to_string_lossy());
        }
    }
}

Other event handlers work similarly. They send PathEvent values that can be used to update a list of paths and their metadata.

To use this watcher, you first retrieve the list of files with walk_serial. It returns a snapshot of the directory and the IgnoreRules collected from the directories below it. Then, you create a watcher with:

let ignore_rules = walk_serial(...)?;
let (watcher, path_event_rec) = make_watcher(ignore_rules)?;

After this, you can create another thread to monitor file system changes:

crossbeam::scope(|s| {
    s.spawn(|_| {
        while let Ok(path_event) = path_event_rec.recv() {
            match path_event {
                PathEvent::Create { .. } => { /* ... */ }
                PathEvent::Delete { .. } => { /* ... */ }
                PathEvent::Update { .. } => { /* ... */ }
            }
        }
    });
});

Recently, I needed a standard way to get the temporary directory. I checked the crate documentation but couldn’t find a proper function for this.

Then I noticed that std::env::temp_dir() returns a PathBuf pointing to the system’s temporary directory. This is a reminder to myself that we don’t always need extra packages for basic functionality.

Another point is that Rust, being a statically typed language, requires far fewer unit tests than dynamic languages. Traditionally, unit tests are used to validate “input consumption” (ensuring input is processed correctly) and “well-formed output” (verifying that the output meets requirements under specific conditions). In Rust, many of these input-output concerns are resolved at compile time through the use of expressive types like structs and enums.

For these reasons, I find myself more inclined toward integration tests that reflect real-world requirements, rather than focusing solely on unit tests in artificial settings.

I was using burntsushi’s ignore crate, which in turn uses globset by the same author. In a directory hierarchy, I was first collecting all .ignore files, then trying to decide which files are ignored. Ripgrep’s from parameter led me to believe this.

However, it looks like it has a bug that doesn’t consider the source directory of the ignore file. Git has different semantics when a/.gitignore and a/b/.gitignore have a line mydir/. In the second case, Git doesn’t ignore a/mydir/, but Ripgrep seems to consider all elements from all files as if they are from the root directory.

In practice, most people may be using ignore files in their root directory, but there are valid use cases (like moving around directories with ignore patterns in them) that are affected by such behavior.

I decided to tackle this. Ripgrep’s globset seems nice and fast, so it was better to base the solution on top of that. My basic use case is walking a directory hierarchy. There are solutions to this like walkdir or jwalk, but they use ripgrep’s gitignore; hence, they have the same bug.

I usually begin by setting up data structures to understand the problem. This allows me to understand the domain formally.

First, what are we trying to accomplish? Basically, we want to decide if a path should be ignored (a) in the results and (b) in the directory traversal. The gitignore specification not only allows ignores, but also has a way to add paths to a whitelist to unignore. Hence, deciding whether to ignore a path is like:

let include = |path| {
    if whitelist_globs.match(path) {
       true
    } else if ignore_globs.match(path) {
       false
    } else {
       true }
}

We also need to decide whether to traverse a directory:

let traverse = |path| {
    if path.is_dir() && !include(path) { false } else { true }
}

We have the following possible MatchResults between the ignore rules and a path:

pub enum MatchResult {
    NoMatch,
    Ignore,
    Whitelist,
}

We need a comparison function to get a MatchResult. One side of the comparison is path, and the other is a set of GlobSets from the globset crate. We’ll have two globsets: one for ignores and the other for whitelists.

pub struct IgnoreRules {
    root: PathBuf,

    patterns: Arc<Vec<Pattern<Glob>>>,

    whitelist_set: Arc<GlobSet>,
    ignore_set: Arc<GlobSet>,
}

The data types for the sets are Arc<GlobSet>, as I planned to make the traversal concurrent for each directory. However, as only one thread will be modifying the rules (one .gitignore or .ignore file per directory), I felt no need to wrap the sets with Mutex.

We also keep track of each pattern separately. patterns is a Pattern vector that contains all globs individually. One reason is to rebuild the GlobSet as we traverse directories: since we cannot add any more rules to the sets once they have been built, we keep the patterns separately to add them to the sets of the child directories.

Another reason is to keep track of the source, as this can be used for debugging or reporting, similar to git check-ignore.

root is the point where this traversal began.

Patterns are defined in a generic way to be instantiated with globs, strings, regexes, or any other type.

pub struct Pattern<T> {
    pattern: T,
    original: String,
    source: Source,
    effect: PatternEffect,
    relativity: PatternRelativity,
    path_kind: PathKind,
}

The rationale for this generic approach is that the patterns need some kind of transformation. They can be read from files as strings, then compiled into globs or regexes.

flowchart LR
     String-->Ok(Glob)-->Glob-->Regex

The source field shows where this pattern was retrieved. It’s an enum with two possible values for the time being:

enum Source {
    File { path: PathBuf, line: usize },
    Global,
}

The gitignore specification says:

An optional prefix “!” which negates the pattern; any matching file excluded by a previous pattern will become included again.

We use the effect field to tag the pattern as either an Ignore or a Whitelist pattern.

pub enum PatternEffect {
    Ignore,
    Whitelist,
}

The spec also makes a distinction between files and directories:

If there is a separator at the end of the pattern then the pattern will only match directories, otherwise the pattern can match both files and directories.

So we use path_kind to reflect this distinction:

pub enum PathKind {
    Any,
    Directory,
}

Some patterns can be relative to the directory they are found in:

If there is a separator at the beginning or middle (or both) of the pattern, then the pattern is relative to the directory level of the particular .gitignore file itself. Otherwise the pattern may also match at any level below the .gitignore level.

And we show this with the relativity field:

pub enum PatternRelativity {
    Anywhere,
    RelativeTo { directory: String },
}

Note that we are converting from the specification to Rust types. In the future, these enums can be extended to cover cases where these patterns are used in other places, or we can merge some of them to reduce complexity. Currently, this level of abstraction seems adequate.

To convert Pattern<String> to Pattern<Glob> (or Pattern<Result<Glob, Error>>), or any other type, we can use a map function similar to Option::map:

    fn map<U, F>(self, f: F) -> Pattern<U>
    where
        F: FnOnce(T) -> U,
    {
        let pat = Pattern::<U> {
            pattern: f(self.pattern),
            original: self.original,
            source: self.source,
            effect: self.effect,
            relativity: self.relativity,
            path_kind: self.path_kind,
        };
        pat
    }

map will make it easy to convert between various types of patterns.

Now, we change the perspective to a higher level. Let’s begin by writing a file tree walker.

pub fn walk_serial(
    given: IgnoreRules,
    dir: &Path,
    walk_options: &WalkOptions,
    sender: &Sender<WalkerResult<PathMetadata>>,
) -> WalkerResult<()> { todo!() }

This function uses the given IgnoreRules and returns files and directories via the channel. PathMetadata is just a bundle of PathBuf and its Metadata.

pub struct PathMetadata {
    path: PathBuf,
    metadata: Metadata,
}

WalkOptions will contain various options about this traversal. Currently, it only has ignore_filename and include_dirs, which set whether only files are sent through the channel, or if directories should be included too.

pub struct WalkOptions<'a> {
    ignore_filename: Option<&'a str>,
    include_dirs: bool,
}

Initially, IgnoreRules contains only the files that should be ignored (or whitelisted) globally. When a directory contains a .gitignore file (or any file name that was set in WalkOptions), it checks for new ignore rules and sends all files through the channel. If a directory is not ignored, it then calls walk_serial recursively with it.

The final lines of the function are as follows:

    for child_dir in child_dirs {
        walk_serial(dir_with_ignores.clone(), &child_dir, walk_options, sender)?;
    }

As it calls itself as a final operation after collecting and sending all child files, this is an example of tail recursion.

The output is sent through a channel because our final goal is to make all this concurrent. Each new directory will create a new thread, and all files will be collected from the channel. At this point, we are creating a serial version of the walker with identical data structures to test it easily.

One gotcha using the channels in the serial version is that the channel size must be enough to contain all file names. Effectively, this means it will be crossbeam_channel::unbounded for walk_serial. We will (ab)use the channel’s Sender<PathMetadata> as a Vec<PathMetadata> in no parallel mode. In parallel settings, using unbounded channels is not recommended; if the consumers of the channel don’t work as quickly, you may fill up the memory, especially for longer-running processes.

walk_serial will basically do four things:

• List all the elements in the given directory.
• Check if there is a new ignore file and update the rules if necessary.
• Filter the elements by ignore rules and send them to the channel.
• Run walk_serial recursively for child directories if they are not ignored.

Read the directory

The first is done using read_dir of Path. It returns a set of elements. Then the list is processed, and elements’ metadata is retrieved. This requires a bit of error handling: read_dir may return an Error, each element in the result can be an Error, and getting metadata may also result in an Error.

    let elements = dir
        .read_dir()
        .map_err(|e| anyhow!("Error reading directory: {:?}, {:?}", dir, e))?;
    let mut child_dirs = Vec::<PathBuf>::new();
    let mut child_paths = Vec::<PathMetadata>::new();

    for entry in elements {
        match entry {
            Err(err) => sender.send(Err(WalkerError::from(anyhow!(
                "Error reading entry in dir {:?} {:?}",
                dir,
                err
            ))))?,
            Ok(entry) => match entry.metadata() {
                Err(err) => sender.send(Err(WalkerError::from(anyhow!(
                    "Error getting metadata {:?} {}",
                    entry,
                    err
                ))))?,
                Ok(md) => {
                    child_paths.push(PathMetadata {
                        path: entry.path(),
                        metadata: md.clone(),
                    });
                    if md.is_dir() {
                        child_dirs.push(entry.path());
                    }
                }
            },
        }
    }

Note that we collect PathMetadata objects in the child_paths vector. If there happens to be an error while retrieving, it’s sent to the caller to ignore or to report to the user.

I usually handle errors first in match expressions because they are usually more straightforward.

Check if there are new ignore rules

After reading all the files in the directory, it checks if there actually is an ignore filename in given and that a file exists with that name in the list we just created. If there is no such file, we can use given rules to decide the ignored elements.

If there is a file that may change the ignore rules, it reads it and updates patterns. Then, if there are new ignore or whitelist rules, the respective globsets are recreated. It checks whether we have new rules because compiling globs is possibly an expensive operation.

Filtering directory elements

Filtering files and directories from the list checks two globsets. If an element matches a whitelist rule, it’s included in the results without checking if it also matches ignore rules. Hence, whitelisting has a priority. Otherwise, it checks the ignore rules, and if they don’t match, the element is included in the results.

If there are directories matching the ignore rules, they are not traversed, and any possible whitelisting rule that was specified within that directory is not taken into consideration.

Listing child directories

We saw how walk_serial calls itself for child directories above. After sending all results to the channel, it calls itself for the child directories and repeats.

Converting patterns from String to Glob

The interesting part about ignoring files is that we should modify patterns into globs to add to a GlobSet. We cannot add the patterns directly to a GlobSet because some of the rules make them context dependent. We parsed the patterns to get PatternRelativity and PathKind, and these also affect how we add the patterns to a GlobSet.

The heart of the conversion is a function that receives a String, filters it, and transforms it to a list of Pattern<Glob>s.

fn content_to_patterns(
    ignore_root: &Path,
    source: Option<&Path>,
    contents: &str,
) -> Vec<Pattern<WalkerResult<Glob>>> {

As we also need to report the line number for patterns, we use enumerate on the iterator:

    let patterns: Vec<Pattern<WalkerResult<Glob>>> = content
        .lines()
        .enumerate()
        // A line starting with # serves as a comment. Put a backslash ("\") in front of the first hash for patterns that begin with a hash.
        .filter(|(_, line)| !(line.trim().is_empty() || line.starts_with("#")))

The filter line is used to check for blank and comment lines. Then we trim the trailing space unless the pattern ends with \.

        .map(|(i, line)| {
            if !line.ends_with("\\ ") {
                (i, line.trim_end())
            } else {
                (i, line)
            }
        })

The function receives the source file name as a parameter. We create the Source using it and the line number:

        .map(|(i, line)| {
            (
                line,
                match source {
                    Some(p) => Source::File {
                        path: p
                            .strip_prefix(ignore_root)
                            .expect("path must be within ignore_root")
                            .to_path_buf(),
                        line: (i + 1).into(),
                    },
                    None => Source::Global,
                },
            )
        })

Then we build the pattern object, update the pattern to add it before the globset, and build the globs. Building the glob may cause errors if it’s not well-formed. They are handled as well.

        .map(|(line, source)| build_pattern(source, line))
        .map(transform_pattern_for_glob)
        .map(|pc| pc.map(|s| Glob::new(&s).map_err(WalkerError::from)))

Two functions are left that may need attention. One is build_pattern that converts a pattern line to a Pattern<String> object. The other is the transform_pattern_for_glob function that modifies Pattern<String> before building the glob.

After writing walk_serial, we’ll use all the machinery to write a parallel version of it.

Parsing pattern strings for pattern objects

We create pattern objects from pattern strings following the spec. There are three basic rules that we will follow:

• If the pattern starts with !, it’s a whitelist rule.
• If the pattern ends with /, it matches only to directories.
• If the pattern contains a non-final /, it’s relative to the directory.

We’ll also check whether the line starts with / and remove that to prevent double-slashes // in patterns. The resulting pattern in the object won’t contain any of the artifacts that change its semantics, and we’ll add them when building the globsets.

Let’s start by checking whether the line starts with !.

    let begin_exclamation = original.starts_with("!");
    let line = if begin_exclamation || original.starts_with("\\!") {
        original[1..].to_owned()
    } else {
        original.to_owned()
    };

Checking the presence of / at the beginning, at the end, and in intermediate positions is also straightforward:

    let end_slash = line.ends_with("/");
    let line = if end_slash {
        &line[..line.len() - 1]
    } else {
        line
    };

    let begin_slash = line.starts_with("/");
    let non_final_slash = if line.len() > 0 {
        line[..line.len() - 1].chars().find(|c| *c == '/').is_some()
    } else {
        false
    };

We check the non-final slash separately from the beginning slash because we’ll remove the initial slash if it exists.

    let line = if begin_slash { &line[1..] } else { line };

Although there is almost 1-1 correspondence between the boolean conditions and the enums, I create enums in separate statements for self-documenting code.

    let effect = if begin_exclamation {
        PatternEffect::Whitelist
    } else {
        PatternEffect::Ignore
    };

    let path_kind = if end_slash {
        PathKind::Directory
    } else {
        PathKind::Any
    };

    let relativity = if non_final_slash {
        PatternRelativity::RelativeTo {
            directory: current_dir.to_owned(),
        }
    } else {
        PatternRelativity::Anywhere
    };

After building these attributes of the pattern and stripping the pattern from the initial and final /, and other semantic-changing elements, we build a pattern:

    let pattern = Pattern::<String> {
        pattern: line.to_owned(),
        original: original.to_owned(),
        source,
        effect,
        relativity,
        path_kind,
    };

Transforming a pattern string to a suitable glob

Now, another transformation is needed to convert Pattern<String> to Pattern<Glob>. In this case, we’ll need to recreate the pattern based on the rules.

The transformation considers relativity and path_kind to generate a glob that can be added to a GlobSet. Note that the function returns not a compiled Pattern<Glob>, but a Pattern<String>, because after this transformation, building the glob requires Glob::new.

fn transform_pattern_for_glob(pattern: Pattern<String>) -> Pattern<String> {
    let anything_anywhere = |p| format!("**/{p}");
    let anything_relative = |p, directory| format!("{directory}/**/{p}");
    let directory_anywhere = |p| format!("**{p}/");
    let directory_relative = |p, directory| format!("{directory}/**/{p}/");

    let transformed_pattern = match (&pattern.path_kind, &pattern.relativity) {
        (PathKind::Any, PatternRelativity::Anywhere) => anything_anywhere(pattern.pattern),
        (PathKind::Any, PatternRelativity::RelativeTo { directory }) => {
            anything_relative(pattern.pattern, directory)
        }
        (PathKind::Directory, PatternRelativity::Anywhere) => directory_anywhere(pattern.pattern),
        (PathKind::Directory, PatternRelativity::RelativeTo { directory }) => {
            directory_relative(pattern.pattern, directory)
        }
    };

    Pattern {
        pattern: transformed_pattern,
        ..pattern
    }
}

I could merge the closures into the match and lower the number of lines, but adding the closures increased self-documentation. Instead of multiple if/else branches, I tend to use match with tuples to have a clearer picture. The corresponding if/else code without closures would be like:

let transformed_pattern = if pattern.path_kind == PathKind::Any {
    if pattern.relativity == PatternRelativity::Anywhere {
        format!("**/{p}")
        } else if ...
        }
...

And in my opinion, this latter format is more error-prone and less readable.

Combining it all together

At this point, we can revisit content_to_patterns again:

fn content_to_patterns(
    ignore_root: &Path,
    source: Option<&Path>,
    content: &str,
) -> Vec<Pattern<Result<Glob>>> {
    watch!(source);
    let patterns: Vec<Pattern<Result<Glob>>> = content
        .lines()
        .enumerate()
        ...
        .map(|(line, source)| build_pattern(source, line))
        .map(transform_pattern_for_glob)
        .map(|pc| pc.map(|s|
            Glob::new(&s).map_err(Error::from)))
        .collect();

    patterns
}

The last step in the transformation is creating the Glob object, as in the highlighted lines above. It runs Pattern<String>.map with Glob::new to get a Pattern<Result<Glob>>. Error and Result are custom types defined within the crate using the thiserror crate.

After we have a vector of Pattern<Result<Glob>>, we report the errors in clear_glob_errors to get Vec<Pattern<Glob>> from this vector. The function is defined as:

fn clear_glob_errors(
    sender: &Sender<Result<PathMetadata>>,
    new_patterns: Vec<Pattern<Result<Glob>>>,
) -> Vec<Pattern<Glob>> {
    let new_glob_patterns: Vec<Pattern<Glob>> = new_patterns
        .into_iter()
        .filter_map(|p| match p.transpose() {
            Ok(p) => Some(p),
            Err(e) => {
                sender
                    .send(Err(Error::from(anyhow!("Error in glob pattern: {:?}", e))))
                    .expect("Error in channel");
                None
            }
        })
        .collect();
    new_glob_patterns
}

The sender is the same channel that we use to send the results. This communication channel is used to report only the errors, not PathMetadata; hence, when we encounter an Err(e) in new_patterns, it’s reported to the same channel that the results are expected. Handling the errors first seems like a common theme here as well.

This is also one of the reasons walk_serial receives a channel instead of a vector. Our usual use case is traversing all the directories in parallel, and without channel-based communication in the serial version, we would need to duplicate this function.

At this point, I completed the walk_serial functionality. I wrote tests. I’m using the test-case crate for writing many tests encompassing different use cases. I’m omitting them here for brevity.

Writing a Parallel Walker

Converting walk_serial to a parallel one is quite trivial. As we already used channels in the serial version, we have all the components of a parallel walker. For the sake of simplicity, I won’t use a thread pool and instead spawn a new thread for each task. One question is what’s the unit task that we’ll use to create tasks.

jwalk uses directories as points to spawn new tasks. That’s a sensible approach, and I’ll use this as well. This means a single directory will be processed by a single thread, even if it contains one million files, and 10 directories will be processed by 10 threads even if each contains one file. In practice, I think we can assume directories are natural chunks for this problem.

The interface for the parallel walker is identical to the serial version. This is to replace one with the other when the need arises.¹

The only difference is in the final loop. walk_serial calls itself recursively at the end, while walk_parallel calls itself in separate threads.

pub fn walk_parallel(
    given: IgnoreRules,
    dir: &Path,
    walk_options: WalkOptions,
    sender: Sender<Result<PathMetadata>>,
) -> Result<()> {
    ....
    crossbeam::scope(|s| {
        for child_dir in child_dirs {
            let dwi = dir_with_ignores.clone();
            let walk_options = walk_options.clone();
            let sender = sender.clone();
            s.spawn(move |_| walk_parallel(dwi, &child_dir.path, walk_options, sender));
        }
    })
    .expect("Error in crossbeam scope in walk_parallel");

    Ok(())
}

Code

The code is soon to be released, and I’ll add a link to it. You can check my GitHub profile if I forget.

1. Update: I decided to change the walk_serial interface to use Vec instead of Sender for simplicity. ↩