🐢 There is a way if we start from the configuration, not just files. The default configuration is a TOML document. We can make it an XvcConfiguration struct and load and store it with confy.

🐇 We have a cascading set of configurations, but maybe we can start from the struct and serialize/deserialize it on demand.

🐲 I don’t think we need to update xvc-config at the moment. It’s working, and we don’t need to alter its inner workings in the near future.

🐢 I agree. We can use the config crate when we need to update and have enough time to work on this.

virtualenv .venv
overlay use .venv/bin/activate.nu

# List issues related to me
export def ghil [] { 
  (gh issue list --search "involves:iesahin" --json title,url) | from json
}

# Add a comment to an issue
export alias ghic = gh issue comment

# Create an issue
export alias ghiC = gh issue create $in

# View an issue
export alias ghiv = gh issue view

# View an issue in the browser
export alias ghiw = gh issue view --web

# List PRs in a table
export def ghpl [] {
  (gh pr list --json "title,url,headRefName" ) | from json
}

# View a PR
export alias ghpv = gh pr view

# View a PR in GitHub
export alias ghpw = gh pr view --web

# Checkout a PR
export alias ghco = gh pr checkout

# Add a comment to a PR
export alias ghpc = gh pr comment

# Show changes in a PR
export alias ghpd = gh pr diff

# Add a review to a PR
export alias ghpR = gh pr review

# Create a PR
export alias ghpC = gh pr create

# Merge a PR to main
export alias ghpM = gh pr merge

# List GH CI runs in a table
export def ghrl [] {
  (gh run list --json conclusion,displayTitle,headBranch,url ) | from json 
}

# View a GH CI run
export alias ghrv = gh run view

# View a GH CI run in the browser
export alias ghrw = gh run view --web 

# View output from a failed run
export alias ghrf = gh run view --log-failed

# Search code and get results in a table
export def ghsc [] { 
  (gh search code $in --json repository,path,textMatches,url) | from json
}

Aliases don’t allow pipes in Nushell, so I used def for those that require them.

When there is an $in parameter in an alias, it expects the input from a pipe. An issue can be created like this, for example:

"Your software sucks" | ghiC

To search for the code copied to the clipboard and get only the URLs, you can use:

pbpaste | ghsc | get url

To open them in different browser windows:

pbpaste | ghsc | get url | each { |u| start $u }

I’ll add a shortcut key for this last one. It’s so quick! :D

🐲 We can work on completions or the GUI.

🐢 Completions are similar to the Homebrew work. It’s in another ecosystem and feels boring.

🐲 We can try to make it in Rust: “Writing shell completions in Rust.”

🐢 There are two solutions: One is clap_complete (https://docs.rs/clap_complete/latest/clap_complete/) and the other is shell_completion (https://github.com/JoshMcguigan/shell_completion). The latter is very new but may be what we need: writing completions only with Rust.

🐲 The shell_completion crate is very bare-bones. It doesn’t have anything, actually. https://github.com/JoshMcguigan/shell_completion/issues/1

🐇 The feature that we need is dynamic completion. We’ll write something similar to ArgValueCompleter for this.

🦊 I think we can just start a new branch.

🐲 We have a branch for Homebrew; should we merge it?

🐢 I think, yes, we can merge it. We’ll fix it if it breaks. It’s a separate workflow file anyway.

🦊 Created the PR.

ghpl
264	add brew tap	add-brew-tap	OPEN	2025-01-03T05:14:45Z

🐢 Merged it.

🐇 Now let’s create a new branch and add clap-complete to Cargo.toml.

🐲 There are some packages that we need to take a look at. thiserror now has v2.

🐢 Upgraded packages and compiled. It works.

🐲 Let’s update the version.

🐢 Done. Now we can add clap-complete.

🦊 Added with the unstable-dynamic command. Can we build it again?

🐢 Built it. Now we can add a completions subcommand to Xvc.

🐲 Is this the right name for this?

🐢 It will output a shell script that we can source in the shell.

🐲 Okay, let’s add the subcommand now.

🐢 The example doesn’t work with the clap builder.

🐇 Let’s search for it.

🐢 I think we’re extending ourselves a bit when trying to add all dynamic features of completion at once. We can just start by adding completion to xvc-test-helper.

🐇 Yep, that’s a better idea. We can have many more examples, and it’s certainly more straightforward.

🐢 One thing I noticed when working yesterday is that using auto-save was actually preventing many of these swap errors.

🐲 It was a bit slow. Can we take a look at a few others?

🐢 https://github.com/okuuva/auto-save.nvim seems a bit more polished.

🐇 Installed it, but haven’t seen an effect yet.

🐢 We may need to restart this.

🐲 Now, we can get into the test-helper completions.

🐢 Done. Added these quickly as you thought. Now adding this to main seems much easier.

🦊 That was a nice approach. Let’s dive into adding this to main.

🐇 Should we go with a separate command or just an option?

🐲 Our commands have distinct initial letters, allowing them to be used with just those letters. Adding another top-level command for this will make c useless.

🐢 I don’t think that’s a valid concern. We can create aliases for all commands and completions shouldn’t have to have an alias. But I agree that completions should not be a top-level command. It will be run once for installation at most.

🦊 I agree. Let’s call it --completions. It will be run as xvc --completions zsh and will print out the completions.

🐢 Okay. Let’s do this.

🦊 No errors left. Let’s install this version.

🐢 We get:

error: 'xvc' requires a subcommand but one was not provided
  [subcommands: file, init, pipeline, storage, root, check-ignore, aliases, help]

Usage: xvc [OPTIONS] <COMMAND>

For more information, try '--help'.

🦊 Umm, okay. I think we don’t have an option to create a command like that. Then we’ll have to print completions with a subcommand.

🐲 The above discussion is now moot. ❌

🐢 We can try to push, but probably it’s not worth it.

🐲 Let’s not lose time on this. I think we can remove the aliases command and replace it with completions, and add single-letter aliases to subcommands.

🐢 We can have an option in the completions command to print aliases. That will work.

🦊 I don’t think people will use the aliases command if we have single-letter command aliases.

🐢 You may be right. No need to try to maintain that at this time.

🐢 Good morning, and I’m fed up with these blink.cmp errors, you know.

🐇 Reinstalling blink works. It’s interesting to rely on such unreliable software.

🐢 Oh, yeah. It’s interesting. Where were we yesterday?

🐲 We decided to rename the aliases command to completions.

🐇 And waiting for rust-analyzer to complete its analysis.

🐢 Uh, yeah. I see.

🦊 The code itself is very short, actually.

    if let Some(shell) = cli_opts.completions {
        let mut cmd = XvcCLI::command();
        generate(shell, &mut cmd, "xvc", &mut io::stdout());
        return Ok(None);
    }

🐲 We’ll use the output! macro instead of writing to io::stdout(), right?

🐢 Yes, we can rely on the usual output system. It will be slower to create an output thread but shouldn’t matter for outputting a shell script.

🐇 It takes a while for rust-analyzer to scan all the directories, it looks like. When I close the project, it just cannot reload it immediately.

🐢 We can use cargo clean from time to time.

🦊 rust-analyzer finished scanning; let’s try to rename AliasesCLI.

🐲 Let’s keep these here; maybe we’ll need them in our scripts:

# Standard Xvc command aliases for longer commands.
alias xls='xvc file list'
alias pvc='xvc pipeline'
alias fvc='xvc file'
alias xvcf='xvc file'
alias xvcft='xvc file track'
alias xvcfl='xvc file list'
alias xvcfs='xvc file send'
alias xvcfb='xvc file bring'
alias xvcfh='xvc file hash'
alias xvcfco='xvc file checkout'
alias xvcfr='xvc file recheck'
alias xvcp='xvc pipeline'
alias xvcpr='xvc pipeline run'
alias xvcps='xvc pipeline step'
alias xvcpsn='xvc pipeline step new'
alias xvcpsd='xvc pipeline step dependency'
alias xvcpso='xvc pipeline step output'
alias xvcpi='xvc pipeline import'
alias xvcpe='xvc pipeline export'
alias xvcpl='xvc pipeline list'
alias xvcpn='xvc pipeline new'
alias xvcpu='xvc pipeline update'
alias xvcpd='xvc pipeline dag'
alias xvcs='xvc storage'
alias xvcsn='xvc storage new'
alias xvcsl='xvc storage list'
alias xvcsr='xvc storage remove'

🦊 We can use them when adding single-letter aliases.

🐇 There is a from_env method for Shell; will we support it?

🦊 I think we can support it. It’s much easier to use if we omit the shell.

🐲 We had aliases in xvc-core, but we need to access XvcCLI from completions. The completions module must be moved to xvc.

🦊 We added the clap_complete dependency to the xvc-pipeline and xvc-file crates, but I don’t think they are necessary. Let’s remove them now.

🐢 Now only the test-helper and xvc crates have the clap_complete dependency.

🐇 Are we ready to test?

🐢 Completions are working. 🎉

🐲 Now we need to update the docs and doc tests, I believe.

🐢 There are also tests to update.

🐇 Tests are running now. In the meantime, can we take a look at the blink configuration?

🐲 When we removed xvc aliases, we also removed pvc, xls, and other aliases. Maybe we can add these to the docs.

🐢 We can put them in the xvc completions reference for now.

🐢 It takes a while for rust-analyzer to finish analyzing the codebase.

🦊 Added aliases for xvc pipeline commands. Do you think we need to repeat root-level flags in xvc-pipeline?

🐢 No need to divert attention, I believe. Also, I still think there is an easier way to do that.

🐇 Okay. Do you think we should add easier subcommands to step? Like, step new becoming xvc p s n instead of xvc p s n? (Wait, that’s the same). I mean, more concise.

🐢 I think we can extend these even to the top-level, but shouldn’t make them visible. It will pollute the help text. We can add xvc fl for file list to avoid the space, but hide these from the help text.

🐲 That’s a good idea, but it will require including sub-crate level modules at the top level. We must test it first.

🐢 Let’s finish up the current changes and release them first.

🐇 By the way, we didn’t add two-letter abbreviations to the xvc storage new subcommands. Do you think we need them?

🐢 When we think about the frequency of these commands, no, I don’t think we need them. Users won’t add a new storage every day.

🐇 By that logic, we shouldn’t need an n for xvc storage new.

🐢 Actually, yeah. Maybe we should remove even s.

🦊 storage list may be useful.

🐢 s is a very common letter, though. We can have other uses for that letter.

🐇 Updating xvc p s dependency options, but it looks like making dependency options subcommands is a better way.

🦊 We didn’t do it because currently we can supply multiple options with a single command. If we go the subcommand route, we’ll have to write all dependencies one by one. Adding multiple subcommands with clap is a bit tricky.

🐢 Umm. Yeah, I see.

🐇 Maybe we can provide a separate command to add multiple dependencies. Like, xvc p s d add 'lines=myfile.csv::10-20; file=myimage.jpg; glob=dir/image-10*'

🐲 That looks like the start of a language. We need a parser for those strings. They will be freeform.

🦊 Another option is to keep the current options and add commands with the same names.

🐢 That will be confusing. The user will have both --param and param, and they will work differently.

🐇 We can have an add command that accepts the current options. Parsing will be done by clap just like now, but for a subcommand of dependency.

🐢 The full command will be something like xvc pipeline step --step-name preprocessing dependency add --params 'params.json::batch_size'

🐇 With shorter commands, it’s like xvc p s -s preprocessing d a --params 'params.json::batch_size', and this doesn’t look like ffmpeg monstrosities.

🐲 In any case, this is a backward-incompatible change. This should wait for v0.7 along with ECS changes.

🐢 ECS changes are not user-visible, but these are. Certainly needs a minor version update.

🐇 Then, okay, let’s keep the current ones for this version and think about updating them in a future version.

🐢 Okay. Let’s finish up and we’ll create a separate invisible subcommand to ask questions to the Xvc repo for completions.

🐇 We completed completions for the common command structure. Now, we need a way to show certain info after certain commands. A tab after xvc p r -p should show pipeline names, for example.

🐢 Umm, yeah. And these should be as quick as possible. They shouldn’t check Git or any other things.

🐇 I looked here and there, and the only working example I found is here: https://github.com/clap-rs/clap/blob/master/clap_complete/tests/testsuite/zsh.rs#L248 in clap_complete tests.

🐢 Let’s try to dive in. We can start by cloning the repo, I believe.

At work, we use Doppler to manage our secrets. It injects environment variables before running commands. I thought, why can’t I do the same with a simple script?

The following script evaluates all password files with env in their names. I keep my environment variables in files named env-aws, etc., in the following format:

export AWS_ACCESS_KEY_ID="123456...."

The following script, which I called rws (short for run-with-secrets), allows me to run a command like rws s3cmd and use pass to inject the variables. So far, I’m happy with it.

#!/bin/zsh

cmd="$@"

fd -F env $HOME/.password-store | while read file ; do
  bb="${file:t:r}"
  eval $(pass show ${bb})
done

exec $cmd

I tried the same approach when testing Python bindings with Pytest:

def test_pipeline_step_dependency(empty_xvc_repo):
    dep_help = empty_xvc_repo.pipeline().step().dependency(help=True)
    expected = """
Usage: xvc pipeline step dependency [OPTIONS] --step-name <STEP_NAME>

Options:
  -s, --step-name <STEP_NAME>
          Name of the step to add the dependency to
"""
    assert dep_help == expected

This doesn’t work because the help text is generated by clap and skips the usual thread-based output handler. All command output and errors in Xvc are returned as strings from the command, except for the help text that’s generated by clap automatically.

There are probably workarounds for this, but I won’t pursue them further as I don’t test the actual functionality in the Python bindings anyway.

I replaced the eprintln!s with println!, but it didn’t make any difference. Maybe we should remove those statements completely.

I can’t really find the place that kills the kernel. The last command to run is xvc file list:

print(xvc_test_data.file().list("test-data/dir-0002"))

and the output it produces is:

[src/output.rs:144:13] &output_str = "SS         131 2024-06-05 08:57:11 41e16be7          test-data/dir-0002/file-0003.bin\nSS         131 2024-06-05 08:57:11 27f0
efd0          test-data/dir-0002/file-0002.bin\nSS         131 2024-06-05 08:57:11 66de5084          test-data/dir-0002/file-0001.bin\nTotal #: 3 Workspace Size:
      393 Cached Size:        6006\n"

print may be causing the crash, but the more likely cause is the command that comes after this:

!ls -l test-data/dir-0001/

I replaced this with lsd, which also failed. Maybe it’s actually a Python crash or bug.

The way to understand is to create a notebook file with only that cell and try to run it.

The ls line runs fine with a new notebook. It even runs on the README file when run at the beginning. The line that makes the kernel crash is:

xvc_test_data.storage().new_s3(name="backup", bucket_name="xvc-test", region="eu-central-1", storage_prefix="xvc-storage")

We can start by removing the new_s3 part.

The storage() method runs fine. It returns an XvcStorage() object, as it should.

When I run storage().list(), it takes a very long time. The bug is likely related to storage().

It looks like the storage object was adding file instead of storage as a subcommand. I’ve fixed it now.

That was the bug. The README notebook now creates the S3 storage.

What was the reason behind this?

Parsing the CLI to the XvcCLI object was perhaps the culprit. Let’s look at it more clearly.

Let’s try xvc file new s3 as a command to see how it behaves.

It says unrecognized subcommand for new.

This is how it should be, but I wonder why it doesn’t work for the XvcCLI parser.

Anyway, it’s already 13:00, so let’s stop here for today.

bind-key -n S-F12 split-window -h -c "#{pane_current_path}" "gt c"

I just hit Shift-F12, and it opens a new pane in the current directory and runs the command.

git rev-parse HEAD

I was using burntsushi’s ignore crate, which in turn uses globset by the same author. In a directory hierarchy, I was first collecting all .ignore files, then trying to decide which files are ignored. Ripgrep’s from parameter led me to believe this.

However, it looks like it has a bug that doesn’t consider the source directory of the ignore file. Git has different semantics when a/.gitignore and a/b/.gitignore have a line mydir/. In the second case, Git doesn’t ignore a/mydir/, but Ripgrep seems to consider all elements from all files as if they are from the root directory.

In practice, most people may be using ignore files in their root directory, but there are valid use cases (like moving around directories with ignore patterns in them) that are affected by such behavior.

I decided to tackle this. Ripgrep’s globset seems nice and fast, so it was better to base the solution on top of that. My basic use case is walking a directory hierarchy. There are solutions to this like walkdir or jwalk, but they use ripgrep’s gitignore; hence, they have the same bug.

I usually begin by setting up data structures to understand the problem. This allows me to understand the domain formally.

First, what are we trying to accomplish? Basically, we want to decide if a path should be ignored (a) in the results and (b) in the directory traversal. The gitignore specification not only allows ignores, but also has a way to add paths to a whitelist to unignore. Hence, deciding whether to ignore a path is like:

let include = |path| {
    if whitelist_globs.match(path) {
       true
    } else if ignore_globs.match(path) {
       false
    } else {
       true }
}

We also need to decide whether to traverse a directory:

let traverse = |path| {
    if path.is_dir() && !include(path) { false } else { true }
}

We have the following possible MatchResults between the ignore rules and a path:

pub enum MatchResult {
    NoMatch,
    Ignore,
    Whitelist,
}

We need a comparison function to get a MatchResult. One side of the comparison is path, and the other is a set of GlobSets from the globset crate. We’ll have two globsets: one for ignores and the other for whitelists.

pub struct IgnoreRules {
    root: PathBuf,

    patterns: Arc<Vec<Pattern<Glob>>>,

    whitelist_set: Arc<GlobSet>,
    ignore_set: Arc<GlobSet>,
}

The data types for the sets are Arc<GlobSet>, as I planned to make the traversal concurrent for each directory. However, as only one thread will be modifying the rules (one .gitignore or .ignore file per directory), I felt no need to wrap the sets with Mutex.

We also keep track of each pattern separately. patterns is a Pattern vector that contains all globs individually. One reason is to rebuild the GlobSet as we traverse directories: since we cannot add any more rules to the sets once they have been built, we keep the patterns separately to add them to the sets of the child directories.

Another reason is to keep track of the source, as this can be used for debugging or reporting, similar to git check-ignore.

root is the point where this traversal began.

Patterns are defined in a generic way to be instantiated with globs, strings, regexes, or any other type.

pub struct Pattern<T> {
    pattern: T,
    original: String,
    source: Source,
    effect: PatternEffect,
    relativity: PatternRelativity,
    path_kind: PathKind,
}

The rationale for this generic approach is that the patterns need some kind of transformation. They can be read from files as strings, then compiled into globs or regexes.

flowchart LR
     String-->Ok(Glob)-->Glob-->Regex

The source field shows where this pattern was retrieved. It’s an enum with two possible values for the time being:

enum Source {
    File { path: PathBuf, line: usize },
    Global,
}

The gitignore specification says:

An optional prefix “!” which negates the pattern; any matching file excluded by a previous pattern will become included again.

We use the effect field to tag the pattern as either an Ignore or a Whitelist pattern.

pub enum PatternEffect {
    Ignore,
    Whitelist,
}

The spec also makes a distinction between files and directories:

If there is a separator at the end of the pattern then the pattern will only match directories, otherwise the pattern can match both files and directories.

So we use path_kind to reflect this distinction:

pub enum PathKind {
    Any,
    Directory,
}

Some patterns can be relative to the directory they are found in:

If there is a separator at the beginning or middle (or both) of the pattern, then the pattern is relative to the directory level of the particular .gitignore file itself. Otherwise the pattern may also match at any level below the .gitignore level.

And we show this with the relativity field:

pub enum PatternRelativity {
    Anywhere,
    RelativeTo { directory: String },
}

Note that we are converting from the specification to Rust types. In the future, these enums can be extended to cover cases where these patterns are used in other places, or we can merge some of them to reduce complexity. Currently, this level of abstraction seems adequate.

To convert Pattern<String> to Pattern<Glob> (or Pattern<Result<Glob, Error>>), or any other type, we can use a map function similar to Option::map:

    fn map<U, F>(self, f: F) -> Pattern<U>
    where
        F: FnOnce(T) -> U,
    {
        let pat = Pattern::<U> {
            pattern: f(self.pattern),
            original: self.original,
            source: self.source,
            effect: self.effect,
            relativity: self.relativity,
            path_kind: self.path_kind,
        };
        pat
    }

map will make it easy to convert between various types of patterns.

Now, we change the perspective to a higher level. Let’s begin by writing a file tree walker.

pub fn walk_serial(
    given: IgnoreRules,
    dir: &Path,
    walk_options: &WalkOptions,
    sender: &Sender<WalkerResult<PathMetadata>>,
) -> WalkerResult<()> { todo!() }

This function uses the given IgnoreRules and returns files and directories via the channel. PathMetadata is just a bundle of PathBuf and its Metadata.

pub struct PathMetadata {
    path: PathBuf,
    metadata: Metadata,
}

WalkOptions will contain various options about this traversal. Currently, it only has ignore_filename and include_dirs, which set whether only files are sent through the channel, or if directories should be included too.

pub struct WalkOptions<'a> {
    ignore_filename: Option<&'a str>,
    include_dirs: bool,
}

Initially, IgnoreRules contains only the files that should be ignored (or whitelisted) globally. When a directory contains a .gitignore file (or any file name that was set in WalkOptions), it checks for new ignore rules and sends all files through the channel. If a directory is not ignored, it then calls walk_serial recursively with it.

The final lines of the function are as follows:

    for child_dir in child_dirs {
        walk_serial(dir_with_ignores.clone(), &child_dir, walk_options, sender)?;
    }

As it calls itself as a final operation after collecting and sending all child files, this is an example of tail recursion.

The output is sent through a channel because our final goal is to make all this concurrent. Each new directory will create a new thread, and all files will be collected from the channel. At this point, we are creating a serial version of the walker with identical data structures to test it easily.

One gotcha using the channels in the serial version is that the channel size must be enough to contain all file names. Effectively, this means it will be crossbeam_channel::unbounded for walk_serial. We will (ab)use the channel’s Sender<PathMetadata> as a Vec<PathMetadata> in no parallel mode. In parallel settings, using unbounded channels is not recommended; if the consumers of the channel don’t work as quickly, you may fill up the memory, especially for longer-running processes.

walk_serial will basically do four things:

• List all the elements in the given directory.
• Check if there is a new ignore file and update the rules if necessary.
• Filter the elements by ignore rules and send them to the channel.
• Run walk_serial recursively for child directories if they are not ignored.

Read the directory

The first is done using read_dir of Path. It returns a set of elements. Then the list is processed, and elements’ metadata is retrieved. This requires a bit of error handling: read_dir may return an Error, each element in the result can be an Error, and getting metadata may also result in an Error.

    let elements = dir
        .read_dir()
        .map_err(|e| anyhow!("Error reading directory: {:?}, {:?}", dir, e))?;
    let mut child_dirs = Vec::<PathBuf>::new();
    let mut child_paths = Vec::<PathMetadata>::new();

    for entry in elements {
        match entry {
            Err(err) => sender.send(Err(WalkerError::from(anyhow!(
                "Error reading entry in dir {:?} {:?}",
                dir,
                err
            ))))?,
            Ok(entry) => match entry.metadata() {
                Err(err) => sender.send(Err(WalkerError::from(anyhow!(
                    "Error getting metadata {:?} {}",
                    entry,
                    err
                ))))?,
                Ok(md) => {
                    child_paths.push(PathMetadata {
                        path: entry.path(),
                        metadata: md.clone(),
                    });
                    if md.is_dir() {
                        child_dirs.push(entry.path());
                    }
                }
            },
        }
    }

Note that we collect PathMetadata objects in the child_paths vector. If there happens to be an error while retrieving, it’s sent to the caller to ignore or to report to the user.

I usually handle errors first in match expressions because they are usually more straightforward.

Check if there are new ignore rules

After reading all the files in the directory, it checks if there actually is an ignore filename in given and that a file exists with that name in the list we just created. If there is no such file, we can use given rules to decide the ignored elements.

If there is a file that may change the ignore rules, it reads it and updates patterns. Then, if there are new ignore or whitelist rules, the respective globsets are recreated. It checks whether we have new rules because compiling globs is possibly an expensive operation.

Filtering directory elements

Filtering files and directories from the list checks two globsets. If an element matches a whitelist rule, it’s included in the results without checking if it also matches ignore rules. Hence, whitelisting has a priority. Otherwise, it checks the ignore rules, and if they don’t match, the element is included in the results.

If there are directories matching the ignore rules, they are not traversed, and any possible whitelisting rule that was specified within that directory is not taken into consideration.

Listing child directories

We saw how walk_serial calls itself for child directories above. After sending all results to the channel, it calls itself for the child directories and repeats.

Converting patterns from String to Glob

The interesting part about ignoring files is that we should modify patterns into globs to add to a GlobSet. We cannot add the patterns directly to a GlobSet because some of the rules make them context dependent. We parsed the patterns to get PatternRelativity and PathKind, and these also affect how we add the patterns to a GlobSet.

The heart of the conversion is a function that receives a String, filters it, and transforms it to a list of Pattern<Glob>s.

fn content_to_patterns(
    ignore_root: &Path,
    source: Option<&Path>,
    contents: &str,
) -> Vec<Pattern<WalkerResult<Glob>>> {

As we also need to report the line number for patterns, we use enumerate on the iterator:

    let patterns: Vec<Pattern<WalkerResult<Glob>>> = content
        .lines()
        .enumerate()
        // A line starting with # serves as a comment. Put a backslash ("\") in front of the first hash for patterns that begin with a hash.
        .filter(|(_, line)| !(line.trim().is_empty() || line.starts_with("#")))

The filter line is used to check for blank and comment lines. Then we trim the trailing space unless the pattern ends with \.

        .map(|(i, line)| {
            if !line.ends_with("\\ ") {
                (i, line.trim_end())
            } else {
                (i, line)
            }
        })

The function receives the source file name as a parameter. We create the Source using it and the line number:

        .map(|(i, line)| {
            (
                line,
                match source {
                    Some(p) => Source::File {
                        path: p
                            .strip_prefix(ignore_root)
                            .expect("path must be within ignore_root")
                            .to_path_buf(),
                        line: (i + 1).into(),
                    },
                    None => Source::Global,
                },
            )
        })

Then we build the pattern object, update the pattern to add it before the globset, and build the globs. Building the glob may cause errors if it’s not well-formed. They are handled as well.

        .map(|(line, source)| build_pattern(source, line))
        .map(transform_pattern_for_glob)
        .map(|pc| pc.map(|s| Glob::new(&s).map_err(WalkerError::from)))

Two functions are left that may need attention. One is build_pattern that converts a pattern line to a Pattern<String> object. The other is the transform_pattern_for_glob function that modifies Pattern<String> before building the glob.

After writing walk_serial, we’ll use all the machinery to write a parallel version of it.

Parsing pattern strings for pattern objects

We create pattern objects from pattern strings following the spec. There are three basic rules that we will follow:

• If the pattern starts with !, it’s a whitelist rule.
• If the pattern ends with /, it matches only to directories.
• If the pattern contains a non-final /, it’s relative to the directory.

We’ll also check whether the line starts with / and remove that to prevent double-slashes // in patterns. The resulting pattern in the object won’t contain any of the artifacts that change its semantics, and we’ll add them when building the globsets.

Let’s start by checking whether the line starts with !.

    let begin_exclamation = original.starts_with("!");
    let line = if begin_exclamation || original.starts_with("\\!") {
        original[1..].to_owned()
    } else {
        original.to_owned()
    };

Checking the presence of / at the beginning, at the end, and in intermediate positions is also straightforward:

    let end_slash = line.ends_with("/");
    let line = if end_slash {
        &line[..line.len() - 1]
    } else {
        line
    };

    let begin_slash = line.starts_with("/");
    let non_final_slash = if line.len() > 0 {
        line[..line.len() - 1].chars().find(|c| *c == '/').is_some()
    } else {
        false
    };

We check the non-final slash separately from the beginning slash because we’ll remove the initial slash if it exists.

    let line = if begin_slash { &line[1..] } else { line };

Although there is almost 1-1 correspondence between the boolean conditions and the enums, I create enums in separate statements for self-documenting code.

    let effect = if begin_exclamation {
        PatternEffect::Whitelist
    } else {
        PatternEffect::Ignore
    };

    let path_kind = if end_slash {
        PathKind::Directory
    } else {
        PathKind::Any
    };

    let relativity = if non_final_slash {
        PatternRelativity::RelativeTo {
            directory: current_dir.to_owned(),
        }
    } else {
        PatternRelativity::Anywhere
    };

After building these attributes of the pattern and stripping the pattern from the initial and final /, and other semantic-changing elements, we build a pattern:

    let pattern = Pattern::<String> {
        pattern: line.to_owned(),
        original: original.to_owned(),
        source,
        effect,
        relativity,
        path_kind,
    };

Transforming a pattern string to a suitable glob

Now, another transformation is needed to convert Pattern<String> to Pattern<Glob>. In this case, we’ll need to recreate the pattern based on the rules.

The transformation considers relativity and path_kind to generate a glob that can be added to a GlobSet. Note that the function returns not a compiled Pattern<Glob>, but a Pattern<String>, because after this transformation, building the glob requires Glob::new.

fn transform_pattern_for_glob(pattern: Pattern<String>) -> Pattern<String> {
    let anything_anywhere = |p| format!("**/{p}");
    let anything_relative = |p, directory| format!("{directory}/**/{p}");
    let directory_anywhere = |p| format!("**{p}/");
    let directory_relative = |p, directory| format!("{directory}/**/{p}/");

    let transformed_pattern = match (&pattern.path_kind, &pattern.relativity) {
        (PathKind::Any, PatternRelativity::Anywhere) => anything_anywhere(pattern.pattern),
        (PathKind::Any, PatternRelativity::RelativeTo { directory }) => {
            anything_relative(pattern.pattern, directory)
        }
        (PathKind::Directory, PatternRelativity::Anywhere) => directory_anywhere(pattern.pattern),
        (PathKind::Directory, PatternRelativity::RelativeTo { directory }) => {
            directory_relative(pattern.pattern, directory)
        }
    };

    Pattern {
        pattern: transformed_pattern,
        ..pattern
    }
}

I could merge the closures into the match and lower the number of lines, but adding the closures increased self-documentation. Instead of multiple if/else branches, I tend to use match with tuples to have a clearer picture. The corresponding if/else code without closures would be like:

let transformed_pattern = if pattern.path_kind == PathKind::Any {
    if pattern.relativity == PatternRelativity::Anywhere {
        format!("**/{p}")
        } else if ...
        }
...

And in my opinion, this latter format is more error-prone and less readable.

Combining it all together

At this point, we can revisit content_to_patterns again:

fn content_to_patterns(
    ignore_root: &Path,
    source: Option<&Path>,
    content: &str,
) -> Vec<Pattern<Result<Glob>>> {
    watch!(source);
    let patterns: Vec<Pattern<Result<Glob>>> = content
        .lines()
        .enumerate()
        ...
        .map(|(line, source)| build_pattern(source, line))
        .map(transform_pattern_for_glob)
        .map(|pc| pc.map(|s|
            Glob::new(&s).map_err(Error::from)))
        .collect();

    patterns
}

The last step in the transformation is creating the Glob object, as in the highlighted lines above. It runs Pattern<String>.map with Glob::new to get a Pattern<Result<Glob>>. Error and Result are custom types defined within the crate using the thiserror crate.

After we have a vector of Pattern<Result<Glob>>, we report the errors in clear_glob_errors to get Vec<Pattern<Glob>> from this vector. The function is defined as:

fn clear_glob_errors(
    sender: &Sender<Result<PathMetadata>>,
    new_patterns: Vec<Pattern<Result<Glob>>>,
) -> Vec<Pattern<Glob>> {
    let new_glob_patterns: Vec<Pattern<Glob>> = new_patterns
        .into_iter()
        .filter_map(|p| match p.transpose() {
            Ok(p) => Some(p),
            Err(e) => {
                sender
                    .send(Err(Error::from(anyhow!("Error in glob pattern: {:?}", e))))
                    .expect("Error in channel");
                None
            }
        })
        .collect();
    new_glob_patterns
}

The sender is the same channel that we use to send the results. This communication channel is used to report only the errors, not PathMetadata; hence, when we encounter an Err(e) in new_patterns, it’s reported to the same channel that the results are expected. Handling the errors first seems like a common theme here as well.

This is also one of the reasons walk_serial receives a channel instead of a vector. Our usual use case is traversing all the directories in parallel, and without channel-based communication in the serial version, we would need to duplicate this function.

At this point, I completed the walk_serial functionality. I wrote tests. I’m using the test-case crate for writing many tests encompassing different use cases. I’m omitting them here for brevity.

Writing a Parallel Walker

Converting walk_serial to a parallel one is quite trivial. As we already used channels in the serial version, we have all the components of a parallel walker. For the sake of simplicity, I won’t use a thread pool and instead spawn a new thread for each task. One question is what’s the unit task that we’ll use to create tasks.

jwalk uses directories as points to spawn new tasks. That’s a sensible approach, and I’ll use this as well. This means a single directory will be processed by a single thread, even if it contains one million files, and 10 directories will be processed by 10 threads even if each contains one file. In practice, I think we can assume directories are natural chunks for this problem.

The interface for the parallel walker is identical to the serial version. This is to replace one with the other when the need arises.¹

The only difference is in the final loop. walk_serial calls itself recursively at the end, while walk_parallel calls itself in separate threads.

pub fn walk_parallel(
    given: IgnoreRules,
    dir: &Path,
    walk_options: WalkOptions,
    sender: Sender<Result<PathMetadata>>,
) -> Result<()> {
    ....
    crossbeam::scope(|s| {
        for child_dir in child_dirs {
            let dwi = dir_with_ignores.clone();
            let walk_options = walk_options.clone();
            let sender = sender.clone();
            s.spawn(move |_| walk_parallel(dwi, &child_dir.path, walk_options, sender));
        }
    })
    .expect("Error in crossbeam scope in walk_parallel");

    Ok(())
}

Code

The code is soon to be released, and I’ll add a link to it. You can check my GitHub profile if I forget.

1. Update: I decided to change the walk_serial interface to use Vec instead of Sender for simplicity. ↩

A Vim Guide for Intermediate Users

:setlocal spell spelllang=en_us

This enables the spell option and configures it to check for US English.

Unrecognized words are highlighted with one of the following:

	SpellBad	word not recognized			|hl-SpellBad|
	SpellCap	word not capitalised			|hl-SpellCap|
	SpellRare	rare word				|hl-SpellRare|
	SpellLocal	wrong spelling for selected region	|hl-SpellLocal|

Vim only checks for spelling; it does not perform grammar checks.

export AWS_ACCESS_KEY_ID="XXXX"
export AWS_SECRET_ACCESS_KEY="YYYY"

you can install the AWS CLI with:

pip3 install --user awscli

and create an S3 bucket from the command line without using a web browser:

aws s3api create-bucket --acl public-read --bucket my-unique-bucket-name --region eu-central-1 --create-bucket-configuration LocationConstraint=eu-central-1

This will create a publicly readable bucket at http://my-unique-bucket-name.s3.amazonaws.com.

Then you can copy your files with:

aws s3 cp my-file.zip s3://my-unique-bucket-name/subdir/my-file.zip

and share the link http://my-unique-bucket-name.s3.amazonaws.com/subdir/my-file.zip with anyone you want.

  In[1]: 10 + 10

 Out[1]: 20.0


           _____
  In[2]: ╲╱ 100

 Out[2]: 10.0


             ┌                  ┐
  In[3]: Max │ 10 , 1 , 21 , -3 │
             └                  ┘

 Out[3]: 21.0


  In[4]: ⟨Emre's Number⟩ ≡  79

 Out[4]: 79.0


           _______________
  In[5]: ╲╱ Emre's Number

 Out[5]: 8.888194417315589


                      2
  In[6]: Emre's Number

 Out[6]: 6241.0


                      Emre's Number
  In[7]: Emre's Number

 Out[7]: 8.1759873707105095e149

It looks a bit heavy for a CLI calculator, as it is written in Haskell and downloads 100+ MB of libraries, but when you need ASCII art to display your calculations or want to use spaces in variable names, it may prove useful.

• I began to use Intention to limit my Twitter time. It allows you to set a limited time (1, 5, 10, or 15 minutes) for yourself and tracks the total time you spend on addictive sites. When this total time is lower than your goal for a period, you get a streak. It looks visually and psychologically nicer than LeechBlock.

• I read the git reset section in the Git book. It details how git reset behaves with its --soft, --mixed, and --hard parameters. The first resets only the HEAD; the second resets both the index and HEAD; and the third resets the working tree and copies files back from the current HEAD to the working tree.

  One important point: Contrasting git checkout master and git reset master: the first moves HEAD to the master branch, while the second moves the current branch to master.

  git reset can also be used to squash commits. Basically, you git reset --mixed to an earlier commit like HEAD~3 and recommit. This creates a new commit, taking HEAD~3 as the parent and skipping HEAD~2 and HEAD~1, resulting in a new HEAD.

• Here are many useful pure Bash functions to be used in scripts. I’m a Zsh person, but writing Bash scripts is more portable, of course.