🐇 It looks like you first need to turn off this ghost text from the blink output. It makes writing insufferable.

🐢 Yep, let’s do that first.

🐇 Now, let’s restart Neovim.

🐢 I don’t know why blink.cmp doesn’t prioritize the emojis I use. Maybe we can just use #tor and #rab for ourselves.

🐇 I think over time it will learn that the emojis I defined in the snippets file should have higher priority, but let’s skip this for now. What do we need to do to add cross-compilation?

🐢 Maybe we can just make the completion menu wait a bit longer. It shows up almost instantly, and I want it to wait for a few more milliseconds.

🐇 Okay, let’s look at the config.

🐢 The configuration file doesn’t seem to have a key for this. Let’s search: blink.cmp.

🐇 I think the culprit is typo resistance; it causes better options to be pushed down: https://cmp.saghen.dev/configuration/reference#fuzzy

🐢 Let’s turn that off.

🐇 There is an error in the configuration file. I don’t know why it’s failing.

🐢 I added emojis to Espanso and checked the error message. The configuration I copied from their docs seems to be broken. The message is:

...share/nvim/lazy/blink.cmp/lua/blink/cmp/config/utils.lua:14: fuzzy.max_items: unexpected field found in configuration

I’ll just delete that line.

🐇 It still doesn’t prioritize snippets, but we can look into this later. Espanso seems to be a better tool for this anyway.

🐢 Yep. Let’s look into cross-compilation support for Rust.

🐇 The well-known option is cross.rs: https://github.com/cross-rs/cross

🐢 We can start with that. Installation is from the Git repository:

$ cargo install cross --git https://github.com/cross-rs/cross
    Updating git repository `https://github.com/cross-rs/cross`
    Updating git submodule `https://github.com/cross-rs/cross-toolchains.git`
  Installing cross v0.2.5 (https://github.com/cross-rs/cross#4090beca)
...
   Installed package `cross v0.2.5 (https://github.com/cross-rs/cross#4090beca)` (executables `cross`, `cross-util`)

🐇 Now we have the cross and cross-util commands. Cross-compilation requires Podman on Linux or Docker on macOS. Do we have Docker?

🐢 It looks like we don’t. Maybe we can just set up a remote build using Podman or GitHub Actions. I saw a crate for that yesterday; I remember saving it somewhere but can’t find it now. Searching again seems easier, which says something about my archival and retrieval habits.

🐇 Maybe later you can add some vector search capabilities to your archive—semantic search.

🐢 Yep, sometime later.

🐇 Now let’s search for “adding rust cross compilation to github actions.”

🐢 I found a link to the action: https://github.com/marketplace/actions/build-rust-projects-with-cross

Let’s look at the example:

jobs:
  release:
    name: Release - ${{ matrix.platform.os-name }}
    strategy:
      matrix:
        platform:
          - os-name: FreeBSD-x86_64
            runs-on: ubuntu-20.04
            target: x86_64-unknown-freebsd
            skip_tests: true

          - os-name: Linux-x86_64
            runs-on: ubuntu-20.04
            target: x86_64-unknown-linux-musl

          - os-name: Linux-aarch64
            runs-on: ubuntu-20.04
            target: aarch64-unknown-linux-musl

          - os-name: Linux-riscv64
            runs-on: ubuntu-20.04
            target: riscv64gc-unknown-linux-gnu

          - os-name: Windows-x86_64
            runs-on: windows-latest
            target: x86_64-pc-windows-msvc

          - os-name: macOS-x86_64
            runs-on: macOS-latest
            target: x86_64-apple-darwin

          # more targets here ...

    runs-on: ${{ matrix.platform.runs-on }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build binary
        uses: houseabsolute/actions-rust-cross@v0
        with:
           command: ${{ matrix.platform.command }}
          target: ${{ matrix.platform.target }}
          args: "--locked --release"
          strip: true
      - name: Publish artifacts and release
        uses: houseabsolute/actions-rust-release@v0
        with:
          executable-name: ubi
          target: ${{ matrix.platform.target }}

🐇 We can just convert the current configuration to this and see if it works.

🐢 Let’s do that. I created .github/workflows/release.yml and will update it.

🐇 Let’s check the results with gh:

$ gh -R iesahin/xvc run list
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525070531	34s	2024-12-28T08:17:28Z

🐢 It says the release has completed.

$ gh -R iesahin/xvc run view 12525070531

X v0.6.13 Release iesahin/xvc#263 · 12525070531
Triggered via pull_request about 3 minutes ago

JOBS
X Release - FreeBSD-x86_64 in 12s (ID 34936257653)
  ✓ Set up job
  ✓ Checkout
  X Build binary
  - Publish artifacts and release
  ✓ Post Build binary
  ✓ Post Checkout
  ✓ Complete job
...

Now let’s look at the failure:

$ gh -R iesahin/xvc run view 12525070531 --log-failed

🐇 Change the order and see what happens. Maybe we should first succeed in a non-cross-compilation build.

$ gh -R iesahin/xvc run list
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525178664	35s	2024-12-28T08:34:18Z
...
$ gh -R iesahin/xvc run view   12525178664 --log-failed
...

🐢 It’s the same error. The usage example is actually much more sophisticated.

🐇 The issue is that we’re asking for command from the matrix, but the matrix doesn’t define it. That’s the second time today an example from the documentation has failed. I’ve added command for each platform. We can also add separate features this way to make platform-specific functionality work.

🐢 Agreed. Let’s look at it once more.

$ gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525253982	41s	2024-12-28T08:46:23Z

$ gh -R iesahin/xvc run view 12525253982 --log-failed
...
Release - macOS-x86_64	Build binary	2024-12-28T08:46:44.8464010Z  [1m [31merror [0m [1m: [0m the lock file /Users/runner/work/xvc/xvc/Cargo.lock needs to be updated but --locked was passed to prevent this

🐇 Ah, that’s a different error. Let’s remove the --locked flag and retry.

$ gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525289176	1m8s	2024-12-28T08:53:23Z

🐢 We’re finally starting to get some good news. Now we’re getting OpenSSL errors. We need feature flags for these platforms or to specify where OpenSSL is. Let’s add bundled-openssl to the failed ones. We also need bundled-sqlite for Windows binaries.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525354179	3m9s	2024-12-28T09:04:55Z

🐇 It looks like the Changes.md file is missing, and it can’t upload the binaries as a release because of this. I’ll set the changes file to CHANGELOG.md.

🐢 It’s weird to fail because of that. I think we should report these errors and possibly send a PR to make that file optional.

gh -R iesahin/xvc run list | rg Release | head -n 1
in_progress		v0.6.13	Release	v0.6.13	pull_request	12525449368	1m38s	2024-12-28T09:18:40Z
...
✓ Release - macOS-x86_64 in 2m39s (ID 34937017695)
✓ Release - macOS-aarch64 in 2m42s (ID 34937017787)
..
ARTIFACTS
xvc-macOS-x86_64.tar.gz
xvc-macOS-arm64.tar.gz

🐢 It looks like Linux-riscv64 has OpenSSL compilation errors. I think we can skip this platform for now. The goal was to add aarch64 for macOS, and that seems to have succeeded.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525504079	3m24s	2024-12-28T09:26:43Z
...
X Release - Linux-x86_64	Build binary	2024-12-28T09:29:16.2262518Z  [0m [1m [38;5;9merror[E0308] [0m [0m [1m: mismatched types [0m
...

🐢 It looks like Linux-x86_64 doesn’t support reflinks. Maybe we can make reflinks an optional feature and add it specifically to macOS and Windows targets.

🐇 I’ve removed reflink from the default features. This is a breaking change, but fortunately we don’t have many users who will be affected by it.

Let’s check the results once more.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525597866	3m39s	2024-12-28T09:42:44Z

🐢 Now turn off the NetBSD target as well.

🐇 FreeBSD and Linux-x86_64 targets are building. Let’s see why the ARM Linux targets are failing.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525643284	4m46s	2024-12-28T09:52:04Z

🐇 It looks like those targets don’t have libsqlite3 installed. Let’s add bundled-sqlite to these targets too.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525734325	4m11s	2024-12-28T10:07:36Z

🐢 Adding Android as a target didn’t work. Let’s remove it for now; it seems to require more work.

🐇 I think we’ll eventually move to using Xvc to distribute binaries. We could build the Android binary on Termux and link it on the releases page or push it as a release artifact.

🐢 I need to learn more about GitHub releases. If we can add artifacts to the release, maybe we can do some of this work locally.

🐇 We can start by looking at the capabilities of gh commands.

🐢 It looks like it’s possible to upload assets to releases. Let’s check how that works.

gh release upload --help

🐇 So basically, we can just upload files to tags. We can list releases and work with them like anything else.

gh -R iesahin/xvc release list

And we can delete releases:

for r in v0.4.2-alpha.8 v0.4.2-alpha.7 v0.4.2-alpha.6 v0.4.2-alpha.5 v0.4.2-alpha.0 v0.4.1-alpha.0; do
  gh -R iesahin/xvc release delete "${r}"
done

🐢 Now let’s list them again.

gh -R iesahin/xvc release list

🐇 The latest one has failed again.

X Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run

🐢 It looks like we’re coming to the end of this session. Under what conditions will we create a release?

🐇 I think it’s better to release only non-alpha tags.

🐢 Then the rule in the workflow will be something like:

on:
  workflow_dispatch:
  push:
    tags:
      - "v*.*.*"
      - "!v.*.*-alpha.*"

🐇 And now we have this result:

✓ v0.6.13 Release iesahin/xvc#263 · 12533498361
Triggered via pull_request about 21 minutes ago

JOBS
✓ Release - FreeBSD-x86_64 in 4m14s
✓ Release - Linux-x86_64 in 4m3s
✓ Release - Linux-aarch64 in 4m50s
✓ Release - Windows-x86_64 in 8m47s
✓ Release - Windows-aarch64 in 6m42s
✓ Release - macOS-x86_64 in 2m37s
✓ Release - macOS-aarch64 in 2m36s

🐢 Nice! Let’s check the release list.

gh -R iesahin/xvc release list

🐇 Why is there no “latest” release?

🐢 We have to tag it first.

🐇 Ah, right. Let’s tag it then.

🐢 I’ve pushed the changes and tagged them with v0.6.13-alpha.5.

🐇 I think it’s possible to make a release today.

🐢 It looks like it, yes.

🐇 We can merge the PR and tag it. Then everything should work.

🐢 We need some cleanup in the YAML files, though.

🐇 We can leave that for the next release.

gh -R iesahin/xvc run list
in_progress		v0.6.13	Rust-CI	v0.6.13	pull_request	12533690474	9m18s	2024-12-29T08:02:29Z
completed	success	Release	Release	v0.6.13-alpha.5	push	12533689099	9m11s	2024-12-29T08:02:19Z

🐢 Now we have another failure in the regular CI. Let’s look at it.

🐇 The issue seems to be in the doc tests:

- Total #: 8 Workspace Size:         276 Cached Size:          19
+ Total #: 8 Workspace Size:         278 Cached Size:          19

🐢 These tests are brittle, but they provide valuable information. Let’s fix it and push again.

🐇 Done. We can also remove some of the watches that produce so many logs.

🐢 I’m a bit ambivalent about them. I thought we could use these watches when debugging, but experience has shown that we need more granular watches during debugging and almost never use these otherwise. Let’s remove some of them.

🐇 We can increase the output for certain commands, but the tracing output doesn’t help much in regular runs. If Xvc gets popular enough that we can’t cope with bug reports, we can always add more watches.

🐢 Another option is to exclude the watch code from the release build, but that won’t change anything for our debug cycles.

🐇 I think removing them is a fair trial. We can always put them back when debugging.

🐢 Watches could also produce regular output instead of tracing. That way we won’t forget to remove them.

🐇 Ah, yep, that’s a good option too.

🐢 We can have a trace! macro similar to the current one for user consumption, and a watch! macro that sends output to stderr.

🐇 Good idea. Let’s do that in the next release.

🐢 Let’s check the tests before pushing this cleanup.

gh -R iesahin/xvc run list
completed	success	v0.6.13	Rust-CI	v0.6.13	pull_request	12533993607	13m49s	2024-12-29T08:47:50Z

🐇 CI succeeded, and the release didn’t run. Let’s do some more cleanup.

gh -R iesahin/xvc run list
in_progress		v0.6.13	Rust-CI	v0.6.13	pull_request	12534416152	43s	2024-12-29T09:57:02Z

🐇 It may be useful, though. It looks like syntactic sugar for a const pointer plus malloc, but implicitness is usually not ideal. It also appears to use the stack.

🐢 I haven’t looked into the details, but it may be useful in some cases. I don’t think it can replace pointer usage if it uses the stack, though. Stack sizes are usually small, and having arrays that can fill them up is not a good approach. Another feature is complex number support. It seems from the lecture that this is also a bit of a half-baked feature.

🐇 What is the difference between this and using a struct with two float fields?

🐢 There are operators (+, -, *, or ==) that support these. Since there is no operator overloading in C, having a separate complex number type may be useful.

🐇 The example looks like double complex cx = 1.0 + 3.0*I; and yes, this may be useful if you’re frequently using complex numbers in your code. But I think it’s unnecessary in most cases. There shouldn’t be such a frequent need for complex numbers, right?

🐢 A good complex number library will probably provide more than the built-in type, such as vectors for these numbers. A dedicated library will still be needed in most cases, I believe.

🐇 Another feature added to C is that struct members can now be initialized by name or index. You can initialize an array like int a[6] = { [3] = 29, [2] = 14 };, and it will initialize only those specific members.

🐢 Ah, this is much more useful than complex numbers for the general case.

🐇 There is also a span syntax: int a[10] = {1, 2, [2 ... 4] = 3, [5] = 30};.

🐢 That is really neat.

🐇 It’s also possible to omit the initial length, so the initializer values determine the size of the array. For example, int a[] = {1, 3, [30] = 55, [9090] = 1010}; will result in an array of length 9091.

🐢 How is this used with structs?

🐇 Instead of [], in struct initializers, we use the dot notation, like struct Point p1 = { .x=0, .y=10 };.

🐢 This is really useful and would keep much of the initialization code simpler.

🐇 There is also a way to initialize arrays of structs, like:

struct points pts[5] = { [0].x = 10, [0].y = 20, [3].x = 100 };

🐢 This feature is a nice addition to C; I really like it.

🐇 It looks, from the coverage page, that our additions to HStore don’t have any tests. Maybe we can add some unit tests to the new joins.

🐢 I don’t find unit tests particularly useful, but let’s use GitHub Copilot to add some for us.

🐇 I added a unit test and a doc test for full_join. I think doc tests have more value; they provide documentation, and we can readily see how to use a function from its docs. It’s better to increase coverage with doc tests.

🐢 There are things I should keep in mind while writing doc tests. Imports must use the full path, not crate. Also, the tested struct doesn’t have implicit imports.

🐇 The ceremony for adding keys and values is a bit too much. It might be worthwhile to add an insert method for anything that implements Into<XvcEntity>.

🐢 That would certainly save time—no more typing .into() for each key! :)

🐇 Pushed to test again. Should we have some way to test coverage locally?

🐢 I don’t think we need to check coverage locally. It’s not worth our time right now.

🐇 Now, while waiting for the tests to complete, what else can we do?

gh -R iesahin/xvc run list
completed failure v0.6.13 Rust-CI v0.6.13 pull_request 12543831360 3m54s 2024-12-30T08:15:06Z
...

🐢 That didn’t take too long. Let’s view the results:

gh -R iesahin/xvc run view 12543831360
...
  X Run Current Dev Tests
...
To see what failed, try: 
View this run on GitHub: https://github.com/iesahin/xvc/actions/runs/12543831360

🐢 The current dev tests are failing for some reason. Let’s run them locally.

🐇 We’re missing llvm-tools-preview locally. How do I install this?

🐢 The command is rustup component add llvm-tools-preview:

info: component 'llvm-tools' for target 'aarch64-apple-darwin' is up to date

🐇 It’s already installed. We just need to set the environment variables.

🐢 Instead, we can just turn off dev tests for the time being. We don’t really need them; our local tests pass.

🐇 Yeah, okay. We don’t need to solve every single bit of these issues right now.

gh -R iesahin/xvc run list
...

🐇 Okay, let’s take a look at the run again.

gh -R iesahin/xvc run list
completed failure v0.6.13 Rust-CI v0.6.13 pull_request 12544020064 3m39s 2024-12-30T08:33:25Z

gh -R iesahin/xvc run view 12544020064
...
  X Test and Coverage
...

gh run view 12544020064 --log-failed
...
Test and Coverage (stable) Test and Coverage 2024-12-30T08:36:58.8911690Z Error: ProcessError { stdout: "", stderr: "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to e1.xvc.dev port 80 after 160 ms: Couldn't connect to server\n" }
...

🐢 We need to start Nginx on the server. We forgot to do that yesterday.

🐇 Ah, right. After that dufs installation. Okay.

🐢 We also need to add a reverse proxy for dufs somehow, but that’s for later.

🐇 For this use case, I don’t think it’s necessary. We can just adjust the port to a non-standard one if we need 443 for something else. Let’s look at the tests again.

🐢 Let’s add another doc test, this time to XvcStore.

cargo test -p xvc-ecs --doc
...
test result: ok. 8 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.87s

🐇 Sent the files again. Waiting for the tests to finish.

🐢 Let’s check the keymaps file in the meantime.

🐇 I tried reading the documentation but didn’t see an error. Maybe we should just set it to non-lazy and disallow remaps.

🐢 We’ve already spent too much time on this.

🐇 Yes, let’s check the tests again.

gh -R iesahin/xvc run list
completed success v0.6.13 Rust-CI v0.6.13 pull_request 12544291068 8m10s 2024-12-30T09:00:42Z
...

🐢 Oh, nice, the merge is ready.

🐇 Patch coverage is still behind the target, though.

🐢 Yeah, but let’s release this one and ensure the next one is better covered. Also, I’m not sure if the doc tests actually affected the coverage report.

🐇 If we look at the coverage page again, we can see.

🐢 It seems codecov.io doesn’t consider coverage for doc tests. That’s a bit weird, but let’s not spend more time on it.

🐇 Sure, let’s merge.

🐢 I think we forgot to bump the versions in the Cargo.toml files. We’ll have to do that in main.

🐇 Oh, yeah. Let’s bump them and tag the release as well.

🐢 Now we can wait for all the files to be produced. What’s next?

🐇 We can release the Python version too. It shouldn’t need any changes.

🐢 Right. Maybe we can add a few tests there as well.

🐇 Let’s bump the version first and see.

🐢 I’ve bumped the versions in Cargo.toml and run maturin develop.

🐇 It seems ready now.

🐢 The main branch is failing, though. The “Publish crates” action is looking for libsqlite3. Let’s investigate.

gh -R iesahin/xvc run list
completed failure Release v0.6.13 Publish Crates v0.6.13 push 12544753359 6m1s 2024-12-30T09:41:56Z

🐢 The issue is that the VM doesn’t have libsqlite3-dev. Let’s add it.

🐇 We need to restart the job manually. Let’s skip tagging this time.

🐢 Some of the packages were already published, and now they’re causing failures because crates.io says they already exist. Maybe we can check if a package is already published before trying.

🐇 Let’s see if we can make cargo publish more forgiving.

🐢 There doesn’t seem to be an easy option. Let’s search for “how to skip published packages in workspace to avoid errors with cargo publish.”

🐇 Claude is hallucinating again. Let’s try a manual approach: how to skip already published packages?

🐢 It might be easier to just add a check. How do we get that info?

🐇 Or we can just move on to the next package if one is already available.

🐢 Let’s push the missing packages manually this time.

🐢 We should have a key to open garden files quickly. What does Fzf-Lua files receive as arguments?

🐇 Let’s check the help page: fzf-lua.

🐢 Before that, maybe we can try to fix why selected lines are not searched in Visual-Line mode?

🐇 Yep, let’s fix our search first.

🐢 How do you set a key in visual line mode in Neovim Lua?

🐇 The abbreviation is V. Let’s try that.

🐢 Looks like we need to restart the session.

🐇 It’s still not working. Let’s check :map.

🐢 It seems it needs more care; I’ll check it later.

👨🏾‍🦲 🐢 Bence kendini biraz daha anlamlı bir işle uğraştırmalısın. 🐇 Ne gibi? 🐢 Belki biraz daha yayın yapmalısın, biraz daha işe başvurmalısın. 🐇 “Meli”, “malı” ile geçiyor ömrümüz.

⌚ 🐢 It looks like we can move most of the daily template to links or commands. We can call it ref/daily. My idea is to fill the page intentionally, without templates.

🐇 It might be better to start with a blank page, yeah. The template makes me a bit nervous. There are too many things to fill in, and most of them aren’t things I like being pushed into doing.

🐢 Let’s start by moving the daily templates to ref/daily. No more daily template.

🐇 Now we can delete the rest of this page. We’ll use the daily links page and maybe have reminders at the end of these sessions.

👨🏽‍⚕️

#!/bin/bash

# Git pre-push hook to check if CHANGELOG.md is included in the push and if the branch is develop

# Get the current branch name

current_branch=$(git rev-parse --abbrev-ref HEAD)

# Check if the branch is develop

if [ "$current_branch" == "main" ]; then
	echo "You are on the main branch. Skipping CHANGELOG.md check."
	exit 0
fi

# remote="$1"
# url="$2"

# Get the list of commits to be pushed
commits=$(git rev-list '@{u}..HEAD')

has_rust_files=$(false)

# TODO: We can iterate to get file list only once
# Get the list of files that are going to be pushed
for commit in $commits; do
	if git diff-tree --no-commit-id --name-only -r "${commit}" | grep -q "\\.rs$"; then
		has_rust_files=$(true)
	fi
done

if [[ ! $has_rust_files ]]; then
	echo "No .rs files in the push, no need to check CHANGELOG"
	exit 0
fi

# Check if CHANGELOG.md is among the files in the commits
for commit in $commits; do
	if git diff-tree --no-commit-id --name-only -r "${commit}" | grep -q "CHANGELOG.md"; then
		echo "CHANGELOG.md is included in the push."
		exit 0
	fi
done

echo "ERROR: CHANGELOG.md is not included in the push."
exit 1

Edit (2025-02-08): Added a check for when only code files are changed.

I tried the same approach when testing Python bindings with Pytest:

def test_pipeline_step_dependency(empty_xvc_repo):
    dep_help = empty_xvc_repo.pipeline().step().dependency(help=True)
    expected = """
Usage: xvc pipeline step dependency [OPTIONS] --step-name <STEP_NAME>

Options:
  -s, --step-name <STEP_NAME>
          Name of the step to add the dependency to
"""
    assert dep_help == expected

This doesn’t work because the help text is generated by clap and skips the usual thread-based output handler. All command output and errors in Xvc are returned as strings from the command, except for the help text that’s generated by clap automatically.

There are probably workarounds for this, but I won’t pursue them further as I don’t test the actual functionality in the Python bindings anyway.

Add the following to your configuration:

vim.opt.conceallevel = 0

There are three types of Office add-ins: Task pane, Content, and Mail. These have different schema definitions at the top of the file. Consequently, it’s not possible to combine these three different types of Office add-ins. If an add-in needs mail permissions, it cannot work in Word, and if it requires document permissions, it cannot work in Outlook.

🐢 The tests for yesterday’s Git integration PR failed. I’ll begin by checking the logs.

🐇 Maybe run the tests locally to see. They might fail on your machine as well. It might be a simple thing.

🐢 Probably, yes. I’ve started the tests now. I shouldn’t forget to run them before testing the PR; it may save some time.

🐇 Ah, yeah, maybe. Yesterday you were already at the end of the workday, so it didn’t matter much. But you can shorten the testing time by reducing the number of files, etc. I think a separate benchmark suite might be good to have. Tests should be shorter; benchmarks should only run for tags.

🐢 Yup. I should shorten the tests. I should make them use a shorter list of files, maybe.

🐇 Local tests have passed. You’ll have to check the logs.

🐢 It seems git diff --name-only --cached returns files not yet added. That’s causing an error in new repositories with stash. Stashing shouldn’t run at all when there are no staged files.

🐇 Git behavior between the local version and the remote version seems different.

🐢 I think the trace should also show the Git version string.

🐇 You should also require a minimum Git version for the commands. You’re using some newer options, and not all users may have them.

🐢 Yep. Let’s see the version on the CI now.

🐢 Ah, I see—the problem is not about versions. It’s that the CI doesn’t configure git config --global user.email and user.name. Commits don’t work without these.

🐇 You should remove the Git version report, then. It’s one more process call for no reason.

🐢 I think it should be in trace!; I can check the verbosity level and call it only if it’s trace.

🐢 I completed the xvc-config docs as well. I think I can merge it before the tests finish, as it’s just a documentation update.

🐇 You seem to be rushing for the release.

🐢 Yeah, I want to really start testing this on the servers.

🐇 Then go ahead; let’s release a new version.

🐇 It looks like you’re back for an evening session. I think it’s time to start using Xvc on your torrent server.

🐢 Ah, yeah. Let’s see how it goes.

🐇 Create a repository for torrents. Then you can add files to it with cache-type = symlink or cache-type = hardlink.

🐢 I think creating a local storage may also help. I can use it to store files to retrieve them later.

🐇 Umm. We don’t have a “garbage collection” facility yet, you know. There are no file deletions at the moment. It may be better to have a repository-to-repository transfer feature, using SSH.

🐢 Yeah, we don’t have SSH storage either. I think this highlights a lack of features.

🐇 It’s possible to mimic Rsync storage with xvc storage new generic, but it doesn’t feel quite the same.

🐢 Then I’m adding these three tickets.

🐇 I think for 0.3.4, the command we add might be xvc file delete. You can work on this and add rsync support as well.

🐢 There is a librsync bindings library for Rust. But it looks like it doesn’t allow transferring file contents between hosts. There is also rusync, which is similar to rsync and implemented in Rust. There is also fast_rsync in pure Rust. It uses MD4 to calculate deltas, though, I think.

🐇 None of these seem to have network capability, though.

🐢 I think it’s better just to use the process for now. We are trying to come up with the simplest solution for now. We are just trying to be more general.

🐇 Yes, I think we can just use the process for the time being.

🐢 Then let’s start by adding it.

🐇 We should also begin to create release notes. GitHub can generate them from PRs. Automatically generated release notes

🐢 Created an issue for that. Now we’re starting xvc storage new rsync #111.

🐇 Ok. Let’s do this.

🐢 It looks like we don’t have many architectural problems.

🐇 You’re forgetting how Exec::cmd works, though. You have those kinds of problems.

🐢 Yeah, I’m figuring that out. Exec::shell requires a string to run on the shell, while Exec::cmd just needs a command name. You have to supply args with another function. I’m writing a closure now to handle this.

🐢 It started working, but it revealed a much bigger problem: .xvc/ is not added to .gitignore in xvc init.

🐇 Wow, that’s a showstopper.

🐢 Yup. I think I should fix it as well.

🐇 On closer glance, I think the problem is not xvc init not modifying .gitignore; it modifies it incorrectly. Maybe putting certain files on a whitelist is a better idea than trying to blacklist everything.

🐢 Yeah, we should define a set of Git-tracked files and directories, whitelist them, and let all other files be ignored.

🐇 Go ahead, then.

🐢 I think I’ve fixed it. There were two problems: the initial gitignore content was wrong, and it was placed in the root of the repository instead of .xvc.

🐇 Maybe putting it directly in the root is better than hiding it in .xvc. What do you think?

🐢 There seems to be an assumption in file track to have a .gitignore file somewhere. I think we should also handle changes in .gitignore files throughout the repository.

🐇 Umm, yes. We should handle .gitignore files as well. But not all .gitignore files are changed by Xvc. How can we make sure they were modified by Xvc?

🐢 I think there are ways to do that, like tracking them somewhere. But I believe we shouldn’t try. We should just get a list of .gitignore files, git add them, and include them in the commit.

🐇 Ok. Let’s write a test for this as well. No .gitignore should appear in git status -s.

🐢 I wrote the test. It fails now. Do you think we should check the output of Git status to determine which files to add?

🐇 That may be a good idea. Git status should already know which files need to be added. We can use that information.

🐢 It looks like pathspec is enough to modify git add behavior. We should be able to write *.gitignore and let all gitignore files be added.

🐇 Let’s try this manually.

🐢 Yep, it works. git add '*.gitignore' adds all .gitignore files in the subdirectories, too.

🐇 Congrats. 👏🥳

🐢 I’m writing the missing documentation for the crates. There are proc_macro not expanded errors all over the code.

🐇 I think you should update rust-analyzer and everything. There must be a command for this.

🐢 I reinstalled RA with :LspInstall and restarted the LSP. It seems to work correctly now.

Ah yeah, when you work for free, you can do so at any time you want, perhaps. You also don’t have team members, and that means when you sit in front of this, you can move it.

Umm, right. Let’s take a look at the outstanding PRs.

You have a documentation PR#93. You also have work that you’ve begun to integrate Git into. I think it’s better to focus on the latter today.

Right. Let’s think about the relationship between Git and Xvc. I believe we should identify a general relation to avoid ending up in a mess like DVC and Git.

Why do you think the DVC and Git relationship is a mess?

They don’t automate common Git operations like a commit after dvc add. There is only auto-stage, and that’s turned off by default. This makes it seem that DVC wants to intervene as little as possible with the user’s Git workflow. That’s understandable. I support this. But on the other hand, they use the .git/ directory itself to store and manage experiments in a custom way that creates custom stash objects for experiments. This is against the principle of minimum intervention.

So this makes it a mess?

The mess, in my opinion, is caused by the second factor. If DVC doesn’t perform any Git operations, that’s alright. It was intended to be VCS-agnostic. Then experiments came and used Git internals in a way that no other similar tool uses.

Git-LFS and Git-Annex seem to use some non-standard mechanisms as well.

Ok. Not no other tool uses, but in a way that no other tool has used.

You know, GitHub PRs are also stored in a similar way. They also use non-standard machinery.

Yeah, but these tools are all Git-specific tools. They accept the dominion of Git, and don’t try to bring any VCS-agnosticism.

And Xvc tries to have this agnosticism?

I believe the initial design of DVC, which aims to be VCS-agnostic or being able to run without a VCS, is valuable. I like the idea behind Git, but the interface and implementation show that it’s a gradual development. There is no library behind it.

Libgit?

Libgit2 is something different. Although it’s said to have some common code, it doesn’t support all features. Git is command-line software with a mix of scripts and compiled executables, and not all code seems to be written in a way that could be used by external tools.

Hmm. The comment you added to the issue says git stash push --staged is not available in libgit2. Can’t you mimic it like DVC does for branches?

I don’t want to depend on Git at that level.

So, you’ll be using the CLI and shell for Git?

Yes, I believe, at the moment, before any performance tests, that this doesn’t matter much. Running Git commands once in a while using the shell shouldn’t make much difference in overall performance.

Then you’ll use it like a command-line tool, like the user?

Yes, and I’ll make it run outside of the usual threads. All Git will be like a sandwich, wrapping around Xvc operations. If there are --git-ref instructions in an xvc command, it will be run before Xvc performs the command, and if there are any changes in Xvc metafiles, they will be committed to the current branch.

Like

graph LR

co["git checkout"] --> xvc
xvc --> cm["git commit"]

The first could be a branch as well. So we have:

graph LR

br["git branch"] --> xvc
co["git checkout"] --> xvc
xvc --> cm["git commit"]

Looks sensible. How will you reflect these in the command line?

With something like xvc --git-checkout my-branch file list

Hmm, and for a branch?

I think instead of different options for branch, checkout or tag, we can have a git-ref option that marks the option as a git reference. It will be checked out, or created as a branch from the current one if it doesn’t exist.

I think creating a branch is not a good idea. It should be explicit. You can just send the --git-ref value to git checkout and perform the Xvc operation. If the user wants to create a branch, I think they can do it themselves.

What about storing the results in a branch? After adding a bunch of files, they may want to store them in another branch, maybe?

That’s sensible. We can have another option, like --to-branch in certain operations.

Or in the xvc command as a general option. In that case, we can change the option names to --from-ref and --to-branch. It will be like:

graph LR

fr["git checkout $(--from-ref)"] --> xvc
xvc --> tb["git checkout --branch $(--to-branch)"]
tb --> co["git add .xvc && git commit -m 'xvc cmd'"]

If no such options are given, xvc will run without branching, right?

Yep. --from-ref and --to-branch options are just shortcuts for user behavior. Any other VCS tool could be used this way. We don’t need to integrate Git at the library level.

This brings up the question of portability, though. When you aim for the software to be portable, you can’t rely on the existence of Git on the host, right?

I think a git.command option in the configuration is a good idea. Xvc will issue a warning if it can’t run the commands.

Will you use the shell to run this command? Otherwise no $PATH configuration is possible, you know.

I believe that could be another option: git.use_shell. If git.command is set to an absolute path, Xvc may use it without the shell. Otherwise, it can use the shell. Running the process directly will make it faster and more secure.

There is also this option to run Xvc in another process. Because we may access Git in the shell that runs Xvc, and if we can access it, maybe we don’t need shell execution in the process.

That’s a cool idea. But I wouldn’t add that extra complexity. Instead, we can try to find the git executable if git.command is not an absolute path. If git.command = /usr/bin/git in the configuration, we use it as is. Otherwise, we can get $PATH or %PATH% from the environment and search for git.command in that to find the exact executable.

It looks like there is a crate called which that does exactly what we are looking for.

Ah, cool. Then we can just use that to find the executable and run it. We don’t need to drop to a shell.

Yep. Let’s go back to implementation now.

🐢 I’ve merged PR#66. Let’s list the outstanding issues now.

65	OPEN	Add Mermaid Support to Netlify Installation
64	OPEN	Fix `xvc-storage` compilation warnings	bug
54	OPEN	Fix all references to `xvc data` in the documentation
51	OPEN	Add a benchmark script to compare Xvc with other tools
50	OPEN	Consider changing `xvc file push` to `xvc file send` and `xvc file pull` to `xvc file retrieve`
49	OPEN	Add `--generic-command` as a dependency type to `xvc pipeline dependency`
48	OPEN	Write rsync example for `xvc storage generic`	documentation
47	OPEN	Write rclone example for `xvc storage generic`	documentation
46	OPEN	Allow to skip init in remotes with `--skip-init` option
45	OPEN	Update `VStore::to_store` to use some internal mechanism to avoid `XvcStore::insert`
43	OPEN	Update to `clap 4.0`
42	OPEN	Clean up xvc crate dependencies
36	OPEN	Add version to released file binaries
33	OPEN	Update `arch/remotes.md` for new naming	documentation
32	OPEN	Remove arc42 sections from the documentation	documentation
29	OPEN	Add documentation for `xvc storage new gcs`	documentation
28	OPEN	`xvc storage new yandex`	enhancement
24	OPEN	Create a logo for Xvc	documentation
23	OPEN	Add a Github action to upload new versions to crates.io	automation
20	OPEN	ref: Add rsync example for `xvc storage new generic`
18	OPEN	Add a new workflow for remote tests	automation
12	OPEN	Create a website for Xvc
5	OPEN	Add storage tests to Github Actions	automation
4	OPEN	fix clippy warnings	bug
1	OPEN	`xvc storage new` for all S3 compatible cloud services supported by `rust-s3`

🐇 As far as I know, you removed the arc42 stubs from the docs yesterday. You can close issue #32.

🐢 I closed it via gh issue close.

🐇 You should add a comment to that.

🐢 Added the comment.

🐇 It’s October 30th. Looking at the logs, you’ve closed some more issues since we last talked.

🐢 I’ve been productive over the last few days. Now, let’s take a look at the open issues.

🐇 I think you have a more pressing problem. Your tests are failing. You need to fix them first before moving to another task.

🐢 Created a new issue. The logs say:

81Z error: 7 targets failed:
2022-10-29T18:18:36.8969467Z ##[error]    `-p xvc-workflow-tests --test test_storage_new_digital_ocean`
2022-10-29T18:18:36.8970697Z     `-p xvc-workflow-tests --test test_storage_new_gcp`
2022-10-29T18:18:36.8971279Z     `-p xvc-workflow-tests --test test_storage_new_generic_rsync`
2022-10-29T18:18:36.8971840Z     `-p xvc-workflow-tests --test test_storage_new_minio`
2022-10-29T18:18:36.8972387Z     `-p xvc-workflow-tests --test test_storage_new_r2`
2022-10-29T18:18:36.8973035Z     `-p xvc-workflow-tests --test test_storage_new_s3`
2022-10-29T18:18:36.8973577Z     `-p xvc-workflow-tests --test test_storage_new_wasabi`
2022-10-29T18:18:36.9368079Z ##[error]The process '/home/runner/.cargo/bin/cargo' failed with exit code 101

🐇 Looking at the logs, you should clean up those older branches.

🐢 Let’s do it now.

🐢 Done. I’ve deleted all branches except main.

🐇 Good. I think the best way is to delete them as soon as you merge them.

🐢 I’ve activated that setting. These were older branches.

🐇 Cool. Now, will you be checking the failing tests?

🐢 I think remote tests should never run if there are other errors. We can run coverage and remote tests as a second step after the first succeed.

🐇 That seems like a neat idea. You want to split the current job into two: one for compiling and testing the non-remote parts, and the second for coverage and storage tests. It won’t waste CI minutes that way?

🐢 On second thought, I think at the moment, that’s not a pressing issue. We should start fixing these tests at once. We can split them later.

🐇 Ok. It looks from the logs like secrets are not being made available to your jobs.

2022-10-29T18:18:17.5039311Z test test_storage_new_digital_ocean ... FAILED
2022-10-29T18:18:17.5039550Z
2022-10-29T18:18:17.5039652Z failures:
2022-10-29T18:18:17.5039784Z
2022-10-29T18:18:17.5040197Z ---- test_storage_new_digital_ocean stdout ----
2022-10-29T18:18:17.5040558Z Error: VarError { source: NotPresent }
2022-10-29T18:18:17.5040758Z
2022-10-29T18:18:17.5040765Z

🐢 I added them now. Let’s wait until the job ends to get a new set of logs.

🐇 You can write some documentation in the meantime.

🐢 I think #82 is a good candidate for this. There must not be too many missing docs in the ECS crate.

🐇 It was about walker and you fixed the ecs. You’re the most absentminded developer here, I believe.

🐢 Ooops, you’re right. I’ll add them together in a single PR.

🐇 In the meantime, the storage testing job has ended with 🔴.

🐢 Checking the raw logs. It looks like we didn’t update the tests to match the current options:

2022-10-30T14:45:50.1058348Z error: Found argument '--storage-prefix' which wasn't expected, or isn't valid in this context
2022-10-30T14:45:50.1059809Z ##[error]Found argument '--storage-prefix' which wasn't expected, or isn't valid in this context
2022-10-30T14:45:50.1061740Z 	If you tried to supply `--storage-prefix` as a value rather than a flag, use `-- --storage-prefix`
2022-10-30T14:45:50.1063220Z
2022-10-30T14:45:50.1063394Z USAGE:
2022-10-30T14:45:50.1064302Z     xvc storage new digital-ocean --name <NAME> --bucket-name <BUCKET_NAME> --region <REGION>

🐇 Is it --storage-prefix or --remote-prefix? Which one is clearer?

🐢 I think updating the tests to conform to the current options is better for now. We can update the options later if desired.

🐇 Another failure is rg. You assume it exists on the testing system.

🐢 ripgrep is available in Ubuntu 20.04, so we can just update the initial package list.

🐇 The same --storage-prefix failure appears in the minio tests.

🐢 Ok. Fixing it.

🐇 S3 tests want to run the new-s3 subcommand. I think we now see why we need these tests in the first place.

🐢 Yeah. Fixing the prefix option, too.

🐇 s3cmd is also required. You should add it, too.

🐢 Ok. I think we’ve added all missing dependencies. I’ll have to convert the mc tests for Minio to use s3cmd.

🐇 Then we can try again. Now, we can get back to documentation.

🐢 Added some more documentation to PR#93. I think it’s better to return to the storage test errors now.

🐇 It looks like you have missing DigitalOcean credentials.

🐢 Let’s take a look.

🐢 I’ve updated the tests to use a config file instead of command-line arguments that would be visible in the logs.

🐇 Good practice. You also need to remove previous logs.

🐢 Maybe there is an option for that.

🐇 It looks like there isn’t. There are masking options; when we add them to secrets, they are masked. But you shouldn’t use them in calls anyway.

The Cargo.toml file can have a [features] section. In this section, features can be defined, and they can enable other features.

[features]
my-feature = []
another-feature = ["my-feature"]

These markers are used in the code as follows:

#[cfg(feature = "my-feature")]
pub mod my_module;

The default features are listed under the default key of the [features] section. If it’s not defined, it’s considered empty, so cargo build supplies no features to the build system by default.

If you don’t want default features added automatically, you can use the --no-default-features option with cargo.

Starting from Rust 1.60, dependencies can be tied to features. When you define a dependency in the [dependencies] section, add optional = true to its options. Then, in [features], use the dep:package syntax.

[dependencies]

pack = { version = "1.0", optional = true }

[features]

my-feature = ["dep:pack"]

This way, Cargo doesn’t compile pack if my-feature is not enabled.

A caveat to using features is that they should be additive. There shouldn’t be a no-std feature to remove the standard library dependency; instead, you should define a std feature and enable it by default. The reason behind this is that Cargo ensures there is a single copy of a package with all the requested features enabled for the entire dependency graph. If you have a no-x feature that removes functionality, it breaks the additive logic of Cargo’s feature resolution.

I was using burntsushi’s ignore crate, which in turn uses globset by the same author. In a directory hierarchy, I was first collecting all .ignore files, then trying to decide which files are ignored. Ripgrep’s from parameter led me to believe this.

However, it looks like it has a bug that doesn’t consider the source directory of the ignore file. Git has different semantics when a/.gitignore and a/b/.gitignore have a line mydir/. In the second case, Git doesn’t ignore a/mydir/, but Ripgrep seems to consider all elements from all files as if they are from the root directory.

In practice, most people may be using ignore files in their root directory, but there are valid use cases (like moving around directories with ignore patterns in them) that are affected by such behavior.

I decided to tackle this. Ripgrep’s globset seems nice and fast, so it was better to base the solution on top of that. My basic use case is walking a directory hierarchy. There are solutions to this like walkdir or jwalk, but they use ripgrep’s gitignore; hence, they have the same bug.

I usually begin by setting up data structures to understand the problem. This allows me to understand the domain formally.

First, what are we trying to accomplish? Basically, we want to decide if a path should be ignored (a) in the results and (b) in the directory traversal. The gitignore specification not only allows ignores, but also has a way to add paths to a whitelist to unignore. Hence, deciding whether to ignore a path is like:

let include = |path| {
    if whitelist_globs.match(path) {
       true
    } else if ignore_globs.match(path) {
       false
    } else {
       true }
}

We also need to decide whether to traverse a directory:

let traverse = |path| {
    if path.is_dir() && !include(path) { false } else { true }
}

We have the following possible MatchResults between the ignore rules and a path:

pub enum MatchResult {
    NoMatch,
    Ignore,
    Whitelist,
}

We need a comparison function to get a MatchResult. One side of the comparison is path, and the other is a set of GlobSets from the globset crate. We’ll have two globsets: one for ignores and the other for whitelists.

pub struct IgnoreRules {
    root: PathBuf,

    patterns: Arc<Vec<Pattern<Glob>>>,

    whitelist_set: Arc<GlobSet>,
    ignore_set: Arc<GlobSet>,
}

The data types for the sets are Arc<GlobSet>, as I planned to make the traversal concurrent for each directory. However, as only one thread will be modifying the rules (one .gitignore or .ignore file per directory), I felt no need to wrap the sets with Mutex.

We also keep track of each pattern separately. patterns is a Pattern vector that contains all globs individually. One reason is to rebuild the GlobSet as we traverse directories: since we cannot add any more rules to the sets once they have been built, we keep the patterns separately to add them to the sets of the child directories.

Another reason is to keep track of the source, as this can be used for debugging or reporting, similar to git check-ignore.

root is the point where this traversal began.

Patterns are defined in a generic way to be instantiated with globs, strings, regexes, or any other type.

pub struct Pattern<T> {
    pattern: T,
    original: String,
    source: Source,
    effect: PatternEffect,
    relativity: PatternRelativity,
    path_kind: PathKind,
}

The rationale for this generic approach is that the patterns need some kind of transformation. They can be read from files as strings, then compiled into globs or regexes.

flowchart LR
     String-->Ok(Glob)-->Glob-->Regex

The source field shows where this pattern was retrieved. It’s an enum with two possible values for the time being:

enum Source {
    File { path: PathBuf, line: usize },
    Global,
}

The gitignore specification says:

An optional prefix “!” which negates the pattern; any matching file excluded by a previous pattern will become included again.

We use the effect field to tag the pattern as either an Ignore or a Whitelist pattern.

pub enum PatternEffect {
    Ignore,
    Whitelist,
}

The spec also makes a distinction between files and directories:

If there is a separator at the end of the pattern then the pattern will only match directories, otherwise the pattern can match both files and directories.

So we use path_kind to reflect this distinction:

pub enum PathKind {
    Any,
    Directory,
}

Some patterns can be relative to the directory they are found in:

If there is a separator at the beginning or middle (or both) of the pattern, then the pattern is relative to the directory level of the particular .gitignore file itself. Otherwise the pattern may also match at any level below the .gitignore level.

And we show this with the relativity field:

pub enum PatternRelativity {
    Anywhere,
    RelativeTo { directory: String },
}

Note that we are converting from the specification to Rust types. In the future, these enums can be extended to cover cases where these patterns are used in other places, or we can merge some of them to reduce complexity. Currently, this level of abstraction seems adequate.

To convert Pattern<String> to Pattern<Glob> (or Pattern<Result<Glob, Error>>), or any other type, we can use a map function similar to Option::map:

    fn map<U, F>(self, f: F) -> Pattern<U>
    where
        F: FnOnce(T) -> U,
    {
        let pat = Pattern::<U> {
            pattern: f(self.pattern),
            original: self.original,
            source: self.source,
            effect: self.effect,
            relativity: self.relativity,
            path_kind: self.path_kind,
        };
        pat
    }

map will make it easy to convert between various types of patterns.

Now, we change the perspective to a higher level. Let’s begin by writing a file tree walker.

pub fn walk_serial(
    given: IgnoreRules,
    dir: &Path,
    walk_options: &WalkOptions,
    sender: &Sender<WalkerResult<PathMetadata>>,
) -> WalkerResult<()> { todo!() }

This function uses the given IgnoreRules and returns files and directories via the channel. PathMetadata is just a bundle of PathBuf and its Metadata.

pub struct PathMetadata {
    path: PathBuf,
    metadata: Metadata,
}

WalkOptions will contain various options about this traversal. Currently, it only has ignore_filename and include_dirs, which set whether only files are sent through the channel, or if directories should be included too.

pub struct WalkOptions<'a> {
    ignore_filename: Option<&'a str>,
    include_dirs: bool,
}

Initially, IgnoreRules contains only the files that should be ignored (or whitelisted) globally. When a directory contains a .gitignore file (or any file name that was set in WalkOptions), it checks for new ignore rules and sends all files through the channel. If a directory is not ignored, it then calls walk_serial recursively with it.

The final lines of the function are as follows:

    for child_dir in child_dirs {
        walk_serial(dir_with_ignores.clone(), &child_dir, walk_options, sender)?;
    }

As it calls itself as a final operation after collecting and sending all child files, this is an example of tail recursion.

The output is sent through a channel because our final goal is to make all this concurrent. Each new directory will create a new thread, and all files will be collected from the channel. At this point, we are creating a serial version of the walker with identical data structures to test it easily.

One gotcha using the channels in the serial version is that the channel size must be enough to contain all file names. Effectively, this means it will be crossbeam_channel::unbounded for walk_serial. We will (ab)use the channel’s Sender<PathMetadata> as a Vec<PathMetadata> in no parallel mode. In parallel settings, using unbounded channels is not recommended; if the consumers of the channel don’t work as quickly, you may fill up the memory, especially for longer-running processes.

walk_serial will basically do four things:

• List all the elements in the given directory.
• Check if there is a new ignore file and update the rules if necessary.
• Filter the elements by ignore rules and send them to the channel.
• Run walk_serial recursively for child directories if they are not ignored.

Read the directory

The first is done using read_dir of Path. It returns a set of elements. Then the list is processed, and elements’ metadata is retrieved. This requires a bit of error handling: read_dir may return an Error, each element in the result can be an Error, and getting metadata may also result in an Error.

    let elements = dir
        .read_dir()
        .map_err(|e| anyhow!("Error reading directory: {:?}, {:?}", dir, e))?;
    let mut child_dirs = Vec::<PathBuf>::new();
    let mut child_paths = Vec::<PathMetadata>::new();

    for entry in elements {
        match entry {
            Err(err) => sender.send(Err(WalkerError::from(anyhow!(
                "Error reading entry in dir {:?} {:?}",
                dir,
                err
            ))))?,
            Ok(entry) => match entry.metadata() {
                Err(err) => sender.send(Err(WalkerError::from(anyhow!(
                    "Error getting metadata {:?} {}",
                    entry,
                    err
                ))))?,
                Ok(md) => {
                    child_paths.push(PathMetadata {
                        path: entry.path(),
                        metadata: md.clone(),
                    });
                    if md.is_dir() {
                        child_dirs.push(entry.path());
                    }
                }
            },
        }
    }

Note that we collect PathMetadata objects in the child_paths vector. If there happens to be an error while retrieving, it’s sent to the caller to ignore or to report to the user.

I usually handle errors first in match expressions because they are usually more straightforward.

Check if there are new ignore rules

After reading all the files in the directory, it checks if there actually is an ignore filename in given and that a file exists with that name in the list we just created. If there is no such file, we can use given rules to decide the ignored elements.

If there is a file that may change the ignore rules, it reads it and updates patterns. Then, if there are new ignore or whitelist rules, the respective globsets are recreated. It checks whether we have new rules because compiling globs is possibly an expensive operation.

Filtering directory elements

Filtering files and directories from the list checks two globsets. If an element matches a whitelist rule, it’s included in the results without checking if it also matches ignore rules. Hence, whitelisting has a priority. Otherwise, it checks the ignore rules, and if they don’t match, the element is included in the results.

If there are directories matching the ignore rules, they are not traversed, and any possible whitelisting rule that was specified within that directory is not taken into consideration.

Listing child directories

We saw how walk_serial calls itself for child directories above. After sending all results to the channel, it calls itself for the child directories and repeats.

Converting patterns from String to Glob

The interesting part about ignoring files is that we should modify patterns into globs to add to a GlobSet. We cannot add the patterns directly to a GlobSet because some of the rules make them context dependent. We parsed the patterns to get PatternRelativity and PathKind, and these also affect how we add the patterns to a GlobSet.

The heart of the conversion is a function that receives a String, filters it, and transforms it to a list of Pattern<Glob>s.

fn content_to_patterns(
    ignore_root: &Path,
    source: Option<&Path>,
    contents: &str,
) -> Vec<Pattern<WalkerResult<Glob>>> {

As we also need to report the line number for patterns, we use enumerate on the iterator:

    let patterns: Vec<Pattern<WalkerResult<Glob>>> = content
        .lines()
        .enumerate()
        // A line starting with # serves as a comment. Put a backslash ("\") in front of the first hash for patterns that begin with a hash.
        .filter(|(_, line)| !(line.trim().is_empty() || line.starts_with("#")))

The filter line is used to check for blank and comment lines. Then we trim the trailing space unless the pattern ends with \.

        .map(|(i, line)| {
            if !line.ends_with("\\ ") {
                (i, line.trim_end())
            } else {
                (i, line)
            }
        })

The function receives the source file name as a parameter. We create the Source using it and the line number:

        .map(|(i, line)| {
            (
                line,
                match source {
                    Some(p) => Source::File {
                        path: p
                            .strip_prefix(ignore_root)
                            .expect("path must be within ignore_root")
                            .to_path_buf(),
                        line: (i + 1).into(),
                    },
                    None => Source::Global,
                },
            )
        })

Then we build the pattern object, update the pattern to add it before the globset, and build the globs. Building the glob may cause errors if it’s not well-formed. They are handled as well.

        .map(|(line, source)| build_pattern(source, line))
        .map(transform_pattern_for_glob)
        .map(|pc| pc.map(|s| Glob::new(&s).map_err(WalkerError::from)))

Two functions are left that may need attention. One is build_pattern that converts a pattern line to a Pattern<String> object. The other is the transform_pattern_for_glob function that modifies Pattern<String> before building the glob.

After writing walk_serial, we’ll use all the machinery to write a parallel version of it.

Parsing pattern strings for pattern objects

We create pattern objects from pattern strings following the spec. There are three basic rules that we will follow:

• If the pattern starts with !, it’s a whitelist rule.
• If the pattern ends with /, it matches only to directories.
• If the pattern contains a non-final /, it’s relative to the directory.

We’ll also check whether the line starts with / and remove that to prevent double-slashes // in patterns. The resulting pattern in the object won’t contain any of the artifacts that change its semantics, and we’ll add them when building the globsets.

Let’s start by checking whether the line starts with !.

    let begin_exclamation = original.starts_with("!");
    let line = if begin_exclamation || original.starts_with("\\!") {
        original[1..].to_owned()
    } else {
        original.to_owned()
    };

Checking the presence of / at the beginning, at the end, and in intermediate positions is also straightforward:

    let end_slash = line.ends_with("/");
    let line = if end_slash {
        &line[..line.len() - 1]
    } else {
        line
    };

    let begin_slash = line.starts_with("/");
    let non_final_slash = if line.len() > 0 {
        line[..line.len() - 1].chars().find(|c| *c == '/').is_some()
    } else {
        false
    };

We check the non-final slash separately from the beginning slash because we’ll remove the initial slash if it exists.

    let line = if begin_slash { &line[1..] } else { line };

Although there is almost 1-1 correspondence between the boolean conditions and the enums, I create enums in separate statements for self-documenting code.

    let effect = if begin_exclamation {
        PatternEffect::Whitelist
    } else {
        PatternEffect::Ignore
    };

    let path_kind = if end_slash {
        PathKind::Directory
    } else {
        PathKind::Any
    };

    let relativity = if non_final_slash {
        PatternRelativity::RelativeTo {
            directory: current_dir.to_owned(),
        }
    } else {
        PatternRelativity::Anywhere
    };

After building these attributes of the pattern and stripping the pattern from the initial and final /, and other semantic-changing elements, we build a pattern:

    let pattern = Pattern::<String> {
        pattern: line.to_owned(),
        original: original.to_owned(),
        source,
        effect,
        relativity,
        path_kind,
    };

Transforming a pattern string to a suitable glob

Now, another transformation is needed to convert Pattern<String> to Pattern<Glob>. In this case, we’ll need to recreate the pattern based on the rules.

The transformation considers relativity and path_kind to generate a glob that can be added to a GlobSet. Note that the function returns not a compiled Pattern<Glob>, but a Pattern<String>, because after this transformation, building the glob requires Glob::new.

fn transform_pattern_for_glob(pattern: Pattern<String>) -> Pattern<String> {
    let anything_anywhere = |p| format!("**/{p}");
    let anything_relative = |p, directory| format!("{directory}/**/{p}");
    let directory_anywhere = |p| format!("**{p}/");
    let directory_relative = |p, directory| format!("{directory}/**/{p}/");

    let transformed_pattern = match (&pattern.path_kind, &pattern.relativity) {
        (PathKind::Any, PatternRelativity::Anywhere) => anything_anywhere(pattern.pattern),
        (PathKind::Any, PatternRelativity::RelativeTo { directory }) => {
            anything_relative(pattern.pattern, directory)
        }
        (PathKind::Directory, PatternRelativity::Anywhere) => directory_anywhere(pattern.pattern),
        (PathKind::Directory, PatternRelativity::RelativeTo { directory }) => {
            directory_relative(pattern.pattern, directory)
        }
    };

    Pattern {
        pattern: transformed_pattern,
        ..pattern
    }
}

I could merge the closures into the match and lower the number of lines, but adding the closures increased self-documentation. Instead of multiple if/else branches, I tend to use match with tuples to have a clearer picture. The corresponding if/else code without closures would be like:

let transformed_pattern = if pattern.path_kind == PathKind::Any {
    if pattern.relativity == PatternRelativity::Anywhere {
        format!("**/{p}")
        } else if ...
        }
...

And in my opinion, this latter format is more error-prone and less readable.

Combining it all together

At this point, we can revisit content_to_patterns again:

fn content_to_patterns(
    ignore_root: &Path,
    source: Option<&Path>,
    content: &str,
) -> Vec<Pattern<Result<Glob>>> {
    watch!(source);
    let patterns: Vec<Pattern<Result<Glob>>> = content
        .lines()
        .enumerate()
        ...
        .map(|(line, source)| build_pattern(source, line))
        .map(transform_pattern_for_glob)
        .map(|pc| pc.map(|s|
            Glob::new(&s).map_err(Error::from)))
        .collect();

    patterns
}

The last step in the transformation is creating the Glob object, as in the highlighted lines above. It runs Pattern<String>.map with Glob::new to get a Pattern<Result<Glob>>. Error and Result are custom types defined within the crate using the thiserror crate.

After we have a vector of Pattern<Result<Glob>>, we report the errors in clear_glob_errors to get Vec<Pattern<Glob>> from this vector. The function is defined as:

fn clear_glob_errors(
    sender: &Sender<Result<PathMetadata>>,
    new_patterns: Vec<Pattern<Result<Glob>>>,
) -> Vec<Pattern<Glob>> {
    let new_glob_patterns: Vec<Pattern<Glob>> = new_patterns
        .into_iter()
        .filter_map(|p| match p.transpose() {
            Ok(p) => Some(p),
            Err(e) => {
                sender
                    .send(Err(Error::from(anyhow!("Error in glob pattern: {:?}", e))))
                    .expect("Error in channel");
                None
            }
        })
        .collect();
    new_glob_patterns
}

The sender is the same channel that we use to send the results. This communication channel is used to report only the errors, not PathMetadata; hence, when we encounter an Err(e) in new_patterns, it’s reported to the same channel that the results are expected. Handling the errors first seems like a common theme here as well.

This is also one of the reasons walk_serial receives a channel instead of a vector. Our usual use case is traversing all the directories in parallel, and without channel-based communication in the serial version, we would need to duplicate this function.

At this point, I completed the walk_serial functionality. I wrote tests. I’m using the test-case crate for writing many tests encompassing different use cases. I’m omitting them here for brevity.

Writing a Parallel Walker

Converting walk_serial to a parallel one is quite trivial. As we already used channels in the serial version, we have all the components of a parallel walker. For the sake of simplicity, I won’t use a thread pool and instead spawn a new thread for each task. One question is what’s the unit task that we’ll use to create tasks.

jwalk uses directories as points to spawn new tasks. That’s a sensible approach, and I’ll use this as well. This means a single directory will be processed by a single thread, even if it contains one million files, and 10 directories will be processed by 10 threads even if each contains one file. In practice, I think we can assume directories are natural chunks for this problem.

The interface for the parallel walker is identical to the serial version. This is to replace one with the other when the need arises.¹

The only difference is in the final loop. walk_serial calls itself recursively at the end, while walk_parallel calls itself in separate threads.

pub fn walk_parallel(
    given: IgnoreRules,
    dir: &Path,
    walk_options: WalkOptions,
    sender: Sender<Result<PathMetadata>>,
) -> Result<()> {
    ....
    crossbeam::scope(|s| {
        for child_dir in child_dirs {
            let dwi = dir_with_ignores.clone();
            let walk_options = walk_options.clone();
            let sender = sender.clone();
            s.spawn(move |_| walk_parallel(dwi, &child_dir.path, walk_options, sender));
        }
    })
    .expect("Error in crossbeam scope in walk_parallel");

    Ok(())
}

Code

The code is soon to be released, and I’ll add a link to it. You can check my GitHub profile if I forget.

1. Update: I decided to change the walk_serial interface to use Vec instead of Sender for simplicity. ↩

class Square extends Shape {}

class Circle extends Shape {}

class Shape {
  Shape();

  factory Shape.fromTypeName(String typeName) {
    if (typeName == 'square') return Square();
    if (typeName == 'circle') return Circle();

    throw ArgumentError('I don\'t recognize $typeName');
  }
}

It warns about highly skilled developers:

“These kinds of high performers are actually low performers when TCO is factored in. Unless you’re a startup where time to market is the highest priority, keep these kinds of developers under close scrutiny with extensive design and code reviews.”

and says continual improvement should be an innate element of all software building practices.

“Continual improvement isn’t a stand alone project. If you make it a stand alone project you’ll eventually find a reason to abandon it entirely because it’s impacting deadlines. It should be something developers are tasked to do all the time during their normal development activities. This is not unlike the power of compound interest.”

In my experience, the following holds too:

“Some employees aren’t aware of the impact they have on other team members and interrupt them frequently with one quick question or worse. Encourage your team to prefer communication in roughly the following order so that interruptions are minimized: e-mail, chat room (if your team isn’t using a chat room yet, they should be), instant message, phone call/dropping by in person.”

But in general, I would not expect a crisis in software development. Software is not like a building or a tool; it’s rather like an orchard you gather value from over time. It surely needs maintenance, but the cost of maintenance and the value it provides should be considered together.

I believe that most errors are preventable in software development, but the (communication/technical/financial) cost of perfect software is usually higher than the value it provides. It’s true that we should aim for particular practices that reduce such cost, like not interrupting team members when they are working, but striving for perfect software does not automatically bring it.

What brings it? Receiving and listening to feedback. A software developer should be open to feedback, be it from the tests they write or users’ bug reports. Our foremost aim should be opening as many feedback channels as possible, in the form of unit tests, integration tests, and shipping results to the users frequently. We can only evaluate the technical debt on top of this feedback.

Bir dizindeki tüm dosyalar: ls *

Bulunduğumuz dizin ve tüm alt dizinlerdeki dosyalar: ls **/*

Son iki haftada değiştirilmiş dosyalar: ls **/*(.mw-2)

Boyutu 100 MB’tan büyük olan dosyalar: ls **/*(.Lm+100)

Sadece okunabilir olan dosyalar: ls **/*(.R)

/etc dizininde dünya tarafından yazılabilir olan dosyalar: ls /etc/**/*(.W)

/etc dizininde son bir haftada değiştirilmiş ve dünya tarafından yazılabilir olan dosyalar: ls /etc/**/*(.Wmw-1)

zsh’in sağladığı bu gibi seçeneklerin yanında dosya adını parçalamak da kolay. Örneğin dosyanın uzantıdan önceki kısmını almak için *(:r) yazıyoruz. Dizindeki jpg dosyalarını ImageMagick’in convert programıyla aynı isimli png dosyalarına çevirmek şöyle mümkün:

for f in **/*.jpg(.) ; do
   convert ${f:r}.jpg ${f:r}.png
done

Burada anlatılanları find yoluyla yapmak da mümkün ancak çok daha uzun sürebiliyor. Bilhassa dosya adlarını parçalamak, dönüştürmek gibi işlemleri yaparken zsh’in nimetleri çok. Tek satırlık bir operasyonla çok güçlü bazı dönüşümler yapabilmek mümkün.

Previously with Pandoc, I was using a simple setup to create RSS feeds. Markdown files were converted to plain, headerless HTML, and they were collected together to build an XML file. The obvious drawback is that all HTML files should be generated by Pandoc; anything that doesn’t fit that route does not appear in the feeds.

However, when I began to use Org Mode for data analysis and other tasks, I stopped using Pandoc. Org Mode has extensive facilities for exporting into HTML and other document formats, so I would not mess with Pandoc for this.

I thought RSS could be produced by parsing HTML files after they are produced. This requires parsing the HTML file, but it’s simple, and there are parsers for all programming languages out there. My previous RSS generator was in Python, and I decided to modify it to fit my needs. I think producing RSS for a static HTML site is a common need, and I tried to solve this problem as simply as possible.

Let’s begin with the ubiquitous shebang line. This tells the system that the script is in Python.

#!/usr/bin/env python

The following are the imports for this script. Apart from PyRSS2Gen, all modules are present in Python 2.7.

import argparse
import codecs
import os
import datetime
from HTMLParser import HTMLParser
import PyRSS2Gen as rssgen
import operator as op
import re
import subprocess as proc

I use Mercurial to track the site’s files. I once thought about using the Mercurial public API to check the status of files, but it proved to be overkill because only the modification time of files is necessary, and retrieving them using a standard command-line call is much simpler. Hence, I removed the following imports for the time being.

# from mercurial import commands as cmd
# from mercurial import hg
# from mercurial import ui as hgui

The following function returns a valid HTML tag string, given the tag and its attributes in a list. HTMLParser sends the tags in a list form, and I use this function to reconvert them to usual HTML tags.

def make_tag(tag, attrs):
    content_list = [ tag ]
    content_list += [ "%s=\"%s\"" % (k, v) for (k, v) in attrs]
    return "<" + " ".join(content_list) + ">"

TitleBodyExtractor is an HTMLParser subclass. It collects the body of a page in a string and also keeps the title. These two are the only requirements. It might be possible to parse meta tags to get publish date and author information as well, but I prefer to keep simple things simple.

class TitleBodyExtractor(HTMLParser):

    def __init__(self):
        HTMLParser.__init__(self)
        self.in_body = False
        self.in_title = False
        self.body = ""
        self.title = ""


    def handle_data(self, data):
        if self.in_body:
            self.body += data
        if self.in_title:
            self.title += data

    def handle_starttag(self, tag, attrs):
        if self.in_body:
            self.body += make_tag(tag, attrs)

        if tag == "body":
            self.in_body = True
        if tag == "title":
            self.in_title = True

    def handle_endtag(self, tag):
        if tag == "body":
            self.in_body = False

        if tag == "title":
            self.in_title = False

        if self.in_body:
            self.body += "</%s>" % (tag)

Getting contents of a file in UTF-8 encoding is a common task. The following two functions retrieve and store the contents in UTF-8 using the codecs module.

def get_content(filename):
    f = codecs.open(filename, "r", "utf-8")
    cont = f.read()
    f.close()
    return cont

def write_content(filename, content):
    f = codecs.open(filename, "w", "utf-8")
    f.write(content)
    f.close()

Mercurial allows running commands for a repository outside of that repository with the -R command-line switch. However, it requires the exact path of the repository and does not accept a child path. The following function finds the repository path of a file by recursively checking whether parent paths contain an .hg/ directory.

def get_repo_path(dir):
    if dir == "/" or dir == "":
        return ""
    if os.path.exists(os.path.join(dir, ".hg")):
        return dir
    else:
        return get_repo_path(os.path.dirname(dir))

The FileObject class keeps the required data of an HTML file. It stores the path, modification time, body, and title.

class FileObject:
    def __init__(self, path, mtime):
        self.path = path
        self.mtime = int(mtime)
        self._body = ""
        self._title = ""

    def parse(self):
        content = get_content(self.path)
        tbe = TitleBodyExtractor()
        tbe.feed(content)
        self._body = tbe.body
        self._title = tbe.title

    def body(self):
        if self._body == "":
            self.parse()
        return self._body

    def title(self):
        if self._title == "":
            self.parse()
        return self._title

    def __str__(self):
        return str(self.path) + " " + str(self.mtime)

The modification time of a file should be retrieved from the Mercurial repository. The following function calls hg log with a specific template, then parses the date to get the last commit time of a file. If the file is not registered to a repository, it simply returns the filesystem modification time.

def get_mtime(full_path):
    if os.path.exists(full_path):
        repo_path = get_repo_path(full_path)
        if repo_path != "":
            logcmd = "/usr/bin/hg log -R %s --template='{date|hgdate}' -l 1 %s " % (repo_path, full_path)
            # print logcmd

            proc_res = proc.check_output(logcmd, shell=True).split()
            if len(proc_res) > 0:
                filetime = int(proc_res[0])
            else:
                filetime = os.path.getmtime(full_path)
        else:
            filetime = os.path.getmtime(full_path)
        return filetime
    else:
        return 0

We need a list of files as FileObject objects, given the directory name and extension. The function also takes a repository path and excludes filenames that match a given regex.

    def file_list(dirname, extension, repo_path, exclude_regex=None):
        results = []
        for root, dirs, files in os.walk(dirname):
            # print "Dirs:", dirs
            for d in dirs:
                if exclude_regex == None or (not re.match(exclude_regex, d)):
                    results += file_list(os.path.join(root, d), extension, repo_path)
                else:
                    print "Skipping", d
            # print "Files:", files
            for f in files:
                if (exclude_regex == None or (not re.match(exclude_regex, f))) and f.endswith(extension):
                    fullname = os.path.join(root, f)
                    filetime = get_mtime(fullname)
                    results.append(FileObject(fullname, filetime))
        return results

Given a local file in the site, we need to create a link that shows the URL of that file relative to the site’s URL. The following function finds the relative path with respect to an input directory and returns the complete URL by concatenating it to the site’s URL.

    def make_link(site_url, input_dir, file_path):
        rel_path = os.path.relpath(file_path, input_dir)
        return site_url + rel_path

Given a FileObject that points to an HTML file, we need a function that builds an RSS item from it. It obtains the URL of the file and fills the rest using the attributes of the FileObject.

    def get_rss_item(file_object, input_dir, site_url):
        the_link = make_link(site_url, input_dir, file_object.path)
        item = rssgen.RSSItem(title=file_object.title(),
                              link=the_link,
                              description=file_object.body(),
                              guid=rssgen.Guid(the_link),
                              pubDate=datetime.datetime.fromtimestamp(file_object.mtime))
        return item

The function that creates an RSS file from the files in a given directory is the topmost function. It takes all variables that are set from the command line and returns the RSS object.

It first lists all files with the given extension (default being .html) in the input directory. Then it compares the modification time of the RSS file with the modification times of these listed files. If there is no previous RSS file or there are newer HTML files, the RSS is generated again.

Note that a certain amount of edit time can be set, so that the script doesn’t consider files as new if they are modified within edit time minutes. This can be set to prevent too frequent generation of files during an edit session.

To generate the RSS, each file is supplied to the previous function and an RSS item is obtained; then these are fed into the RSS2 function of PyRSS2Gen to get the resulting object.

def generate_rss(input_dir, extension = ".html", output = "rss/rss.xml", site_title = "Title", site_description = "Description", max_items = 20, site_url = "http://example.com", edit_time=0, exclude_regex = None):
        if not site_url.endswith("/"):
            site_url += "/"
        files = file_list(input_dir, extension, get_repo_path(input_dir), exclude_regex)
        rssmtime = get_mtime(output)
        files_up = [f for f in files if f.mtime >= (rssmtime + edit_time)]
        if len(files_up) > 0:
            print "Before", files_up
            files_up.sort(key=lambda x: x.mtime, reverse=True)
            print "After", files_up
            rss_items = [get_rss_item(fo, input_dir, site_url) for fo in files_up[:max_items]]
            rssobj = rssgen.RSS2(title = site_title,
                                 link = site_url,
                                 description = site_description,
                                 lastBuildDate = datetime.datetime.now(),
                                 items = rss_items)
            return rssobj
        return None

The main function uses argparse to handle options. The input directory, site title, site URL, and number of items are mandatory; other options have sensible default values.

The function also builds the exclude_regex object to supply to file listings. The regex is built here from the supplied string, and all other functions use this compiled regex.

After generating the RSS, it writes the file with the write_xml function.

    def main():

        parser = argparse.ArgumentParser(description='Generate RSS feed from a set of HTML files')

        parser.add_argument('--input-dir', help="input directory", required=True)
        parser.add_argument("--extension", help="file extension to collect", default=".html")
        parser.add_argument("--output", help="output filename to write the results", default="rss/rss.xml")
        parser.add_argument('--title', help="title of the RSS feed", required=True)
        parser.add_argument("--description", help="site description", default="")
        parser.add_argument("--items", help="max items included in the feed", required=True, type=int)
        parser.add_argument("--site-url", help="site url of items", required=True)
        parser.add_argument("--exclude-regex", help="regex to set skipped files", default="")
        parser.add_argument("--edit-time", help="minutes to wait before putting an item into rss", default=0, type=int)

        args = vars(parser.parse_args())

        if args["exclude_regex"] == "":
            exclude_regex = None
        else:
            exclude_regex = re.compile(args["exclude_regex"])

        rssresults = generate_rss(args["input_dir"],
                                  args["extension"],
                                  args["output"],
                                  args["title"],
                                  args["description"],
                                  args["items"],
                                  args["site_url"],
                                  args["edit_time"],
                                  exclude_regex)

        if rssresults != None:
            rssresults.write_xml(open(args["output"], "w"))



    if __name__ == "__main__":
        main()

You can get the resulting Python script from rss-generator.py.