🐢 Now, I have two classes. This one is a copy of the ONNX Patcher that’s used to convert ColQwen2 to ONNX format. But this one is valid only for the underlying model, that is Qwen2VLForConditionalGeneration, and ColQwen2 also has some modifications to call it.

🐇 What are these modifications?

🐢 The forward method looks something like this:

ColQwen2

    def forward(self, *args, **kwargs) -> torch.Tensor:
        kwargs.pop("output_hidden_states", None)

        # Handle the custom "pixel_values" input obtained with `ColQwen2Processor` through unpadding
        if "pixel_values" in kwargs:
            offsets = kwargs["image_grid_thw"][:, 1] * kwargs["image_grid_thw"][:, 2]  # (batch_size,)
            kwargs["pixel_values"] = torch.cat(
                [pixel_sequence[:offset] for pixel_sequence, offset in zip(kwargs["pixel_values"], offsets)],
                dim=0,
            )

        position_ids, rope_deltas = self.get_rope_index(
            input_ids=kwargs["input_ids"],
            image_grid_thw=kwargs.get("image_grid_thw", None),
            video_grid_thw=None,
            attention_mask=kwargs.get("attention_mask", None),
        )
        last_hidden_states = self.inner_forward(
            *args, **kwargs, position_ids=position_ids, use_cache=False, output_hidden_states=True
        )  # (batch_size, sequence_length, hidden_size)

        proj = self.custom_text_proj(last_hidden_states)  # (batch_size, sequence_length, dim)

        # L2 normalization
        proj = proj / proj.norm(dim=-1, keepdim=True)  # (batch_size, sequence_length, dim)
        proj = proj * kwargs["attention_mask"].unsqueeze(-1)  # (batch_size, sequence_length, dim)

        if "pixel_values" in kwargs and self.mask_non_image_embeddings:
            # Pools only the image embeddings
            image_mask = (kwargs["input_ids"] == self.config.image_token_id).unsqueeze(-1)
            proj = proj * image_mask
        return proj

🐇 How about the inner_forward method? Does it just directly call Qwen2VLForConditionalGeneration’s forward?

🐢 No, it has some conditionals:

        if inputs_embeds is None:
            inputs_embeds = self.model.embed_tokens(input_ids)
            if pixel_values is not None:
                pixel_values = pixel_values.type(self.visual.get_dtype())
                image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw)
                image_mask = (input_ids == self.config.image_token_id).unsqueeze(-1).expand_as(inputs_embeds)
                image_embeds = image_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
                inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)

            if pixel_values_videos is not None:
                pixel_values_videos = pixel_values_videos.type(self.visual.get_dtype())
                video_embeds = self.visual(pixel_values_videos, grid_thw=video_grid_thw)
                video_mask = (input_ids == self.config.video_token_id).unsqueeze(-1).expand_as(inputs_embeds)
                video_embeds = video_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
                inputs_embeds = inputs_embeds.masked_scatter(video_mask, video_embeds)

            if attention_mask is not None:
                attention_mask = attention_mask.to(inputs_embeds.device)

        outputs = self.model(
            input_ids=None,
            position_ids=position_ids,
            attention_mask=attention_mask,
            past_key_values=past_key_values,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_states = outputs[0]
        return hidden_states

🦊 The patcher only modifies the past_key_values_args and moves them to a DynamicCache. There are no other changes for patching.

🐢 But the return types are also different. Qwen2VLForConditionalGeneration.forward returns a Union[Tuple, Qwen2VLCausalLMOutputWithPast], but ColQwen2.forward returns a torch.Tensor.

🐇 How is that torch.Tensor calculated from the return type of Qwen2VLForConditionalGeneration? ❓

🦊 There are multiple return types in Qwen2VLForConditionalGeneration. It’s determined by the return_dict argument.

        if not return_dict:
            output = (logits,) + outputs[1:]
            return (loss,) + output if loss is not None else output

        return Qwen2VLCausalLMOutputWithPast(
            loss=loss,
            logits=logits,
            past_key_values=outputs.past_key_values,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
            rope_deltas=self.rope_deltas,
        )

🐇 What’s that argument in ColQwen2?

🐢 It’s passed from the caller; it’s a kwarg.

🐇 Is there any modification to this in the ONNX patcher?

🐢 Nope.

🐇 Then we can consider it as the default. What’s the default?

🐢 The default is None. Hence it returns (logits,) + outputs[1:].

🐇 Then, logits are the first parameter by default.

🐢 Yes, we can assume this.

🐇 What does ColQwen2 do with these logits?

🦊 By the way, it calls inner_forward with use_cache=False and output_hidden_states=True. What do these change in Qwen2VLForConditionalGeneration?

🐢 These hidden states are actually output. In ColQwen2.forward, the inner_forward call is actually:

        last_hidden_states = self.inner_forward(
            *args, **kwargs, position_ids=position_ids, use_cache=False, output_hidden_states=True
        )  # (batch_size, sequence_length, hidden_size)

and the output is the hidden states.

🦊 It then runs:

        proj = self.custom_text_proj(last_hidden_states)  # (batch_size, sequence_length, dim)

and calculates a set of projections.

🐢 custom_set_proj is defined as:

        self.custom_text_proj = nn.Linear(self.model.config.hidden_size, self.dim)

hence these projections are fully connected layer calculations.

🦊 It returns these after L2 normalization and checks if there are pixel_values to consider.

🐢 In this case, the output from ColQwen2 is this single tensor.

🐇 Ok. Then it’s actually simpler than wrapping up the whole Qwen2VLForConditionalGeneration.

🐢 It looks so, yes. But the example for ColQwen2 also has a post-processing step:

scores = processor.score_multi_vector(query_embeddings, image_embeddings)

🐇 I think we can leave that post-processing for the time being. We only need multi-vector embeddings for FastEmbed.

🐢 Maybe it’s convertible to a forward method that we can use with ONNX.

🐇 Let’s see how this scoring works, then.

🐢 score_multi_vector receives two tensors or tensor lists and compares query vectors with passage vectors. It’s a rather straightforward implementation that requires Torch, but not Transformers.

🐇 In theory, we can also convert this to an ONNX model.

🦊 We can also write a custom model for this.

🐢 It has two for loops for comparisons. Can we convert all of these?

🦊 The loop indices can be seen as dynamic_axes, and it’s possible to convert the whole thing as a Torch model, then use torch.onnx.export just as we do for the model itself.

🐢 I see, but I don’t think that’s what we must do now.

🐇 Yes, let’s skip that for the time being and convert the model itself. We’ll have multivectors for patches and queries at the end.

🐢 So, we’ll keep the processing part, send BatchFeature objects that are output from the processor, and send this to two models: one for images and one for text.

🐇 Yep, that’s the plan. In the end, we’ll have two ONNX models that require BatchFeatures.

🐢 Then, we’ll modify past_key_values in this argument to use DynamicCache.

🐇 Yes, that’s alright. We can start by moving past_key_values_converter to a method in PatchedColQwen2.

🐢 past_key_values wasn’t used much, so I completely removed it. I also began to use processor outputs as dummy input in the exporter. However, when using BatchFeature, we get:

RuntimeError: Only tuples, lists and Variables are supported as JIT inputs/outputs. Dictionaries and strings are also accepted, but their usage is not recommended. Here, received an input of unsupported type: BatchFeature

🐇 So, we can try Dynamo first, I think. If that doesn’t work, we can just collect items as tensors and build up a BatchFeature inside the patcher.

🐢 Let’s try dynamo=True for this first.

🦊 This time, it’s about BatchFeature again, but the error is different: KeyError: 'Indexing with integers is not available when using Python based feature extractors'

🐇 In this case, we can just create BatchFeature inside the patcher.

PatchedColQwen2

class PatchedColQwen2(ColQwen2):
    def forward(self, *args):
        (
            input_ids,
            inputs_embeds,
            attention_mask,
            position_ids,
            *past_key_values_args,
        ) = args
        # Convert past_key_values list to DynamicCache
        if len(past_key_values_args) == 0:
            past_key_values = None
        else:
            past_key_values = DynamicCache()
            for i in range(self.config.num_hidden_layers):
                key = past_key_values_args.pop(0)
                value = past_key_values_args.pop(0)
                past_key_values.update(key_states=key, value_states=value, layer_idx=i)

        breakpoint()
        o = super().forward(
            input_ids=input_ids,
            inputs_embeds=inputs_embeds,
            attention_mask=attention_mask,
            # position_ids=position_ids,
            past_key_values=past_key_values,
        )

        flattened_past_key_values_outputs = {
            "logits": o.logits,
        }
        output_past_key_values: DynamicCache = o.past_key_values
        for i, (key, value) in enumerate(
            zip(output_past_key_values.key_cache, output_past_key_values.value_cache)
        ):
            flattened_past_key_values_outputs[f"present.{i}.key"] = key
            flattened_past_key_values_outputs[f"present.{i}.value"] = value

        return flattened_past_key_values_outputs

🐇 It should only be needed to update the dependency versions in Cargo.toml, right?

🐢 Let’s start with that.

🐇 We have interface changes regarding aliases; let’s start using uv for building.

🦊 Added requirements to pyproject.toml by running:

uv add -r requirements.txt

virtualenv .venv
overlay use .venv/bin/activate.nu

🐇 I think we can start by improving xvc.py. I mean, releasing. Yesterday we finished our work with the Rust library.

🐢 Then we can start to look at the GUI, I believe. We can replace that 3-column view with a table and preview. It will be much easier that way.

🐇 Yep. Let’s finish and release the Python version first. Then we’ll go on to the GUI.

🐢 Let’s take a look at the PRs first.

ghpl 
36	Bump pyo3 from 0.22.2 to 0.23.3	dependabot/cargo/pyo3-0.23.3	OPEN	2024-12-04T03:52:03Z

gh pr close 36
✓ Closed pull request iesahin/xvc.py#36 (Bump pyo3 from 0.22.2 to 0.23.3)

🐢 We have already upgraded to pyo3 0.23. No need for this. Let’s create a PR for the current branch.

git push --set-upstream origin v0.6.13

ghpC --fill
https://github.com/iesahin/xvc.py/pull/37

🐇 We can also tag and push the tags.

🐢 I’d like to have some more coverage for certain parts of the code. Let’s add some tests.

🐇 It looks like we mainly lack the storage tests. They need configuration to add keys to GitHub, and we already skip some of these even in the Rust code.

🐢 Umm, I see. We also need a way to measure coverage. Could we do this with Codecov, I wonder?

🐇 I found an example here: https://github.com/codecov/example-python/blob/main/.github/workflows/ci.yml. It needs coverage and pytest-cov in the requirements.

🐢 Let’s try this then.

🐇 Added coverage.yml file. It’s simpler than the other GitHub action. Need to update the token now.

🐢 There is an issue installing the requirements.

🐇 I forgot to add sudo to apt-get. Will take care of it now.

🐢 Let’s check the run.

ghrl
in_progress		Release v0.6.13	coverage	v0.6.13	pull_request	12556590550	2m31s	2024-12-31T06:59:58Z
in_progress		Release v0.6.13	publish-to-pypi	v0.6.13	pull_request	12556590548	2m31s	2024-12-31T06:59:58Z
...

🐇 It takes a while and we still didn’t add the CODECOV_TOKEN.

🐢 Let’s add it, and after that, we need to take a look at this blink configuration. It adds letters after the selection.

🐇 Added the secret and configured xvc.py for coverage. There is an error with the build, though. Maybe the command we should be using is maturin develop instead of build to make Xvc available for the environment.

🐢 Let’s update and try it then.

🐇 maturin develop requires a virtual environment.

🐢 I checked the options for build, and I think there is an option, but let’s search first.

🐇 I searched, but it looks like we can just pass an --out directory or install xvc from the target/wheels/ directory. The second option requires less maintenance.

🐢 Okay. Let’s add a step to the action then.

🐇 Now, let’s wait for the run to finish with gh run watch.

🐢 It failed. Let’s take a look at the logs:

ghrl
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12556939836	3m26s	2024-12-31T07:38:54Z

gh run view 12556939836 --log-failed

🐇 It looks like some of the tests are failing. Let’s run the tests locally.

🐢 We should run with the --forked option, and it looks like we have a test to update with the xvc file list.

🐇 Updated the test, and I noticed we forgot to supply the new --show-directories option in the Python interface.

🐢 Yep. Passing these options as command-line options in strings is not robust. It’s very easy to forget things. I think we should start using CLI structs directly, but it’s not time yet.

🐇 I agree. It’s one of the goals for building a GUI, actually.

🐢 The tests failed again.

ghrl | rg failure
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12557031091	3m49s	2024-12-31T07:51:19Z

gh run view 12557031091 --log-failed

🐇 There is a Git error now. We need to add a Git user and email to the action.

🐢 Added those and watching the results again now.

🐇 Why do you think the publish action always works? It can only run with the main branch, I think. No need to run it with other pushes.

🐢 Yes, let’s configure it now.

ghrl | rg failure
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12557083393	3m21s	2024-12-31T07:59:18Z

gh run view 12557083393 --log-failed

🐇 It looks like we also need xvc-test-helper in the path. Let’s cargo install it and add it to the path.

🐢 Added .cargo/bin to the path like:

- name: Add cargo bin to PATH
  run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH

and installed the helper with cargo install xvc-test-helper.

🐇 By the way, GitHub Copilot is hallucinating about a method to update the path.

🐢 I searched and it may not be hallucinating. We can use the ::add-path:: command with echo, it looks like. This is new to me.

🐇 The tests failed again.

ghrl | rg failure
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12557208723	3m34s	2024-12-31T08:12:09Z

gh run view 12557208723 --log-failed

🐢 file().list() had a mistake, and we need to install rg for the tests.

🐇 Watching the test run. In the meantime, maybe we can review…

🐢 Failed again.

ghrl | rg failure
completed	failure	Release v0.6.13	coverage	v0.6.13	pull_request	12557294105	3m46s	2024-12-31T08:21:17Z

gh run view 12557294105 --log-failed

🐇 Added db.commit() to two places in the code. This should pass now.

🐢 Yeah!

ghrl | head -n 1
completed	success	Release v0.6.13	coverage	v0.6.13	pull_request	12557509836	3m46s	2024-12-31T08:44:33Z

gh run view 12557509836

✓ v0.6.13 coverage iesahin/xvc.py#37 · 12557509836
Triggered via pull_request about 4 minutes ago

JOBS
✓ linux in 3m36s (ID 35010253915)

ANNOTATIONS
! ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
linux: .github#1


For more information about the job, try: gh run view --job=35010253915
View this run on GitHub: https://github.com/iesahin/xvc.py/actions/runs/12557509836

🐇 I can’t see a coverage report on codecov.io, though.

🐢 It says no coverage report is generated. Let’s test to generate XML files locally.

🐇 The option in the docs seems incorrect. --cov-branch doesn’t produce anything.

🐢 Let’s wait for the run again.

🐇 Should we add a badge to the README?

🐢 It won’t show much, but yeah, let’s make it.

🐇 The results are in and it shows 100% coverage. This means it doesn’t actually test anything.

🐢 We need Rust coverage for this. Let’s search for it.

🐇 I found this: https://github.com/cjermain/rust-python-coverage. It runs cargo llvm-cov with the project and measures test coverage. But we don’t have any Rust tests.

🐢 It looks like we don’t need Rust tests. cargo llvm-cov can check coverage with the Python as well.

$ cargo llvm-cov show-env --export-prefix
export RUSTFLAGS=" -C instrument-coverage --cfg coverage --cfg trybuild_no_target"
export LLVM_PROFILE_FILE="/home/.../rust-python-coverage/target/rust-python-coverage-%m.profraw"
export CARGO_INCREMENTAL="0"
export CARGO_LLVM_COV_TARGET_DIR="/home/.../rust-python-coverage/target"

🐢 Let’s take a look at what remained for 0.6.13.

🐇 I think there is nothing left. Python must be published when we merged the PR.

🐢 Let’s take a look by searching xvc python.

🐇 The PyPI page is https://pypi.org/project/xvc/ and it still reports the version as 0.6.11. There must be something.

🐢 Now let’s take a look at

https://github.com/iesahin/xvc.py

🐇 The run seems to be OK though.

https://github.com/iesahin/xvc.py/actions/runs/12557780141/job/35010938462

🐢 Maybe the version in pyproject.toml is still 0.6.11 and we forgot to update it?

tmux new-window -c $HOME/github.com/iesahin/xvc.py/ nvim

🐇 There is no version string in ~/github.com/iesahin/xvc.py/pyproject.toml. It’s dynamic.

🐢 Then, let’s try to publish from local now.

🐇 The iex username requires email verification.

🐢 It looks like I publish xvc through the iesahin account, not iex. Maybe I can add both accounts to the project.

🐇 There seem to be no errors on the PyPI site.

🐢 Updating the token. Let’s add the token to pass to run maturin publish.

🐇 We’re receiving invalid or non-existent authentication information. Upgraded maturin to see if it fixes the issue.

🐢 We can also double-check the key.

🐇 Uploaded successfully from local. Let’s check the job again.

🐢 The release job was skipped because we didn’t tag after the merge. https://github.com/iesahin/xvc.py/actions/runs/12557780141/job/35011303045 That looks like the reason.

🐇 I should be more careful which is run and which is skipped.

🐢 Maybe we can relax the condition. We do this rarely. Maybe republishing is alright?

🐇 Yep, removed that. The jobs are running now. Let’s watch them to see what happens when we publish some of the packages.

🐢 In the meantime, let’s experiment with searching commands file with fzf-lua.

🐇 It requires more experimentation, but we can start from https://github.com/ibhagwan/fzf-lua/wiki/Advanced#interactive-shell-command.

🐢 The xvc publish jobs failed, btw.

ghrl
completed	failure	Remove if condition from release	publish-to-pypi	v0.6.13	push	12568863716	15m26s	2025-01-01T08:30:03Z

gh run view 12568863716 --log-failed
...
Release	Run actions/download-artifact@v4.1.7	        Please ensure that your artifact is not expired and the artifact was uploaded using a compatible version of toolkit/upload-artifact.
...

🐇 It was using an older version of upload-artifact.

🐢 Rerunning the job and it looks like it runs for both push to main and tag with v0.6.13. We can turn off push to main, I believe.

🐇 There was a missing upload-artifact again. Fixed and updated the tags.

ghrl completed failure update upload-artifacts publish-to-pypi main push 12569060915 14m39s 2025-01-01T08:57:22Z

ghrf 12569060915

🐢 There are conflicts with uploaded artifacts now. We may need to clean up the artifacts manually.

🐇 The conflicts were not about inter-workflow names. The names were conflicting because all files were named wheel. I added the platform and target to the names to avoid conflicts.

🐢 Let’s wait then. Maybe it will work this time. Could you search for a Lua console for Neovim?

🐇 Let’s try this one: return { “yarospace/lua-console.nvim”, lazy = true, keys = “`”, opts = {}, }

🐢 I couldn’t make it run but won’t spend much time ATM. How about the jobs?

ghrl
completed	failure	update upload artifact names	publish-to-pypi	v0.6.13	push	12569298414	13m57s	2025-01-01T09:27:00Z

ghrf 12569298414
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9180510Z ##[group]Run actions/download-artifact@v4
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9182098Z with:
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9182859Z   name: wheels
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9183843Z   merge-multiple: false
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9184817Z   repository: iesahin/xvc.py
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9185820Z   run-id: 12569298414
Release	Run actions/download-artifact@v4	2025-01-01T09:40:52.9186903Z ##[endgroup]
Release	Run actions/download-artifact@v4	2025-01-01T09:40:53.1937617Z Downloading single artifact
Release	Run actions/download-artifact@v4	2025-01-01T09:40:53.4694769Z ##[error]Unable to download artifact(s): Artifact not found for name: wheels
Release	Run actions/download-artifact@v4	        Please ensure that your artifact is not expired and the artifact was uploaded using a compatible version of toolkit/upload-artifact.
Release	Run actions/download-artifact@v4	        For more information, visit the GitHub Artifacts FAQ: https://github.com/actions/toolkit/blob/main/packages/artifact/docs/faq.md

🐇 Now the download doesn’t work.

🐢 Update it to download using patterns.

🐇 Did so and let’s take a look at the jobs again.

ghrl
completed	failure	added pattern to download	.github/workflows/publish.yml	main	push	12569456859	0s	2025-01-01T09:49:19Z

ghrv 12569456859

X main .github/workflows/publish.yml · 12569456859
Triggered via push about 3 minutes ago

X This run likely failed because of a workflow file issue.

For more information, see: https://github.com/iesahin/xvc.py/actions/runs/12569456859

🐢 The line with the pattern was reported as broken.

🐇 Removed it and recommitted.

ghrl
✅ completed	success	remove pattern to download all	publish-to-pypi	v0.6.13	push	12569509435	14m41s	2025-01-01T09:58:49Z

🐢 And now this completes the v0.6.13 release.

🐇 We’ll see how it will work next time.

🐢 Yep. Let’s move on to the GUI for now. Is that okay with you?

I tried the same approach when testing Python bindings with Pytest:

def test_pipeline_step_dependency(empty_xvc_repo):
    dep_help = empty_xvc_repo.pipeline().step().dependency(help=True)
    expected = """
Usage: xvc pipeline step dependency [OPTIONS] --step-name <STEP_NAME>

Options:
  -s, --step-name <STEP_NAME>
          Name of the step to add the dependency to
"""
    assert dep_help == expected

This doesn’t work because the help text is generated by clap and skips the usual thread-based output handler. All command output and errors in Xvc are returned as strings from the command, except for the help text that’s generated by clap automatically.

There are probably workarounds for this, but I won’t pursue them further as I don’t test the actual functionality in the Python bindings anyway.

The single-loader allows only one instance of Xvc to be run in a single process. This is no problem for the shell, but it looks like it won’t be possible to use multiple Xvc instances in a single Python process.

It’s possible to overcome this with an elaborate multiprocessing setup in the wrapper, but I won’t bother with it for now.

I replaced the eprintln!s with println!, but it didn’t make any difference. Maybe we should remove those statements completely.

I can’t really find the place that kills the kernel. The last command to run is xvc file list:

print(xvc_test_data.file().list("test-data/dir-0002"))

and the output it produces is:

[src/output.rs:144:13] &output_str = "SS         131 2024-06-05 08:57:11 41e16be7          test-data/dir-0002/file-0003.bin\nSS         131 2024-06-05 08:57:11 27f0
efd0          test-data/dir-0002/file-0002.bin\nSS         131 2024-06-05 08:57:11 66de5084          test-data/dir-0002/file-0001.bin\nTotal #: 3 Workspace Size:
      393 Cached Size:        6006\n"

print may be causing the crash, but the more likely cause is the command that comes after this:

!ls -l test-data/dir-0001/

I replaced this with lsd, which also failed. Maybe it’s actually a Python crash or bug.

The way to understand is to create a notebook file with only that cell and try to run it.

The ls line runs fine with a new notebook. It even runs on the README file when run at the beginning. The line that makes the kernel crash is:

xvc_test_data.storage().new_s3(name="backup", bucket_name="xvc-test", region="eu-central-1", storage_prefix="xvc-storage")

We can start by removing the new_s3 part.

The storage() method runs fine. It returns an XvcStorage() object, as it should.

When I run storage().list(), it takes a very long time. The bug is likely related to storage().

It looks like the storage object was adding file instead of storage as a subcommand. I’ve fixed it now.

That was the bug. The README notebook now creates the S3 storage.

What was the reason behind this?

Parsing the CLI to the XvcCLI object was perhaps the culprit. Let’s look at it more clearly.

Let’s try xvc file new s3 as a command to see how it behaves.

It says unrecognized subcommand for new.

This is how it should be, but I wonder why it doesn’t work for the XvcCLI parser.

Anyway, it’s already 13:00, so let’s stop here for today.

Another issue is the restart script. The if in that script that restarts the server doesn’t work; it always restarts the notebook server.

I removed some println! statements from xvc.py. There doesn’t seem to be anything in the library related to outputs.

Let’s search for how to check if a jupyter-lab command with the port 7979 runs in the background.

It looks like the bug is in the condition; it’s not -s, it’s -z.

Oops, yeah.

Let’s do a bit of tidying and check if the script is fixed.

I have a clippy warning with a new function that says these usually don’t take self as a parameter. This is for the pipeline new command, and it receives a self as an XvcPipeline object. It seems best to turn off the clippy warning for this.

I allowed two clippy warnings, and the script seems to work fine.

Let’s go on to copying the content from the Xvc README to the notebook.

There is an issue with the run-after-commit script. When Xvc commits the changes, the command we give is run again.

The issue is that git initializes the directory in the test-data/ directory, while xvc works in the current directory. I think we can either git init in the current directory or xvc init in test-data.

We’re already deleting .git and .xvc directories in the start-readme script. I think it may be easier to update git init to just initialize in the current directory.

Yep, let’s do it that way.

I see there are still extra outputs from the commands. We need to deal with this first.

I removed them, but there are still pink outputs. These are from dbg! statements, it looks like, or we’re initializing the output thread incorrectly.

Fixed those as well. I wrote up the xvc file list command examples as well. Now we have issues with xvc storage new s3 not running, and not even showing any debug output. We’ll deal with it in the next devlog, though.

Can't locate IPC/Cmd.pm in @INC (@INC contains: /home/runner/work/xvc.py/xvc.py/target/x86_64-unknown-linux-gnu/release/build/openssl-sys-844a96d66ae533b1/out/openssl-build/build/src/util/perl /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /home/runner/work/xvc.py/xvc.py/target/x86_64-unknown-linux-gnu/release/build/openssl-sys-844a96d66ae533b1/out/openssl-build/build/src/external/perl/Text-Template-1.56/lib)

The issue seemed to be a missing Perl package. Building rust-openssl now appears to require the perl-core package.

I added apt-get update && apt-get install perl-core to the CI configuration, but that didn’t work.

However, as this GitHub issue comment suggests, Maturin with its manylinux support uses different kinds of Docker containers to build the packages. We must detect whether it is a CentOS or Debian-based container and install the missing packages accordingly.

The Building wheels step in the configuration should look similar to:

- name: Build wheels
  uses: PyO3/maturin-action@v1
  with:
    target: ${{ matrix.target }}
    manylinux: auto
    args: --release --out dist
    before-script-linux: |
      # If we're running on RHEL/CentOS, install needed packages.
      if command -v yum &> /dev/null; then
          yum update -y && yum install -y perl-core openssl openssl-devel pkgconfig libatomic

          # If we're running on i686, we need to symlink libatomic
          # in order to build openssl with the -latomic flag.
          if [[ ! -d "/usr/lib64" ]]; then
              ln -s /usr/lib/libatomic.so.1 /usr/lib/libatomic.so
          fi
      else
          # If we're running on a Debian-based system.
          apt update -y && apt-get install -y libssl-dev openssl pkg-config
      fi

This will run the specified script before executing the Maturin build command in the container, ensuring all missing packages are installed.

What do I mean by OOP? Mainly the classical style, where you define classes with members and methods and use them to model the solution you’re working on. It was supposed that this way was superior to the procedural style, where you write procedures that modify global state. I believe the rage against procedural programming was because of this global state. Somehow, in all domains, we have to maintain state, and having a global state makes the reusability of procedures/functions almost impossible.

I can understand how the reaction against global state led to something like Java’s “everything is a class” creed. In order to contain global state, you need classes that contain parts of it and interact through methods. The idea is simple: if we forbid global state and keep partial state within structs (or classes), and mutate it with methods, we get reusable software. Once upon a time, I believed in this, too.

In an ideal world where we could write software in Smalltalk instead of C++, I’d probably not write this post. Actually, this post isn’t about C++ or Java, either. They have their places; they solve real problems and were probably necessary steps to arrive at Go or Rust. We now have better ideas thanks to the “everything should be a class” worldview.

The problem, in my experience, is applying these classical classes to interpreted languages. In C++, although it became like a Leviathan with arms for many different paradigms, there is some care for the cost of abstraction. If you know the tool, you can get away with classes. Java also seems to try to make abstractions cost zero. So if you use these languages, it’s a matter of modeling, habit, or domain suitability. I still think, over the long term, OOP increases the maintenance burden, but like most ideas in our profession, this is not rigorously tested.

However, when it comes to multi-paradigm interpreted languages like Python, objects begin to hurt. The problem, in my opinion, is that object-oriented design is in conflict with the Principle of Locality.

The Principle of Locality (PoL) is probably the most important empirical idea in software design. It’s directly related to the Pareto Principle, Zipf’s Law, and other well-known notions. Our CPUs, disks, search engines, and content delivery networks depend on it to cache artifacts for reuse. We know it works because a CPU with a larger L1 cache is faster; we add more caches to CPUs and disks to increase their speed.

The conflict between the PoL and OOP arises when classes include data that is not directly related to the problem at hand. The problem at hand, for example, might be finding the maximum of one million numbers or performing transformations on some variables. But if these are members of a class, they bring unusable data to the locality. If the variable I’m transforming is a member of a class, when I access it through the object, all other members—be they 3, 30, or 300—are also referenced. Thus, looping over objects ends up polluting the locality with all other members of the class.

In compiled languages, the advantage of iterators over plain loops is to overcome this. However, in interpreted languages, no one writes specific iterators for their member variables. This means when I have:

for obj in my_objects:
    obj.do_something(a)

All the members of obj are now in the loop. This is against the PoL.

Another problem I see with OOP is its alienation from the basic elements of software. When you begin to work with objects that do not directly correspond to anything in computer/software architecture, it becomes more or less a castle in the clouds. You have to tie this castle—i.e., the class hierarchy—somehow to the ground. The ground is the CPU, memory, disk, cache, etc. Compilers may do a good job of this, or they may not, but this detachment from the basic elements of computing machinery is the cause of what I call Object-Oriented Brain Damage.

When you begin to believe that the objects in your program are real, you try to express your problem using more objects. Classes are like kipple. They proliferate all the time. You add classes, then you add classes to create classes, then you add some base class to derive classes, then you try to fix these problems with patterns, and so on.

But this whole enterprise doesn’t have anything to do with the real tools we have. Our tools are processors, memory, disks, screens, printers, etc. When we detach the problem from these basics, it doesn’t become more solvable. We only create an ideal version of our understanding that requires additional attention to teach others, to document, etc.

It’s true that we need abstractions over the tools—we cannot just read bytes from disk, process in CPU, and print to screen. These all must be abstracted. But in my experience, OOP is not a good way to abstract these tools. Instead, it tries to abstract the problem at hand in an arbitrary way that is supposed to solve the problem. Yet this solution itself becomes a problem that must be fit onto the ground.

Databases and ORMs are good examples of this. Databases and Entity-Relationship theory are well-understood abstractions. We know how to use them across multiple CPUs, multiple disks, and multiple machines across multiple continents. ORMs are not like this; they are supposed to correspond to databases and provide that cozy object-oriented feeling. However, any system that depends on ORMs learns that these are not identical to databases. They don’t have the same capabilities and performance, and over the long run, depending on an ORM causes more problems than just writing SQL queries. Because an ORM does not consider the tools we have and their limitations, it tries to fit an idealistic world view onto databases. When it doesn’t scale, we think this performance degradation is natural.

No, it’s not natural. When your models load whole rows from billions of records just to access a single field for a calculation, you deplete the cache space quickly, and it doesn’t scale. OOP might just be an educational tool, but even in this regard, I believe it causes most of the brain damage we see in enterprise software.

First, you need to register a new bot with @BotFather and get an API token. Then, run pip3 install --user telegram-send and prepare a config file at ~/.config/telegram-send.conf:

[telegram]
token = <TOKEN_YOU_GET_FROM_BOT_FATHER>
chat_id = <CHAT_OR_USER_ID>

You’ll need to start a conversation with the bot and find your user ID (which is identical to the chat ID for the conversation you start with the bot).

After that, you can send yourself messages from the CLI like this:

telegram-send "Hello Telegram."

You can also send Markdown-formatted messages, audio, stickers, and more using various command-line options:

usage: telegram-send [-h] [--format {text,markdown,html}] [--stdin] [--pre]
                     [--disable-web-page-preview] [--silent] [-c]
                     [--configure-channel] [--configure-group]
                     [-f FILE [FILE ...]] [-i IMAGE [IMAGE ...]]
                     [-s STICKER [STICKER ...]]
                     [--animation ANIMATION [ANIMATION ...]]
                     [--video VIDEO [VIDEO ...]] [--audio AUDIO [AUDIO ...]]
                     [-l LOCATION [LOCATION ...]]
                     [--caption CAPTION [CAPTION ...]] [--config CONF] [-g]
                     [--file-manager] [--clean] [--timeout TIMEOUT]
                     [--version]
                     [message [message ...]]

This motivated me to learn Docker and gcloud rather quickly.

I was able to create a new Docker container in a short time. However, as I don’t want to share my Git SSH key publicly, I needed a way to put an SSH private key (~/.ssh/id_rsa) into this new container securely, without leaving any trace.

I tried a few things, but in the end, I decided to do something like:

FROM python:3.5-stretch as intermediate

# add credentials on build
RUN mkdir /root/.ssh/
# To use docker --build-arg, you can uncomment the following two lines and comment out the COPY line below.
# ARG SSH_PRIVATE_KEY
# RUN echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_rsa
COPY application_sshkey /root/.ssh/id_rsa
RUN chmod 0600 /root/.ssh/id_rsa

# make sure your domain is accepted
RUN touch /root/.ssh/known_hosts
RUN ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts

RUN git clone git@bitbucket.org:username/application /root/application

FROM python:3.5-stretch
COPY --from=intermediate /root/application /root/application

RUN pip3 install -r /root/application/requirements.txt

EXPOSE 9090/tcp

WORKDIR /root/application/

CMD python3 manage.py runserver 0.0.0.0:9090

Here, application_sshkey is a file I created using ssh-keygen and granted read-only access to on Bitbucket.

As you can see, the Dockerfile has two FROM statements. It creates a container to clone the application repository. Then it starts again with a new container and copies the repository to this new container. This way, it is not possible to peek into the private key using the docker history command.

By the way, I included two methods in the Dockerfile because gcloud builds does not accept a --build-arg parameter similar to docker build. I’m sure there are other workarounds for passing secrets to gcloud builds, but instead of digging for them, I found a solution that works for both Docker and Google Cloud Run.

For some UTF-8 files, there is a BOM marker at the beginning of the file. The script (as a true quick and dirty solution) doesn’t check for the presence of such a marker and cannot detect the Title element if it exists.

I added an fm = fm.strip('\ufeff') line to clear the BOM marker from a line if it exists.

There is an editor called bvi to edit binary files in Hex format, similar to the vi editor.

I’m building a face recognizer with dlib’s frontal face detector, and today, I noticed that some of the results return negative coordinates in face detection. This means the detected face is partial, although it’s a bit of a stretch to use negative coordinates for this.

My code wasn’t checking for negative coordinates and was building the NumPy array incorrectly. However, after reading the issue, although I’m still not convinced that it’s a good approach, I added a few if statements and the problem was solved.

It’s hard to tell for those who use their editors with mouse clicks on pretty icons, but once you catch this virus called doing everything from the keyboard, it becomes attached to your digital (from digitus, finger) psyche, making it impossible to leave behind. I even purchased an Android keyboard for my phablet just to ssh to my home and write with Emacs. Consider that!

Today I stumbled upon several posts regarding favorite Emacs packages and wanted to list mine.

org-mode

I’m not writing this in org-mode because I observed that I can detail, micro-revise, and lose focus while writing because of its extensive functionality. Instead, I’m using Zsh’s vared to enter lines to an org-file when I write first drafts. But editing, keeping, following, creating plots, literate programming—doing everything that can be done with text—is possible in org-mode.

I have F11 bound to org-capture and F12 to org-agenda-list. F9 F9 stores links wherever I like, and I can use C-c C-l in org buffers to paste them.

w3m

When I want to search Google but don’t like the idea of running a huge browser session just to look up an answer on StackOverflow, I use this.

undo-tree

It shows the versions created by undo/redo of a buffer in a tree format. Emacs’s default undo may be cumbersome at times, but this package is terrific.

wc-mode

To count words in a file. When I forget this package, I use C-x h M-| wc as a substitute.

auctex

Once upon a time, this mode led me to migrate from LyX to LaTeX itself.

ESS

Emacs Speaks Statistics is an R frontend. I’m not using it daily, but it’s much better than the R command-line client.

Uniquify

Makes buffer names unique. If you have files with identical names, this is much better than having names like myfile.txt<1>, myfile.txt<2>, etc.

recentf-mode

A list of recent files.

smex

ido for M-x. Standard M-x selects commands by their prefix; with smex, you can filter them by any substring. This makes a difference for those with poor memory.

ahg

I use Mercurial more than Git, and ahg-mode seems like a good alternative to standard vc- commands.

projectile

It considers directories under version control as projects and defines functions like find-file, multi-occur, and recentf for them. It’s a very simple and non-intrusive way to have project management.

elpy

For Python programming, elpy provides completion and much more functionality.

ack-and-a-half

An ack mode, a hybrid version of the other two.

xgtags

A GNU Global mode for C++ code tagging.

The feature is called patch histogram, and it represents the histogram of 3x3 patches of black and white pixels. We collect all 3x3 patches and count their frequencies.

A 3x3 patch for a binary image contains $2^9 = 512$ different combinations. For each of these combinations, we assign a unique ID. I wrote the implementation in Python; here is a lookup table that converts all possible 3x3 patches to their IDs.

patch_histogram_dict = {}
for f in range(0, 512):
    i22 = f & 1
    i21 = (f >> 1) & 1
    i20 = (f >> 2) & 1
    i12 = (f >> 3) & 1
    i11 = (f >> 4) & 1
    i10 = (f >> 5) & 1
    i02 = (f >> 6) & 1
    i01 = (f >> 7) & 1
    i00 = (f >> 8) & 1
    patch_histogram_dict[ma(i00, i01, i02, i10, i11, i12, i20, i21, i22)] = f

I was looking for a skeleton feature that can be applied to components after a medial axis transform. However, in this version, we will use binarized connected component images directly and explore the skeletal version later.

Above, we used a function ma to convert a set of numbers to a tuple of tuples. It’s as simple as:

def ma(i00, i01, i02, i10, i11, i12, i20, i21, i22):
    return ((i00, i01, i02),
            (i10, i11, i12),
            (i20, i21, i22))

The first step in feature generation is marking each pixel of the image with its ID. Then, a simple np.histogram call generates the histogram.

for i in range(1, rows - 1):
    for j in range(1, cols - 1):
        s = framed[(i-1):(i+2), (j-1):(j+2)]
        marks[i-1, j-1] = patch_dict[ma(s[0, 0], s[0, 1], s[0, 2],
                                        s[1, 0], s[1, 1], s[1, 2],
                                        s[2, 0], s[2, 1], s[2, 2])]

framed is a copy of the original image with a 1-pixel frame, so that boundary pixels are also counted without much boundary checking. patch_dict is a parameter that the function receives. Its default value is the dictionary that we created above, but any kind of patch dictionary can be used.

The feature is generated using a call to numpy.histogram as follows:

histogram = numpy.histogram(marked_points,
                            hist_range,
                            density=True,
                            bins=bins)

marked_points are the results of the pixel counting loop above. hist_range is the minimum and maximum value for the patch dictionary. The density parameter is set to True so that the histogram is normalized to 1, making the feature size-invariant. bins is a parameter we can set freely, but by default, it’s equal to the range of elements.

Comparisons can be performed using standard histogram comparison methods, such as EMD (Earth Mover’s Distance) or Chi-Square distance metrics.

Previously with Pandoc, I was using a simple setup to create RSS feeds. Markdown files were converted to plain, headerless HTML, and they were collected together to build an XML file. The obvious drawback is that all HTML files should be generated by Pandoc; anything that doesn’t fit that route does not appear in the feeds.

However, when I began to use Org Mode for data analysis and other tasks, I stopped using Pandoc. Org Mode has extensive facilities for exporting into HTML and other document formats, so I would not mess with Pandoc for this.

I thought RSS could be produced by parsing HTML files after they are produced. This requires parsing the HTML file, but it’s simple, and there are parsers for all programming languages out there. My previous RSS generator was in Python, and I decided to modify it to fit my needs. I think producing RSS for a static HTML site is a common need, and I tried to solve this problem as simply as possible.

Let’s begin with the ubiquitous shebang line. This tells the system that the script is in Python.

#!/usr/bin/env python

The following are the imports for this script. Apart from PyRSS2Gen, all modules are present in Python 2.7.

import argparse
import codecs
import os
import datetime
from HTMLParser import HTMLParser
import PyRSS2Gen as rssgen
import operator as op
import re
import subprocess as proc

I use Mercurial to track the site’s files. I once thought about using the Mercurial public API to check the status of files, but it proved to be overkill because only the modification time of files is necessary, and retrieving them using a standard command-line call is much simpler. Hence, I removed the following imports for the time being.

# from mercurial import commands as cmd
# from mercurial import hg
# from mercurial import ui as hgui

The following function returns a valid HTML tag string, given the tag and its attributes in a list. HTMLParser sends the tags in a list form, and I use this function to reconvert them to usual HTML tags.

def make_tag(tag, attrs):
    content_list = [ tag ]
    content_list += [ "%s=\"%s\"" % (k, v) for (k, v) in attrs]
    return "<" + " ".join(content_list) + ">"

TitleBodyExtractor is an HTMLParser subclass. It collects the body of a page in a string and also keeps the title. These two are the only requirements. It might be possible to parse meta tags to get publish date and author information as well, but I prefer to keep simple things simple.

class TitleBodyExtractor(HTMLParser):

    def __init__(self):
        HTMLParser.__init__(self)
        self.in_body = False
        self.in_title = False
        self.body = ""
        self.title = ""


    def handle_data(self, data):
        if self.in_body:
            self.body += data
        if self.in_title:
            self.title += data

    def handle_starttag(self, tag, attrs):
        if self.in_body:
            self.body += make_tag(tag, attrs)

        if tag == "body":
            self.in_body = True
        if tag == "title":
            self.in_title = True

    def handle_endtag(self, tag):
        if tag == "body":
            self.in_body = False

        if tag == "title":
            self.in_title = False

        if self.in_body:
            self.body += "</%s>" % (tag)

Getting contents of a file in UTF-8 encoding is a common task. The following two functions retrieve and store the contents in UTF-8 using the codecs module.

def get_content(filename):
    f = codecs.open(filename, "r", "utf-8")
    cont = f.read()
    f.close()
    return cont

def write_content(filename, content):
    f = codecs.open(filename, "w", "utf-8")
    f.write(content)
    f.close()

Mercurial allows running commands for a repository outside of that repository with the -R command-line switch. However, it requires the exact path of the repository and does not accept a child path. The following function finds the repository path of a file by recursively checking whether parent paths contain an .hg/ directory.

def get_repo_path(dir):
    if dir == "/" or dir == "":
        return ""
    if os.path.exists(os.path.join(dir, ".hg")):
        return dir
    else:
        return get_repo_path(os.path.dirname(dir))

The FileObject class keeps the required data of an HTML file. It stores the path, modification time, body, and title.

class FileObject:
    def __init__(self, path, mtime):
        self.path = path
        self.mtime = int(mtime)
        self._body = ""
        self._title = ""

    def parse(self):
        content = get_content(self.path)
        tbe = TitleBodyExtractor()
        tbe.feed(content)
        self._body = tbe.body
        self._title = tbe.title

    def body(self):
        if self._body == "":
            self.parse()
        return self._body

    def title(self):
        if self._title == "":
            self.parse()
        return self._title

    def __str__(self):
        return str(self.path) + " " + str(self.mtime)

The modification time of a file should be retrieved from the Mercurial repository. The following function calls hg log with a specific template, then parses the date to get the last commit time of a file. If the file is not registered to a repository, it simply returns the filesystem modification time.

def get_mtime(full_path):
    if os.path.exists(full_path):
        repo_path = get_repo_path(full_path)
        if repo_path != "":
            logcmd = "/usr/bin/hg log -R %s --template='{date|hgdate}' -l 1 %s " % (repo_path, full_path)
            # print logcmd

            proc_res = proc.check_output(logcmd, shell=True).split()
            if len(proc_res) > 0:
                filetime = int(proc_res[0])
            else:
                filetime = os.path.getmtime(full_path)
        else:
            filetime = os.path.getmtime(full_path)
        return filetime
    else:
        return 0

We need a list of files as FileObject objects, given the directory name and extension. The function also takes a repository path and excludes filenames that match a given regex.

    def file_list(dirname, extension, repo_path, exclude_regex=None):
        results = []
        for root, dirs, files in os.walk(dirname):
            # print "Dirs:", dirs
            for d in dirs:
                if exclude_regex == None or (not re.match(exclude_regex, d)):
                    results += file_list(os.path.join(root, d), extension, repo_path)
                else:
                    print "Skipping", d
            # print "Files:", files
            for f in files:
                if (exclude_regex == None or (not re.match(exclude_regex, f))) and f.endswith(extension):
                    fullname = os.path.join(root, f)
                    filetime = get_mtime(fullname)
                    results.append(FileObject(fullname, filetime))
        return results

Given a local file in the site, we need to create a link that shows the URL of that file relative to the site’s URL. The following function finds the relative path with respect to an input directory and returns the complete URL by concatenating it to the site’s URL.

    def make_link(site_url, input_dir, file_path):
        rel_path = os.path.relpath(file_path, input_dir)
        return site_url + rel_path

Given a FileObject that points to an HTML file, we need a function that builds an RSS item from it. It obtains the URL of the file and fills the rest using the attributes of the FileObject.

    def get_rss_item(file_object, input_dir, site_url):
        the_link = make_link(site_url, input_dir, file_object.path)
        item = rssgen.RSSItem(title=file_object.title(),
                              link=the_link,
                              description=file_object.body(),
                              guid=rssgen.Guid(the_link),
                              pubDate=datetime.datetime.fromtimestamp(file_object.mtime))
        return item

The function that creates an RSS file from the files in a given directory is the topmost function. It takes all variables that are set from the command line and returns the RSS object.

It first lists all files with the given extension (default being .html) in the input directory. Then it compares the modification time of the RSS file with the modification times of these listed files. If there is no previous RSS file or there are newer HTML files, the RSS is generated again.

Note that a certain amount of edit time can be set, so that the script doesn’t consider files as new if they are modified within edit time minutes. This can be set to prevent too frequent generation of files during an edit session.

To generate the RSS, each file is supplied to the previous function and an RSS item is obtained; then these are fed into the RSS2 function of PyRSS2Gen to get the resulting object.

def generate_rss(input_dir, extension = ".html", output = "rss/rss.xml", site_title = "Title", site_description = "Description", max_items = 20, site_url = "http://example.com", edit_time=0, exclude_regex = None):
        if not site_url.endswith("/"):
            site_url += "/"
        files = file_list(input_dir, extension, get_repo_path(input_dir), exclude_regex)
        rssmtime = get_mtime(output)
        files_up = [f for f in files if f.mtime >= (rssmtime + edit_time)]
        if len(files_up) > 0:
            print "Before", files_up
            files_up.sort(key=lambda x: x.mtime, reverse=True)
            print "After", files_up
            rss_items = [get_rss_item(fo, input_dir, site_url) for fo in files_up[:max_items]]
            rssobj = rssgen.RSS2(title = site_title,
                                 link = site_url,
                                 description = site_description,
                                 lastBuildDate = datetime.datetime.now(),
                                 items = rss_items)
            return rssobj
        return None

The main function uses argparse to handle options. The input directory, site title, site URL, and number of items are mandatory; other options have sensible default values.

The function also builds the exclude_regex object to supply to file listings. The regex is built here from the supplied string, and all other functions use this compiled regex.

After generating the RSS, it writes the file with the write_xml function.

    def main():

        parser = argparse.ArgumentParser(description='Generate RSS feed from a set of HTML files')

        parser.add_argument('--input-dir', help="input directory", required=True)
        parser.add_argument("--extension", help="file extension to collect", default=".html")
        parser.add_argument("--output", help="output filename to write the results", default="rss/rss.xml")
        parser.add_argument('--title', help="title of the RSS feed", required=True)
        parser.add_argument("--description", help="site description", default="")
        parser.add_argument("--items", help="max items included in the feed", required=True, type=int)
        parser.add_argument("--site-url", help="site url of items", required=True)
        parser.add_argument("--exclude-regex", help="regex to set skipped files", default="")
        parser.add_argument("--edit-time", help="minutes to wait before putting an item into rss", default=0, type=int)

        args = vars(parser.parse_args())

        if args["exclude_regex"] == "":
            exclude_regex = None
        else:
            exclude_regex = re.compile(args["exclude_regex"])

        rssresults = generate_rss(args["input_dir"],
                                  args["extension"],
                                  args["output"],
                                  args["title"],
                                  args["description"],
                                  args["items"],
                                  args["site_url"],
                                  args["edit_time"],
                                  exclude_regex)

        if rssresults != None:
            rssresults.write_xml(open(args["output"], "w"))



    if __name__ == "__main__":
        main()

You can get the resulting Python script from rss-generator.py.