emre şahin's digital garden 🍃 - devlog

devlog 32

Emre Şahin — 2025-07-12T16:10:36+00:00

This dialog is about converting ColQwen2 to the ONNX format.

🐢 Now, I have two classes. This one is a copy of the ONNX Patcher that’s used to convert ColQwen2 to ONNX format. But this one is valid only for the underlying model, that is Qwen2VLForConditionalGeneration, and ColQwen2 also has some modifications to call it.

🐇 What are these modifications?

🐢 The forward method looks something like this:

ColQwen2

    def forward(self, *args, **kwargs) -> torch.Tensor:
        kwargs.pop("output_hidden_states", None)

        # Handle the custom "pixel_values" input obtained with `ColQwen2Processor` through unpadding
        if "pixel_values" in kwargs:
            offsets = kwargs["image_grid_thw"][:, 1] * kwargs["image_grid_thw"][:, 2]  # (batch_size,)
            kwargs["pixel_values"] = torch.cat(
                [pixel_sequence[:offset] for pixel_sequence, offset in zip(kwargs["pixel_values"], offsets)],
                dim=0,
            )

        position_ids, rope_deltas = self.get_rope_index(
            input_ids=kwargs["input_ids"],
            image_grid_thw=kwargs.get("image_grid_thw", None),
            video_grid_thw=None,
            attention_mask=kwargs.get("attention_mask", None),
        )
        last_hidden_states = self.inner_forward(
            *args, **kwargs, position_ids=position_ids, use_cache=False, output_hidden_states=True
        )  # (batch_size, sequence_length, hidden_size)

        proj = self.custom_text_proj(last_hidden_states)  # (batch_size, sequence_length, dim)

        # L2 normalization
        proj = proj / proj.norm(dim=-1, keepdim=True)  # (batch_size, sequence_length, dim)
        proj = proj * kwargs["attention_mask"].unsqueeze(-1)  # (batch_size, sequence_length, dim)

        if "pixel_values" in kwargs and self.mask_non_image_embeddings:
            # Pools only the image embeddings
            image_mask = (kwargs["input_ids"] == self.config.image_token_id).unsqueeze(-1)
            proj = proj * image_mask
        return proj

🐇 How about the inner_forward method? Does it just directly call Qwen2VLForConditionalGeneration’s forward?

🐢 No, it has some conditionals:

        if inputs_embeds is None:
            inputs_embeds = self.model.embed_tokens(input_ids)
            if pixel_values is not None:
                pixel_values = pixel_values.type(self.visual.get_dtype())
                image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw)
                image_mask = (input_ids == self.config.image_token_id).unsqueeze(-1).expand_as(inputs_embeds)
                image_embeds = image_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
                inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)

            if pixel_values_videos is not None:
                pixel_values_videos = pixel_values_videos.type(self.visual.get_dtype())
                video_embeds = self.visual(pixel_values_videos, grid_thw=video_grid_thw)
                video_mask = (input_ids == self.config.video_token_id).unsqueeze(-1).expand_as(inputs_embeds)
                video_embeds = video_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
                inputs_embeds = inputs_embeds.masked_scatter(video_mask, video_embeds)

            if attention_mask is not None:
                attention_mask = attention_mask.to(inputs_embeds.device)

        outputs = self.model(
            input_ids=None,
            position_ids=position_ids,
            attention_mask=attention_mask,
            past_key_values=past_key_values,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        hidden_states = outputs[0]
        return hidden_states

🦊 The patcher only modifies the past_key_values_args and moves them to a DynamicCache. There are no other changes for patching.

🐢 But the return types are also different. Qwen2VLForConditionalGeneration.forward returns a Union[Tuple, Qwen2VLCausalLMOutputWithPast], but ColQwen2.forward returns a torch.Tensor.

🐇 How is that torch.Tensor calculated from the return type of Qwen2VLForConditionalGeneration? ❓

🦊 There are multiple return types in Qwen2VLForConditionalGeneration. It’s determined by the return_dict argument.


        if not return_dict:
            output = (logits,) + outputs[1:]
            return (loss,) + output if loss is not None else output

        return Qwen2VLCausalLMOutputWithPast(
            loss=loss,
            logits=logits,
            past_key_values=outputs.past_key_values,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
            rope_deltas=self.rope_deltas,
        )

🐇 What’s that argument in ColQwen2?

🐢 It’s passed from the caller; it’s a kwarg.

🐇 Is there any modification to this in the ONNX patcher?

🐢 Nope.

🐇 Then we can consider it as the default. What’s the default?

🐢 The default is None. Hence it returns (logits,) + outputs[1:].

🐇 Then, logits are the first parameter by default.

🐢 Yes, we can assume this.

🐇 What does ColQwen2 do with these logits?

🦊 By the way, it calls inner_forward with use_cache=False and output_hidden_states=True. What do these change in Qwen2VLForConditionalGeneration?

🐢 These hidden states are actually output. In ColQwen2.forward, the inner_forward call is actually:

        last_hidden_states = self.inner_forward(
            *args, **kwargs, position_ids=position_ids, use_cache=False, output_hidden_states=True
        )  # (batch_size, sequence_length, hidden_size)

and the output is the hidden states.

🦊 It then runs:

        proj = self.custom_text_proj(last_hidden_states)  # (batch_size, sequence_length, dim)

and calculates a set of projections.

🐢 custom_set_proj is defined as:

        self.custom_text_proj = nn.Linear(self.model.config.hidden_size, self.dim)

hence these projections are fully connected layer calculations.

🦊 It returns these after L2 normalization and checks if there are pixel_values to consider.

🐢 In this case, the output from ColQwen2 is this single tensor.

🐇 Ok. Then it’s actually simpler than wrapping up the whole Qwen2VLForConditionalGeneration.

🐢 It looks so, yes. But the example for ColQwen2 also has a post-processing step:

scores = processor.score_multi_vector(query_embeddings, image_embeddings)

🐇 I think we can leave that post-processing for the time being. We only need multi-vector embeddings for FastEmbed.

🐢 Maybe it’s convertible to a forward method that we can use with ONNX.

🐇 Let’s see how this scoring works, then.

🐢 score_multi_vector receives two tensors or tensor lists and compares query vectors with passage vectors. It’s a rather straightforward implementation that requires Torch, but not Transformers.

🐇 In theory, we can also convert this to an ONNX model.

🦊 We can also write a custom model for this.

🐢 It has two for loops for comparisons. Can we convert all of these?

🦊 The loop indices can be seen as dynamic_axes, and it’s possible to convert the whole thing as a Torch model, then use torch.onnx.export just as we do for the model itself.

🐢 I see, but I don’t think that’s what we must do now.

🐇 Yes, let’s skip that for the time being and convert the model itself. We’ll have multivectors for patches and queries at the end.

🐢 So, we’ll keep the processing part, send BatchFeature objects that are output from the processor, and send this to two models: one for images and one for text.

🐇 Yep, that’s the plan. In the end, we’ll have two ONNX models that require BatchFeatures.

🐢 Then, we’ll modify past_key_values in this argument to use DynamicCache.

🐇 Yes, that’s alright. We can start by moving past_key_values_converter to a method in PatchedColQwen2.

🐢 past_key_values wasn’t used much, so I completely removed it. I also began to use processor outputs as dummy input in the exporter. However, when using BatchFeature, we get:

RuntimeError: Only tuples, lists and Variables are supported as JIT inputs/outputs. Dictionaries and strings are also accepted, but their usage is not recommended. Here, received an input of unsupported type: BatchFeature

🐇 So, we can try Dynamo first, I think. If that doesn’t work, we can just collect items as tensors and build up a BatchFeature inside the patcher.

🐢 Let’s try dynamo=True for this first.

🦊 This time, it’s about BatchFeature again, but the error is different: KeyError: 'Indexing with integers is not available when using Python based feature extractors'

🐇 In this case, we can just create BatchFeature inside the patcher.

PatchedColQwen2


class PatchedColQwen2(ColQwen2):
    def forward(self, *args):
        (
            input_ids,
            inputs_embeds,
            attention_mask,
            position_ids,
            *past_key_values_args,
        ) = args
        # Convert past_key_values list to DynamicCache
        if len(past_key_values_args) == 0:
            past_key_values = None
        else:
            past_key_values = DynamicCache()
            for i in range(self.config.num_hidden_layers):
                key = past_key_values_args.pop(0)
                value = past_key_values_args.pop(0)
                past_key_values.update(key_states=key, value_states=value, layer_idx=i)

        breakpoint()
        o = super().forward(
            input_ids=input_ids,
            inputs_embeds=inputs_embeds,
            attention_mask=attention_mask,
            # position_ids=position_ids,
            past_key_values=past_key_values,
        )

        flattened_past_key_values_outputs = {
            "logits": o.logits,
        }
        output_past_key_values: DynamicCache = o.past_key_values
        for i, (key, value) in enumerate(
            zip(output_past_key_values.key_cache, output_past_key_values.value_cache)
        ):
            flattened_past_key_values_outputs[f"present.{i}.key"] = key
            flattened_past_key_values_outputs[f"present.{i}.value"] = value

        return flattened_past_key_values_outputs

devlog 30

Emre Şahin — 2025-05-11T18:17:52+00:00

🐢 Let’s discuss how to move Xvc forward—maybe we can write a post to Reddit and the Rust forum in the meantime. 🐇 I think the next step is rclone remotes. It will allow us to use all remote storages supported by Rclone, which is a nice feature. 🐢 Don’t you think we need to publish the current version to Reddit and the forum? 🐇 We can do that as well. 🦊 Adding rclone remote must be a straightforward task. 🐢 We need to understand rclone paths, but overall, yes. We’ll just need to get the remote name, like drive://, and a path, like my-xvc-storage, and build paths with these. 🐇 What are the commands? 🐢 We need to learn how to upload files from local to remote and how to download these files. We can also list the files and get files as well. 🐲 How about adding a paths.txt to folders in remotes to show which paths the files in 0.jpg belong to? This will change the remote cache structure a bit. We will have a reverse index of files and they will be findable. 🐢 What’s the reason for this? 🐲 When I upload a file to Drive with only the content hash, I lose track of the actual path. This is not desirable. We can add a file to the directory, called paths.txt, to get the paths for a file. 🐢 This may prove to be a feat, though; adding these XvcPaths to a file requires a lookup. 🐇 Maybe a JSON file? It might be possible to look up a path with a JSON file, and it will be easier to parse. 🐲 I don’t think the issue is about parsing, though. We can just have a plain text file that lists the paths. It’s a text file, which is the most compatible across all storages. 🐢 Storages, you mean. 🐲 Ugh, yeah. If I have a file called Alan Watts but I only have the content, this file will be immensely useful. 🐢 This makes XvcCachePath and XvcPath coupled. Architecture-wise, it may not be a good thing, though. 🐇 Also, there may be common storages for multiple repositories. 🐲 Umm, that’s a good point. I don’t think the architecture will be much compromised, though. We already keep the file paths and their cache paths somewhere. 🐢 Cache paths are generated from the content, but any number of paths can point to a single path in the cache. If I have 1 million copies of the same file, will I add all these files to the paths.txt you mentioned? 🐲 That’s a good point too. We can have a limit, like 1,000 or something, not to make these files too big. 🐢 Instead of this, we can store the output of xvc file list at the storage root and allow looking up the files that way. 🐲 It has the same problem, though; if we have a million files, their list will be too large. 🐢 There can be a manual command, like xvc file index --to storage, that will show content hashes and paths of each file. We can also add URLs to files if possible. 🐲 No one will use it when it’s manual, though. 🐢 We can add functionality to update this index when we send a file, though.
🐲 So, after each send, we’ll update the index for the repository on that storage. Is that correct? 🐢 Not after each send. After each send session, maybe. 🐇 We can have an incremental way of updating the index, like we do in ECS? 🐢 It will be overkill for this functionality and add too much noise to the storage. 🐲 Let’s keep this discussion here, but I also want to have an index merge or index cleanup mechanism for the entity generator and the ECS. 🐢 We can have a “merge indices” functionality in ECS. That will remove all older entity-generator files and merge all store files. 🐇 Removing older entity files is easy, but what about merging the store files? 🐢 It’s easy too. We’ll just load all event logs from the directory, remove all other files, and save the event log to a file. 🐇 Will this be manual or automatic? 🐢 I think the first version can be manual, something like xvc fsck merge-store-files or something like that. We can notify the user if the number of files is > 10,000 or something like that. I don’t think we need to make it automatic unless we measure the impact of these files. There is no point in trying to do it at every command. 🐇 Then we’ll have two new commands for the next version? 🐢 I think we can just add rclone remote now and release it, then make changes in the ECS for this new xvc fsck command. 🐇 Can the name be doctor or something? Or util? Or can we add a top-level merge indices command? 🐢 xvc doctor seems like a better alternative. We can have a diagnose subcommand as well to check for possible inconsistencies. xvc doctor merge-store-files is a better command. 🐲 Will we use d for this command? 🐢 No need to add a single-letter command for this, I believe. It shouldn’t be required to run frequently. 🐇 Hmm, ok. What do we need to know for rclone remote? 🐲 I noticed we don’t have the xvc storage remove command implemented yet. Maybe we can start from that. 🐢 Hmm, yeap. Let’s start by implementing that first. We can add the rclone command next. 🐇 Will we use a feature flag for rclone? It will run the command only with an external binary. 🐢 It’s better to have a feature flag. I think we can add a feature flag for rsync remote as well. 🐇 We can use the generic one to update the feature flag. 🐢 I think the only two items of information we need for rclone are the remote name and the remote directory. Will we make these required?

devlog 20

Emre Şahin — 2025-02-01T09:27:50+00:00

🐢 We have an error in the publish action. Let’s fix it and rerun it.

🐇 There are two errors. One resulted from forgetting sudo when installing dependencies. The other was the incorrect name for the OpenSSL library. It should be libssl-dev instead of openssl-dev. Maybe we can link the command list in the docs.

🐢 Oops, yes, we should at least keep that in mind.

🐇 I’m checking the most popular crates. getrandom retrieves a random number from the system. It looks much lighter than the rand crate.

🦊 We can use once_cell to initialize XvcEntityCounter. We currently use Once for this purpose.

🐇 I don’t think it will provide any better features.

🦊 For that case, yes, no better features. But the interface is something like:

impl OnceCell {
    const fn new() -> OnceCell { ... }
    fn set(&self, value: T) -> Result<(), T> { ... }
    fn get(&self) -> Option<&T> { ... }
}

And this makes, for example, working with XvcRoot much easier. We are passing Arc>> everywhere. This is a heavy price when we only use it in a read-only manner. We can prevent most of these, when we use a read lock, by using OnceCell.

🐇 XvcConfig can benefit from this as well. We don’t update the config during runs.

🐢 Why do we want to assign it, though? We currently have a config field in XvcRootInner, and we get a reference to it with the config() method.

🐇 Ok. Let’s skip this for now. No need to worry before measuring the performance impact.

devlog 19

Emre Şahin — 2025-01-29T09:19:56+00:00

🐇 Let’s start by checking the GitHub Actions results.

$ ghrl | first
╭──────────────┬─────────────────────────────────────────────────────────╮
│ conclusion   │ failure                                                 │
│ displayTitle │ Add CLI completions                                     │
│ headBranch   │ clap-complete-16608                                     │
│ url          │ https://github.com/iesahin/xvc/actions/runs/13007688178 │
╰──────────────┴─────────────────────────────────────────────────────────╯

🐢 Although I turned off most of the watches, logs are still so large that it’s not possible to view them from the interface. Downloaded the log archive.

🦊 We can have different GitHub Actions steps for each test. Claude can help write such a repeating set of steps.

🐇 We can at least separate z_test_docs to see if integration tests or that fails.

🐢 We’ll have to add caching for test artifacts to upload them for coverage. I’m not sure we really need to add that complexity to the process just to avoid downloading the logs.

🐇 We can actually move all to Xvc. We need a GitHub Action to run an Xvc pipeline.

🐢 Eventually yes, we should move our testing to Xvc itself. Now, the logs show that there are differences in z_test_docs actually.

🐇 When I run the command below, it passes. We may have a different config in GitHub Actions.

$ XVC_TRYCMD_TESTS=storage,file,pipeline,core,start TRYCMD=overwrite rws cargo test --features test-ci -p xvc --test z_test_docs
test z_doc_tests ... ok
...
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 15.34s

🐢 We don’t have an XVC_TRYCMD_TESTS=storage,file,pipeline,core,start definition in GitHub Actions; let’s add it, bump the version, and try again.

$ cargo set-version "0.6.14-alpha.10"
   Upgrading xvc from 0.6.14-alpha.9 to 0.6.14-alpha.10
...

🦊 We could also update run-tests.zsh to get a quick response.

🐢 Yep, let’s do that as well.

🐇 Dev tests pass, but there are differences in the documents still. For some reason, the local run-tests.zsh doesn’t update xvc/book/src/ref/xvc-storage.md. It only has the help text as a reference, but it’s not updated with the aliases, and it breaks the CI. This is weird but a small issue. Fixed it manually.

$ ghrl | first 2
╭───┬────────────┬───────────────────┬────────────────────┬────────────────────╮
│ # │ conclusion │   displayTitle    │     headBranch     │        url         │
├───┼────────────┼───────────────────┼────────────────────┼────────────────────┤
│ 0 │            │ Add CLI           │ clap-complete-1660 │ https://github.com │
│   │            │ completions       │ 8                  │ /iesahin/xvc/actio │
│   │            │                   │                    │ ns/runs/1302770108 │
│   │            │                   │                    │ 2                  │
│ 1 │ success    │ Add CLI           │ clap-complete-1660 │ https://github.com │
│   │            │ completions       │ 8                  │ /iesahin/xvc/actio │
│   │            │                   │                    │ ns/runs/1302755462 │
│   │            │                   │                    │ 2                  │
╰───┴────────────┴───────────────────┴────────────────────┴────────────────────╯

🐢 The earlier one has passed. It looks like we’re ready to merge. Let’s update the CHANGELOG.md for release.

🐇 Setting the release version:

cargo set-version "0.6.14"
   Upgrading xvc from 0.6.14-alpha.10 to 0.6.14
...

🐢 Let’s check the CI

$ ghrl | first 2

╭───┬────────────┬───────────────────┬────────────────────┬────────────────────╮
│ # │ conclusion │   displayTitle    │     headBranch     │        url         │
├───┼────────────┼───────────────────┼────────────────────┼────────────────────┤
│ 0 │ success    │ Add CLI           │ clap-complete-1660 │ https://github.com │
│   │            │ completions       │ 8                  │ /iesahin/xvc/actio │
│   │            │                   │                    │ ns/runs/1302770108 │
│   │            │                   │                    │ 2                  │
│ 1 │ success    │ Add CLI           │ clap-complete-1660 │ https://github.com │
│   │            │ completions       │ 8                  │ /iesahin/xvc/actio │
│   │            │                   │                    │ ns/runs/1302755462 │
│   │            │                   │                    │ 2                  │
╰───┴────────────┴───────────────────┴────────────────────┴────────────────────╯

🐇 And merge:

$ ghpM --body $"(open CHANGELOG.md | lines | skip 2 | take 7)" --subject "Add completions" --squash

🐢 Tagged main and pushed. Packages should be built in a few minutes.

devlog 18

Emre Şahin — 2025-01-28T10:07:28+00:00

🐇 There is no xvc completions command anymore. Let’s remove the test.

🐢 Instead, we can make it test with environment variables for each of these shells.

🐇 It looks like the test fails with wait_status.

cargo test -p xvc --test test_completions
...
failures:

---- test_completions stdout ----
...
ExitStatus(unix_wait_status(512))

thread 'test_completions' panicked at lib/tests/common/mod.rs:46:5:
Command failed: Command { cmd: "/Users/iex/github.com/iesahin/xvc/target/debug/xvc", stdin: None, timeout: None }
...

🐢 The failure is in the line:

    let mut cmd = Command::cargo_bin("xvc").unwrap();

So it probably waits for user input to complete, but it doesn’t have any and cannot complete it, so it returns an error.

🦊 Let’s comment the earlier test out and check if other tests work.

🐢 Yes, they fail the same. When there are no commands to run, xvc returns an error. Maybe we can change this behavior or handle the error in the Command::cargo_bin line above.

🐇 I checked the code and we don’t handle the “no arguments” case anywhere. It looks like the behavior to return an error code is inherited from clap.

🐢 Updated the error handling code to report a more descriptive message from the source.

cargo test -p xvc --test test_completions
...
failures:

---- test_completions stdout ----
Output { status: ExitStatus(unix_wait_status(25856)), stdout: "", stderr: "\nthread 'main' panicked at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/debug_asserts.rs:341:13:\nCommand pipeline: command `export` alias `l` is duplicated\nstack backtrace:\n   0: rust_begin_unwind\n             at /rustc/b1a7dfb91106018f47ed9dc9b27aee1977682868/library/std/src/panicking.rs:692:5\n   1: core::panicking::panic_fmt\n             at /rustc/b1a7dfb91106018f47ed9dc9b27aee1977682868/library/core/src/panicking.rs:75:14\n   2: clap_builder::builder::debug_asserts::assert_app\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/debug_asserts.rs:341:13\n   3: clap_builder::builder::command::Command::_build_self\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4173:13\n   4: clap_builder::builder::command::Command::_build_recursive\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4076:9\n   5: clap_builder::builder::command::Command::_build_recursive\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4078:13\n   6: clap_builder::builder::command::Command::build\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_builder-4.5.20/src/builder/command.rs:4071:9\n   7: clap_complete::env::CompleteEnv::try_complete_\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_complete-4.5.40/src/env/mod.rs:229:9\n   8: clap_complete::env::CompleteEnv::try_complete\n             at /Users/iex/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/clap_complete-4.5.40/src/env/mod.rs:210:9\n   9: xvc::main\n             at ./src/main.rs:18:30\n  10: core::ops::function::FnOnce::call_once\n             at... [truncated]
ExitStatus(unix_wait_status(25856))

...

error: test failed, to rerun pass `-p xvc --test test_completions`

🐢 The real issue is when we assert the successful run, at this line:

            assert!(output.status.success(), "Command failed: {:?}", prepared);

🐇 If you look at the error message carefully, you’ll see that this is a different error. We added an alias to xvc pipeline export with l, which is a duplicate.

🐢 Oops, yeah.

cargo test -p xvc --test test_completions
...
test test_completions ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.81s

🐇 There are some warnings in compilation. Let’s fix these.

cargo build
   Compiling xvc-storage v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/storage)
   Compiling xvc-file v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/file)
   Compiling xvc-pipeline v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/pipeline)
   Compiling xvc v0.6.14-alpha.8 (/Users/iex/github.com/iesahin/xvc/lib)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.28s

🐢 Finished the build without warnings. Now we can run the whole test suite again.

🐇 Tests are running fine. Updated some error messages and types, and doc tests also pass. There is an issue with the Rsync storage ref.

🐢 It looks like we try to elide output with [...], while the proper format is [..].

🐇 Replaced them with ... that will elide multiple lines.

🐢 Pushed changes to the server.

devlog 17

Emre Şahin — 2025-01-27T16:07:11+00:00

🐢 We are progressing towards adding all store completions. In the meantime, I switched to nushell from zsh. Completions for nushell require a different crate and it doesn’t have dynamic completions yet.

🐇 We can just complete the completions for this version and add JSON output for all commands to use in nushell. In the meantime, dynamic completions support for nushell is likely to be added.

🐢 Umm, yep. Then we’ll continue to run tests with zsh for the current version.

🐇 run-tests is a zsh file, and we will continue to use it. We can switch to nushell eventually as we add more compatibility with it, but let’s not do this now.

🐢 Where were we the last time for completions?

🐇 We added xvc_path_completers. Let’s check the TODO list now.

🐢 We still need tracked_targets completions in a few places.

🐇 Added those.

🐢 Now we have some strum completers. Let’s finish these up as well.

🐇 Ok. They are done as well.

🐢 Now, let’s start adding a storage_identifier completer. It should read all XvcStorage records and list their names.

🐇 Added a storage_identifier completer. Let’s add it to all places where we use storage_identifiers.

🐢 Now, we need completers for pipeline and step names. These are straightforward.

🐇 Added these. Let’s fill up where they are needed.

🐢 I want to skip some of the fine-grained completions in this version. We can have a specific completer for --params options for files and params inside, for example. Or a special completer for directories.

🐇 Let’s not forget these. Reading YAML files and extracting hyperparameter keys for the prompt would be a really good feature for the user.

🐢 I agree. These are good features in general, but we need to ship the current version as soon as possible.

🐇 Let’s check the TODO comments once more.

$ rg 'TODO:' 
...
- ✅ pipeline/src/pipeline/api/update.rs:    /// TODO: Add a repository_dirs completer (11:08)
- ✅ file/src/remove/mod.rs:    /// TODO: Add a storage_identifier completer (11:08)
- ✅ file/src/bring/mod.rs:    /// TODO: Add a storage_identifier completer (11:08)
...

🦊 We can use xvc_path_completer for the tracked directory completer for the time being. For pipeline update, we can use a ValueHint instead. It’s not necessary to use a custom completion.

🐢 For the xvc file copy destination, we can have a file or a directory that we track and is not available, or we don’t track and is available. It’s a similar situation to xvc_path_completer, but we also need to check the local paths. It requires some more care.

🐇 Now we can begin to run the tests.

🐢 Is this building?

🐇 It should.

🐢 We also have one bug. xvc shouldn’t print help text when the COMPLETE environment variable is set.

🐇 Umm, yeah, that’s a blocker.

🐢 Fixed it. We can bump the version and push the changes.

Devlog 11: Homebrew Taps and Automation

Emre Şahin — 2025-01-19T09:37:37+00:00

🐢 Let’s move slowly. Let’s create a brew project first.

🦊 There is a brew create command, but could we really copy it from someone else?

🐢 I read the espanso example, and I’ll start by adding a repository. Welcome https://github.com/iesahin/homebrew-xvc

🦊 We can search to automate this for Rust stuff first. Maybe it’s easier to automate than to do it manually with examples.

🐢 Let’s search “how to automate homebrew tap formula updates with github actions”

🐲 There is a shell script that updates the brew repo with a shell script. It fires the action from the main repository with a command like:

gh workflow run release.yml -f version=${{ env.VERSION }} -R itspriddle/homebrew-slack-notify

🐢 It looks a little brittle, though.

🐲 There is also a GitHub action: https://github.com/marketplace/actions/homebrew-tap but it doesn’t look very popular. There is another one https://github.com/marketplace/actions/bump-homebrew-formula but this is for formulas, or the default settings are those.

🐢 There is an example https://github.com/marketplace/actions/bump-homebrew-formula#examples for taps as well.

🐲 This one is simpler, and probably we can just add this first: https://github.com/marketplace/actions/homebrew-bump-formula Most of the fields are optional.

🐇 I think this is much easier than the other. Let’s start with this.

🐢 Now let’s add a token to the Xvc repo. ✅

🦊 Add a branch to Xvc and add the release action file.

🐢 Added the release action file ~/github.com/iesahin/xvc/.github/workflows/homebrew.yml, but we still don’t have a tap; maybe we can just add one.

🦊 Can you check espanso’s for example?

🐢 It has a cask and is in the core. We need a tap example.

🐲 I believe we need to clone homebrew-xvc and run gh workflow run.

🐢 Yes, let’s clone the repo and try to run it manually.

🐇 We don’t have a formula yet.

🦊 What’s the directory structure of a Homebrew tap project?

🐢 It’s something like this, but we don’t need tests for this, I believe.

.
├── Formula
│   └── .rb
├── LICENSE
├── README.md
└── .github
    └── workflows
        └── tests.yml

🐇 What goes into the Formula?

🦊 Let’s take a look at ripgrep’s example:

class Ripgrep < Formula
  desc "Search tool like grep and The Silver Searcher"
  homepage "https://github.com/BurntSushi/ripgrep"
  url "https://github.com/BurntSushi/ripgrep/archive/refs/tags/14.1.1.tar.gz"
  sha256 "4dad02a2f9c8c3c8d89434e47337aa654cb0e2aa50e806589132f186bf5c2b66"
  license "Unlicense"
  head "https://github.com/BurntSushi/ripgrep.git", branch: "master"

  livecheck do
    url :stable
    strategy :github_latest
  end

  bottle do
    sha256 cellar: :any,                 arm64_sequoia:  "b8bf5e73c9c9b441de067ec86ac167b071ecc2078dcb1d89d2cebbb151feab35"
    sha256 cellar: :any,                 arm64_sonoma:   "47b9c3515c866b147f0e98735cab165d6471b9f28fab1ba2c57e59c43da5c10b"
    sha256 cellar: :any,                 arm64_ventura:  "e14a94e84c028ff53c1be3b106fdeb5aca4d7c893a819e7fb967e0719b946a28"
    sha256 cellar: :any,                 arm64_monterey: "ad8dc4ab475c84e2a1e60f5b3107f52dd59e33f84a08284b19681d8b98508fd7"
    sha256 cellar: :any,                 sonoma:         "71d434eeabc2af220285b037f7264563ce9bc77a41af35eabe2213276a37ec2b"
    sha256 cellar: :any,                 ventura:        "0cdb547c696992d08c6613c40934218964f4a061b5413c4b2f013c3f0c3ed253"
    sha256 cellar: :any,                 monterey:       "2ce54302e4524ad28389aca5a16333d4193128e911de2881e6b0e953559d89cd"
    sha256 cellar: :any_skip_relocation, x86_64_linux:   "97d7cbd33b4d0ed09551e3dbc07f830d3df018c2aefbb2222a12ccfb829aae30"
  end

  depends_on "asciidoctor" => :build
  depends_on "pkgconf" => :build
  depends_on "rust" => :build
  depends_on "pcre2"

  def install
    system "cargo", "install", "--features", "pcre2", *std_cargo_args

    generate_completions_from_executable(bin/"rg", "--generate", shell_parameter_format: "complete-")
    (man1/"rg.1").write Utils.safe_popen_read(bin/"rg", "--generate", "man")
  end

  test do
    (testpath/"Hello.txt").write("Hello World!")
    system bin/"rg", "Hello World!", testpath
  end
end

🐇 What’s in that tar file?

wget https://github.com/BurntSushi/ripgrep/archive/refs/tags/14.1.1.tar.gz 
--2025-01-03 06:28:07--  https://github.com/BurntSushi/ripgrep/archive/refs/tags/14.1.1.tar.gz
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/BurntSushi/ripgrep/tar.gz/refs/tags/14.1.1 [following]
--2025-01-03 06:28:08--  https://codeload.github.com/BurntSushi/ripgrep/tar.gz/refs/tags/14.1.1
Resolving codeload.github.com (codeload.github.com)... 140.82.121.10
Connecting to codeload.github.com (codeload.github.com)|140.82.121.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘14.1.1.tar.gz’

     0K .......... .......... .......... .......... ..........  355K
    50K .......... .......... .......... .......... .......... 1.05M
   100K .......... .......... .......... .......... .......... 1.35M
   150K .......... .......... .......... .......... .......... 3.61M
   200K .......... .......... .......... .......... ..........  876K
   250K .......... .......... .......... .......... .......... 6.50M
   300K .......... .......... .......... .......... .......... 1.15M
   350K .......... .......... .......... .......... .......... 1.65M
   400K .......... .......... .......... .......... .......... 4.88M
   450K .......... .......... .......... .......... .......... 1.22M
   500K .......... .......... .......... .......... .......... 1.56M
   550K .......... .......                                     3.62M=0.5s

2025-01-03 06:28:09 (1.21 MB/s) - ‘14.1.1.tar.gz’ saved [581402]

mv 14.1.1.tar.gz $HOME/Downloads/ripgrep-14.1.1.tar.gz

tar xvzf $HOME/Downloads/ripgrep-14.1.1.tar.gz

🐢 It’s a source distribution. It doesn’t contain any binaries.

🦊 We can use the same URL format, it looks. Let’s try this:

wget https://github.com/iesahin/xvc/archive/refs/tags/0.6.13.tar.gz 
--2025-01-03 06:34:00--  https://github.com/iesahin/xvc/archive/refs/tags/0.6.13.tar.gz
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/iesahin/xvc/tar.gz/refs/tags/0.6.13 [following]
--2025-01-03 06:34:01--  https://codeload.github.com/iesahin/xvc/tar.gz/refs/tags/0.6.13
Resolving codeload.github.com (codeload.github.com)... 140.82.121.9
Connecting to codeload.github.com (codeload.github.com)|140.82.121.9|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2025-01-03 06:34:01 ERROR 404: Not Found.

🐢 Our source should be in another location.

🐇 We had an unpublished release. Didn’t we release the binaries a few days ago?

🐢 Umm, there must be something about the gh command where we set draft=false, but seemingly it didn’t work out.

🐇 Ok, the URL is something like:

wget https://github.com/iesahin/xvc/archive/refs/tags/v0.6.13.tar.gz

🐢 Ok, this works. We’ll use this one for the URL.

🐇 What should we put for SHA256?

🦊 Let’s check ripgrep’s again.

🐢 Downloading and using xvc file hash (alias xvcfh):

wget https://github.com/BurntSushi/ripgrep/archive/refs/tags/14.1.1.tar.gz 
...

xvc file hash -a sha2 14.1.1.tar.gz
4dad02a2f9c8c3c8d89434e47337aa654cb0e2aa50e806589132f186bf5c2b66	14.1.1.tar.gz

🐢 Yep, this matches the source.

🦊 Then we can just use the same.

🐇 I want to learn how to download to $TMPDIR with wget.

🐢 The -P option is used for this.

wget -P $TMPDIR https://github.com/iesahin/xvc/archive/refs/tags/v0.6.13.tar.gz 
--2025-01-03 06:55:02--  https://github.com/iesahin/xvc/archive/refs/tags/v0.6.13.tar.gz
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/iesahin/xvc/tar.gz/refs/tags/v0.6.13 [following]
--2025-01-03 06:55:03--  https://codeload.github.com/iesahin/xvc/tar.gz/refs/tags/v0.6.13
Resolving codeload.github.com (codeload.github.com)... 140.82.121.9
Connecting to codeload.github.com (codeload.github.com)|140.82.121.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘/var/folders/gf/18gghtz15_zf0cr8v6dymglw0000gn/T/v0.6.13.tar.gz’
...
2025-01-03 06:55:07 (2.59 MB/s) - ‘/var/folders/gf/18gghtz15_zf0cr8v6dymglw0000gn/T/v0.6.13.tar.gz’ saved [8785449]

xvcfh -a sha2 $TMPDIR/v0.6.13.tar.gz
01bee5d840eefec7be1f52cc75546e1ffd7e332dfac83d875655f84e61e3a9f6	/var/folders/gf/18gghtz15_zf0cr8v6dymglw0000gn/T//v0.6.13.tar.gz

🐇 According to documentation, bottles are produced by Brew itself, and the documentation around bottling taps is limited.

🦊 We can just start with the source distribution. We can test it now.

🐢 The source distribution seems to work, but it requires Rust, and Brew downloads everything related to it.

🐇 I think that may be enough for the time being. With 0.6.14, we can work on providing bottles.

🐢 Umm, yes. I’m already bored with this stuff.

Devlog 10: Codecov, Doctests, and Cargo Publish Struggles

Emre Şahin — 2025-01-19T09:30:49+00:00

🐢 Now we’re going into the real deep work. The only failure was the Codecov patch in Xvc. Let’s see what needs to be done.

🐇 It looks, from the coverage page, that our additions to HStore don’t have any tests. We can add some unit tests to new joins, maybe.

🐢 I don’t find unit tests particularly useful, but let’s use GitHub Copilot to add tests for us.

🐇 Added a unit test and a doc test for full_join, and I think doc tests have more value. They provide documentation, and we can readily see how to use a function from its docs. Better to increase coverage with doc tests.

🐢 There are points that I should learn while writing doc tests. The imports must use the full path, not crate::. The tested struct also doesn’t have implicit imports.

🐇 The ceremony of adding keys and values is a bit too much. It may be worthwhile to add insert for any Into.

🐢 It will certainly save time if we don’t have to type .into() for each key :)

🐇 Pushed to test again. Should we have some means to test coverage locally?

🐢 I don’t think we need to consider coverage locally. It’s not worth our time.

🐇 Now while waiting for tests to be completed, what can we do?

gh -R iesahin/xvc run list
completed	failure	v0.6.13	Rust-CI	v0.6.13	pull_request	12543831360	3m54s	2024-12-30T08:15:06Z
...

🐢 I don’t think it took too much time. Let’s view the results:

gh -R iesahin/xvc run view 12543831360
...
  X Run Current Dev Tests
...
To see what failed, try: 
View this run on GitHub: https://github.com/iesahin/xvc/actions/runs/12543831360

🐢 The current dev tests fail for some reason. Let’s run these locally.

🐇 We’re missing llvm-tools-preview locally. How do we install this?

🐢 The command is:

rustup component add llvm-tools-preview
info: component 'llvm-tools' for target 'aarch64-apple-darwin' is up to date

🐇 It’s already installed. We need to set the environment variables.

🐢 Instead, we can just turn off dev tests for the time being. We don’t need them. Our local tests pass.

🐇 Yeah, ok, we don’t need to solve each and every bit of these issues.

gh -R iesahin/xvc run list

… 🐇 Ok. Let’s take a look at the run again.

gh -R iesahin/xvc run list
completed	failure	v0.6.13	Rust-CI	v0.6.13	pull_request	12544020064	3m39s	2024-12-30T08:33:25Z

gh -R iesahin/xvc run view 12544020064
...
  X Test and Coverage
...

gh run view 12544020064 --log-failed
...
Test and Coverage (stable)	Test and Coverage	2024-12-30T08:36:58.8911690Z Error: ProcessError { stdout: "", stderr: "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to e1.xvc.dev port 80 after 160 ms: Couldn't connect to server\n" }
...

🐢 We need to start nginx on the server. We forgot it yesterday.

🐇 Ah, yeah. After adding that dufs installation. Ok.

🐢 We also need to add a reverse proxy to dufs somehow, but this is for later.

🐇 For the use case, I don’t think it’s necessary. We can just adjust the port to a non-standard one if we need to use 443 for another thing, but let’s take a look at the tests again.

🐢 Let’s add another doc test. This time to XvcStore.

cargo test -p xvc-ecs --doc
...
test result: ok. 8 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.87s

🐇 Sent the files again. Waiting for tests to finish.

🐢 Let’s check the keymaps file in the meantime.

🐇 I tried to read some documentation but didn’t see an error. Maybe we should just set it to non-lazy and avoid allowing remaps.

🐢 We already spent too much time with this.

🐇 Yes, let’s take a look at the tests again.

gh -R iesahin/xvc run list
completed	success	v0.6.13	Rust-CI	v0.6.13	pull_request	12544291068	8m10s	2024-12-30T09:00:42Z
...

🐢 Oh, yeah, the merge is ready.

🐇 Patch coverage is still behind the target, though.

🐢 Yeah, but let’s release this one and make the next better covered. Also, I’m not sure if the doc tests had any effect on coverage.

🐇 If we look at the coverage page again, we can see if the doc tests had any effect.

🐢 It seems codecov.io doesn’t consider coverage for doc tests. This is a bit weird, but let’s not spend more time on this.

🐇 Sure, let’s merge.

🐢 I think we forgot to bump the version in Cargo.toml. We’ll have to do that in main.

🐇 Oh, yeah. Let’s bump it and tag as well.

🐢 Now, we can wait for all files to be produced. What will we do next?

🐇 We can just release the Python version as well. It shouldn’t need any changes.

🐢 Umm, right. Maybe we can add a few tests as well.

🐇 Let’s bump the version first and see.

🐢 Bumped versions in Cargo.toml and ran maturin develop.

🐇 It seems ready now.

🐢 The main fails, though. The “Publish Crates” action looks for libsqlite3. Let’s take a look at it.

gh -R iesahin/xvc run list
completed	failure	Release v0.6.13	Publish Crates	v0.6.13	push	12544753359	6m1s	2024-12-30T09:41:56Z

🐢 The issue is that the VM doesn’t have libsqlite3-dev. Let’s add it.

🐇 We need to start the job manually again. Let’s not tag this time.

🐢 Some of the packages were already published. Now they break. crates.io says they’re already published. Maybe we can check if a package is already published.

🐇 Let’s check if we can make cargo publish more forgiving.

🐢 There doesn’t seem to be an option. Let’s search “how to skip published packages in the workspace to avoid errors with cargo publish”.

🐇 Claude is bullshitting again. Let’s try a manual approach: how to skip already published packages.

🐢 It may be easier to just add a check if the package is published. How do we get the info?

🐇 Or we can just go on to the next package if the package is already available.

🐢 Let’s do this manually this time.

devlog 9

Emre Şahin — 2025-01-04T13:03:16+00:00

🐢 Today I’m planning to add cross-compilation to the Xvc 0.6.13 branch to provide more platform support.

🐇 It looks like you first need to turn off this ghost text from the blink output. It makes writing insufferable.

🐢 Yep, let’s do that first.

🐇 Now, let’s restart Neovim.

🐢 I don’t know why blink.cmp doesn’t prioritize the emojis I use. Maybe we can just use #tor and #rab for ourselves.

🐇 I think over time it will learn that the emojis I defined in the snippets file should have higher priority, but let’s skip this for now. What do we need to do to add cross-compilation?

🐢 Maybe we can just make the completion menu wait a bit longer. It shows up almost instantly, and I want it to wait for a few more milliseconds.

🐇 Okay, let’s look at the config.

🐢 The configuration file doesn’t seem to have a key for this. Let’s search: blink.cmp.

🐇 I think the culprit is typo resistance; it causes better options to be pushed down: https://cmp.saghen.dev/configuration/reference#fuzzy

🐢 Let’s turn that off.

🐇 There is an error in the configuration file. I don’t know why it’s failing.

🐢 I added emojis to Espanso and checked the error message. The configuration I copied from their docs seems to be broken. The message is:

...share/nvim/lazy/blink.cmp/lua/blink/cmp/config/utils.lua:14: fuzzy.max_items: unexpected field found in configuration

I’ll just delete that line.

🐇 It still doesn’t prioritize snippets, but we can look into this later. Espanso seems to be a better tool for this anyway.

🐢 Yep. Let’s look into cross-compilation support for Rust.

🐇 The well-known option is cross.rs: https://github.com/cross-rs/cross

🐢 We can start with that. Installation is from the Git repository:

$ cargo install cross --git https://github.com/cross-rs/cross
    Updating git repository `https://github.com/cross-rs/cross`
    Updating git submodule `https://github.com/cross-rs/cross-toolchains.git`
  Installing cross v0.2.5 (https://github.com/cross-rs/cross#4090beca)
...
   Installed package `cross v0.2.5 (https://github.com/cross-rs/cross#4090beca)` (executables `cross`, `cross-util`)

🐇 Now we have the cross and cross-util commands. Cross-compilation requires Podman on Linux or Docker on macOS. Do we have Docker?

🐢 It looks like we don’t. Maybe we can just set up a remote build using Podman or GitHub Actions. I saw a crate for that yesterday; I remember saving it somewhere but can’t find it now. Searching again seems easier, which says something about my archival and retrieval habits.

🐇 Maybe later you can add some vector search capabilities to your archive—semantic search.

🐢 Yep, sometime later.

🐇 Now let’s search for “adding rust cross compilation to github actions.”

🐢 I found a link to the action: https://github.com/marketplace/actions/build-rust-projects-with-cross

Let’s look at the example:

jobs:
  release:
    name: Release - ${{ matrix.platform.os-name }}
    strategy:
      matrix:
        platform:
          - os-name: FreeBSD-x86_64
            runs-on: ubuntu-20.04
            target: x86_64-unknown-freebsd
            skip_tests: true

          - os-name: Linux-x86_64
            runs-on: ubuntu-20.04
            target: x86_64-unknown-linux-musl

          - os-name: Linux-aarch64
            runs-on: ubuntu-20.04
            target: aarch64-unknown-linux-musl

          - os-name: Linux-riscv64
            runs-on: ubuntu-20.04
            target: riscv64gc-unknown-linux-gnu

          - os-name: Windows-x86_64
            runs-on: windows-latest
            target: x86_64-pc-windows-msvc

          - os-name: macOS-x86_64
            runs-on: macOS-latest
            target: x86_64-apple-darwin

          # more targets here ...

    runs-on: ${{ matrix.platform.runs-on }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build binary
        uses: houseabsolute/actions-rust-cross@v0
        with:
           command: ${{ matrix.platform.command }}
          target: ${{ matrix.platform.target }}
          args: "--locked --release"
          strip: true
      - name: Publish artifacts and release
        uses: houseabsolute/actions-rust-release@v0
        with:
          executable-name: ubi
          target: ${{ matrix.platform.target }}

🐇 We can just convert the current configuration to this and see if it works.

🐢 Let’s do that. I created .github/workflows/release.yml and will update it.

🐇 Let’s check the results with gh:

$ gh -R iesahin/xvc run list
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525070531	34s	2024-12-28T08:17:28Z

🐢 It says the release has completed.

$ gh -R iesahin/xvc run view 12525070531

X v0.6.13 Release iesahin/xvc#263 · 12525070531
Triggered via pull_request about 3 minutes ago

JOBS
X Release - FreeBSD-x86_64 in 12s (ID 34936257653)
  ✓ Set up job
  ✓ Checkout
  X Build binary
  - Publish artifacts and release
  ✓ Post Build binary
  ✓ Post Checkout
  ✓ Complete job
...

Now let’s look at the failure:

$ gh -R iesahin/xvc run view 12525070531 --log-failed

🐇 Change the order and see what happens. Maybe we should first succeed in a non-cross-compilation build.

$ gh -R iesahin/xvc run list
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525178664	35s	2024-12-28T08:34:18Z
...
$ gh -R iesahin/xvc run view   12525178664 --log-failed
...

🐢 It’s the same error. The usage example is actually much more sophisticated.

🐇 The issue is that we’re asking for command from the matrix, but the matrix doesn’t define it. That’s the second time today an example from the documentation has failed. I’ve added command for each platform. We can also add separate features this way to make platform-specific functionality work.

🐢 Agreed. Let’s look at it once more.

$ gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525253982	41s	2024-12-28T08:46:23Z

$ gh -R iesahin/xvc run view 12525253982 --log-failed
...
Release - macOS-x86_64	Build binary	2024-12-28T08:46:44.8464010Z  [1m [31merror [0m [1m: [0m the lock file /Users/runner/work/xvc/xvc/Cargo.lock needs to be updated but --locked was passed to prevent this

🐇 Ah, that’s a different error. Let’s remove the --locked flag and retry.

$ gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525289176	1m8s	2024-12-28T08:53:23Z

🐢 We’re finally starting to get some good news. Now we’re getting OpenSSL errors. We need feature flags for these platforms or to specify where OpenSSL is. Let’s add bundled-openssl to the failed ones. We also need bundled-sqlite for Windows binaries.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525354179	3m9s	2024-12-28T09:04:55Z

🐇 It looks like the Changes.md file is missing, and it can’t upload the binaries as a release because of this. I’ll set the changes file to CHANGELOG.md.

🐢 It’s weird to fail because of that. I think we should report these errors and possibly send a PR to make that file optional.

gh -R iesahin/xvc run list | rg Release | head -n 1
in_progress		v0.6.13	Release	v0.6.13	pull_request	12525449368	1m38s	2024-12-28T09:18:40Z
...
✓ Release - macOS-x86_64 in 2m39s (ID 34937017695)
✓ Release - macOS-aarch64 in 2m42s (ID 34937017787)
..
ARTIFACTS
xvc-macOS-x86_64.tar.gz
xvc-macOS-arm64.tar.gz

🐢 It looks like Linux-riscv64 has OpenSSL compilation errors. I think we can skip this platform for now. The goal was to add aarch64 for macOS, and that seems to have succeeded.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525504079	3m24s	2024-12-28T09:26:43Z
...
X Release - Linux-x86_64	Build binary	2024-12-28T09:29:16.2262518Z  [0m [1m [38;5;9merror[E0308] [0m [0m [1m: mismatched types [0m
...

🐢 It looks like Linux-x86_64 doesn’t support reflinks. Maybe we can make reflinks an optional feature and add it specifically to macOS and Windows targets.

🐇 I’ve removed reflink from the default features. This is a breaking change, but fortunately we don’t have many users who will be affected by it.

Let’s check the results once more.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525597866	3m39s	2024-12-28T09:42:44Z

🐢 Now turn off the NetBSD target as well.

🐇 FreeBSD and Linux-x86_64 targets are building. Let’s see why the ARM Linux targets are failing.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525643284	4m46s	2024-12-28T09:52:04Z

🐇 It looks like those targets don’t have libsqlite3 installed. Let’s add bundled-sqlite to these targets too.

gh -R iesahin/xvc run list | rg Release | head -n 1
completed	failure	v0.6.13	Release	v0.6.13	pull_request	12525734325	4m11s	2024-12-28T10:07:36Z

🐢 Adding Android as a target didn’t work. Let’s remove it for now; it seems to require more work.

🐇 I think we’ll eventually move to using Xvc to distribute binaries. We could build the Android binary on Termux and link it on the releases page or push it as a release artifact.

🐢 I need to learn more about GitHub releases. If we can add artifacts to the release, maybe we can do some of this work locally.

🐇 We can start by looking at the capabilities of gh commands.

🐢 It looks like it’s possible to upload assets to releases. Let’s check how that works.

gh release upload --help

🐇 So basically, we can just upload files to tags. We can list releases and work with them like anything else.

gh -R iesahin/xvc release list

And we can delete releases:

for r in v0.4.2-alpha.8 v0.4.2-alpha.7 v0.4.2-alpha.6 v0.4.2-alpha.5 v0.4.2-alpha.0 v0.4.1-alpha.0; do
  gh -R iesahin/xvc release delete "${r}"
done

🐢 Now let’s list them again.

gh -R iesahin/xvc release list

🐇 The latest one has failed again.

X Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run

🐢 It looks like we’re coming to the end of this session. Under what conditions will we create a release?

🐇 I think it’s better to release only non-alpha tags.

🐢 Then the rule in the workflow will be something like:

on:
  workflow_dispatch:
  push:
    tags:
      - "v*.*.*"
      - "!v.*.*-alpha.*"

🐇 And now we have this result:

✓ v0.6.13 Release iesahin/xvc#263 · 12533498361
Triggered via pull_request about 21 minutes ago

JOBS
✓ Release - FreeBSD-x86_64 in 4m14s
✓ Release - Linux-x86_64 in 4m3s
✓ Release - Linux-aarch64 in 4m50s
✓ Release - Windows-x86_64 in 8m47s
✓ Release - Windows-aarch64 in 6m42s
✓ Release - macOS-x86_64 in 2m37s
✓ Release - macOS-aarch64 in 2m36s

🐢 Nice! Let’s check the release list.

gh -R iesahin/xvc release list

🐇 Why is there no “latest” release?

🐢 We have to tag it first.

🐇 Ah, right. Let’s tag it then.

🐢 I’ve pushed the changes and tagged them with v0.6.13-alpha.5.

🐇 I think it’s possible to make a release today.

🐢 It looks like it, yes.

🐇 We can merge the PR and tag it. Then everything should work.

🐢 We need some cleanup in the YAML files, though.

🐇 We can leave that for the next release.

gh -R iesahin/xvc run list
in_progress		v0.6.13	Rust-CI	v0.6.13	pull_request	12533690474	9m18s	2024-12-29T08:02:29Z
completed	success	Release	Release	v0.6.13-alpha.5	push	12533689099	9m11s	2024-12-29T08:02:19Z

🐢 Now we have another failure in the regular CI. Let’s look at it.

🐇 The issue seems to be in the doc tests:

- Total #: 8 Workspace Size:         276 Cached Size:          19
+ Total #: 8 Workspace Size:         278 Cached Size:          19

🐢 These tests are brittle, but they provide valuable information. Let’s fix it and push again.

🐇 Done. We can also remove some of the watches that produce so many logs.

🐢 I’m a bit ambivalent about them. I thought we could use these watches when debugging, but experience has shown that we need more granular watches during debugging and almost never use these otherwise. Let’s remove some of them.

🐇 We can increase the output for certain commands, but the tracing output doesn’t help much in regular runs. If Xvc gets popular enough that we can’t cope with bug reports, we can always add more watches.

🐢 Another option is to exclude the watch code from the release build, but that won’t change anything for our debug cycles.

🐇 I think removing them is a fair trial. We can always put them back when debugging.

🐢 Watches could also produce regular output instead of tracing. That way we won’t forget to remove them.

🐇 Ah, yep, that’s a good option too.

🐢 We can have a trace! macro similar to the current one for user consumption, and a watch! macro that sends output to stderr.

🐇 Good idea. Let’s do that in the next release.

🐢 Let’s check the tests before pushing this cleanup.

gh -R iesahin/xvc run list
completed	success	v0.6.13	Rust-CI	v0.6.13	pull_request	12533993607	13m49s	2024-12-29T08:47:50Z

🐇 CI succeeded, and the release didn’t run. Let’s do some more cleanup.

gh -R iesahin/xvc run list
in_progress		v0.6.13	Rust-CI	v0.6.13	pull_request	12534416152	43s	2024-12-29T09:57:02Z

devlog 8

Emre Şahin — 2025-01-04T12:41:16+00:00

🐢 The only failure was the patch coverage in Xvc. Let’s see what needs to be done.

🐇 It looks, from the coverage page, that our additions to HStore don’t have any tests. Maybe we can add some unit tests to the new joins.

🐢 I don’t find unit tests particularly useful, but let’s use GitHub Copilot to add some for us.

🐇 I added a unit test and a doc test for full_join. I think doc tests have more value; they provide documentation, and we can readily see how to use a function from its docs. It’s better to increase coverage with doc tests.

🐢 There are things I should keep in mind while writing doc tests. Imports must use the full path, not crate. Also, the tested struct doesn’t have implicit imports.

🐇 The ceremony for adding keys and values is a bit too much. It might be worthwhile to add an insert method for anything that implements Into.

🐢 That would certainly save time—no more typing .into() for each key! :)

🐇 Pushed to test again. Should we have some way to test coverage locally?

🐢 I don’t think we need to check coverage locally. It’s not worth our time right now.

🐇 Now, while waiting for the tests to complete, what else can we do?

gh -R iesahin/xvc run list
completed failure v0.6.13 Rust-CI v0.6.13 pull_request 12543831360 3m54s 2024-12-30T08:15:06Z
...

🐢 That didn’t take too long. Let’s view the results:

gh -R iesahin/xvc run view 12543831360
...
  X Run Current Dev Tests
...
To see what failed, try: 
View this run on GitHub: https://github.com/iesahin/xvc/actions/runs/12543831360

🐢 The current dev tests are failing for some reason. Let’s run them locally.

🐇 We’re missing llvm-tools-preview locally. How do I install this?

🐢 The command is rustup component add llvm-tools-preview:

info: component 'llvm-tools' for target 'aarch64-apple-darwin' is up to date

🐇 It’s already installed. We just need to set the environment variables.

🐢 Instead, we can just turn off dev tests for the time being. We don’t really need them; our local tests pass.

🐇 Yeah, okay. We don’t need to solve every single bit of these issues right now.

gh -R iesahin/xvc run list
...

🐇 Okay, let’s take a look at the run again.

gh -R iesahin/xvc run list
completed failure v0.6.13 Rust-CI v0.6.13 pull_request 12544020064 3m39s 2024-12-30T08:33:25Z

gh -R iesahin/xvc run view 12544020064
...
  X Test and Coverage
...

gh run view 12544020064 --log-failed
...
Test and Coverage (stable) Test and Coverage 2024-12-30T08:36:58.8911690Z Error: ProcessError { stdout: "", stderr: "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (7) Failed to connect to e1.xvc.dev port 80 after 160 ms: Couldn't connect to server\n" }
...

🐢 We need to start Nginx on the server. We forgot to do that yesterday.

🐇 Ah, right. After that dufs installation. Okay.

🐢 We also need to add a reverse proxy for dufs somehow, but that’s for later.

🐇 For this use case, I don’t think it’s necessary. We can just adjust the port to a non-standard one if we need 443 for something else. Let’s look at the tests again.

🐢 Let’s add another doc test, this time to XvcStore.

cargo test -p xvc-ecs --doc
...
test result: ok. 8 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.87s

🐇 Sent the files again. Waiting for the tests to finish.

🐢 Let’s check the keymaps file in the meantime.

🐇 I tried reading the documentation but didn’t see an error. Maybe we should just set it to non-lazy and disallow remaps.

🐢 We’ve already spent too much time on this.

🐇 Yes, let’s check the tests again.

gh -R iesahin/xvc run list
completed success v0.6.13 Rust-CI v0.6.13 pull_request 12544291068 8m10s 2024-12-30T09:00:42Z
...

🐢 Oh, nice, the merge is ready.

🐇 Patch coverage is still behind the target, though.

🐢 Yeah, but let’s release this one and ensure the next one is better covered. Also, I’m not sure if the doc tests actually affected the coverage report.

🐇 If we look at the coverage page again, we can see.

🐢 It seems codecov.io doesn’t consider coverage for doc tests. That’s a bit weird, but let’s not spend more time on it.

🐇 Sure, let’s merge.

🐢 I think we forgot to bump the versions in the Cargo.toml files. We’ll have to do that in main.

🐇 Oh, yeah. Let’s bump them and tag the release as well.

🐢 Now we can wait for all the files to be produced. What’s next?

🐇 We can release the Python version too. It shouldn’t need any changes.

🐢 Right. Maybe we can add a few tests there as well.

🐇 Let’s bump the version first and see.

🐢 I’ve bumped the versions in Cargo.toml and run maturin develop.

🐇 It seems ready now.

🐢 The main branch is failing, though. The “Publish crates” action is looking for libsqlite3. Let’s investigate.

gh -R iesahin/xvc run list
completed failure Release v0.6.13 Publish Crates v0.6.13 push 12544753359 6m1s 2024-12-30T09:41:56Z

🐢 The issue is that the VM doesn’t have libsqlite3-dev. Let’s add it.

🐇 We need to restart the job manually. Let’s skip tagging this time.

🐢 Some of the packages were already published, and now they’re causing failures because crates.io says they already exist. Maybe we can check if a package is already published before trying.

🐇 Let’s see if we can make cargo publish more forgiving.

🐢 There doesn’t seem to be an easy option. Let’s search for “how to skip published packages in workspace to avoid errors with cargo publish.”

🐇 Claude is hallucinating again. Let’s try a manual approach: how to skip already published packages?

🐢 It might be easier to just add a check. How do we get that info?

🐇 Or we can just move on to the next package if one is already available.

🐢 Let’s push the missing packages manually this time.

🐢 We should have a key to open garden files quickly. What does Fzf-Lua files receive as arguments?

🐇 Let’s check the help page: fzf-lua.

🐢 Before that, maybe we can try to fix why selected lines are not searched in Visual-Line mode?

🐇 Yep, let’s fix our search first.

🐢 How do you set a key in visual line mode in Neovim Lua?

🐇 The abbreviation is V. Let’s try that.

🐢 Looks like we need to restart the session.

🐇 It’s still not working. Let’s check :map.

🐢 It seems it needs more care; I’ll check it later.

👨🏾‍🦲 🐢 Bence kendini biraz daha anlamlı bir işle uğraştırmalısın. 🐇 Ne gibi? 🐢 Belki biraz daha yayın yapmalısın, biraz daha işe başvurmalısın. 🐇 “Meli”, “malı” ile geçiyor ömrümüz.

⌚ 🐢 It looks like we can move most of the daily template to links or commands. We can call it ref/daily. My idea is to fill the page intentionally, without templates.

🐇 It might be better to start with a blank page, yeah. The template makes me a bit nervous. There are too many things to fill in, and most of them aren’t things I like being pushed into doing.

🐢 Let’s start by moving the daily templates to ref/daily. No more daily template.

🐇 Now we can delete the rest of this page. We’ll use the daily links page and maybe have reminders at the end of these sessions.

👨🏽‍⚕️

devlog 7

Emre Şahin — 2024-08-06T06:43:01+00:00

I noticed that I often forget to update the Xvc CHANGELOG. To fix this, I added a pre-push hook that checks the files I’m pushing. If the CHANGELOG is not among them and there are changes to Rust files, it prevents the push. I hope this will remind me to update the logs more frequently.

#!/bin/bash

# Git pre-push hook to check if CHANGELOG.md is included in the push and if the branch is develop

# Get the current branch name

current_branch=$(git rev-parse --abbrev-ref HEAD)

# Check if the branch is develop

if [ "$current_branch" == "main" ]; then
	echo "You are on the main branch. Skipping CHANGELOG.md check."
	exit 0
fi

# remote="$1"
# url="$2"

# Get the list of commits to be pushed
commits=$(git rev-list '@{u}..HEAD')

has_rust_files=$(false)

# TODO: We can iterate to get file list only once
# Get the list of files that are going to be pushed
for commit in $commits; do
	if git diff-tree --no-commit-id --name-only -r "${commit}" | grep -q "\\.rs$"; then
		has_rust_files=$(true)
	fi
done

if [[ ! $has_rust_files ]]; then
	echo "No .rs files in the push, no need to check CHANGELOG"
	exit 0
fi

# Check if CHANGELOG.md is among the files in the commits
for commit in $commits; do
	if git diff-tree --no-commit-id --name-only -r "${commit}" | grep -q "CHANGELOG.md"; then
		echo "CHANGELOG.md is included in the push."
		exit 0
	fi
done

echo "ERROR: CHANGELOG.md is not included in the push."
exit 1

Edit (2025-02-08): Added a check for when only code files are changed.

devlog 6

Emre Şahin — 2024-07-12T08:53:14+00:00

I have a habit of testing against the CLI’s help string output. It allows me to keep the documentation up to date and makes me aware of any undocumented options. When new features are added, the help text changes and the test fails, which prompts me to add those options to the documentation.

I tried the same approach when testing Python bindings with Pytest:

def test_pipeline_step_dependency(empty_xvc_repo):
    dep_help = empty_xvc_repo.pipeline().step().dependency(help=True)
    expected = """
Usage: xvc pipeline step dependency [OPTIONS] --step-name 

Options:
  -s, --step-name 
          Name of the step to add the dependency to
"""
    assert dep_help == expected

This doesn’t work because the help text is generated by clap and skips the usual thread-based output handler. All command output and errors in Xvc are returned as strings from the command, except for the help text that’s generated by clap automatically.

There are probably workarounds for this, but I won’t pursue them further as I don’t test the actual functionality in the Python bindings anyway.

devlog 5

Emre Şahin — 2024-07-08T20:35:24+00:00

While writing Xvc tests for Python, I hit an error caused by the ECS single-load protection.

The single-loader allows only one instance of Xvc to be run in a single process. This is no problem for the shell, but it looks like it won’t be possible to use multiple Xvc instances in a single Python process.

It’s possible to overcome this with an elaborate multiprocessing setup in the wrapper, but I won’t bother with it for now.

devlog 4

Emre Şahin — 2024-06-05T10:04:54+00:00

Let’s start by looking at the debug output issue. We can start by replacing the eprintln! macros with println!, perhaps.

I replaced the eprintln!s with println!, but it didn’t make any difference. Maybe we should remove those statements completely.

I can’t really find the place that kills the kernel. The last command to run is xvc file list:

print(xvc_test_data.file().list("test-data/dir-0002"))

and the output it produces is:

[src/output.rs:144:13] &output_str = "SS         131 2024-06-05 08:57:11 41e16be7          test-data/dir-0002/file-0003.bin\nSS         131 2024-06-05 08:57:11 27f0
efd0          test-data/dir-0002/file-0002.bin\nSS         131 2024-06-05 08:57:11 66de5084          test-data/dir-0002/file-0001.bin\nTotal #: 3 Workspace Size:
      393 Cached Size:        6006\n"

print may be causing the crash, but the more likely cause is the command that comes after this:

!ls -l test-data/dir-0001/

I replaced this with lsd, which also failed. Maybe it’s actually a Python crash or bug.

The way to understand is to create a notebook file with only that cell and try to run it.

The ls line runs fine with a new notebook. It even runs on the README file when run at the beginning. The line that makes the kernel crash is:

xvc_test_data.storage().new_s3(name="backup", bucket_name="xvc-test", region="eu-central-1", storage_prefix="xvc-storage")

We can start by removing the new_s3 part.

The storage() method runs fine. It returns an XvcStorage() object, as it should.

When I run storage().list(), it takes a very long time. The bug is likely related to storage().

It looks like the storage object was adding file instead of storage as a subcommand. I’ve fixed it now.

That was the bug. The README notebook now creates the S3 storage.

What was the reason behind this?

Parsing the CLI to the XvcCLI object was perhaps the culprit. Let’s look at it more clearly.

Let’s try xvc file new s3 as a command to see how it behaves.

It says unrecognized subcommand for new.

This is how it should be, but I wonder why it doesn’t work for the XvcCLI parser.

Anyway, it’s already 13:00, so let’s stop here for today.

devlog 2

Emre Şahin — 2024-05-26T17:29:09+00:00

We were debugging Python bindings for Xvc. xvc.file.track enters an infinite loop when given a non-existent path.

To make debugging easier, we can add file deletions in the ./start-readme script to the notebook itself.

Instead, I’ll run my watcher run-after-commit.sh to ensure that the files are deleted after commits.

That may also work; you can use both as well.

I noticed cli-opts pass --no-system-config etc. by default. Let’s deal with this first.

I fixed that.

I’m testing whether we are in the directory that we should be in with xvc.root("--absolute"), but it looks like it’s not possible to pass a string argument to the command.

Let me take a look at this.

It looks like there have been some changes in optional parameter handling in PyO3. You may need to deal with keyword arguments with decorators.

Now, let’s try xvc.file().track() once more with dir-0001/.

It seems to be working now. With an existing directory, it doesn’t show an error.

What does xvc.file().list() show?

It shows a single string with \n in it. It looks like we need to handle this in the output thread.

I added a replace("\\n", "\n") to output_str at the end, but it didn’t make a difference. I tested in the notebook with list_result.split("\n"), and these \n characters are indeed CRLF. So Jupyter shows CRLF in strings with \n, and this is not something we should try to fix, I think.

You can search how to show \n characters in a Jupyter notebook with CRLF.

The output string should be fed into repr, as in:

# Define a string with CRLF characters
text_crlf = "Hello\r\nWorld\r\nThis is a test string."

# Use repr to show the \r\n characters explicitly
print(repr(text_crlf))

I tested this, and GPT misleads. repr is when you want to show \n, not vice versa. When I print(list_result), it prints the results properly.

It looks like we don’t need to make this a priority now. We can tell the user in the notebook that the commands are intentionally returning strings, and they can process or print them however they want.

Devlog 1: XVC Root and Python Bindings Debugging

Emre Şahin — 2024-05-24T10:12:50+00:00

I should have a dispatch method that receives an XvcRootOpt and runs a command with it. The dispatcher can also update the XvcRoot from None to Some(XvcRoot) in some cases, so it should receive a mutable XvcRootOpt or return one after receiving ownership.

The issue with having a mutable element is that we need to update XvcRoot from Arc to Arc>, which will cause almost all machinery around XvcRoot to require locking the object first. This is too large a refactoring.

However, I can write an xvc_root! macro to replace xvc_root.read() or whatever is required to minimize the code changes. XvcRoot can also be a wrapper object, but that’s too much fuss, and the responsibility may be misplaced.

Let’s update the type and let the dispatcher update XvcRootInner only when necessary. Let’s see what will require updating.

We have a bug in the Python bindings where xvc.file.track() enters an infinite loop.

There can be multiple reasons, but are you sure that the xvc CLI works with the same command?

Let’s start testing by creating another notebook server.

The notebook creates an Xvc repo successfully and returns the root with xvc.root(), but xvc.file().track("test-data/dir-0001/") never finishes.

It’s likely that this is caused by something in the background threads.

The CLI command completes successfully, but we may be passing the command incorrectly. Let’s try dir-0001 only.

Even if an incorrect path is provided, it shouldn’t enter an infinite loop.

When I provide the correct path, dir-0001, it returns. There may be something going on with finding the root of the Xvc repo.

It returns, but it’s also giving an error that it cannot find a repository. I should take a closer look.

The Xvc struct wasn’t implementing Debug, so I added it.

Let’s add it to others, XvcFile, etc.

Added it. Recompiling to get more info when we run track with an incorrect path.

You can also make it run after a commit to avoid starting a new server if one is already running.

Added conditionals, and now it runs the server if none is running in the background.

PORT=7979
if [[ ! -s "$(ps ax | rg -v ps | rg jupyter-lab | rg $PORT)" ]] ; then
  jupyter lab --port=$PORT --notebook-dir=Readme/ &
  open http://localhost:${PORT}/lab/workspaces/auto-H/tree/Readme.ipynb
fi

devlog

Emre Şahin — 2023-06-12T15:04:19+00:00

I’ve read a few interesting ideas here.
Identifying whether an uploaded document is a template could be a useful feature.
- This is a basic classification task. We can also find ways to extract named entities from the documents and remove them to create a template. Then, we can ask the user to provide values for these entities to generate a complete document.

Xvc Devlog - 221109

Emre Şahin — 2022-11-10T09:26:00+00:00

🐇 How do you want to proceed from here, Mr. 🐢?

🐢 I think I can implement Rsync today. Looking at ssh2-rs, though, I think we can implement file transfer without relying on rsync. It might be easier to implement everything within the code.

🐇 Then you should rename the issue to new ssh.

🐢 Fair. There is also the ssh_rs crate, but it doesn’t have full support for the protocol. Instead, we can have another command, like xvc storage new ssh, that uses libssh2 via the crate mentioned above. It has some limitations with OpenSSH on macOS.

🐇 From the crate’s README, it looks like you can enable the vendored-openssl feature to compile it statically.

🐢 Let’s go ahead then. It’s better to compile it behind a feature flag, though.

🐇 Yup. Rsync can be separate. I think for now you can implement rsync via Exec::cmd and make new ssh a new issue.

🐢 I’ll copy this conversation there.

🐇 Now, let’s start implementing rsync.

🐢 Do we really want to hide it behind a feature flag? It doesn’t bring any extra complexity to generic, for example—just using the commands and returning the errors.

🐇 I think so. If the user doesn’t have rsync on their system, they’ll just get errors. We don’t need to make the implementation optional, but the tests might be.

🐢 OK.

Xvc Devlog - 221108

Emre Şahin — 2022-11-09T10:42:00+00:00

🐇 I think we can begin by checking GitHub PRs. What do you have today?

🐢 The tests for yesterday’s Git integration PR failed. I’ll begin by checking the logs.

🐇 Maybe run the tests locally to see. They might fail on your machine as well. It might be a simple thing.

🐢 Probably, yes. I’ve started the tests now. I shouldn’t forget to run them before testing the PR; it may save some time.

🐇 Ah, yeah, maybe. Yesterday you were already at the end of the workday, so it didn’t matter much. But you can shorten the testing time by reducing the number of files, etc. I think a separate benchmark suite might be good to have. Tests should be shorter; benchmarks should only run for tags.

🐢 Yup. I should shorten the tests. I should make them use a shorter list of files, maybe.

🐇 Local tests have passed. You’ll have to check the logs.

🐢 It seems git diff --name-only --cached returns files not yet added. That’s causing an error in new repositories with stash. Stashing shouldn’t run at all when there are no staged files.

🐇 Git behavior between the local version and the remote version seems different.

🐢 I think the trace should also show the Git version string.

🐇 You should also require a minimum Git version for the commands. You’re using some newer options, and not all users may have them.

🐢 Yep. Let’s see the version on the CI now.

🐢 Ah, I see—the problem is not about versions. It’s that the CI doesn’t configure git config --global user.email and user.name. Commits don’t work without these.

🐇 You should remove the Git version report, then. It’s one more process call for no reason.

🐢 I think it should be in trace!; I can check the verbosity level and call it only if it’s trace.

🐢 I completed the xvc-config docs as well. I think I can merge it before the tests finish, as it’s just a documentation update.

🐇 You seem to be rushing for the release.

🐢 Yeah, I want to really start testing this on the servers.

🐇 Then go ahead; let’s release a new version.

🐇 It looks like you’re back for an evening session. I think it’s time to start using Xvc on your torrent server.

🐢 Ah, yeah. Let’s see how it goes.

🐇 Create a repository for torrents. Then you can add files to it with cache-type = symlink or cache-type = hardlink.

🐢 I think creating a local storage may also help. I can use it to store files to retrieve them later.

🐇 Umm. We don’t have a “garbage collection” facility yet, you know. There are no file deletions at the moment. It may be better to have a repository-to-repository transfer feature, using SSH.

🐢 Yeah, we don’t have SSH storage either. I think this highlights a lack of features.

🐇 It’s possible to mimic Rsync storage with xvc storage new generic, but it doesn’t feel quite the same.

🐢 Then I’m adding these three tickets.

🐇 I think for 0.3.4, the command we add might be xvc file delete. You can work on this and add rsync support as well.

🐢 There is a librsync bindings library for Rust. But it looks like it doesn’t allow transferring file contents between hosts. There is also rusync, which is similar to rsync and implemented in Rust. There is also fast_rsync in pure Rust. It uses MD4 to calculate deltas, though, I think.

🐇 None of these seem to have network capability, though.

🐢 I think it’s better just to use the process for now. We are trying to come up with the simplest solution for now. We are just trying to be more general.

🐇 Yes, I think we can just use the process for the time being.

🐢 Then let’s start by adding it.

🐇 We should also begin to create release notes. GitHub can generate them from PRs. Automatically generated release notes

🐢 Created an issue for that. Now we’re starting xvc storage new rsync #111.

🐇 Ok. Let’s do this.

Xvc Devlog - 221107

Emre Şahin — 2022-11-08T09:57:00+00:00

🐇 Welcome to the November 7th issue of Xvc Devlog. In the previous devlog, we began to implement Git integration. How is it going, Mr. Tortoise?

🐢 It looks like we don’t have many architectural problems.

🐇 You’re forgetting how Exec::cmd works, though. You have those kinds of problems.

🐢 Yeah, I’m figuring that out. Exec::shell requires a string to run on the shell, while Exec::cmd just needs a command name. You have to supply args with another function. I’m writing a closure now to handle this.

🐢 It started working, but it revealed a much bigger problem: .xvc/ is not added to .gitignore in xvc init.

🐇 Wow, that’s a showstopper.

🐢 Yup. I think I should fix it as well.

🐇 On closer glance, I think the problem is not xvc init not modifying .gitignore; it modifies it incorrectly. Maybe putting certain files on a whitelist is a better idea than trying to blacklist everything.

🐢 Yeah, we should define a set of Git-tracked files and directories, whitelist them, and let all other files be ignored.

🐇 Go ahead, then.

🐢 I think I’ve fixed it. There were two problems: the initial gitignore content was wrong, and it was placed in the root of the repository instead of .xvc.

🐇 Maybe putting it directly in the root is better than hiding it in .xvc. What do you think?

🐢 There seems to be an assumption in file track to have a .gitignore file somewhere. I think we should also handle changes in .gitignore files throughout the repository.

🐇 Umm, yes. We should handle .gitignore files as well. But not all .gitignore files are changed by Xvc. How can we make sure they were modified by Xvc?

🐢 I think there are ways to do that, like tracking them somewhere. But I believe we shouldn’t try. We should just get a list of .gitignore files, git add them, and include them in the commit.

🐇 Ok. Let’s write a test for this as well. No .gitignore should appear in git status -s.

🐢 I wrote the test. It fails now. Do you think we should check the output of Git status to determine which files to add?

🐇 That may be a good idea. Git status should already know which files need to be added. We can use that information.

🐢 It looks like pathspec is enough to modify git add behavior. We should be able to write *.gitignore and let all gitignore files be added.

🐇 Let’s try this manually.

🐢 Yep, it works. git add '*.gitignore' adds all .gitignore files in the subdirectories, too.

🐇 Congrats. 👏🥳

🐢 I’m writing the missing documentation for the crates. There are proc_macro not expanded errors all over the code.

🐇 I think you should update rust-analyzer and everything. There must be a command for this.

🐢 I reinstalled RA with :LspInstall and restarted the LSP. It seems to work correctly now.