I replaced the eprintln!s with println!, but it didn’t make any difference. Maybe we should remove those statements completely.

I can’t really find the place that kills the kernel. The last command to run is xvc file list:

print(xvc_test_data.file().list("test-data/dir-0002"))

and the output it produces is:

[src/output.rs:144:13] &output_str = "SS         131 2024-06-05 08:57:11 41e16be7          test-data/dir-0002/file-0003.bin\nSS         131 2024-06-05 08:57:11 27f0
efd0          test-data/dir-0002/file-0002.bin\nSS         131 2024-06-05 08:57:11 66de5084          test-data/dir-0002/file-0001.bin\nTotal #: 3 Workspace Size:
      393 Cached Size:        6006\n"

print may be causing the crash, but the more likely cause is the command that comes after this:

!ls -l test-data/dir-0001/

I replaced this with lsd, which also failed. Maybe it’s actually a Python crash or bug.

The way to understand is to create a notebook file with only that cell and try to run it.

The ls line runs fine with a new notebook. It even runs on the README file when run at the beginning. The line that makes the kernel crash is:

xvc_test_data.storage().new_s3(name="backup", bucket_name="xvc-test", region="eu-central-1", storage_prefix="xvc-storage")

We can start by removing the new_s3 part.

The storage() method runs fine. It returns an XvcStorage() object, as it should.

When I run storage().list(), it takes a very long time. The bug is likely related to storage().

It looks like the storage object was adding file instead of storage as a subcommand. I’ve fixed it now.

That was the bug. The README notebook now creates the S3 storage.

What was the reason behind this?

Parsing the CLI to the XvcCLI object was perhaps the culprit. Let’s look at it more clearly.

Let’s try xvc file new s3 as a command to see how it behaves.

It says unrecognized subcommand for new.

This is how it should be, but I wonder why it doesn’t work for the XvcCLI parser.

Anyway, it’s already 13:00, so let’s stop here for today.

🐢 I’ve merged PR#66. Let’s list the outstanding issues now.

65	OPEN	Add Mermaid Support to Netlify Installation
64	OPEN	Fix `xvc-storage` compilation warnings	bug
54	OPEN	Fix all references to `xvc data` in the documentation
51	OPEN	Add a benchmark script to compare Xvc with other tools
50	OPEN	Consider changing `xvc file push` to `xvc file send` and `xvc file pull` to `xvc file retrieve`
49	OPEN	Add `--generic-command` as a dependency type to `xvc pipeline dependency`
48	OPEN	Write rsync example for `xvc storage generic`	documentation
47	OPEN	Write rclone example for `xvc storage generic`	documentation
46	OPEN	Allow to skip init in remotes with `--skip-init` option
45	OPEN	Update `VStore::to_store` to use some internal mechanism to avoid `XvcStore::insert`
43	OPEN	Update to `clap 4.0`
42	OPEN	Clean up xvc crate dependencies
36	OPEN	Add version to released file binaries
33	OPEN	Update `arch/remotes.md` for new naming	documentation
32	OPEN	Remove arc42 sections from the documentation	documentation
29	OPEN	Add documentation for `xvc storage new gcs`	documentation
28	OPEN	`xvc storage new yandex`	enhancement
24	OPEN	Create a logo for Xvc	documentation
23	OPEN	Add a Github action to upload new versions to crates.io	automation
20	OPEN	ref: Add rsync example for `xvc storage new generic`
18	OPEN	Add a new workflow for remote tests	automation
12	OPEN	Create a website for Xvc
5	OPEN	Add storage tests to Github Actions	automation
4	OPEN	fix clippy warnings	bug
1	OPEN	`xvc storage new` for all S3 compatible cloud services supported by `rust-s3`

🐇 As far as I know, you removed the arc42 stubs from the docs yesterday. You can close issue #32.

🐢 I closed it via gh issue close.

🐇 You should add a comment to that.

🐢 Added the comment.

🐇 It’s October 30th. Looking at the logs, you’ve closed some more issues since we last talked.

🐢 I’ve been productive over the last few days. Now, let’s take a look at the open issues.

🐇 I think you have a more pressing problem. Your tests are failing. You need to fix them first before moving to another task.

🐢 Created a new issue. The logs say:

81Z error: 7 targets failed:
2022-10-29T18:18:36.8969467Z ##[error]    `-p xvc-workflow-tests --test test_storage_new_digital_ocean`
2022-10-29T18:18:36.8970697Z     `-p xvc-workflow-tests --test test_storage_new_gcp`
2022-10-29T18:18:36.8971279Z     `-p xvc-workflow-tests --test test_storage_new_generic_rsync`
2022-10-29T18:18:36.8971840Z     `-p xvc-workflow-tests --test test_storage_new_minio`
2022-10-29T18:18:36.8972387Z     `-p xvc-workflow-tests --test test_storage_new_r2`
2022-10-29T18:18:36.8973035Z     `-p xvc-workflow-tests --test test_storage_new_s3`
2022-10-29T18:18:36.8973577Z     `-p xvc-workflow-tests --test test_storage_new_wasabi`
2022-10-29T18:18:36.9368079Z ##[error]The process '/home/runner/.cargo/bin/cargo' failed with exit code 101

🐇 Looking at the logs, you should clean up those older branches.

🐢 Let’s do it now.

🐢 Done. I’ve deleted all branches except main.

🐇 Good. I think the best way is to delete them as soon as you merge them.

🐢 I’ve activated that setting. These were older branches.

🐇 Cool. Now, will you be checking the failing tests?

🐢 I think remote tests should never run if there are other errors. We can run coverage and remote tests as a second step after the first succeed.

🐇 That seems like a neat idea. You want to split the current job into two: one for compiling and testing the non-remote parts, and the second for coverage and storage tests. It won’t waste CI minutes that way?

🐢 On second thought, I think at the moment, that’s not a pressing issue. We should start fixing these tests at once. We can split them later.

🐇 Ok. It looks from the logs like secrets are not being made available to your jobs.

2022-10-29T18:18:17.5039311Z test test_storage_new_digital_ocean ... FAILED
2022-10-29T18:18:17.5039550Z
2022-10-29T18:18:17.5039652Z failures:
2022-10-29T18:18:17.5039784Z
2022-10-29T18:18:17.5040197Z ---- test_storage_new_digital_ocean stdout ----
2022-10-29T18:18:17.5040558Z Error: VarError { source: NotPresent }
2022-10-29T18:18:17.5040758Z
2022-10-29T18:18:17.5040765Z

🐢 I added them now. Let’s wait until the job ends to get a new set of logs.

🐇 You can write some documentation in the meantime.

🐢 I think #82 is a good candidate for this. There must not be too many missing docs in the ECS crate.

🐇 It was about walker and you fixed the ecs. You’re the most absentminded developer here, I believe.

🐢 Ooops, you’re right. I’ll add them together in a single PR.

🐇 In the meantime, the storage testing job has ended with 🔴.

🐢 Checking the raw logs. It looks like we didn’t update the tests to match the current options:

2022-10-30T14:45:50.1058348Z error: Found argument '--storage-prefix' which wasn't expected, or isn't valid in this context
2022-10-30T14:45:50.1059809Z ##[error]Found argument '--storage-prefix' which wasn't expected, or isn't valid in this context
2022-10-30T14:45:50.1061740Z 	If you tried to supply `--storage-prefix` as a value rather than a flag, use `-- --storage-prefix`
2022-10-30T14:45:50.1063220Z
2022-10-30T14:45:50.1063394Z USAGE:
2022-10-30T14:45:50.1064302Z     xvc storage new digital-ocean --name <NAME> --bucket-name <BUCKET_NAME> --region <REGION>

🐇 Is it --storage-prefix or --remote-prefix? Which one is clearer?

🐢 I think updating the tests to conform to the current options is better for now. We can update the options later if desired.

🐇 Another failure is rg. You assume it exists on the testing system.

🐢 ripgrep is available in Ubuntu 20.04, so we can just update the initial package list.

🐇 The same --storage-prefix failure appears in the minio tests.

🐢 Ok. Fixing it.

🐇 S3 tests want to run the new-s3 subcommand. I think we now see why we need these tests in the first place.

🐢 Yeah. Fixing the prefix option, too.

🐇 s3cmd is also required. You should add it, too.

🐢 Ok. I think we’ve added all missing dependencies. I’ll have to convert the mc tests for Minio to use s3cmd.

🐇 Then we can try again. Now, we can get back to documentation.

🐢 Added some more documentation to PR#93. I think it’s better to return to the storage test errors now.

🐇 It looks like you have missing DigitalOcean credentials.

🐢 Let’s take a look.

🐢 I’ve updated the tests to use a config file instead of command-line arguments that would be visible in the logs.

🐇 Good practice. You also need to remove previous logs.

🐢 Maybe there is an option for that.

🐇 It looks like there isn’t. There are masking options; when we add them to secrets, they are masked. But you shouldn’t use them in calls anyway.