Xvc Devlog 221030

πŸ‡ We’ll start by checking the repository today. What are the most important issues?

🐒 I merged PR#66 . Let’s list the outstanding issues now.

65	OPEN	Add Mermaid Support to Netlify Installation
64	OPEN	Fix `xvc-storage` compilation warnings	bug	
54	OPEN	Fix all references to `xvc data` in the documentation	
51	OPEN	Add a benchmark script to compare Xvc with other tools
50	OPEN	Consider changing `xvc file push` to `xvc file send` and `xvc file pull` to `xvc file retrieve`
49	OPEN	Add `--generic-command` as a dependency type to `xvc pipeline dependency`
48	OPEN	Write rsync example for `xvc storage generic`	documentation	
47	OPEN	Write rclone example for `xvc storage generic`	documentation
46	OPEN	Allow to skip init in remotes with `--skip-init` option		
45	OPEN	Update `VStore::to_store` to use some internal mechanism to avoid `XvcStore::insert`
43	OPEN	Update to `clap 4.0`
42	OPEN	Clean up xvc crate dependencies
36	OPEN	Add version to released file binaries
33	OPEN	Update `arch/remotes.md` for new naming	documentation
32	OPEN	Remove arc42 sections from the documentation	documentation
29	OPEN	Add documentation for `xvc storage new gcs`	documentation
28	OPEN	`xvc storage new yandex`	enhancement	
24	OPEN	Create a logo for Xvc	documentation
23	OPEN	Add a Github action to upload new versions to crates.io	automation	
20	OPEN	ref: Add rsync example for `xvc storage new generic`
18	OPEN	Add a new workflow for remote tests	automation
12	OPEN	Create a website for Xvc
5	OPEN	Add storage tests to Github Actions	automation	
4	OPEN	fix clippy warnings	bug	
1	OPEN	`xvc storage new` for all S3 compatible cloud services supported by `rust-s3`

πŸ‡ AFAIK, you removed arc42 stubs from the docs yesterday. You can close 32.

🐒 I closed it via gh issue close.

πŸ‡ You should add a comment to that.

🐒 Added the comment.

πŸ‡ It’s October 30th. Looking at the logs, you closed some more issues since we last talked.

🐒 I’m productive for the last few days. Now, let’s take a look at the open issues.

πŸ‡ I think you have a more pressing problem. Your tests fail. You need to fix them first, before moving to another task.

🐒 Created a new issue. The logs say

81Z error: 7 targets failed:
2022-10-29T18:18:36.8969467Z ##[error]    `-p xvc-workflow-tests --test test_storage_new_digital_ocean`
2022-10-29T18:18:36.8970697Z     `-p xvc-workflow-tests --test test_storage_new_gcp`
2022-10-29T18:18:36.8971279Z     `-p xvc-workflow-tests --test test_storage_new_generic_rsync`
2022-10-29T18:18:36.8971840Z     `-p xvc-workflow-tests --test test_storage_new_minio`
2022-10-29T18:18:36.8972387Z     `-p xvc-workflow-tests --test test_storage_new_r2`
2022-10-29T18:18:36.8973035Z     `-p xvc-workflow-tests --test test_storage_new_s3`
2022-10-29T18:18:36.8973577Z     `-p xvc-workflow-tests --test test_storage_new_wasabi`
2022-10-29T18:18:36.9368079Z ##[error]The process '/home/runner/.cargo/bin/cargo' failed with exit code 101

πŸ‡ Looking at the logs, you should clean up those older branches.

🐒 Let’s do it now.

🐒 Done. I’ve deleted all branches except main.

πŸ‡ Good. I think the best way is to delete them as soon as you merge.

🐒 I activated that. These are older branches.

πŸ‡ Cool. Now will be checking the failing tests?

🐒 I think remote tests should never run if there are other errors. We can run coverage and remote tests as a second step after the first succeed.

πŸ‡ That seems a neat idea. You want to split the current job into two. One for compiling and testing the non-remote parts. The second for coverage and storage tests. It won’t spend the minutes that way?

🐒 On second thought, I think at the moment, that’s not a pressing issue. We should start to fix these tests at once. We can split them later.

πŸ‡ Ok. It looks from the logs that secrets are not made available to your jobs?

2022-10-29T18:18:17.5039311Z test test_storage_new_digital_ocean ... FAILED
2022-10-29T18:18:17.5039550Z 
2022-10-29T18:18:17.5039652Z failures:
2022-10-29T18:18:17.5039784Z 
2022-10-29T18:18:17.5040197Z ---- test_storage_new_digital_ocean stdout ----
2022-10-29T18:18:17.5040558Z Error: VarError { source: NotPresent }
2022-10-29T18:18:17.5040758Z 
2022-10-29T18:18:17.5040765Z

🐒 I added them now. Let’s wait until the job ends to get a new set of logs.

πŸ‡ You can write some documentation in the meantime.

🐒 I think #82 is a good candidate for this. There must not be too many missing docs in the ECS crate.

πŸ‡ It was about walker and you fixed the ecs. You’re the most absentminded developer here, I believe.

🐒 Ooops, you’re right. I’ll add them together in a single PR.

πŸ‡ In the meantime storage testing job has ended with πŸ”΄.

🐒 Checking the raw logs. It looks we didn’t update the tests with the options:

2022-10-30T14:45:50.1058348Z error: Found argument '--storage-prefix' which wasn't expected, or isn't valid in this context
2022-10-30T14:45:50.1059809Z ##[error]Found argument '--storage-prefix' which wasn't expected, or isn't valid in this context
2022-10-30T14:45:50.1061740Z 	If you tried to supply `--storage-prefix` as a value rather than a flag, use `-- --storage-prefix`
2022-10-30T14:45:50.1063220Z 
2022-10-30T14:45:50.1063394Z USAGE:
2022-10-30T14:45:50.1064302Z     xvc storage new digital-ocean --name <NAME> --bucket-name <BUCKET_NAME> --region <REGION>

πŸ‡ Is it --storage-prefix or --remote-prefix. Which one is more clear?

🐒 I think updating the tests to conform to options is better for now. We can update the options later if desired.

πŸ‡ Another failure is rg. You assume it exists on the testing system.

🐒 ripgrep is available in Ubuntu 20, so we can just update the initial package list.

πŸ‡ The same --storage-prefix failure is in minio tests.

🐒 Ok. Fixing.

πŸ‡ S3 tests wants to run new-s3 subcommand. I think we now see why we need these tests in the first place.

🐒 Yeah. Fixing the prefix option too.

πŸ‡ s3cmd is also required. You should add it too.

🐒 Ok. I think we added all missing files. I’ll have to convert mc tests for minio to use s3cmd

πŸ‡ Then, we can try again. Now, we can get back to documentation.

🐒 Added some more documentation to PR#93. I think it’s better to return the storage test errors now.

πŸ‡ It looks you have missing Digital Ocean credentials.

🐒 Let’s take a look.

🐒 I’ve updated the tests to use a config file instead of command line arguments visible in the logs.

πŸ‡ Good practice. You also need to remove previous logs.

🐒 Maybe there is an option for that.

πŸ‡ It looks not. There are masking options. When we add them to secrets, they are masked. But you shouldn’t use them in calls anyway.