Xvc Devlog 221030
π Weβll start by checking the repository today. What are the most important issues?
π’ I merged PR#66 . Letβs list the outstanding issues now.
65 OPEN Add Mermaid Support to Netlify Installation
64 OPEN Fix `xvc-storage` compilation warnings bug
54 OPEN Fix all references to `xvc data` in the documentation
51 OPEN Add a benchmark script to compare Xvc with other tools
50 OPEN Consider changing `xvc file push` to `xvc file send` and `xvc file pull` to `xvc file retrieve`
49 OPEN Add `--generic-command` as a dependency type to `xvc pipeline dependency`
48 OPEN Write rsync example for `xvc storage generic` documentation
47 OPEN Write rclone example for `xvc storage generic` documentation
46 OPEN Allow to skip init in remotes with `--skip-init` option
45 OPEN Update `VStore::to_store` to use some internal mechanism to avoid `XvcStore::insert`
43 OPEN Update to `clap 4.0`
42 OPEN Clean up xvc crate dependencies
36 OPEN Add version to released file binaries
33 OPEN Update `arch/remotes.md` for new naming documentation
32 OPEN Remove arc42 sections from the documentation documentation
29 OPEN Add documentation for `xvc storage new gcs` documentation
28 OPEN `xvc storage new yandex` enhancement
24 OPEN Create a logo for Xvc documentation
23 OPEN Add a Github action to upload new versions to crates.io automation
20 OPEN ref: Add rsync example for `xvc storage new generic`
18 OPEN Add a new workflow for remote tests automation
12 OPEN Create a website for Xvc
5 OPEN Add storage tests to Github Actions automation
4 OPEN fix clippy warnings bug
1 OPEN `xvc storage new` for all S3 compatible cloud services supported by `rust-s3`
π AFAIK, you removed arc42 stubs from the docs yesterday. You can close 32.
π’ I closed it via gh issue close.
π You should add a comment to that.
π’ Added the comment.
π Itβs October 30th. Looking at the logs, you closed some more issues since we last talked.
π’ Iβm productive for the last few days. Now, letβs take a look at the open issues.
π I think you have a more pressing problem. Your tests fail. You need to fix them first, before moving to another task.
π’ Created a new issue. The logs say
81Z error: 7 targets failed:
2022-10-29T18:18:36.8969467Z ##[error] `-p xvc-workflow-tests --test test_storage_new_digital_ocean`
2022-10-29T18:18:36.8970697Z `-p xvc-workflow-tests --test test_storage_new_gcp`
2022-10-29T18:18:36.8971279Z `-p xvc-workflow-tests --test test_storage_new_generic_rsync`
2022-10-29T18:18:36.8971840Z `-p xvc-workflow-tests --test test_storage_new_minio`
2022-10-29T18:18:36.8972387Z `-p xvc-workflow-tests --test test_storage_new_r2`
2022-10-29T18:18:36.8973035Z `-p xvc-workflow-tests --test test_storage_new_s3`
2022-10-29T18:18:36.8973577Z `-p xvc-workflow-tests --test test_storage_new_wasabi`
2022-10-29T18:18:36.9368079Z ##[error]The process '/home/runner/.cargo/bin/cargo' failed with exit code 101
π Looking at the logs, you should clean up those older branches.
π’ Letβs do it now.
π’ Done. Iβve deleted all branches except main.
π Good. I think the best way is to delete them as soon as you merge.
π’ I activated that. These are older branches.
π Cool. Now will be checking the failing tests?
π’ I think remote tests should never run if there are other errors. We can run coverage and remote tests as a second step after the first succeed.
π That seems a neat idea. You want to split the current job into two. One for compiling and testing the non-remote parts. The second for coverage and storage tests. It wonβt spend the minutes that way?
π’ On second thought, I think at the moment, thatβs not a pressing issue. We should start to fix these tests at once. We can split them later.
π Ok. It looks from the logs that secrets are not made available to your jobs?
2022-10-29T18:18:17.5039311Z test test_storage_new_digital_ocean ... FAILED
2022-10-29T18:18:17.5039550Z
2022-10-29T18:18:17.5039652Z failures:
2022-10-29T18:18:17.5039784Z
2022-10-29T18:18:17.5040197Z ---- test_storage_new_digital_ocean stdout ----
2022-10-29T18:18:17.5040558Z Error: VarError { source: NotPresent }
2022-10-29T18:18:17.5040758Z
2022-10-29T18:18:17.5040765Z
π’ I added them now. Letβs wait until the job ends to get a new set of logs.
π You can write some documentation in the meantime.
π’ I think #82 is a good candidate for this. There must not be too many missing docs in the ECS crate.
π It was about walker and you fixed the ecs. Youβre the most absentminded developer here, I believe.
π’ Ooops, youβre right. Iβll add them together in a single PR.
π In the meantime storage testing job has ended with π΄.
π’ Checking the raw logs. It looks we didnβt update the tests with the options:
2022-10-30T14:45:50.1058348Z error: Found argument '--storage-prefix' which wasn't expected, or isn't valid in this context
2022-10-30T14:45:50.1059809Z ##[error]Found argument '--storage-prefix' which wasn't expected, or isn't valid in this context
2022-10-30T14:45:50.1061740Z If you tried to supply `--storage-prefix` as a value rather than a flag, use `-- --storage-prefix`
2022-10-30T14:45:50.1063220Z
2022-10-30T14:45:50.1063394Z USAGE:
2022-10-30T14:45:50.1064302Z xvc storage new digital-ocean --name <NAME> --bucket-name <BUCKET_NAME> --region <REGION>
π Is it --storage-prefix or --remote-prefix. Which one is more clear?
π’ I think updating the tests to conform to options is better for now. We can update the options later if desired.
π Another failure is rg. You assume it exists on the testing system.
π’ ripgrep is available in Ubuntu 20, so we can just update the initial package list.
π The same --storage-prefix failure is in minio tests.
π’ Ok. Fixing.
π S3 tests wants to run new-s3 subcommand. I think we now see why we need these tests in the first place.
π’ Yeah. Fixing the prefix option too.
π s3cmd is also required. You should add it too.
π’ Ok. I think we added all missing files. Iβll have to convert mc tests for minio to use s3cmd
π Then, we can try again. Now, we can get back to documentation.
π’ Added some more documentation to PR#93. I think itβs better to return the storage test errors now.
π It looks you have missing Digital Ocean credentials.
π’ Letβs take a look.
π’ Iβve updated the tests to use a config file instead of command line arguments visible in the logs.
π Good practice. You also need to remove previous logs.
π’ Maybe there is an option for that.
π It looks not. There are masking options. When we add them to secrets, they are masked. But you shouldnβt use them in calls anyway.