To make debugging easier, we can add file deletions in the ./start-readme script to the notebook itself.

Instead, I’ll run my watcher run-after-commit.sh to ensure that the files are deleted after commits.

That may also work; you can use both as well.

I noticed cli-opts pass --no-system-config etc. by default. Let’s deal with this first.

I fixed that.

I’m testing whether we are in the directory that we should be in with xvc.root("--absolute"), but it looks like it’s not possible to pass a string argument to the command.

Let me take a look at this.

It looks like there have been some changes in optional parameter handling in PyO3. You may need to deal with keyword arguments with decorators.

Now, let’s try xvc.file().track() once more with dir-0001/.

It seems to be working now. With an existing directory, it doesn’t show an error.

What does xvc.file().list() show?

It shows a single string with \n in it. It looks like we need to handle this in the output thread.

I added a replace("\\n", "\n") to output_str at the end, but it didn’t make a difference. I tested in the notebook with list_result.split("\n"), and these \n characters are indeed CRLF. So Jupyter shows CRLF in strings with \n, and this is not something we should try to fix, I think.

You can search how to show \n characters in a Jupyter notebook with CRLF.

The output string should be fed into repr, as in:

# Define a string with CRLF characters
text_crlf = "Hello\r\nWorld\r\nThis is a test string."

# Use repr to show the \r\n characters explicitly
print(repr(text_crlf))

I tested this, and GPT misleads. repr is when you want to show \n, not vice versa. When I print(list_result), it prints the results properly.

It looks like we don’t need to make this a priority now. We can tell the user in the notebook that the commands are intentionally returning strings, and they can process or print them however they want.

The issue with having a mutable element is that we need to update XvcRoot from Arc<XvcRootInner> to Arc<RwLock<XvcRootInner>>, which will cause almost all machinery around XvcRoot to require locking the object first. This is too large a refactoring.

However, I can write an xvc_root! macro to replace xvc_root.read() or whatever is required to minimize the code changes. XvcRoot can also be a wrapper object, but that’s too much fuss, and the responsibility may be misplaced.

Let’s update the type and let the dispatcher update XvcRootInner only when necessary. Let’s see what will require updating.

We have a bug in the Python bindings where xvc.file.track() enters an infinite loop.

There can be multiple reasons, but are you sure that the xvc CLI works with the same command?

Let’s start testing by creating another notebook server.

The notebook creates an Xvc repo successfully and returns the root with xvc.root(), but xvc.file().track("test-data/dir-0001/") never finishes.

It’s likely that this is caused by something in the background threads.

The CLI command completes successfully, but we may be passing the command incorrectly. Let’s try dir-0001 only.

Even if an incorrect path is provided, it shouldn’t enter an infinite loop.

When I provide the correct path, dir-0001, it returns. There may be something going on with finding the root of the Xvc repo.

It returns, but it’s also giving an error that it cannot find a repository. I should take a closer look.

The Xvc struct wasn’t implementing Debug, so I added it.

Let’s add it to others, XvcFile, etc.

Added it. Recompiling to get more info when we run track with an incorrect path.

You can also make it run after a commit to avoid starting a new server if one is already running.

Added conditionals, and now it runs the server if none is running in the background.

PORT=7979
if [[ ! -s "$(ps ax | rg -v ps | rg jupyter-lab | rg $PORT)" ]] ; then
  jupyter lab --port=$PORT --notebook-dir=Readme/ &
  open http://localhost:${PORT}/lab/workspaces/auto-H/tree/Readme.ipynb
fi

🐢 I’ve merged PR#66. Let’s list the outstanding issues now.

65	OPEN	Add Mermaid Support to Netlify Installation
64	OPEN	Fix `xvc-storage` compilation warnings	bug
54	OPEN	Fix all references to `xvc data` in the documentation
51	OPEN	Add a benchmark script to compare Xvc with other tools
50	OPEN	Consider changing `xvc file push` to `xvc file send` and `xvc file pull` to `xvc file retrieve`
49	OPEN	Add `--generic-command` as a dependency type to `xvc pipeline dependency`
48	OPEN	Write rsync example for `xvc storage generic`	documentation
47	OPEN	Write rclone example for `xvc storage generic`	documentation
46	OPEN	Allow to skip init in remotes with `--skip-init` option
45	OPEN	Update `VStore::to_store` to use some internal mechanism to avoid `XvcStore::insert`
43	OPEN	Update to `clap 4.0`
42	OPEN	Clean up xvc crate dependencies
36	OPEN	Add version to released file binaries
33	OPEN	Update `arch/remotes.md` for new naming	documentation
32	OPEN	Remove arc42 sections from the documentation	documentation
29	OPEN	Add documentation for `xvc storage new gcs`	documentation
28	OPEN	`xvc storage new yandex`	enhancement
24	OPEN	Create a logo for Xvc	documentation
23	OPEN	Add a Github action to upload new versions to crates.io	automation
20	OPEN	ref: Add rsync example for `xvc storage new generic`
18	OPEN	Add a new workflow for remote tests	automation
12	OPEN	Create a website for Xvc
5	OPEN	Add storage tests to Github Actions	automation
4	OPEN	fix clippy warnings	bug
1	OPEN	`xvc storage new` for all S3 compatible cloud services supported by `rust-s3`

🐇 As far as I know, you removed the arc42 stubs from the docs yesterday. You can close issue #32.

🐢 I closed it via gh issue close.

🐇 You should add a comment to that.

🐢 Added the comment.

🐇 It’s October 30th. Looking at the logs, you’ve closed some more issues since we last talked.

🐢 I’ve been productive over the last few days. Now, let’s take a look at the open issues.

🐇 I think you have a more pressing problem. Your tests are failing. You need to fix them first before moving to another task.

🐢 Created a new issue. The logs say:

81Z error: 7 targets failed:
2022-10-29T18:18:36.8969467Z ##[error]    `-p xvc-workflow-tests --test test_storage_new_digital_ocean`
2022-10-29T18:18:36.8970697Z     `-p xvc-workflow-tests --test test_storage_new_gcp`
2022-10-29T18:18:36.8971279Z     `-p xvc-workflow-tests --test test_storage_new_generic_rsync`
2022-10-29T18:18:36.8971840Z     `-p xvc-workflow-tests --test test_storage_new_minio`
2022-10-29T18:18:36.8972387Z     `-p xvc-workflow-tests --test test_storage_new_r2`
2022-10-29T18:18:36.8973035Z     `-p xvc-workflow-tests --test test_storage_new_s3`
2022-10-29T18:18:36.8973577Z     `-p xvc-workflow-tests --test test_storage_new_wasabi`
2022-10-29T18:18:36.9368079Z ##[error]The process '/home/runner/.cargo/bin/cargo' failed with exit code 101

🐇 Looking at the logs, you should clean up those older branches.

🐢 Let’s do it now.

🐢 Done. I’ve deleted all branches except main.

🐇 Good. I think the best way is to delete them as soon as you merge them.

🐢 I’ve activated that setting. These were older branches.

🐇 Cool. Now, will you be checking the failing tests?

🐢 I think remote tests should never run if there are other errors. We can run coverage and remote tests as a second step after the first succeed.

🐇 That seems like a neat idea. You want to split the current job into two: one for compiling and testing the non-remote parts, and the second for coverage and storage tests. It won’t waste CI minutes that way?

🐢 On second thought, I think at the moment, that’s not a pressing issue. We should start fixing these tests at once. We can split them later.

🐇 Ok. It looks from the logs like secrets are not being made available to your jobs.

2022-10-29T18:18:17.5039311Z test test_storage_new_digital_ocean ... FAILED
2022-10-29T18:18:17.5039550Z
2022-10-29T18:18:17.5039652Z failures:
2022-10-29T18:18:17.5039784Z
2022-10-29T18:18:17.5040197Z ---- test_storage_new_digital_ocean stdout ----
2022-10-29T18:18:17.5040558Z Error: VarError { source: NotPresent }
2022-10-29T18:18:17.5040758Z
2022-10-29T18:18:17.5040765Z

🐢 I added them now. Let’s wait until the job ends to get a new set of logs.

🐇 You can write some documentation in the meantime.

🐢 I think #82 is a good candidate for this. There must not be too many missing docs in the ECS crate.

🐇 It was about walker and you fixed the ecs. You’re the most absentminded developer here, I believe.

🐢 Ooops, you’re right. I’ll add them together in a single PR.

🐇 In the meantime, the storage testing job has ended with 🔴.

🐢 Checking the raw logs. It looks like we didn’t update the tests to match the current options:

2022-10-30T14:45:50.1058348Z error: Found argument '--storage-prefix' which wasn't expected, or isn't valid in this context
2022-10-30T14:45:50.1059809Z ##[error]Found argument '--storage-prefix' which wasn't expected, or isn't valid in this context
2022-10-30T14:45:50.1061740Z 	If you tried to supply `--storage-prefix` as a value rather than a flag, use `-- --storage-prefix`
2022-10-30T14:45:50.1063220Z
2022-10-30T14:45:50.1063394Z USAGE:
2022-10-30T14:45:50.1064302Z     xvc storage new digital-ocean --name <NAME> --bucket-name <BUCKET_NAME> --region <REGION>

🐇 Is it --storage-prefix or --remote-prefix? Which one is clearer?

🐢 I think updating the tests to conform to the current options is better for now. We can update the options later if desired.

🐇 Another failure is rg. You assume it exists on the testing system.

🐢 ripgrep is available in Ubuntu 20.04, so we can just update the initial package list.

🐇 The same --storage-prefix failure appears in the minio tests.

🐢 Ok. Fixing it.

🐇 S3 tests want to run the new-s3 subcommand. I think we now see why we need these tests in the first place.

🐢 Yeah. Fixing the prefix option, too.

🐇 s3cmd is also required. You should add it, too.

🐢 Ok. I think we’ve added all missing dependencies. I’ll have to convert the mc tests for Minio to use s3cmd.

🐇 Then we can try again. Now, we can get back to documentation.

🐢 Added some more documentation to PR#93. I think it’s better to return to the storage test errors now.

🐇 It looks like you have missing DigitalOcean credentials.

🐢 Let’s take a look.

🐢 I’ve updated the tests to use a config file instead of command-line arguments that would be visible in the logs.

🐇 Good practice. You also need to remove previous logs.

🐢 Maybe there is an option for that.

🐇 It looks like there isn’t. There are masking options; when we add them to secrets, they are masked. But you shouldn’t use them in calls anyway.

• ends_with in Path is different from ends_with in str: If you try to check whether a Path is a directory by its final character, you’ll find that path.ends_with("/") returns false.

• Path::strip_prefix also consumes the final slash: If you have a directory marker at the end, such as /Users/emre/mydir/, and pass it to strip_prefix("/Users/emre"), you’ll get mydir instead of mydir/.

Instead, we use parameter passing to the database engine. For example:

SELECT * FROM people WHERE name = ?

We send this query and the parameters separately to the database. Most modern database systems support this approach.

In SQLite 3 with Python, we use it like this:

query = "SELECT * FROM people WHERE name = ?"
params = (name,)
db_result = cursor.execute(query, params)

However, when debugging, we may need to see the actual queries sent to the database—for instance, when data types are important or when we suspect a column is receiving a string instead of an integer.

In these cases, rather than manually reconstructing the query, we can use the set_trace_callback feature available in Python 3.3 and later:

connection.set_trace_callback(print)

The argument can be any function (such as print or a logger function) or None to disable tracing. This makes it easy to integrate with Python’s standard logging module.