I first tried to use the rust-fsm library, but it became apparent that Xvc pipeline steps’ states are tied to XvcRoot. That is, if we are to check the presence of a file or the value of a parameter, we have to do it relative to the repository root. The root directory should be taken into consideration in every transition.

I checked the code and noticed that the FSM is actually very simple. I copied it, added &XvcRoot to the transition and output functions in the trait definition, and implemented it for XvcOutput, XvcDependency, and XvcStep.

These are the constituents of a pipeline. A pipeline is composed of an XvcStep that defines a command, and each step can have multiple XvcDependency and XvcOutput definitions. For each of these structs, I’ve added fields that represent their current state.

For example, an XvcOutput can be Missing, Found, Old, or Ok. An XvcStep that produces this XvcOutput doesn’t check the dependency content hash if an output is missing. However, if an XvcOutput is Found, the XvcStepStateMachine checks the XvcDependency states and their modification times to see if they have changed since the output was generated. An XvcStep is invalidated when an XvcOutput is Missing or an XvcDependency has changed after the last command run.

Unlike DVC, I added the ability for an XvcStep to depend on other XvcSteps. They communicate through outputs. I’ve added XvcDependency::Step(XvcStep) to the XvcDependency definition. I’m also planning XvcDependency::Pipeline(XvcPipeline) to allow steps to depend on other pipelines, so that pipelines can be run in order.

Currently, the following are included as XvcDependency:

• File: A (binary or text) file in the repository. If the metadata (size and modification time) or the content changes, the dependent step becomes invalidated.
• Directory: A directory that contains files. If a file is added to or removed from the directory, or any of the files are changed, the associated step becomes invalidated.
• Glob: A glob such as my-data/*.png. If the list of files changes or their content has changed, the associated step becomes invalidated.
• Parameter: Xvc can parse YAML, TOML, and JSON files and get the values of variables. It’s possible to define these (hyper)parameters as dependencies.
• URL: An HTTPS URL, which is checked first by metadata and then by content to see whether it has changed.
• Step: A previously defined step; if it’s invalidated, the depending step also becomes invalidated.

Additionally, I’m planning to add the following items to Xvc as dependencies:

• Lines {path, begin, end}: Lines in a text file. This can be used for general-purpose input tracking. If the given lines in a file are changed, the dependent step becomes invalidated.
• Regex {path, regex}: If the regular expression result on the file changes, the dependent stage becomes invalidated.
• Pipeline { name }: If any of the steps in a pipeline is invalidated, the pipeline is also invalidated, or the step that depends on this pipeline becomes invalidated.

Each of these dependencies is checked minimally; that is, when their size on disk is detected to have changed, they are considered changed without checking the content hash. It needs a very detailed state machine to track the changes without bugs.

I’ve noticed that if I can write such a state machine, most of the I/O operations can be done in parallel. If two steps do not depend on each other in the dependency graph, they can be run in parallel. The state machine’s granularity allows this.

🐇 LGTM. I think you can also link to commits here when they occur.

🐢 I think that’s a bit too much work, but PRs may be linked.

🐇 Ok. Let’s start with the PRs. What are you working on these days?

🐢 I started fixing warnings in issue #3 and am now fixing missing documentation warnings.

🐇 Do you really think that we’ll have a use for [RMNStore] in the future?

🐢 I’m not sure. Possibly any M-N relationship can be tracked by two 1-N relationships. Under the hood, this is what RMNStore does actually.

🐇 Maybe we should just remove it.

🐢 I want to save it for the time being. If [xvc-ecs] is used in other places, it may be needed.

🐇 Ok. Probably YAGNI, but let’s keep it for the time being.

🐢 Writing documentation can be a debugging method, too. I think I’ve found the reason for a bug here.

🐇 Um, how could it lead to a bug?

🐢 Insert means a new event happened. But when converting a VStore to an XvcStore, new events are not happening. We are simply converting from one store to another.

🐇 That means they are added to XvcStore::current events, not XvcStore::previous, and when save_dir is called, they are saved.

🐢 Yeah, that was the bug. From time to time, I was seeing duplicate JSON files.

🐇 Then maybe it’s better to limit all insert usage.

🐢 Yeah, I should review that, too.

🐇 Added a bug report.

🐢 I’m installing cargo-geiger to show that there is no unsafe code.

🐇 Do you think you’ll maintain Xvc that way? That it won’t require any unsafe code?

🐢 Maintain the forbidden(unsafe) status? I think so. Xvc doesn’t seem to need any unsafe blocks by itself. Most of the dependencies may need it, though.

🐇 Working on it, and it will be merged today.

🐢 I worked on that s3cmd bug yesterday, and it turned out that my modification of the configuration was the cause.

🐇 You were thinking that it’s about R2 all along?

🐢 I should suspect my custom config more.

🐇 Anyway, congratulations 🌠 that Xvc now has Cloudflare R2 support.

In Go, it is possible to embed structs within other structs:

type person struct {
    first string
    last string
    age   int
}

type employee struct {
    person
    employee_id int
    salary      int
}

We can instantiate the employee type as follows:

e := employee{
    person: person{
        first: "Ali",
        last:  "Yeşil",
        age:   33,
    },
    employee_id: 001,
    salary:      4500,
}

We can then access the fields of the embedded struct as if they were fields of the outer struct:

// e.first is a shorthand for e.person.first
fmt.Println(e.first) 

One can also use anonymous structs without explicitly defining a type:

z := struct {
    a string
    b int
}{
    a: "aaa",
    b: 123,
}

If these won’t be reused elsewhere, using anonymous structs can keep the code clean.