🔺 So, who is the typical user of Xvc? Who is the probable user?

🔹 I’m thinking of a user with large amounts of data who will be training machine learning models. They need to version this data.

🔺 What do you mean by data in this context?

🔹 The data in this context consists of files. Image files, audio files, text files—a whole bunch of files. Lots and lots of files. They may be organized into directories; they may be reserved for training, verification, and testing; and they may be stored in multiple locations.

🔺 So, not database files? Are you saying we can’t version the data in databases?

🔹 If the database you’re thinking of is something like SQLite, then it is possible to version it. It’s just another file. You can track the file with Xvc and move back and forth between versions to go back in time. But I doubt that’s the best way.

🔺 Why?

🔹 A database is an alternative to the file system for storing data. You can version the data by adding a timestamp or a version description and track individual records. In this case, you can go back and forth between versions by selecting a subset of the records. To my mind, it doesn’t make sense to replace the database file itself just to track the data version.

🔺 For tracking, yes, there may be other ways to version the data. Databases themselves often provide this functionality. Except for SQLite, replacing database files doesn’t work seamlessly. Databases maintain extra indices and auxiliary files for performance, and simply replacing the database files would likely cause them to fail. However, these databases can be sources of data in data pipelines. When you have, for example, a database table as a source, you should be able to define it as a dependency in the pipelines.

🔹 That sounds worthwhile, right.

🔺 So, the user could define a database table as a dependency. Suppose you have a table that contains a lot of data, and you want to update your ML model whenever this data changes. This must be a very common use case.

🔹 It seems so, yes. It’s more difficult than file system dependencies, though. At the very least, it must be customized per database engine. We need different connections for different databases.

🔺 This is something that can be abstracted. All databases have tables and queries. We can have a database connection layer to retrieve a table, check whether it has changed, and invalidate it as a dependency if it has. The user is responsible for getting the actual data and doing whatever they need to do with it anyway.

🔹 It looks like a nice model. We can also adapt it to SQL queries. When queries are run, they produce a result, and if this result has changed, we can consider the dependency invalidated. It would be useful to run a simple query to check whether a complex join has changed and update the models accordingly.

🔺 What would be the performance implications? A query would need to be run in each execution of the pipeline, I think.

🔹 Yep. If we define a database table dependency as xvc pipeline step dependency --database-table, then it should connect to the database during each xvc pipeline run and check whether the table has changed. A table change can be detected by selecting all records or via database internals. It’s not that big a deal if the table or query doesn’t produce too many results.

🔺 Maybe it’s possible to generalize this behavior to the command line as well. Xvc could run a command to check whether something has changed—like ls -R my-data—and if the result differs from the previous run, it can trigger a command. This could be the generic way to handle pipeline steps.

🔹 That sounds like a great idea. We can talk about this tomorrow.

Another problem we face is establishing a connection between an executable file or library and its source code. We normally don’t add executable files to version control, as they are produced from code files. A common solution to this is writing version information to an “About” page or something similar.

When I was using Subversion some 15 years ago, I would create hooks to change the code files for the necessary versioning info, but this is not a recommended approach in Git because of its distributed nature. I have never tried it, but it would likely create more problems than it solves. The recommended way is to use the build system’s facilities to retrieve the versioning information and add it to the necessary places.

While developing the C library for dervaze, I wanted to add descriptive versioning information, as the library will also contain wordlists, and more words will be added over time.

In CMake, it’s possible to set versioning information and supply it through compiler options. It’s also possible to replace strings formatted as @CHANGE_THIS@ in source files. To supply version information to the executable, you can use these facilities:

set (DERVAZE_VERSION_MAJOR 1)
set (DERVAZE_VERSION_MINOR 0)
string(TIMESTAMP DERVAZE_TIMESTAMP "%y%m%d%H%M%S")
# current branch
execute_process(
  COMMAND git rev-parse --abbrev-ref HEAD
  WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
  OUTPUT_VARIABLE DERVAZE_GIT_BRANCH
  OUTPUT_STRIP_TRAILING_WHITESPACE
)

# abbreviated commit hash
execute_process(
  COMMAND git log -1 --format=%h
  WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
  OUTPUT_VARIABLE DERVAZE_GIT_COMMIT_HASH
  OUTPUT_STRIP_TRAILING_WHITESPACE
)

This information can be supplied to the C files by creating a header file that will be used as a template.

#define DERVAZE_VERSION_MAJOR       "@DERVAZE_VERSION_MAJOR@"
#define DERVAZE_VERSION_MINOR       "@DERVAZE_VERSION_MINOR@"
#define DERVAZE_TIMESTAMP           "@DERVAZE_TIMESTAMP@"
#define DERVAZE_LIB_GIT_BRANCH      "@DERVAZE_GIT_BRANCH@"
#define DERVAZE_LIB_GIT_COMMIT_HASH "@DERVAZE_GIT_COMMIT_HASH@"

Suppose this file is named version.h.in; the following command creates the actual version.h for each build.

# configure a header file to pass some of the CMake settings
# to the source code
configure_file (
  "${PROJECT_SOURCE_DIR}/version.h.in"
  "${PROJECT_SOURCE_DIR}/version.h"
  )

It’s also possible to write this file only during the build by using ${PROJECT_BINARY_DIR}/version.h as the second argument, but in my experience, keeping such a file in the source directory is sometimes needed by build tools. When you keep it in the source directory, it’s better to ignore the generated version.h by adding it to .gitignore.