#!/bin/bash

# Git pre-push hook to check if CHANGELOG.md is included in the push and if the branch is develop

# Get the current branch name

current_branch=$(git rev-parse --abbrev-ref HEAD)

# Check if the branch is develop

if [ "$current_branch" == "main" ]; then
	echo "You are on the main branch. Skipping CHANGELOG.md check."
	exit 0
fi

# remote="$1"
# url="$2"

# Get the list of commits to be pushed
commits=$(git rev-list '@{u}..HEAD')

has_rust_files=$(false)

# TODO: We can iterate to get file list only once
# Get the list of files that are going to be pushed
for commit in $commits; do
	if git diff-tree --no-commit-id --name-only -r "${commit}" | grep -q "\\.rs$"; then
		has_rust_files=$(true)
	fi
done

if [[ ! $has_rust_files ]]; then
	echo "No .rs files in the push, no need to check CHANGELOG"
	exit 0
fi

# Check if CHANGELOG.md is among the files in the commits
for commit in $commits; do
	if git diff-tree --no-commit-id --name-only -r "${commit}" | grep -q "CHANGELOG.md"; then
		echo "CHANGELOG.md is included in the push."
		exit 0
	fi
done

echo "ERROR: CHANGELOG.md is not included in the push."
exit 1

Edit (2025-02-08): Added a check for when only code files are changed.

This script merges all windows from all Tmux sessions into a single session. You must specify the target session as a parameter. If you save the script as merge-tmux-sessions.zsh, you can run it like:

merge-tmux-sessions.zsh my-current-session

It will then move all windows into the session you specify.

#!/bin/zsh

# set -vuex
#
#  Specify the target session

target_session=$1

# Get the list of all sessions

sessions=$(tmux list-sessions -F '#S')

# Iterate through each session

for session in ${(@f)sessions}; do
    if [ "$session" != "$target_session" ]; then
        # Get the list of windows in the current session
        windows=$(tmux list-windows -t $session -F '#I')

        # Move each window to the target session
        for window in ${(@f)windows}; do
            tmux move-window -s ${session}:${window} -t ${target_session}
        done
    fi

done

I needed a way to quickly open URLs in the Kagi Summarizer from my terminal. I wrote a small shell function to do that:

ksum() {
  local urlenc=$(jq -rn --arg x "$1" '$x|@uri')
  open "https://kagi.com/summarizer/index.html?target_language=&summary=takeaway&url=${urlenc}"
}

🐢 The tests for yesterday’s Git integration PR failed. I’ll begin by checking the logs.

🐇 Maybe run the tests locally to see. They might fail on your machine as well. It might be a simple thing.

🐢 Probably, yes. I’ve started the tests now. I shouldn’t forget to run them before testing the PR; it may save some time.

🐇 Ah, yeah, maybe. Yesterday you were already at the end of the workday, so it didn’t matter much. But you can shorten the testing time by reducing the number of files, etc. I think a separate benchmark suite might be good to have. Tests should be shorter; benchmarks should only run for tags.

🐢 Yup. I should shorten the tests. I should make them use a shorter list of files, maybe.

🐇 Local tests have passed. You’ll have to check the logs.

🐢 It seems git diff --name-only --cached returns files not yet added. That’s causing an error in new repositories with stash. Stashing shouldn’t run at all when there are no staged files.

🐇 Git behavior between the local version and the remote version seems different.

🐢 I think the trace should also show the Git version string.

🐇 You should also require a minimum Git version for the commands. You’re using some newer options, and not all users may have them.

🐢 Yep. Let’s see the version on the CI now.

🐢 Ah, I see—the problem is not about versions. It’s that the CI doesn’t configure git config --global user.email and user.name. Commits don’t work without these.

🐇 You should remove the Git version report, then. It’s one more process call for no reason.

🐢 I think it should be in trace!; I can check the verbosity level and call it only if it’s trace.

🐢 I completed the xvc-config docs as well. I think I can merge it before the tests finish, as it’s just a documentation update.

🐇 You seem to be rushing for the release.

🐢 Yeah, I want to really start testing this on the servers.

🐇 Then go ahead; let’s release a new version.

🐇 It looks like you’re back for an evening session. I think it’s time to start using Xvc on your torrent server.

🐢 Ah, yeah. Let’s see how it goes.

🐇 Create a repository for torrents. Then you can add files to it with cache-type = symlink or cache-type = hardlink.

🐢 I think creating a local storage may also help. I can use it to store files to retrieve them later.

🐇 Umm. We don’t have a “garbage collection” facility yet, you know. There are no file deletions at the moment. It may be better to have a repository-to-repository transfer feature, using SSH.

🐢 Yeah, we don’t have SSH storage either. I think this highlights a lack of features.

🐇 It’s possible to mimic Rsync storage with xvc storage new generic, but it doesn’t feel quite the same.

🐢 Then I’m adding these three tickets.

🐇 I think for 0.3.4, the command we add might be xvc file delete. You can work on this and add rsync support as well.

🐢 There is a librsync bindings library for Rust. But it looks like it doesn’t allow transferring file contents between hosts. There is also rusync, which is similar to rsync and implemented in Rust. There is also fast_rsync in pure Rust. It uses MD4 to calculate deltas, though, I think.

🐇 None of these seem to have network capability, though.

🐢 I think it’s better just to use the process for now. We are trying to come up with the simplest solution for now. We are just trying to be more general.

🐇 Yes, I think we can just use the process for the time being.

🐢 Then let’s start by adding it.

🐇 We should also begin to create release notes. GitHub can generate them from PRs. Automatically generated release notes

🐢 Created an issue for that. Now we’re starting xvc storage new rsync #111.

🐇 Ok. Let’s do this.

i3-msg workspace "n"

Nowadays, I’m using rclone. It has become my primary way of using Drive on Linux after that half-baked sync application. I discovered that rclone supports server-side MD5 hashes for files. I decided to write a script to delete the duplicate files in Drive.

First, I obtain the MD5 hashes of all files in Drive using:

$ rclone md5hash drive:/ > $HOME/Google-Drive-md5-$(date +%F).txt

This may take some time depending on the number of files, but it finished more quickly than I expected.

The file contents look like this:

39044094333de4a47d7478227cfa22a9  Facebin/vgg-clean/clea_duvall/00000501.jpg
398d44f0b32d1b4855e838754c2c49fc  Facebin/vgg-clean/Bob_Barker/00000445.jpg
c2090f9412b93d71fed884bb45b26518  Facebin/vgg-clean/Adam_Goldberg/00000432.jpg
cd6971f809a534ab02ee1b03eb6c1183  CHECK Uploads/Google Photos/2017/06/IMG_1461.JPG
1fb80cc9ef622410557cb42c4abf26a8  Facebin/vgg-clean/Angell_Conwell/00000510.jpg
160793fd8f24fcf27efb9a2b3698a9c8  Facebin/makimface-artifacts/dataset-images/user-test-v4/00056--/img-5c1fdc5d4fd82c1e12a7d49937d7f47b.png
bcddf6ebbc1eed83950d908c2ccbac4a  Facebin/vgg-clean/danny_pino/00000102.jpg
eb2746a9559e7a93fd0002f4bfd90517  Facebin/makimface-artifacts/dataset-images/user-test/00009--/ds-767cd16bfdf4e31860d597708a586979.png
9605d69aaf5fb83d309f10f9e2544630  Facebin/vgg-clean/Adam_Beach/00000953.jpg
68e6d3ec3cca8a728a78fb60c9a1ddba  Facebin/vgg-clean/corey_stoll/00000137.jpg

The first 32 characters are the MD5 hash of the file, and the rest of the line is the file path.

You’ll probably see some blanks for the MD5 hashes of certain files. It’s important to remove these from the list, as these are your Google Docs, spreadsheets, etc.

$ grep '[^0-9a-f]' $HOME/Google-Drive-md5-$(date +%F).txt > $HOME/Google-Drive-md5-cleaned.txt

Next, we sort the file using:

$ sort $HOME/Google-Drive-md5-cleaned.txt > $HOME/Google-Drive-md5-sorted.txt

We keep all intermediate files because you might want to review the differences after the cleanup.

Then, we write the following script and run it:

#!/bin/zsh

PREV_MD5=""
PREV_PATH=""
CURRENT_MD5=""
CURRENT_PATH=""
MD5_FILE=$HOME/Google-Drive-md5-sorted.txt
cat $MD5_FILE | while read current_line ; do
    # echo $current_line
    CURRENT_MD5=$(echo "$current_line" | cut -c -32)
    CURRENT_PATH=$(echo "$current_line" | cut -c 35-)
    # echo $CURRENT_MD5
    # echo "$CURRENT_PATH"
    if [[ "$CURRENT_MD5" == "$PREV_MD5" ]] ; then
        echo "EQUAL: $CURRENT_MD5 $PREV_MD5"
        echo "DELETE: drive:/$CURRENT_PATH"
        rclone -v delete "drive:/$CURRENT_PATH"
    else
        PREV_MD5=$CURRENT_MD5
        PREV_PATH=$CURRENT_PATH
    fi
done

You can save this script as google-drive-delete-duplicates.sh and run:

$ chmod +x google-drive-delete-duplicates.sh
$ ./google-drive-delete-duplicates.sh

The script checks each line one by one; if two duplicate MD5 hashes are found consecutively, the second file is deleted. It keeps only one of the duplicates, even if there are more than two copies.

I have gained about 200GB by running this script.

First, you need to register a new bot with @BotFather and get an API token. Then, run pip3 install --user telegram-send and prepare a config file at ~/.config/telegram-send.conf:

[telegram]
token = <TOKEN_YOU_GET_FROM_BOT_FATHER>
chat_id = <CHAT_OR_USER_ID>

You’ll need to start a conversation with the bot and find your user ID (which is identical to the chat ID for the conversation you start with the bot).

After that, you can send yourself messages from the CLI like this:

telegram-send "Hello Telegram."

You can also send Markdown-formatted messages, audio, stickers, and more using various command-line options:

usage: telegram-send [-h] [--format {text,markdown,html}] [--stdin] [--pre]
                     [--disable-web-page-preview] [--silent] [-c]
                     [--configure-channel] [--configure-group]
                     [-f FILE [FILE ...]] [-i IMAGE [IMAGE ...]]
                     [-s STICKER [STICKER ...]]
                     [--animation ANIMATION [ANIMATION ...]]
                     [--video VIDEO [VIDEO ...]] [--audio AUDIO [AUDIO ...]]
                     [-l LOCATION [LOCATION ...]]
                     [--caption CAPTION [CAPTION ...]] [--config CONF] [-g]
                     [--file-manager] [--clean] [--timeout TIMEOUT]
                     [--version]
                     [message [message ...]]

Another problem we face is establishing a connection between an executable file or library and its source code. We normally don’t add executable files to version control, as they are produced from code files. A common solution to this is writing version information to an “About” page or something similar.

When I was using Subversion some 15 years ago, I would create hooks to change the code files for the necessary versioning info, but this is not a recommended approach in Git because of its distributed nature. I have never tried it, but it would likely create more problems than it solves. The recommended way is to use the build system’s facilities to retrieve the versioning information and add it to the necessary places.

While developing the C library for dervaze, I wanted to add descriptive versioning information, as the library will also contain wordlists, and more words will be added over time.

In CMake, it’s possible to set versioning information and supply it through compiler options. It’s also possible to replace strings formatted as @CHANGE_THIS@ in source files. To supply version information to the executable, you can use these facilities:

set (DERVAZE_VERSION_MAJOR 1)
set (DERVAZE_VERSION_MINOR 0)
string(TIMESTAMP DERVAZE_TIMESTAMP "%y%m%d%H%M%S")
# current branch
execute_process(
  COMMAND git rev-parse --abbrev-ref HEAD
  WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
  OUTPUT_VARIABLE DERVAZE_GIT_BRANCH
  OUTPUT_STRIP_TRAILING_WHITESPACE
)

# abbreviated commit hash
execute_process(
  COMMAND git log -1 --format=%h
  WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
  OUTPUT_VARIABLE DERVAZE_GIT_COMMIT_HASH
  OUTPUT_STRIP_TRAILING_WHITESPACE
)

This information can be supplied to the C files by creating a header file that will be used as a template.

#define DERVAZE_VERSION_MAJOR       "@DERVAZE_VERSION_MAJOR@"
#define DERVAZE_VERSION_MINOR       "@DERVAZE_VERSION_MINOR@"
#define DERVAZE_TIMESTAMP           "@DERVAZE_TIMESTAMP@"
#define DERVAZE_LIB_GIT_BRANCH      "@DERVAZE_GIT_BRANCH@"
#define DERVAZE_LIB_GIT_COMMIT_HASH "@DERVAZE_GIT_COMMIT_HASH@"

Suppose this file is named version.h.in; the following command creates the actual version.h for each build.

# configure a header file to pass some of the CMake settings
# to the source code
configure_file (
  "${PROJECT_SOURCE_DIR}/version.h.in"
  "${PROJECT_SOURCE_DIR}/version.h"
  )

It’s also possible to write this file only during the build by using ${PROJECT_BINARY_DIR}/version.h as the second argument, but in my experience, keeping such a file in the source directory is sometimes needed by build tools. When you keep it in the source directory, it’s better to ignore the generated version.h by adding it to .gitignore.

Therefore, I decided that a small Bash script using rsync and find would work better. It monitors my entire home directory and backs up recent files.

The following script does exactly that:

#!/bin/bash

if [ "x$1" = "x" ] ; then
    TARGET=/media/augustus/backup-recent-`hostname`
else
    TARGET=$1
fi

PERIOD=15

mkdir -p $TARGET

# Delete files older than $PERIOD days
find $TARGET -ctime +$PERIOD -print -delete

# Copy files newer than $PERIOD under ~. Ignore files under .hg
for d in ~/*/ ; do
    find $d -path '*/.hg/*' -prune -o -type f -ctime -$PERIOD -print  -exec rsync -aRv {} $TARGET/ \;
done

It checks whether a command-line option is provided as the target; otherwise, it sets a default target. /media/augustus/ is an NFS mount in my case, but you can specify any path.

PERIOD is the number of days that the script considers recent. Currently, it backs up files changed in the last 15 days.

The script deletes older files from the backup. Since I use other solutions for long-term storage, I don’t want to keep them here, especially since this script runs via cron every two hours.

Note that the script only checks directories under the home directory, so files located directly in the home directory are not backed up.

It also skips files under .hg directories. You can add more -prune options to the find command to exclude other irrelevant directories from your backup.

Previously with Pandoc, I was using a simple setup to create RSS feeds. Markdown files were converted to plain, headerless HTML, and they were collected together to build an XML file. The obvious drawback is that all HTML files should be generated by Pandoc; anything that doesn’t fit that route does not appear in the feeds.

However, when I began to use Org Mode for data analysis and other tasks, I stopped using Pandoc. Org Mode has extensive facilities for exporting into HTML and other document formats, so I would not mess with Pandoc for this.

I thought RSS could be produced by parsing HTML files after they are produced. This requires parsing the HTML file, but it’s simple, and there are parsers for all programming languages out there. My previous RSS generator was in Python, and I decided to modify it to fit my needs. I think producing RSS for a static HTML site is a common need, and I tried to solve this problem as simply as possible.

Let’s begin with the ubiquitous shebang line. This tells the system that the script is in Python.

#!/usr/bin/env python

The following are the imports for this script. Apart from PyRSS2Gen, all modules are present in Python 2.7.

import argparse
import codecs
import os
import datetime
from HTMLParser import HTMLParser
import PyRSS2Gen as rssgen
import operator as op
import re
import subprocess as proc

I use Mercurial to track the site’s files. I once thought about using the Mercurial public API to check the status of files, but it proved to be overkill because only the modification time of files is necessary, and retrieving them using a standard command-line call is much simpler. Hence, I removed the following imports for the time being.

# from mercurial import commands as cmd
# from mercurial import hg
# from mercurial import ui as hgui

The following function returns a valid HTML tag string, given the tag and its attributes in a list. HTMLParser sends the tags in a list form, and I use this function to reconvert them to usual HTML tags.

def make_tag(tag, attrs):
    content_list = [ tag ]
    content_list += [ "%s=\"%s\"" % (k, v) for (k, v) in attrs]
    return "<" + " ".join(content_list) + ">"

TitleBodyExtractor is an HTMLParser subclass. It collects the body of a page in a string and also keeps the title. These two are the only requirements. It might be possible to parse meta tags to get publish date and author information as well, but I prefer to keep simple things simple.

class TitleBodyExtractor(HTMLParser):

    def __init__(self):
        HTMLParser.__init__(self)
        self.in_body = False
        self.in_title = False
        self.body = ""
        self.title = ""


    def handle_data(self, data):
        if self.in_body:
            self.body += data
        if self.in_title:
            self.title += data

    def handle_starttag(self, tag, attrs):
        if self.in_body:
            self.body += make_tag(tag, attrs)

        if tag == "body":
            self.in_body = True
        if tag == "title":
            self.in_title = True

    def handle_endtag(self, tag):
        if tag == "body":
            self.in_body = False

        if tag == "title":
            self.in_title = False

        if self.in_body:
            self.body += "</%s>" % (tag)

Getting contents of a file in UTF-8 encoding is a common task. The following two functions retrieve and store the contents in UTF-8 using the codecs module.

def get_content(filename):
    f = codecs.open(filename, "r", "utf-8")
    cont = f.read()
    f.close()
    return cont

def write_content(filename, content):
    f = codecs.open(filename, "w", "utf-8")
    f.write(content)
    f.close()

Mercurial allows running commands for a repository outside of that repository with the -R command-line switch. However, it requires the exact path of the repository and does not accept a child path. The following function finds the repository path of a file by recursively checking whether parent paths contain an .hg/ directory.

def get_repo_path(dir):
    if dir == "/" or dir == "":
        return ""
    if os.path.exists(os.path.join(dir, ".hg")):
        return dir
    else:
        return get_repo_path(os.path.dirname(dir))

The FileObject class keeps the required data of an HTML file. It stores the path, modification time, body, and title.

class FileObject:
    def __init__(self, path, mtime):
        self.path = path
        self.mtime = int(mtime)
        self._body = ""
        self._title = ""

    def parse(self):
        content = get_content(self.path)
        tbe = TitleBodyExtractor()
        tbe.feed(content)
        self._body = tbe.body
        self._title = tbe.title

    def body(self):
        if self._body == "":
            self.parse()
        return self._body

    def title(self):
        if self._title == "":
            self.parse()
        return self._title

    def __str__(self):
        return str(self.path) + " " + str(self.mtime)

The modification time of a file should be retrieved from the Mercurial repository. The following function calls hg log with a specific template, then parses the date to get the last commit time of a file. If the file is not registered to a repository, it simply returns the filesystem modification time.

def get_mtime(full_path):
    if os.path.exists(full_path):
        repo_path = get_repo_path(full_path)
        if repo_path != "":
            logcmd = "/usr/bin/hg log -R %s --template='{date|hgdate}' -l 1 %s " % (repo_path, full_path)
            # print logcmd

            proc_res = proc.check_output(logcmd, shell=True).split()
            if len(proc_res) > 0:
                filetime = int(proc_res[0])
            else:
                filetime = os.path.getmtime(full_path)
        else:
            filetime = os.path.getmtime(full_path)
        return filetime
    else:
        return 0

We need a list of files as FileObject objects, given the directory name and extension. The function also takes a repository path and excludes filenames that match a given regex.

    def file_list(dirname, extension, repo_path, exclude_regex=None):
        results = []
        for root, dirs, files in os.walk(dirname):
            # print "Dirs:", dirs
            for d in dirs:
                if exclude_regex == None or (not re.match(exclude_regex, d)):
                    results += file_list(os.path.join(root, d), extension, repo_path)
                else:
                    print "Skipping", d
            # print "Files:", files
            for f in files:
                if (exclude_regex == None or (not re.match(exclude_regex, f))) and f.endswith(extension):
                    fullname = os.path.join(root, f)
                    filetime = get_mtime(fullname)
                    results.append(FileObject(fullname, filetime))
        return results

Given a local file in the site, we need to create a link that shows the URL of that file relative to the site’s URL. The following function finds the relative path with respect to an input directory and returns the complete URL by concatenating it to the site’s URL.

    def make_link(site_url, input_dir, file_path):
        rel_path = os.path.relpath(file_path, input_dir)
        return site_url + rel_path

Given a FileObject that points to an HTML file, we need a function that builds an RSS item from it. It obtains the URL of the file and fills the rest using the attributes of the FileObject.

    def get_rss_item(file_object, input_dir, site_url):
        the_link = make_link(site_url, input_dir, file_object.path)
        item = rssgen.RSSItem(title=file_object.title(),
                              link=the_link,
                              description=file_object.body(),
                              guid=rssgen.Guid(the_link),
                              pubDate=datetime.datetime.fromtimestamp(file_object.mtime))
        return item

The function that creates an RSS file from the files in a given directory is the topmost function. It takes all variables that are set from the command line and returns the RSS object.

It first lists all files with the given extension (default being .html) in the input directory. Then it compares the modification time of the RSS file with the modification times of these listed files. If there is no previous RSS file or there are newer HTML files, the RSS is generated again.

Note that a certain amount of edit time can be set, so that the script doesn’t consider files as new if they are modified within edit time minutes. This can be set to prevent too frequent generation of files during an edit session.

To generate the RSS, each file is supplied to the previous function and an RSS item is obtained; then these are fed into the RSS2 function of PyRSS2Gen to get the resulting object.

def generate_rss(input_dir, extension = ".html", output = "rss/rss.xml", site_title = "Title", site_description = "Description", max_items = 20, site_url = "http://example.com", edit_time=0, exclude_regex = None):
        if not site_url.endswith("/"):
            site_url += "/"
        files = file_list(input_dir, extension, get_repo_path(input_dir), exclude_regex)
        rssmtime = get_mtime(output)
        files_up = [f for f in files if f.mtime >= (rssmtime + edit_time)]
        if len(files_up) > 0:
            print "Before", files_up
            files_up.sort(key=lambda x: x.mtime, reverse=True)
            print "After", files_up
            rss_items = [get_rss_item(fo, input_dir, site_url) for fo in files_up[:max_items]]
            rssobj = rssgen.RSS2(title = site_title,
                                 link = site_url,
                                 description = site_description,
                                 lastBuildDate = datetime.datetime.now(),
                                 items = rss_items)
            return rssobj
        return None

The main function uses argparse to handle options. The input directory, site title, site URL, and number of items are mandatory; other options have sensible default values.

The function also builds the exclude_regex object to supply to file listings. The regex is built here from the supplied string, and all other functions use this compiled regex.

After generating the RSS, it writes the file with the write_xml function.

    def main():

        parser = argparse.ArgumentParser(description='Generate RSS feed from a set of HTML files')

        parser.add_argument('--input-dir', help="input directory", required=True)
        parser.add_argument("--extension", help="file extension to collect", default=".html")
        parser.add_argument("--output", help="output filename to write the results", default="rss/rss.xml")
        parser.add_argument('--title', help="title of the RSS feed", required=True)
        parser.add_argument("--description", help="site description", default="")
        parser.add_argument("--items", help="max items included in the feed", required=True, type=int)
        parser.add_argument("--site-url", help="site url of items", required=True)
        parser.add_argument("--exclude-regex", help="regex to set skipped files", default="")
        parser.add_argument("--edit-time", help="minutes to wait before putting an item into rss", default=0, type=int)

        args = vars(parser.parse_args())

        if args["exclude_regex"] == "":
            exclude_regex = None
        else:
            exclude_regex = re.compile(args["exclude_regex"])

        rssresults = generate_rss(args["input_dir"],
                                  args["extension"],
                                  args["output"],
                                  args["title"],
                                  args["description"],
                                  args["items"],
                                  args["site_url"],
                                  args["edit_time"],
                                  exclude_regex)

        if rssresults != None:
            rssresults.write_xml(open(args["output"], "w"))



    if __name__ == "__main__":
        main()

You can get the resulting Python script from rss-generator.py.