Gemini proposed this:

git diff --word-diff=plain -- "*.md" | rg -o '\{\+(.*?)\+\}' | str replace -r '\{\+|\+\}' '' | str join " " | str words | length

What I ended up doing:

git diff --word-diff=plain -- "*.md" | rg -o '\{\+(.*?)\+\}' | str replace -a "{+" "" | str replace -a "+}" "" | str replace -a ' ' "\n" | lines | length

I notice LLMs tend to err on the side of complexity. I don’t know if this is due to their training to spit out as many tokens as possible, but using regular expressions when a simple string replacement may suffice is a good sign that they may be adding more complexity than necessary.

Gemini proposed this:

git diff --word-diff=plain -- "*.md" | rg -o '\{\+(.*?)\+\}' | str replace -r '\{\+|\+\}' '' | str join " " | str words | length

What I ended up doing:

git diff --word-diff=plain -- "*.md" | rg -o '\{\+(.*?)\+\}' | str replace -a "{+" "" | str replace -a "+}" "" | str replace -a ' ' "\n" | lines | length

I notice LLMs tend to err on the side of complexity. I don’t know if this is due to their training to spit out as many tokens as possible, but using regular expressions when a simple string replacement may suffice is a good sign that they may be adding more complexity than necessary.

Gemini proposed this:

git diff --word-diff=plain -- "*.md" | rg -o '\{\+(.*?)\+\}' | str replace -r '\{\+|\+\}' '' | str join " " | str words | length

What I ended up doing:

git diff --word-diff=plain -- "*.md" | rg -o '\{\+(.*?)\+\}' | str replace -a "{+" "" | str replace -a "+}" "" | str replace -a ' ' "\n" | lines | length

I notice LLMs tend to err on the side of complexity. I don’t know if this is due to their training to spit out as many tokens as possible, but using regular expressions when a simple string replacement may suffice is a good sign that they may be adding more complexity than necessary.

Gemini proposed this:

git diff --word-diff=plain -- "*.md" | rg -o '\{\+(.*?)\+\}' | str replace -r '\{\+|\+\}' '' | str join " " | str words | length

What I ended up doing:

git diff --word-diff=plain -- "*.md" | rg -o '\{\+(.*?)\+\}' | str replace -a "{+" "" | str replace -a "+}" "" | str replace -a ' ' "\n" | lines | length

I notice LLMs tend to err on the side of complexity. I don’t know if this is due to their training to spit out as many tokens as possible, but using regular expressions when a simple string replacement may suffice is a good sign that they may be adding more complexity than necessary.

• ends_with in Path is different from ends_with in str: If you try to check whether a Path is a directory by its final character, you’ll find that path.ends_with("/") returns false.

• Path::strip_prefix also consumes the final slash: If you have a directory marker at the end, such as /Users/emre/mydir/, and pass it to strip_prefix("/Users/emre"), you’ll get mydir instead of mydir/.

Recently, I needed a standard way to get the temporary directory. I checked the crate documentation but couldn’t find a proper function for this.

Then I noticed that std::env::temp_dir() returns a PathBuf pointing to the system’s temporary directory. This is a reminder to myself that we don’t always need extra packages for basic functionality.

class Square extends Shape {}

class Circle extends Shape {}

class Shape {
  Shape();

  factory Shape.fromTypeName(String typeName) {
    if (typeName == 'square') return Square();
    if (typeName == 'circle') return Circle();

    throw ArgumentError('I don\'t recognize $typeName');
  }
}

The first one is used when there is a single expression to write:

(a, b) => a + b

The other is used when you need to write multiple statements in an anonymous function:

(a, b) { 
  return a + b; 
}

Python doesn’t have C-style static variables natively. (Although it supports class variables, which can be used for a similar purpose in OOP.) However, since functions are also objects in Python, it’s possible to embed variables inside the function. An elegant solution on Stack Overflow creates a decorator for static variables.

def static_vars(**kwargs):
    def decorate(func):
        for k in kwargs:
            setattr(func, k, kwargs[k])
        return func
    return decorate

@static_vars(counter=0)
def foo():
    foo.counter += 1
    print(f"Counter is {foo.counter}")

Basics

R objects have attributes which can be observed using the attributes() function.

<- is the assignment operator.

: is used to create integer sequences. For example, 1:4 results in 1 2 3 4.

The c() function (concatenate) can be used to create vectors from different kinds of objects:

• c(TRUE, FALSE) creates a logical vector.
• c(1+3i, 4+8i, 3-5i) creates a complex vector.

Type coercion happens if different kinds of objects are mixed.

as.* functions can be used to coerce data types. For example, as.numeric(TRUE) returns 1.

matrix(ncol = 3, nrow = 4) creates a matrix.

cbind() and rbind() are other options to create a matrix from vectors by binding them as columns or rows.

factors are categorized data, like male/female. They are created using the factor() function. The table() and unclass() functions can also be used to get information or change the factor into a numeric table.

The levels parameter in the factor() function can be used to determine the factor-to-number correspondence.

is.nan() and is.na() functions are used to check whether vector values are NaN or NA.

data.frames are used to store tabular data like matrices. Unlike matrices, they can store different types of data in each column (e.g., the first column can be numeric, the second a factor, and the third logical).

data.frames are usually created using the read.table() or read.csv() functions. Each row has a name which can be accessed by row.names(). A data frame can be converted to a matrix with data.matrix().

nrow() and ncol() functions can be used to get the number of rows and columns.

str() and summary() functions provide concise information about a data structure.

Use getwd() to report the current working directory, and setwd() to change it.

The ls() function displays the names of objects in your workspace:

> x <- 10
> y <- 50
> z <- c("three", "blind", "mice")
> f <- function(n, p) sqrt(p * (1 - p) / n)
> ls()
[1] "f" "x" "y" "z"

The rm() function permanently removes one or more objects from the workspace.

Scripts

R executes the .Rprofile script when it starts. The location of .Rprofile depends on your platform; on Linux/Unix, it is typically in your home directory: ~/.Rprofile.

The source() function instructs R to read a text file and execute its contents:

source("myScript.R")

On the command line, this can be run as:

$ R CMD BATCH /home/jim/psych/adoldrug/partyuse1.R

Managing various objects used in R can be challenging. Sorting objects into sensible directory structures can help. You may wish to keep a directory of R scripts that change the working directory to suit the task they perform.

par(ask=TRUE) requires you to hit Enter before each plot is displayed.

readline("Press <Enter> to continue") presents a prompt to the user.

Vectors

Vectors are created like v <- c(1.1, 2.2, 3.3). Vectors can be used in arithmetic expressions, such as x <- v + 2 * w. A shorter vector is cycled until it reaches the length of the longer vector in arithmetic expressions.

range() returns the minimum and maximum elements of a vector.

sort() sorts a vector in increasing order.

sqrt(-17) returns NaN, but sqrt(-17+0i) returns a complex result.

Regular sequences are generated by the : operator. 4:10 returns [4, 5, 6, 7, 8, 9, 10]. This is syntactic sugar for the seq() function, which can also specify step size and length.

The rep() function repeats supplied elements to create a vector.

Arrays

If z is a vector with 1500 elements (e.g., z <- 1:1500), then dim(z) <- c(3, 5, 100) makes it a 3D array with those dimensions.

Another way to create an array is x <- array(1:20, dim=c(4, 5)).

Matrices

Two matrices A and B can be multiplied using A %*% B.

A linear equation of the form b <- A %*% x can be solved using solve(A, b).

Lists

A list can be created using the list() function. List elements don’t have to be of the same type; they can be anything from characters to vectors.

> mylist <- list(name="Fred", no.children=3, child.ages=c(4, 7, 9))

Components can be accessed by index like mylist[[1]] or by component name like mylist$no.children or mylist[["no.children"]].

Lists are similar to structs in other languages. The c() function can be used to concatenate lists.

Arbitrary Functions

An arbitrary function (similar to a lambda) can be created as f <- function(x, y) x + y.

Statistical Functions for Discrete Distributions

The library(distrEx) provides functions E(), var(), and sd() to calculate mean, variance, and standard deviation.

Uniform random events can be emulated with the sample() function. It has three parameters:

1. The range of values to select from.
2. size: the number of events.
3. replace: whether to sample with replacement.

Examples:

• 1000 dice rolls: sample(6, size=1000, replace=TRUE)
• 50 random numbers from 1000 to 2000: sample(1000:2000, size=50, replace=TRUE)
• Flip a fair coin 100 times: sample(c("H", "T"), size=100, replace=TRUE)

Reading and Writing Data

read.table() and read.csv() read tabular data from text files.

readLines() reads lines of text.

source() and dget() read R code files.

load() and unserialize() are used to read binary objects.

dump() and dput() are the inverses of source() and dget(). They include the object’s metadata in the output.

file() is used to open file connections. gzfile() opens gzipped files, and bzfile() opens bzip2 files. The url() command opens a connection to a web page.

read.table

read.table() is the primary function for importing data.

• file: name of the file or connection.
• header: boolean indicating if the file has a header row.
• sep: the field separator (comma, tab, etc.).
• colClasses: a vector of column data types. Specifying this can make R significantly faster.
• nrows: the number of rows in the dataset.
• comment.char: character that starts a comment.
• skip: number of lines to skip from the beginning.

read.csv() is a wrapper for read.table() with the default separator set to a comma.

initial <- read.table("datatable.txt", nrows = 100)
classes <- sapply(initial, class)
tabAll <- read.table("datatable.txt", colClasses = classes)

Plotting

plot(x, y) plots the values in x against y. Additional parameters can configure visual settings.

Use the density() function to approximate sample density, and lines() to draw it:

hist(x, prob=T)
lines(density(x))

Installing R packages

Method 1: Install from source

Download the package (e.g., mypkg) and run this in the shell:

$ R CMD INSTALL mypkg -l /my/own/R-packages/

Method 2: Install from CRAN

Run this in the R console:

> install.packages("mypkg", lib="/my/own/R-packages/")

Load the library

> library("mypkg", lib.loc="/my/own/R-packages/")

Statistics

Density

Approximate sample density and draw it:

> hist(x, prob=T)
> lines(density(x))

~ notation for relations between variables

R uses a special notation for describing relationships between variables. Suppose you assume a linear model for a variable $y$, predicted from variables $x_1, x_2, \dots, x_n$:

$y \sim x_1 + x_2 + \dots + x_n$

Statisticians refer to $y$ as the dependent variable and $x_i$ as the independent variables. In R, this is represented as a formula object.

Working with data

Creating a Data Frame

> points <- data.frame(label=c("Low", "Mid", "High"),
                       lbound=c(0, 0.67, 1.64),
                       ubound=c(0.674, 1.64, 2.33))

The print() function

Allows you to specify the number of printed digits:

> print(pi, digits=4)

The cat() function

Does not provide direct control over formatting. Use format() before calling cat():

> cat(format(pi, digits=4), "\n")

The list.files() function

Shows the contents of your working directory.

The write.csv() function

> write.csv(x, file="filename.csv", row.names=FALSE)

Factor analysis

Available via factanal() in the stats package:

factanal(x, factors, data = NULL, covmat = NULL, n.obs = NA,
          subset, na.action, start = NULL,
          scores = c("none", "regression", "Bartlett"),
          rotation = "varimax", control = NULL, ...)

PCA

Principal Components Analysis (PCA) breaks a set of correlated variables into uncorrelated variables. Available via prcomp().

Distributions in R

• Binomial: binom (n = trials, p = probability)
• Geometric: geom (p = probability)
• Hypergeometric: hyper (m = white balls, n = black balls, k = balls drawn)
• Negative Binomial: nbinom
• Poisson: pois (lambda = mean)
• Beta: beta
• Cauchy: cauchy
• Chi-squared: chisq (df = degrees of freedom)
• Exponential: exp
• F: f
• Gamma: gamma
• Log-normal: lnorm
• Logistic: logis
• Normal: norm
• Student’s t: t
• Uniform: unif
• Weibull: weibull
• Wilcoxon: wilcox

Combination calculation

Calculating combinations (n choose k) is done via the choose() function:

> choose(5, 3)
[1] 10

Generating combinations

Use the combn() function to generate all combinations:

> combn(items, k)

Selecting $n$ items from a vector

> sample(vec, n)

dotplot() in lattice

The dotplot() function in library(lattice) is useful for displaying labeled quantitative values.

Correlation

Correlation ranges between -1 and 1:

• 1: perfect positive linear relationship.
• 0: no correlation.
• -1: perfect negative linear relationship.

Bootstrapping

Bootstrapping is a technique for estimating the bias/variance of an estimator by repeatedly resampling with replacement. In R, use the boot() function in the boot package.

paste()

The paste() function concatenates multiple character vectors into a single vector.

Chi-squared test

Used for statistical tests of categorical data, such as goodness of fit and independence.

Plotting the regression line

> plot(x, y)
> abline(lm.result)

Coefficients of regression

The coef() function returns a vector of coefficients:

> coef(lm.result)

ANOVA

Analysis of Variance (ANOVA) compares means for more than two independent samples.

Regression analysis

Used for modeling the relationship between a response variable ($y$) and one or more predictors ($x$).

• $p=1$: simple regression.
• $p>1$: multiple regression.

i.i.d.

Independent and identically distributed.

Linear models

Despite the name, linear models are flexible and can model curved relationships if predictors are transformed.

Failing to reject the null hypothesis

Failing to reject the null hypothesis doesn’t mean you “accept” it; it may just mean there is insufficient data or obscured relationships (e.g., due to outliers).

Lurking variables

A lurking variable $Z$ might be the real driver behind an observed relationship between $x$ and $y$.

Statistical inference assumptions

Data must be independent and identically distributed (i.i.d.).

My current project on word spotting in historical documents is fairly complete in functionality, but I decided that searching for word images on page images concurrently would be better for speeding it up. I’m already using Boost for much of the functionality, and instead of creating a dependency on the not-yet-mature C++11 support in various compilers, I decided to use boost::thread.

Suppose we have a functor such as:

class Search_t
{
   public:
       Search_t(Document d) { ... };
       SearchResult operator()(SearchItem i) { ... };
};

And we want to use this functor in multiple threads. We can’t simply do the following:

std::vector<SearchResult> results;
Search_t search(document);
// search_items is a vector<SearchItem> and si is an iterator over it.
for (si = search_items.begin(); si != search_items.end(); ++si)
{
   boost::thread task(boost::bind(search, *si));
   results.push_back(task); // ERROR!
}

This is because task does not return a SearchResult.

Instead, we need to store results within the object and retrieve them after they are generated.

I didn’t want to change the interface of Search_t because multithreading should be optional, and other parts of the program may depend on this interface. Instead, a wrapper class that runs these threads with a similar interface seemed like a better solution.

class SearchMT_t
{
   boost::shared_ptr<std::vector<boost::unique_future<SearchResult> > > futures_;

   public:
   SearchMT_t(Document d) :
   /* The most important assumption here is that Search_t does not alter
      the Document object's state in any way. Otherwise, we need to ensure that 
      document_ is accessed by only a single thread at a time using mutexes. */
   document_(d),
   futures_(new std::vector<boost::unique_future<SearchResult> >)
   {};

   void operator()(SearchItem si)
   {
       /* If you are sure that there won't be any race conditions
       between Search_t threads during search, you can move the following line
       to the constructor and use a single object for all searches. */
       Search_t search(document_);

       boost::packaged_task<SearchResult> search_task(std::bind(search, si));
       futures_->push_back(search_task.get_future());
       boost::thread task(boost::move(search_task));
   };

   std::vector<SearchResult> results()
   {
      std::vector<SearchResult> results;

      /* Wait for all threads to complete their work. */
      boost::wait_for_all(futures_->begin(), futures_->end());

      for(int i = 0; i < futures_->size(); ++i)
      {
          results.push_back((*futures_)[i].get());
      }

      return results;
   }
};

This way, it becomes much more straightforward to use multithreading in a loop:

std::vector<SearchResult> results;
SearchMT_t search(document);
// search_items is a vector<SearchItem> and si is an iterator over it.
for (si = search_items.begin(); si != search_items.end(); ++si)
{
    search(*si);
}

results = search.results();

We kept the Search_t class intact and used a much simpler approach in the loop.