This motivated me to learn Docker and gcloud rather quickly.

I was able to create a new Docker container in a short time. However, as I don’t want to share my Git SSH key publicly, I needed a way to put an SSH private key (~/.ssh/id_rsa) into this new container securely, without leaving any trace.

I tried a few things, but in the end, I decided to do something like:

FROM python:3.5-stretch as intermediate

# add credentials on build
RUN mkdir /root/.ssh/
# To use docker --build-arg, you can uncomment the following two lines and comment out the COPY line below.
# ARG SSH_PRIVATE_KEY
# RUN echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_rsa
COPY application_sshkey /root/.ssh/id_rsa
RUN chmod 0600 /root/.ssh/id_rsa

# make sure your domain is accepted
RUN touch /root/.ssh/known_hosts
RUN ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts

RUN git clone git@bitbucket.org:username/application /root/application

FROM python:3.5-stretch
COPY --from=intermediate /root/application /root/application

RUN pip3 install -r /root/application/requirements.txt

EXPOSE 9090/tcp

WORKDIR /root/application/

CMD python3 manage.py runserver 0.0.0.0:9090

Here, application_sshkey is a file I created using ssh-keygen and granted read-only access to on Bitbucket.

As you can see, the Dockerfile has two FROM statements. It creates a container to clone the application repository. Then it starts again with a new container and copies the repository to this new container. This way, it is not possible to peek into the private key using the docker history command.

By the way, I included two methods in the Dockerfile because gcloud builds does not accept a --build-arg parameter similar to docker build. I’m sure there are other workarounds for passing secrets to gcloud builds, but instead of digging for them, I found a solution that works for both Docker and Google Cloud Run.

Fortunately, there is a way to use the ~/.ssh/config file to specify different keys for different target URLs.

# Personal account - the default config
Host github.com
   HostName github.com
   User git
   IdentityFile ~/.ssh/id_rsa

# Work Account
Host work.github.com
   HostName github.com
   User git
   IdentityFile ~/.ssh/id_rsa_work

Now you can create another SSH key in ~/.ssh/id_rsa_work and add it to your work account. When you configure a repository, you need to specify git@work.github.com:user/repo.git as the repository address rather than git@github.com:user/repo.git.

pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

I noticed that pip has a --default-timeout parameter that can be configured to avoid this issue:

pip --default-timeout=1000 install package-name

Therefore, I decided that a small Bash script using rsync and find would work better. It monitors my entire home directory and backs up recent files.

The following script does exactly that:

#!/bin/bash

if [ "x$1" = "x" ] ; then
    TARGET=/media/augustus/backup-recent-`hostname`
else
    TARGET=$1
fi

PERIOD=15

mkdir -p $TARGET

# Delete files older than $PERIOD days
find $TARGET -ctime +$PERIOD -print -delete

# Copy files newer than $PERIOD under ~. Ignore files under .hg
for d in ~/*/ ; do
    find $d -path '*/.hg/*' -prune -o -type f -ctime -$PERIOD -print  -exec rsync -aRv {} $TARGET/ \;
done

It checks whether a command-line option is provided as the target; otherwise, it sets a default target. /media/augustus/ is an NFS mount in my case, but you can specify any path.

PERIOD is the number of days that the script considers recent. Currently, it backs up files changed in the last 15 days.

The script deletes older files from the backup. Since I use other solutions for long-term storage, I don’t want to keep them here, especially since this script runs via cron every two hours.

Note that the script only checks directories under the home directory, so files located directly in the home directory are not backed up.

It also skips files under .hg directories. You can add more -prune options to the find command to exclude other irrelevant directories from your backup.