Blog

We recently moved to a mountainside, and while we are theoretically within the service area of our local cable internet provider, we have been waiting for them to get back to us about installation for over two weeks.

During that time, I’ve been doing my development work using the LTE on my phone (thanks unlimited data from Visible!). However, LTE isn’t really the fastest thing in the world, especially when you’re trying to pull a bunch of docker images as you’re testing multiple local kind clusters.

While I was waiting for another cluster to spin up, I decided to look into running a local Docker mirror. Docker publishes a recipe for this, but calling it a recipe feels a little misleading – there’s no code to copy and paste!

So here’s my recipe:

1. Get a registry container running:

   $ docker run --detach --env "REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io" --publish=5000:5000 --restart=always --volume "$HOME/docker-registry:/var/lib/registry" --name registry registry:2
   

2. Update your docker daemon JSON configuration to include the following:

   {
     "registry-mirrors": ["http://localhost:5000"]
   }
   
   
   

I use fish as my go-to shell – it’s fast, the syntax is more sane than bash, and there are a lot of great plugins for it.

One plugin I use a lot is jethrokuan’s port of z, which allows for very quick directory jumping.

Unfortunately, sometimes I reorganize my directories, and z can get confused and try to jump into a no longer existent directory.

No worries! z thought of that, and provides the z --clean command which removes nonexistent directories from its lists.

But I never remember to run that. Wouldn’t it be nice if I could just have that run automatically every two weeks or so?

I recently revisited a Python module that I developed called singletons. When I set it up, I tried to follow best practices for lots of things, including using tox with Travis CI to automatically run tests upon push. I used a cookiecutter template called cookiecutter-pylibrary, which set a lot of sensible defaults. And then I took a job where I didn’t do much Python at all.

Well, I’m finally getting back into Python (yay!), and decided to revisit this library. It seems the community is converging on poetry for packaging and depdency management rolled into one elegant tool, and having tried it out a bit, I have to say it’s quite nice. I decided to migrate my project to use this instead of setup.py, and while I was at it I decided to get rid of a lot of extraneous files and make the development and deployment process more streamlined.

I did, however, run into some hiccups getting everything set up to work with the way I do development, so I’m documenting my process here (if only to help my future self).

Python version management with asdf

First of all, there’s Python version management. Once upon a time I used pyenv, but I hated having to install a whole bunch of disparate tools for each programming language I used. Now I use asdf, which lets me use a single command to manage basically every programming language. If you haven’t set up asdf already, here’s a quickstart:

# install asdf and common dependencies
$ brew install asdf \
  coreutils automake autoconf openssl \
  libyaml readline libxslt libtool unixodbc \
  unzip curl

# set up asdf with python
$ asdf plugin add python
$ asdf install python 3.8.0

# install additional versions as necessary
$ asdf install python 3.7.5
$ asdf install python 3.6.9

What asdf does is add itself to your path, so that when you run python (or python3 or python3.8), it will use the version installed by asdf. Awesome! But there’s one caveat – it only uses those versions if you tell it to.

Using asdf versions of Python

asdf does give you the option of specifying a global version of a particular interpreter/compiler to use. However, given that OSX includes a system version of python (and some tools may expect that to function normally), I didn’t want to replace it system-wide. So my solution is to do the following.

In each folder where I’m doing python development, I run an asdf local python command. This creates a file called .tool-versions (which you should probably add to a global gitignore file). asdf refers to this file, and looks up the file hierarchy to find one, to determine which version of python to use.

For example, if I want to use Python 3.8.0, I would run the following:

$ asdf local python 3.8.0

The special trick for tox

tox requires multiple versions of Python to be installed. Using asdf, you have multiple versions installed, but they aren’t normally exposed to the current shell. Enter – multiple versions!

You can use the following command to expose multiple versions of Python in the current directory:

$ asdf local python 3.8.0 3.7.5 3.6.9

This will use 3.8.0 by default (if you just run python), but it will also put python3.7 and python3.6 symlinks in your path so you can run those too (which is exactly what tox is looking for).

Installing tox and poetry

Lastly, just to be safe, you should ensure that each of those asdf versions of python have the bare minimum of dependencies. Namely, tox and poetry.

$ pip3.8 install tox poetry
$ pip3.7 install tox poetry
$ pip3.6 install tox poetry

One other thing – asdf might miss the fact that you’ve installed tox and poetry, so you can run the following to force it to pick up on that:

$ asdf reshim python

Now you should be able to run tox normally!

Travis CI

Last of all, getting Travis to work with all this. It’s actually much simpler than it used to be. With an appropriate tox setup, you can keep your Travis configuration very simple:

.travis.yml

language: python
python:
  - "3.6"
  - "3.7"
  - "3.8"
before_install:
  - pip install poetry
install:
  - pip install tox-travis
script:
  - tox

tox.ini

[tox]
isolated_build = true
envlist = py36,py37,py38
skip_missing_interpreters = true

[testenv]
whitelist_externals = poetry
commands =
  poetry install -v --extras "eventlet gevent"
  poetry run pytest {posargs} tests/

Also, if you have other build stages, like docs, linting, etc., things will become a little more complicated, but hopefully still manageable!

Note that the poetry install command includes some extras. Chances are your library doesn’t have these, but I have some tests that use them. You can probably just do poetry install -v for most situations.

Bonus:

You can update pip for each environment to hide some annoying warnings:

$ pip3.8 install --upgrade pip
$ pip3.7 install --upgrade pip
$ pip3.6 install --upgrade pip

Also, by default, poetry creates virtualenvs in your user directory (~). I prefer to keep my virtualenvs close to the project files, and poetry has an option to support this.

$ poetry config settings.virtualenvs.in-project true
# or if you are running poetry 1.0
$ poetry config virtualenvs.in-project true