An Incomplete Guide to Docker

Install docker on Armbian

To install latest docker, apt repository need to be updated, add docker official repository as follows:

$ sudo apt install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -

$ sudo apt-key fingerprint 0EBFCD88

$ sudo add-apt-repository \
   "deb [arch=arm64] https://download.docker.com/linux/debian \
   buster stable"

$ sudo apt update

NOTE:
The instruction in official doc does not work for Armbian, there is no docker binaries for Focal release, change $(lsb_release -cs) to buster to make it works.

Install docker engine:

$ sudo apt install docker-ce docker-ce-cli containerd.io=1.2.13-2
sudo usermod -aG docker $(whoami)
sudo reboot

The latest containerd has segmentation issue, so I pick version 1.2.13-2, other version besides 1.3.7 may also work, version string can be queried with:

$ apt-cache madison containerd.io
containerd.io |    1.3.7-1 | https://download.docker.com/linux/debian buster/stable arm64 Packages
containerd.io |   1.2.13-2 | https://download.docker.com/linux/debian buster/stable arm64 Packages
[...]

Using mirror to save time

For restricted network, using mirrors may speed up the docker pull process, thus save a lot of time, this can be done by either pass --registry-mirrors option to dockerd in /lib/systemd/system/docker.service, or add registry-mirrors KV-pair to daemon.json.

cat << EOF | sudo tee -a /etc/docker/daemon.json
{
   "registry-mirrors": [
       "https://hub-mirror.c.163.com",
       "https://docker.mirrors.ustc.edu.cn"
  ],
  "debug": true
}
EOF

If more than one mirrors are added, dockerd will try to use the first one, if it returns with errors or timed out, then next one will be used, and so forth, if all fails, it falls to the official repo.

After daemon configuration changing file , docker must be restarted:

sudo systemctl daemon-reload
sudo systemctl restart docker

The mirrors will be shown in the Registry Mirrors in the docker info output:

$ docker info | grep -A5 'Registry Mirrors'
WARNING: No swap limit support
 Registry Mirrors:
  https://hub-mirror.c.163.com/
  https://docker.mirrors.ustc.edu.cn/
 Live Restore Enabled: false

Now try to pull hello-world to see if it works:

$ docker pull hello-world

$ journalctl -fu docker.service
[...]
Oct 28 13:30:59 arm-64 dockerd[3300]: time="2020-10-28T13:30:59.176672354Z" level=debug msg="Calling POST /v1.40/images/create?fromImage=hello-world&tag=latest"
Oct 28 13:30:59 arm-64 dockerd[3300]: time="2020-10-28T13:30:59.343914257Z" level=debug msg="hostDir: /etc/docker/certs.d/hub-mirror.c.163.com"
Oct 28 13:30:59 arm-64 dockerd[3300]: time="2020-10-28T13:30:59.395723604Z" level=debug msg="Trying to pull hello-world from https://hub-mirror.c.163.com/ v2"
Oct 28 13:31:02 arm-64 dockerd[3300]: time="2020-10-28T13:31:02.553016126Z" level=debug msg="Pulling ref from V2 registry: hello-world:latest"
Oct 28 13:31:02 arm-64 dockerd[3300]: time="2020-10-28T13:31:02.553385090Z" level=debug msg="docker.io/library/hello-world:latest resolved to a manifestList object with 9 entries; looking for a unknown/arm64 match"
[...]

Private registry server deployment

A registry is an instance of the registry image, and runs within Docker, hosting docker images in local network saves a lot of time when running docker on several machines, the following example shows how to run a private registry on local host supporting multiple architecture with Image manifest.

Julio Suarez has a blog post on Deploying a Multi-Arch Docker Registry with much details.

I am going to setup local registry on Ubuntu, and running docker image on Phicomm N1, below command will registry in background listening to port 5000, and saves all the images to /mnt/registry, there are two options to achieve this, the first one is:

$ docker run -d \
    -p 5000:5000 \
    --restart=always \
    --name registry \
    -v /mnt/registry:/var/lib/registry \
    -e REGISTRY_STORAGE_DELETE_ENABLED=true \
    registry:2

Another way to deploy registry is using compose file:

cat << EOF >> $HOME/docker/docker-compose.yml
registry:
  restart: always
  image: registry:2
  ports:
    - 5000:5000
  environment:
    REGISTRY_STORAGE_DELETE_ENABLED: "true"
  volumes:
    - /mnt/registry:/var/lib/registry
EOF

$ docker-compose up -d

The container running status can be checked with:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
c9176f53cbb2        registry:2          "/entrypoint.sh /etc…"   1 hours ago         Up 56 minutes       0.0.0.0:5000->5000/tcp   registry

To pull arm64 image from docker hub, experimental feature has to be enabled, add "experimental": true to /etc/docker/daemon.json and restart docker service.

Now push docker images to localhost, jellyfin will be running on ubuntu, and other two are supposed to run on arm64 box:

# pull images from docker hub or mirror registry

$ docker pull jellyfin/jellyfin
$ docker image tag jellyfin/jellyfin localhost:5000/jellyfin/jellyfin-amd64
$ docker push localhost:5000/jellyfin/jellyfin-amd64
$ docker rmi jellyfin/jellyfin
$ docker rmi localhost:5000/jellyfin/jellyfin-amd64

$ docker pull --platform arm64 jellyfin/jellyfin
$ docker image tag jellyfin/jellyfin localhost:5000/jellyfin/jellyfin-arm64
$ docker push localhost:5000/jellyfin/jellyfin-arm64
$ docker rmi jellyfin/jellyfin
$ docker rmi localhost:5000/jellyfin/jellyfin-arm64

$ docker pull pihole/pihole
$ docker image tag pihole/pihole localhost:5000/pihole/pihole-amd64
$ docker push localhost:5000/pihole/pihole-amd64
$ docker rmi pihole/pihole
$ docker rmi localhost:5000/pihole/pihole-amd64

$ docker pull --platform arm64 pihole/pihole
$ docker image tag pihole/pihole localhost:5000/pihole/pihole-arm64
$ docker push localhost:5000/pihole/pihole-arm64
$ docker rmi pihole/pihole
$ docker rmi localhost:5000/pihole/pihole-arm64

$ docker pull homeassistant/home-assistant:stable
$ docker image tag homeassistant/home-assistant:stable localhost:5000/homeassistant/home-assistant-amd64:stable
$ docker push localhost:5000/homeassistant/home-assistant-amd64:stable
$ docker rmi homeassistant/home-assistant:stable
$ docker rmi localhost:5000/homeassistant/home-assistant-amd64:stable

$ docker pull --platform arm64 homeassistant/home-assistant:stable
$ docker image tag homeassistant/home-assistant:stable localhost:5000/homeassistant/home-assistant-arm64:stable
$ docker push localhost:5000/homeassistant/home-assistant-arm64:stable
$ docker rmi homeassistant/home-assistant:stable
$ docker rmi localhost:5000/homeassistant/home-assistant-arm64:stable

$ docker system prune

The repositories can be listed with docker registry HTTP API:

$ curl http://localhost:5000/v2/_catalog
{"repositories":["homeassistant/home-assistant-amd64","homeassistant/home-assistant-arm64","jellyfin/jellyfin-amd64","pihole/pihole-amd64","pihole/pihole-arm64","ubuntu"]}

$ curl http://localhost:5000/v2/pihole/pihole/tags/list
{"name":"pihole/pihole","tags":["latest"]}

The final step is to create manifest for each image, this need to enable experimental in docker config file $HOME/.docker/config.json:

{
    "experimental": "enabled"
}

The syntax for creating manifest file is:

Usage: docker manifest create MANIFEST_LIST MANIFEST [MANIFEST…]

The manifest for tags we created previously can be inspected:

$ docker manifest inspect --insecure localhost:5000/jellyfin/jellyfin-arm64
$ docker manifest inspect --insecure localhost:5000/pihole/pihole-arm64
$ docker manifest inspect --insecure localhost:5000/homeassistant/home-assistant-arm64:stable

Create manifest for each repository and push them to registry:

$ docker manifest create --insecure localhost:5000/jellyfin/jellyfin localhost:5000/jellyfin/jellyfin-arm64 localhost:5000/jellyfin/jellyfin-amd64
$ docker manifest push --insecure localhost:5000/jellyfin/jellyfin

$ docker manifest create --insecure localhost:5000/pihole/pihole localhost:5000/pihole/pihole-arm64 localhost:5000/pihole/pihole-amd64
$ docker manifest push --insecure localhost:5000/pihole/pihole

$ docker manifest create --insecure localhost:5000/homeassistant/home-assistant:stable localhost:5000/homeassistant/home-assistant-arm64:stable localhost:5000/homeassistant/home-assistant-amd64:stable
$ docker manifest push --insecure localhost:5000/homeassistant/home-assistant:stable

Now we can can see the two architectures in the newly created manifest:

$ docker manifest inspect --insecure localhost:5000/jellyfin/jellyfin
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1165,
         "digest": "sha256:010e197041008ffa00d2c94f31b87d64666e9a93a9cf3314ce7f87134b801a5f",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1376,
         "digest": "sha256:0826942c3085471463bb5cd30d9bd413449498bf5fb0c381d70cd7409e317d01",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

This usually not needed, unless you do NOT want the registry any more:

$ docker container stop registry && docker container rm -v registry

Last but not least, image tag can also pushed from another machine running docker, enable this by adding "insecure-registries" : ["ip.add.re.ss:5000"], to daemon.json

$ docker image tag jellyfin/jellyfin:stable ip.add.re.ss:5000/jellyfin/jellyfin-arm64:stable
$ docker push ip.add.re.ss:5000/jellyfin/jellyfin-arm64:stable

Troubleshooting

segmentation in containerd

After a reboot from freshly installation of docker, docker service failed to start, the log says, the dependency failed to start:

sudo systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com

Oct 28 05:50:05 arm-64 systemd[1]: Dependency failed for Docker Application Container Engine.
Oct 28 05:50:05 arm-64 systemd[1]: docker.service: Job docker.service/start failed with result 'dependency'.

From /lib/systemd/system/docker.service we see the most probable dependency is containerd:

BindsTo=containerd.service

Here BindsTo means dockerd depends on containerd, refer to description of systemd directives.

And when I check its version, it crashes due to segfault:

$ containerd --version
unexpected fault address 0xaaaad6c4f920
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0xaaaad6c4f920 pc=0xaaaad56fe85c]

goroutine 1 [running, locked to thread]:
runtime.throw(0xaaaad6401a57, 0x5)
	/usr/local/go/src/runtime/panic.go:774 +0x54 fp=0x400007bdd0 sp=0x400007bda0 pc=0xaaaad571b2cc
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:401 +0x394 fp=0x400007be00 sp=0x400007bdd0 pc=0xaaaad5730fcc
runtime.evacuate_fast64(0x400007bf08, 0xaaaad57294a0, 0xaaaad771b3e0)
	/usr/local/go/src/runtime/map_fast64.go:409 +0x114 fp=0x400007bec0 sp=0x400007be10 pc=0xaaaad56fe85c
runtime: unexpected return pc for runtime.doInit called from 0x300000002

The solution for this issue is to downgrade containerd to a lower version.

server gave HTTP response to HTTPS client

After the local registry setup completed, I try to test it by creating jellyfin service with compose file, I’ve noticed the pull speed was not as expected, dockerd complains about server gave HTTP response to HTTPS client:

dockerd[7266]: time="2020-10-29T23:48:39.573558660+08:00" level=warning msg="Error getting v2 registry: Get https://ip.add.re.ss:5000/v2/: http: server gave HTTP response to HTTPS client"

Most people got this resolved by adding

"insecure-registries" : ["ip.add.re.ss:5000"],

to file /etc/docker/daemon.json

This is not the case for me, I already done this before, double checking this file reveals the problem, before the ip address, it’s NOT http, it must be a copy and paste issue.