When building a Docker container image, in most cases, the machine where you build it on runs on the same CPU architecture as where you eventually want to use it. Today, more and more, there is a need for deploying containers on a different architecture as where it was built on. The most common scenario would be to build on your regular x86 desktop and to deploy to an ARM variant.
Docker images and containers are standardized and are substantive as they contain everything your application needs in a single image with multiple layers. This enables portability and maximum flexibility in terms of where to deploy them, independent of the underlying host OS (Linux, Windows, macOS).
One limitation though, is that the container image needs to be built for the target architecture. For example, let’s say you want to build a container image on your desktop PC that will ultimately run on an embedded device like a Raspberry Pi. The desktop is most likely running on a x86_64 CPU but your Raspberry Pi is using an ARM-based CPU architecture. In this case, when using the regular steps to build a Docker image, the result is in an unusable image for your Raspberry Pi as it doesn’t understand the ARM architecture.
While ARM becomes more popular, especially after the recent announcements from Apple regarding Apple Silicon, I figured it was a good time to write an article about this.
Building images for non-x86 architecture
There are a few options to overcome this challenge:
- Develop on the target platform itself
- Use pre-built images, for example from Dockerhub
- Use Emulation like Docker buildx or binfmt and QEMU
The first option is the easiest to work with but limits you to the resources of the target device. For a raspberry Pi, this is still feasible but for slower embedded devices or some edge compute equipment, this might be a pain.
The second option greatly limits you in terms of customization and flexibility unless you plan to perform a standard task that has a pre-built image available.
The last and third option is what we will be looking at, and since I’m mainly focusing on Linux, the rest of the article will describe using binfmt and QEMU to accomplish this task.
QEMU and binfmt
QEMU is an open source project that is a fast processor emulator. It supports emulation of many CPU architectures like ARM, alpha, MIPS, PowerPC, SPARC, s390x, … QEMU has the option to do user mode emulation. In this mode, QEMU doesn’t emulate a full VM but only the foreign CPU itself. To execute foreign code in the emulated CPU, it captures the syscalls and forwards them to the host kernel. This way, we are interfacing the native kernel in the same way as any native piece of software.
Binfmt is a subsystem that is part of the Linux kernel. It adds kernel support for miscellaneous Binary Formats. This piece will recognize arbitrary executable file formats and pass them to specific user space application such as emulators and virtual machines. For the purpose of this article, binfmt will be responsible for instructing the kernel to run non-x86 binaries with QEMU.
Prepare developer machine
As mentioned, the most common scenario would be to develop on a x86-based machine running Linux, Windows or Mac OS (for now) and building a container image for an ARM-based device.
The goal of the article is to build a small Apline-based container running a tiny Python script, service a simple webpage.
Starting point for our developer machine is a minimal install of Debian 10. This way the below steps should work on most existing installations. The target will be an ARM64v8 device (aarch64 Linux architecture), similar to a Raspberry Pi.
As you can see, the architecture of the machine we use to develop the Docker container image is x86-based:
jensd@deb10:~$ uname -m x86_64
Our target platform is ARM-based:
jensd@arm:~$ uname -m aarch64
The first step, if not done already, would be to install Docker on the freshly installed Debian machine. I’ll try to keep this short as it’s not the purpose of the article.
Install prerequisites for installing Docker from the separate repo:
jensd@deb10:~$ sudo apt-get update … jensd@deb10:~$ sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common Reading package lists… Done Building dependency tree … Need to get 15.6 MB of archives. After this operation, 48.3 MB of additional disk space will be used. Do you want to continue? [Y/n] y … Processing triggers for man-db (2.8.5-2) … Processing triggers for dbus (1.12.20-0+deb10u1) …
Install the GPG key and add the repository for Docker:
jensd@deb10:~$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add - OK jensd@deb10:~$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
Install the Docker packages:
jensd@deb10:~$ sudo apt-get update … jensd@deb10:~$ sudo apt-get install docker-ce docker-ce-cli Reading package lists… Done Building dependency tree … Need to get 159 MB of archives. After this operation, 693 MB of additional disk space will be used. Do you want to continue? [Y/n] y … Processing triggers for man-db (2.8.5-2) … Processing triggers for libc-bin (2.28-10) …
In order to make things comfortable, it’s handy to allow our non-root user to access/run Docker commands:
jensd@deb10:~$ sudo usermod -a -G docker jensd jensd@deb10:~$ newgrp docker
To test, we can run a small Alpine image and double check as which CPU architecture this runs as a Docker container:
jensd@deb10:~$ docker run -i -t alpine /bin/uname -m Unable to find image 'alpine:latest' locally latest: Pulling from library/alpine df20fa9351a1: Pull complete Digest: sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 Status: Downloaded newer image for alpine:latest x86_64
As you can see, the output, as expected, is x86_64, which matches with the same architecture of our build machine.
Install QEMU and binfmt
Next, we need to install binfmt and the QEMU emulation binaries:
jensd@deb10:~$ sudo apt-get install qemu-user qemu-user-static binfmt-support Reading package lists… Done Building dependency tree … Need to get 31.6 MB of archives. After this operation, 301 MB of additional disk space will be used. Do you want to continue? [Y/n] y … Processing triggers for man-db (2.8.5-2) … Processing triggers for libc-bin (2.28-10) …
After the installation, we can find the static QEMU binaries in /usr/bin/qemu*static:
jensd@deb10:~$ ls -1 /usr/bin/qemu-*static /usr/bin/qemu-aarch64_be-static /usr/bin/qemu-aarch64-static /usr/bin/qemu-alpha-static /usr/bin/qemu-armeb-static /usr/bin/qemu-arm-static /usr/bin/qemu-cris-static /usr/bin/qemu-hppa-static /usr/bin/qemu-i386-static /usr/bin/qemu-m68k-static /usr/bin/qemu-microblazeel-static /usr/bin/qemu-microblaze-static /usr/bin/qemu-mips64el-static /usr/bin/qemu-mips64-static /usr/bin/qemu-mipsel-static /usr/bin/qemu-mipsn32el-static /usr/bin/qemu-mipsn32-static /usr/bin/qemu-mips-static /usr/bin/qemu-nios2-static /usr/bin/qemu-or1k-static /usr/bin/qemu-ppc64abi32-static /usr/bin/qemu-ppc64le-static /usr/bin/qemu-ppc64-static /usr/bin/qemu-ppc-static /usr/bin/qemu-riscv32-static /usr/bin/qemu-riscv64-static /usr/bin/qemu-s390x-static /usr/bin/qemu-sh4eb-static /usr/bin/qemu-sh4-static /usr/bin/qemu-sparc32plus-static /usr/bin/qemu-sparc64-static /usr/bin/qemu-sparc-static /usr/bin/qemu-tilegx-static /usr/bin/qemu-x86_64-static /usr/bin/qemu-xtensaeb-static /usr/bin/qemu-xtensa-static
As you can see, there is plenty of architectures we can emulate using QEMU!
As a test, we can try to run the same Alpine image as we did as a test after installing docker, built for arm64v8, on our x86 box.
jensd@deb10:~$ docker run -v /usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static -i -t arm64v8/alpine /bin/uname -m Unable to find image 'arm64v8/alpine:latest' locally latest: Pulling from arm64v8/alpine b538f80385f9: Pull complete Digest: sha256:3b3f647d2d99cac772ed64c4791e5d9b750dd5fe0b25db653ec4976f7b72837c Status: Downloaded newer image for arm64v8/alpine:latest aarch64
As you can see from the uname -m output, this Docker container is executing binaries compiled for aarch64, although it is physically running on our x86-based machine.
Build a test container image
Now that we got the necessary tools installed, we can start building a simple Docker image and prepare it for deployment on a different architecture.
The goal of this simple Docker image is to run a tiny Python based webserver that will tell us on which architecture it is running.
This is how the python script (webserver.py) looks like:
from http.server import BaseHTTPRequestHandler, HTTPServer import platform class S(BaseHTTPRequestHandler): def _set_headers(self): self.send_response(200) self.send_header('Content-type', 'text/html') self.end_headers() def do_GET(self): self._set_headers() r="<html><body><h1>Webserver on: " + str(platform.machine()) + "</h1></body></html>" self.wfile.write(bytes(r,"utf8")) def run(server_class=HTTPServer, handler_class=S, port=9000): server_address = ('', port) httpd = server_class(server_address, handler_class) print ('Starting webserver on ' + str(platform.machine()) + ' architecture') httpd.serve_forever() if __name__ == "__main__": run()
This script will listen on port 9000 and will, when a GET request comes in, return a simple HTML page saying Webserver on <architecture>.
First we’ll build the image for our developer machine, then we’ll make the necessary adjustments to build the same for another architecture. This way, you can see what changes are required to adapt/port one to another.
Dockerfile to create this image:
jensd@deb10:~$ cat Dockerfile-x86_64 FROM alpine RUN apk add --no-cache python3 COPY webserver.py /webserver.py EXPOSE 9000 ENTRYPOINT ["python3", "/webserver.py"]
At this point, we can build the image to test on the x86_64 dev machine:
jensd@deb10:~$ docker build -t web1 -f Dockerfile-x86_64 . Sending build context to Docker daemon 26.11kB Step 1/5 : FROM alpine ---> a24bb4013296 Step 2/5 : RUN apk add --no-cache python3 ---> Running in db9d393c7dcd fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz (1/10) Installing libbz2 (1.0.8-r1) (2/10) Installing expat (2.2.9-r1) (3/10) Installing libffi (3.3-r2) (4/10) Installing gdbm (1.13-r1) (5/10) Installing xz-libs (5.2.5-r0) (6/10) Installing ncurses-terminfo-base (6.2_p20200523-r0) (7/10) Installing ncurses-libs (6.2_p20200523-r0) (8/10) Installing readline (8.0.4-r0) (9/10) Installing sqlite-libs (3.32.1-r0) (10/10) Installing python3 (3.8.5-r0) Executing busybox-1.31.1-r16.trigger OK: 53 MiB in 24 packages Removing intermediate container db9d393c7dcd ---> 9430f28cd7a0 Step 3/5 : COPY webserver.py /webserver.py ---> 557a4c8115a0 Step 4/5 : EXPOSE 9000 ---> Running in 0b896bf63ac3 Removing intermediate container 0b896bf63ac3 ---> fbcc355442e9 Step 5/5 : ENTRYPOINT ["python3", "/webserver.py"] ---> Running in 8b8a5805f3d8 Removing intermediate container 8b8a5805f3d8 ---> 316a4c95080a Successfully built 316a4c95080a Successfully tagged web1:latest
Once the image is built, we can see it in the repository and we can check for which architecture it got built:
jensd@deb10:~$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE web1 latest 316a4c95080a 57 seconds ago 49MB arm64v8/alpine latest 62ee0e9f8440 4 months ago 5.32MB alpine latest a24bb4013296 4 months ago 5.57MB hello-world latest bf756fb1ae65 9 months ago 13.3kB jensd@deb10:~$ docker inspect web1 | grep Arch "Architecture": "amd64",
We can now run the image and start a container from it:
jensd@deb10:~$ docker run --rm -ti -p 9000:9000 web1 Starting webserver on x86_64 architecture 172.17.0.1 - - [15/Oct/2020 07:49:10] "GET / HTTP/1.1" 200 -
In another session, we can query the webserver on port 9000 and see what it returns on a GET request:
jensd@deb10:~$ curl localhost:9000 <html><body><h1>Webserver on: x86_64</h1></body></html>
Or the same in a brower:
As expected, we get a response that we’re running on an x86_64 platform. Makes sense as so far we haven’t made any adjustments to change our image/container for anything else that the developer machine.
Build the test image for non-x86
At this point, we went through all the steps to build a “regular” Docker image for the same platform as where we built it on. Now we’ll look into the changes required in order to build for another architecture.
The only change we need to do is on our Dockerfile. We need to copy the static QEMU binary for the target architecture to our container and we need to use a base image that suits the same architecture.
As we need to copy that QEMU file to the image on build time we need to make sure it is available in the build context .:
jensd@deb10:~$ cp /usr/bin/qemu-aarch64-static .
Taking the above example and adapting it for aarch64 or arm64v8:
jensd@deb10:~$ cat Dockerfile-aarch64 FROM arm64v8/alpine COPY qemu-aarch64-static /usr/bin RUN apk add --no-cache python3 COPY webserver.py /webserver.py EXPOSE 9000 ENTRYPOINT ["python3", "/webserver.py"]
This is all it takes to adapt the Dockerfile and build the image for our target platform: arm64v8/aarch64:
jensd@deb10:~$ docker build -t web2 -f Dockerfile-aarch64 .
Sending build context to Docker daemon 6.657MB
Step 1/6 : FROM arm64v8/alpine
Step 2/6 : COPY qemu-aarch64-static /usr/bin
Step 3/6 : RUN apk add --no-cache python3
---> Running in 9e01d3ca8e32
(1/10) Installing libbz2 (1.0.8-r1)
(2/10) Installing expat (2.2.9-r1)
(3/10) Installing libffi (3.3-r2)
(4/10) Installing gdbm (1.13-r1)
(5/10) Installing xz-libs (5.2.5-r0)
(6/10) Installing ncurses-terminfo-base (6.2_p20200523-r0)
(7/10) Installing ncurses-libs (6.2_p20200523-r0)
(8/10) Installing readline (8.0.4-r0)
(9/10) Installing sqlite-libs (3.32.1-r0)
(10/10) Installing python3 (3.8.5-r0)
OK: 52 MiB in 24 packages
Removing intermediate container 9e01d3ca8e32
Step 4/6 : COPY webserver.py /webserver.py
Step 5/6 : EXPOSE 9000
---> Running in 6dbd9aff033f
Removing intermediate container 6dbd9aff033f
Step 6/6 : ENTRYPOINT ["python3", "/webserver.py"]
---> Running in 29067efc8955
Removing intermediate container 29067efc8955
Successfully built 21c13086b01c
Successfully tagged web2:latest
Checking the architecture listed with Docker inspect for web2, shows us the following:
jensd@deb10:~$ docker inspect web2 | grep Arch
Running the image and start a container:
jensd@deb10:~$ docker run --rm -ti -p 9000:9000 web2
Starting webserver on aarch64 architecture
172.17.0.1 - - [15/Oct/2020 08:48:30] "GET / HTTP/1.1" 200 -
When doing the same test, to query the webserver, as with the native image for x86_64, we can see that this is really running as on ARM:
jensd@deb10:~$ curl localhost:9000 <html><body><h1>Webserver on: aarch64</h1></body></html>
And the same using a browser:
At this point, we can save/export the image and deploy it on a machine/device that is having the aarch64 CPU architecutre:
jensd@deb10:~$ docker save -o web_aarch64.tar web2
jensd@deb10:~$ ls web_aarch64.tar
After saving the image to web_aarch64.tar, we can copy it to our ARM device, load it and run it as if it was built on the same box:
jensd@arm:~$ scp email@example.com:/home/jensd/web_aarch64.tar . The authenticity of host '192.168.0.186 (192.168.0.186)' can't be established. ECDSA key fingerprint is SHA256:7S17jIT9fotQOdYjasxLpan9LHvcs7rj23eRtLFSWbU. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.0.186' (ECDSA) to the list of known hosts. firstname.lastname@example.org's password: web_aarch64.tar 100% 55MB 60.0MB/s 00:00 jensd@arm:~$ docker load -i web_aarch64.tar f0256daf202a: Loading layer 6.631MB/6.631MB 4dc695e29258: Loading layer 45.21MB/45.21MB 4dd6fec867f3: Loading layer 2.56kB/2.56kB Loaded image: web2:latest jensd@arm:~$ docker run --rm -ti -p 9000:9000 web2 Starting webserver on aarch64 architecture 192.168.0.30 - - [15/Oct/2020 08:54:11] "GET / HTTP/1.1" 200 -
This results in the same webpage being displayed and architecture aarch64 as output.