BUILD ALL THE IMAGES for ARM with k3s

TL;DR

This is a guide about building native Docker images for ARM by using a Raspberry Pi powered k3s cluster as CI/CD workers for gitlab. There’s a sample repository that demonstrates the basic usage:

https://gitlab.com/mariuspod/arm_ci_cd_example

Prerequesites

For this project I’m going to use an ARM-based k3s cluster like described in this excellent blog post from Alex Ellis: https://blog.alexellis.io/test-drive-k3s-on-raspberry-pi/

Once you have the cluster up and running, you can install helm for ARM by following the instructions from here: https://yoshwata-blog.netlify.com/posts/install-helm-to-k3s/

With helm we will install gitlab-runner, which is gitlab’s opensource project that runs the build jobs and can be configured to push back the results into a container registry. Gitlab-runners can be easily integrated into both a self-hosted or a managed gitlab instance. Unfortunately, the official helm chart for gitlab-runner doesn’t support ARM yet, but that’s an easy fix. Currently, this is still a bit experimental so you can checkout the chart from my git repository below and install it into your k3s by providing a custom <GITLAB_RUNNER_TOKEN> which you can simply retrieve from Gitlab in Settings -> CI/CD -> Runners:

$ git clone https://gitlab.com/mariuspod/gitlab-runner && cd gitlab-runner && helm install --set runnerRegistrationToken=<GITLAB_RUNNER_TOKEN> --name gitlab-runner .

Helm will output the status of the service deployment and you can check on your k3s if the gitlab-runner pod has been successfully started:

$ kubectl get po | grep gitlab-runner
gitlab-runner-gitlab-runner-dfdf7c79-tvxs7   1/1     Running   0          8d

CI/CD + ARM = ?

Okay, so we have a gitlab-runner pod running inside our self-hosted k3s cluster backed by several Raspberry Pis, quite neat. But what is this all good for other than for playing around with our k3s stack and having a good excuse to not use existing services like balena.io, create cross-compile builds with qemu or even ‚docker build‘-it directly on the Raspberry Pi ?

  • It’s a good usecase for k3s
  • It’s a good usecase for gitlab CI/CD
  • Target hardware can be used for building the software for it, so no extra costs apply
  • You have more time left for development because builds are automated
  • Artifacts are stored in gitlab’s container registry which is tied to the git repository of the project and can be easily deployed into the k3s cluster
  • It’s for free & fun 🙂

So the main motivator behind this was to have a dev-friendly CI/CD build pipeline for ARM builds that has a low management overhead. Whenever git-pushing new code, it should be packaged and built automatically so that it can be later deployed into the k3s ARM cluster without any hassle. In that way, we can focus on the what instead of the how of our software development cycle. This is pretty common for most developers in the x86_64 archictecture today, but for the ARM world it’s not as comfortable, yet. Basically, I always ended up moving my codebase to an ARM device and developing from there, git pushing from there and it always felt wrong.

So I decided to implement the CI/CD pipeline for ARM to make it easier for me to develop new software that could be run on my Raspberry Pis. I only had to port some relevant projects to ARM, which wasn’t too bad. It’s still a bit experimental, but now it works 🙂

Finally, the flow that I use looks like:

$ git push // at my dev machine, e.g. x86_64

Gitlab + k3s are doing the build magic

$ helm upgrade ... // at my k3s cluster, e.g. armv7l

… the dots

Photo by Maria Bobrova on Unsplash

Finally, we can Setup the CI/CD pipeline definition for gitlab.

Gitlab has created a great toolchain for automatically running build jobs for your projects. When you activate it, you can run tests, build your app into Docker images and deploy the packaged artifacts to target systems with ease. The configuration of a pipeline is done via the .gitlab-ci.yml inside your gitlab project.

As an example here’s the .gitlab-ci.yml from the example repository which I’ve created for this post:

stages:
  - build

build-image:
  stage: build
  image:
    name: registry.gitlab.com/mariuspod/kaniko/executor-debug-arm:latest
    entrypoint: [""]
  tags:
    - arm
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG

The pipeline has one stage called ‚build‘. In the job ‚build-image‘ which is assigned to the ‚build‘ stage we assign a custom kaniko image ported to ARM that is used by the gitlab-runner as the execution context.

The tag ‚arm‘ specifies that this project should be run on gitlab-runners with the ‚arm‘ tag only, so that no other runner without that tag can pick up your build job and fail because of an architecture mismatch.

The scripts which are executed persist the gitlab credentials for the container registry within the build context and execute the kaniko executor by passing it a Dockerfile to build and a destination registry to push back the results once the build is successful.

Basically the file is very similar to the .gitlab-ci.yml template for building Docker images using the kaniko executor from the gitlab documentation. I only had to change the image that can be used by the kubernetes executor and replace it with my ARM-compatible one.

DONE

We’ve started with an ARM-based k3s cluster and with the power of gitlab-runners and some ARM-ported versions for the gitlab-runner-helm-chart, gitlab-runner and kaniko we were able to create an ARM-based CI/CD build pipeline that runs on gitlab and k3s and builds ARM-compatible Docker images with a simple ‚git push‘ command from a developer machine! Here’s the build job output for the example respository.

Isn’t that cool ? I wish I had more time to code stuff, but theoretically I have more time now 🙂

I hope you had some fun reading this technical blog post. If you have time you can try it out and build something awesome with it. If you enjoyed it, you can leave me some comments. I’m also really happy to hear how you’re building your Docker images for ARM 🙂

//mariuspod