Categories
Automation CI/CD DevOps

Running minikube in Docker container

In the last article we saw how to run Jenkins Docker in Docker Agent. In addition to its obvious use case of building and pushing Docker images, I also noted that such an agent allows to run minikube inside. Today, I’ll show how to run minikube in Docker container. Of course, we’ll use docker in docker client and daemon images for that.

If you later find this article useful take a look at the disclaimer for information on how to thank me.

Table Of Contents
  1. Introduction
  2. Minikube in Docker demo
  3. Alternatives? Ah, yes, there's podman
  4. Summary

Introduction

If all you need is step by step procedure for running minikube inside Docker container, jump straight to the demo. Otherwise, if you want to understand the motivation better, keep reading.

Docker in Docker Jenkins agent is great for Jenkins which is running in Kubernetes and provisions its agents on demand. We also saw such an agent main use case – building docker images and pushing them to some registry. That’s cool. However, what if you want to take your CI/CD to the next level and deploy helm charts as part of the release pipeline.

Deploy and test helm charts as part of CI/CD

Why would you want to do that? For example, for testing chart’s successful deployment before you release it i.e push it to some registry. In addition, you may want to run some tests after deployment, e.g. acceptance tests triggered my magic command helm test. You can, of course, skip these steps, push the probably not working chart and know about it only when it reached production. Even if the deployment succeeds, but some basic sanity test fails after deployment, it’s bad, right? Yes, it’s better to have some automated test running before the actual release. This test will run on a successfully deployed chart in CI/CD environment during release pipeline.

What is this environment? There are many options. Today, I’ll show the option of minikube cluster started in docker container. We’ll deploy the chart there and run acceptance tests using helm test. That will give the idea of what could run during release of helm charts if the same commands will run in Docker in Docker Jenkins agent.

Minikube in Docker container options

We’ll see 2 options:

  • minikube in Docker daemon in Docker container
  • minikube in Docker client in Docker container. This container will, of course, connect to another Docker daemon in Docker container

See Docker Client or Docker Daemon in Docker? if you are confused.

Enough talking. Let’s see these 2 options in action.

Minikube in Docker demo

Demo Prerequisites

Today, you’ll only need Docker for running images.

minikube in Docker daemon container

It’s quite easy to run minikube in docker daemon container. Let’s do that:

$ docker run --name dind -d --privileged docker:20.10.17- dind 
$ docker exec -it dind sh
/ # wget https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
/ # mv minikube-linux-amd64 minikube
/ # chmod +x minikube 
/ # ./minikube start --force
...
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
/ # ./minikube kubectl -- run --image=hello-world
/ # ./minikube kubectl -- logs pod/hello

Hello from Docker!
...

minikube in Docker client container

Note that --force flag starts minikube using docker driver as root. This is undesirable as minikube warns us.

Let’s now run minikube in docker client container which will connect to docker daemon container.

Firstly, let’s start docker daemon container as before, yet now let’s connect it to docker network and give it network alias of dind. 

docker run --name dind -d --privileged --network dind --network-alias dind -e DOCKER_TLS_CERTDIR=""  docker:dind

We set DOCKER_TLS_CERTDIR to "" in order to not require docker client to authenticate against the daemon. Inspect the logs for making sure the daemon started successfully or add health check which will do that for you.

$ docker logs -f dind
...
time="2022-08-09T06:30:23.481053486Z" level=info msg="Daemon has completed initialization"
time="2022-08-09T06:30:23.560717962Z" level=info msg="API listen on /var/run/docker.sock"
time="2022-08-09T06:30:23.567636047Z" level=info msg="API listen on [::]:2375"

Now, let’s start docker client container in the same network, enter its shell and make sure it can connect to the daemon:

docker run --name dind-client -it  --network dind  -e DOCKER_HOST="tcp://dind:2375"docker sh
/ # docker version
...
Client:
 Version:           20.10.17
...
Server: Docker Engine - Community
 Engine:
  Version:          20.10.17

It works! Pay attention to DOCKER_HOST environment variable which we set to the hostname and port of the deamon.

You can now repeat the same commands we ran in Docker daemon container in order to start minikube.

So, we started minikube in docker client container as well. Now let’s put minikube to some use by deploying helm chart to it and running some acceptance tests.

Deploy helm chart to minikube in docker and test it

# create dedicated namespace where Jenkins helm chart will be installed
./minikube kubectl create namespace jenkins
# install helm
wget https://get.helm.sh/helm-v3.9.2-linux-amd64.tar.gz
tar -zxvf helm-v3.9.2-linux-amd64.tar.gz 
mv linux-amd64/helm /usr/local/bin/helm
# install jenkins chart
helm repo add jenkinsci https://charts.jenkins.io
helm repo update
helm install jenkins -n jenkins jenkinsci/jenkins
  • After Jenkins images are downloaded and deployed (it may take several minutes) you will see below status:
./minikube kubectl -- get all -n jenkins
pod/jenkins-0   2/2     Running   0          2m57s

NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S
)          AGE
service/jenkins         NodePort    10.100.21.112   <none>        8080:3
0653/TCP   2m57s
service/jenkins-agent   ClusterIP   10.96.252.104   <none>        50000/
TCP        2m57s

NAME                       READY   AGE
statefulset.apps/jenkins   1/1     2m57s
helm test jenkins -n jenkins
NAME: jenkins
LAST DEPLOYED: Tue Aug  9 06:59:11 2022
NAMESPACE: jenkins
STATUS: deployed
REVISION: 1
TEST SUITE:     jenkins-tests
Last Started:   Tue Aug  9 07:04:05 2022
Last Completed: Tue Aug  9 07:04:05 2022
Phase:          Succeeded
TEST SUITE:     jenkins-ui-test-ybu3s
Last Started:   Tue Aug  9 07:04:05 2022
Last Completed: Tue Aug  9 07:05:28 2022
Phase:          Succeeded

Wow. That was cool. Now, imagine you would want to do the same in CI/CD pipelines. Makes sense?

Alternatives? Ah, yes, there’s podman

Yes, there’s Podman which uses the same CLI as Docker. It’s indeed lightweight and doesn’t require a daemon Docker needs. And it may be a good idea to run minikube in a container with podman installed. However, I didn’t succeed to run minikube there. Moreover, from my understanding podman is still not widely adopted. Have a look, for instance, at the number of questions at Stack Overflow tagged with podman and docker tags and make conclusions yourself.

I’ve also explored Podman Jenkins agent you may find interesting.

Summary

That’s it about running minikube in Docker container. You may also find useful my older Kubernetes and CI/CD articles useful. As always, feel free to share. If you found this article useful, take a look at the disclaimer for information on how to thank me.

Recommended Kubernetes courses on Pluralsight:

Sign up using this link to get exclusive discounts like 50% off your first month or 15% off an annual subscription)

Recommended Kubernetes books on Amazon: