alihan/microk8s
1
0
Fork 0
mirror of https://github.com/ubuntu/microk8s.git synced 2021-05-23 02:23:41 +03:00
Code Issues Projects Releases Wiki Activity

Remove user guide from docs (#471)

Browse Source 
* Remove user guide from docs
* Keep community file
This commit is contained in:
Konstantinos Tsakalozos
2019-05-20 12:41:31 +02:00
committed by GitHub
parent c08c7027eb
commit 8cbb4d30f5
5 changed files with 2 additions and 854 deletions
Download Patch File Download Diff File Expand all files Collapse all files

191
README.md
View File

@@ -4,202 +4,15 @@
<img src="/docs/images/certified_kubernetes_color-222x300.png" align="right" width="200px">Kubernetes in a [snap](https://snapcraft.io/) that you can run locally.
## User Guide
Snaps are frequently updated to match each release of Kubernetes. The quickest way to get started is to install directly from the snap store. You can install MicroK8s and let it update to the latest stable upstream Kubernetes release with:
You can install MicroK8s with the latest stable upstream Kubernetes release with:
```
snap install microk8s --classic
```
Alternatively, you can select a MicroK8s channel that will follow a specific Kubernetes release series. For example, you install MicroK8s and let it follow the `v1.12` series with:
```
snap install microk8s --classic --channel=1.12/stable
```
You can read more on the MicroK8s release channels in the [Release Channels and Upgrades](docs/release-channels.md) doc.
At any point you can check MicroK8s' availability with:
```
microk8s.status
```
During installation you can use the `--wait-ready` flag to wait for the kubernetes services to initialise:
```
microk8s.status --wait-ready
```
> In order to install MicroK8s make sure
> - you go through the [list of ports](docs/ports.md) that need to be available
### Accessing Kubernetes
To avoid colliding with a `kubectl` already installed and to avoid overwriting any existing Kubernetes configuration file, MicroK8s adds a `microk8s.kubectl` command, configured to exclusively access the new MicroK8s install. When following instructions online, make sure to prefix `kubectl` with `microk8s.`.
```
microk8s.kubectl get nodes
microk8s.kubectl get services
```
If you do not already have a `kubectl` installed you can alias `microk8s.kubectl` to `kubectl` using the following command
```
snap alias microk8s.kubectl kubectl
```
This measure can be safely reverted at anytime by doing
```
snap unalias kubectl
```
If you already have `kubectl` installed and you want to use it to access the MicroK8s deployment you can export the cluster's config with:
```
microk8s.kubectl config view --raw > $HOME/.kube/config
```
Note: The API server on port 8080 is listening on all network interfaces. In its kubeconfig file, MicroK8s is using the loopback interface, as you can see with `microk8s.kubectl config view`. The `microk8s.config` command will output a kubeconfig with the host machine's IP (instead of the 127.0.0.1) as the API server endpoint.
For more information on using MicroK8s see the [official docs](https://microk8s.io/docs/).
### Kubernetes Addons
MicroK8s installs a barebones upstream Kubernetes. This means just the `api-server`, `controller-manager`, `scheduler`, `kubelet`, `cni`, `kube-proxy` are installed and run. Additional services like `kube-dns` and `dashboard` can be run using the `microk8s.enable` command
```
microk8s.enable dns dashboard
```
These addons can be disabled at anytime using the `disable` command
```
microk8s.disable dashboard dns
```
With `microk8s.status` you can see the list of available addons and which ones are currently enabled. You can find the addon manifests and/or scripts under `${SNAP}/actions/`, with `${SNAP}` pointing by default to `/snap/microk8s/current`.
#### List of Available Addons
- **dns**: Deploy kube dns. This addon may be required by others thus we recommend you always enable it. In environments where the external dns servers `8.8.8.8` and `8.8.4.4` are blocked you will need to update the upstream dns servers in `microk8s.kubectl -n kube-system edit configmap/kube-dns` after enabling the addon.
- **rbac**: Enable RBAC (role-based access control) authorization mode. **NOTE**: Most of the other addons will not work with the RBAC addon, since they are not RBAC enabled.
- **dashboard**: Deploy kubernetes dashboard as well as grafana and influxdb. To access grafana point your browser to the url reported by `microk8s.kubectl cluster-info`.
- **storage**: Create a default storage class. This storage class makes use of the hostpath-provisioner pointing to a directory on the host. Persistent volumes are created under `${SNAP_COMMON}/default-storage`. Upon disabling this addon you will be asked if you want to delete the persistent volumes created.
- **ingress**: Create an ingress controller.
- **gpu**: Expose GPU(s) to MicroK8s by enabling the nvidia runtime and nvidia-device-plugin-daemonset. Requires NVIDIA drivers to already be installed on the host system.
- **istio**: Deploy the core [Istio](https://istio.io/) services. You can use the `microk8s.istioctl` command to manage your deployments.
- **registry**: Deploy an image private registry and expose it on `localhost:32000`. The storage addon will be enabled as part of this addon. See [the registry documentation](docs/working.md) for more details.
- **metrics-server**: Deploy the [Metrics Server](https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/#metrics-server).
- **prometheus**: Deploy the [Prometheus Operator](https://github.com/coreos/prometheus-operator) v0.25.
- **fluentd**: Deploy [Elasticsearch-Kibana-Fluentd](https://kubernetes.io/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/) logging and monitoring solution.
- **jaeger**: Deploy the [Jaeger Operator](https://github.com/jaegertracing/jaeger-operator) v1.8.2 in the "simplest" configuration.
- **linkerd**: Deploy linkerd2 [Linkerd](https://linkerd.io/2/overview/) service mesh. By default proxy auto inject is not enabled. To enable auto proxy injection, simply use `microk8s.enable linkerd:proxy-auto-inject`. If you need to pass more arguments, separate them with `;` and enclose the addons plus arguments with double quotes. Example: `microk8s.enable "linkerd:proxy-auto-inject;tls=optional;skip-outbound-ports=1234,3456"`. User `microk8s.linkerd` command to interact with Linkerd.
### Stopping and Restarting MicroK8s
You may wish to temporarily shutdown MicroK8s when not in use without un-installing it.
MicroK8s can be shutdown with:
```
microk8s.stop
```
MicroK8s can be restarted later with:
```
microk8s.start
```
### Removing MicroK8s
Before removing MicroK8s, use `microk8s.reset` to stop all running pods.
```
microk8s.reset
snap remove microk8s
```
### Configuring MicroK8s Services
The following systemd services will be running in your system:
- **snap.microk8s.daemon-apiserver**, is the [kube-apiserver](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) daemon started using the arguments in `${SNAP_DATA}/args/kube-apiserver`
- **snap.microk8s.daemon-controller-manager**, is the [kube-controller-manager](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/) daemon started using the arguments in `${SNAP_DATA}/args/kube-controller-manager`
- **snap.microk8s.daemon-scheduler**, is the [kube-scheduler](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/) daemon started using the arguments in `${SNAP_DATA}/args/kube-scheduler`
- **snap.microk8s.daemon-kubelet**, is the [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) daemon started using the arguments in `${SNAP_DATA}/args/kubelet`
- **snap.microk8s.daemon-proxy**, is the [kube-proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) daemon started using the arguments in `${SNAP_DATA}/args/kube-proxy`
- **snap.microk8s.daemon-containerd**, is the [containerd](https://containerd.io/) daemon started using the configuration in `${SNAP_DATA}/args/containerd` and `${SNAP_DATA}/args/containerd-template.toml`.
- **snap.microk8s.daemon-etcd**, is the [etcd](https://coreos.com/etcd/docs/latest/v2/configuration.html) daemon started using the arguments in `${SNAP_DATA}/args/etcd`
Normally, `${SNAP_DATA}` points to `/var/snap/microk8s/current`.
To reconfigure a service you will need to edit the corresponding file and then restart the respective daemon. For example:
```
echo '-l=debug' | sudo tee -a /var/snap/microk8s/current/args/containerd
sudo systemctl restart snap.microk8s.daemon-containerd.service
```
### Deploy Behind a Proxy
To let MicroK8s use a proxy enter the proxy details in `${SNAP_DATA}/args/containerd-env` and restart the containerd daemon service with:
```
sudo systemctl restart snap.microk8s.daemon-containerd.service
```
## Troubleshooting
To troubleshoot a non-functional MicroK8s deployment, start by running the `microk8s.inspect` command. This command performs a set of tests against MicroK8s and collects traces and logs in a report tarball. In case any of the aforementioned daemons are failing you will be urged to look at the respective logs with `journalctl -u snap.microk8s.<daemon>.service`. `microk8s.inspect` may also make suggestions on potential issues it may find. If you do not manage to resolve the issue you are facing please file a [bug](https://github.com/ubuntu/microk8s/issues) attaching the inspection report tarball.
Some common problems and solutions are listed below.
### My dns and dashboard pods are CrashLooping.
The [Kubenet](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) network plugin used by MicroK8s creates a `cbr0` interface when the first pod is created. If you have `ufw` enabled, you'll need to allow traffic on this interface:
`sudo ufw allow in on cbr0 && sudo ufw allow out on cbr0`
### My pods can't reach the internet or each other (but my MicroK8s host machine can).
Make sure packets to/from the pod network interface can be forwarded
to/from the default interface on the host via the iptables tool. As shown below such changes can be made persistent via installation of the iptables-persistent package:
```
sudo iptables -P FORWARD ACCEPT
sudo apt-get install iptables-persistent
```
or, if using `ufw`:
`sudo ufw default allow routed`
The MicroK8s inspect command can be used to check the firewall configuration:
`microk8s.inspect`
A warning will be shown if the firewall is not forwarding traffic.
### My log collector is not collecting any logs.
By default container logs are located in `/var/log/pods/{id}`. You have to mount this location in your log collector for that to work. Following is an example diff for [fluent-bit](https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds.yaml):
```diff
@@ -36,6 +36,9 @@
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
+ - name: varlibdockercontainers
+ mountPath: /var/snap/microk8s/common/var/lib/containerd/
+ readOnly: true
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
terminationGracePeriodSeconds: 10
@@ -45,7 +48,7 @@
path: /var/log
- name: varlibdockercontainers
hostPath:
- path: /var/lib/docker/containers
+ mountPath: /var/snap/microk8s/common/var/lib/containerd/
- name: fluent-bit-config
configMap:
name: fluent-bit-config
```
## Building from source

69
docs/containerd-template.toml
View File

@@ -1,69 +0,0 @@
oom_score = 0
[grpc]
uid = 0
gid = 0
max_recv_message_size = 16777216
max_send_message_size = 16777216
[debug]
address = ""
uid = 0
gid = 0
[metrics]
address = "127.0.0.1:1338"
grpc_histogram = false
[cgroup]
path = ""
[plugins]
[plugins.cgroups]
no_prometheus = false
[plugins.cri]
stream_server_address = ""
stream_server_port = "10010"
enable_selinux = false
sandbox_image = "k8s.gcr.io/pause:3.1"
stats_collect_period = 10
systemd_cgroup = false
enable_tls_streaming = false
max_container_log_line_size = 16384
[plugins.cri.containerd]
snapshotter = "overlayfs"
no_pivot = false
[plugins.cri.containerd.default_runtime]
runtime_type = "io.containerd.runtime.v1.linux"
runtime_engine = ""
runtime_root = ""
[plugins.cri.containerd.untrusted_workload_runtime]
runtime_type = ""
runtime_engine = ""
runtime_root = ""
[plugins.cri.cni]
bin_dir = "${SNAP}/opt/cni/bin"
conf_dir = "${SNAP_DATA}/args/cni-network"
conf_template = ""
[plugins.cri.registry]
[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins.cri.registry.mirrors."local.insecure-registry.io"]
endpoint = ["http://localhost:32000"]
[plugins.cri.registry.mirrors."10.141.241.175:32000"]
endpoint = ["http://10.141.241.175:32000"]
[plugins.diff-service]
default = ["walking"]
[plugins.linux]
shim = "containerd-shim"
runtime = "${RUNTIME}"
runtime_root = ""
no_shim = false
shim_debug = true
[plugins.scheduler]
pause_threshold = 0.02
deletion_threshold = 0
mutation_threshold = 100
schedule_delay = "0s"
startup_delay = "100ms"

58
docs/ports.md
View File

@@ -1,58 +0,0 @@
# Services Exposed and Ports Used
MicroK8s is meant to be used for local development thus certain security issues are not addressed at this point. Here we present the ports and sockets each service uses as well as the default authorisation and authentication configuration.
Services can be placed in two groups based on the network interface they bind to. Services binding to the localhost interface are only available from within the host and we take no action to limit access to them. Services binding to the default host interface are available from outside the host and thus we enforce some form of access restrictions.
### Services Binding to the Default Host Interface
Port | Service | Access Restrictions
--- | --- | ---
16443 | API server | SSL encrypted. Clients need to present a valid password from a [Static Password File](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#authentication-strategies).
10250 | kubelet | Anonymous authentication is disabled. [X509 client certificate](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/) is required.
10255 | kubelet | Read only port for the Kubelet.
random | kube-proxy | One random port per hosted service is opened as we use `--proxy-mode=userspace` for compatibility reasons.
If you remove `--proxy-mode` from `/var/snap/microk8s/current/args/kube-proxy` and `sudo systemctl restart snap.microk8s.daemon-proxy` kube-proxy will stop exposing the cluster hosted services.
### Services Binding to the localhost Interface
Port | Service | Description
--- | --- | ---
8080 | API server | Port for insecure communication to the API server
10248 | kubelet | Localhost healthz endpoint.
10249 | kube-proxy | Port for the metrics server to serve on.
10251 | kube-schedule | Port on which to serve HTTP insecurely.
10252 | kube-controller | Port on which to serve HTTP insecurely.
10256 | kube-proxy | Port to bind the health check server.
Note that this is not an exhaustive list of ports used.
### Containerd and etcd
Both these services are exposed through unix sockets.
Service | Socket
--- | ---
containerd | unix:///var/snap/microk8s/common/run/containerd.sock
etcd | unix://etcd.socket:2379
## Authentication and Authorization
Upon deployment MicroK8s creates a CA, a signed server certificate and a service account key file. These files are stored under `/var/microk8s/current/certs`. Kubelet and the API server are aware of the same CA and so the signed server certificate is used by the API server to authenticate with kubelet (`--kubelet-client-certificate`). Clients talking to the secure port of the API server (`16443`) have to also be aware of the CA (`certificate-authority-data` in user kubeconfig).
Authentication of users is done with a [Static Password File](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#authentication-strategies) also generated at first MicroK8s deployment. Password tokens and usernames are stored in the `basic_token.csv` file available under `/var/snap/microk8s/current/credentials/`. Also under `/var/snap/microk8s/current/credentials/` you can find the `client.config` kubeconfig file used by `microk8s.kubectl`.
Currently all requests coming from authenticated sources are authorized as we have configured the api-server with `--authorization-mode=AlwaysAllow`.
## References
- Authentication strategies: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#authentication-strategies
- kubelet: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
- kube-proxy: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/
- kube-scheduler: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/
- kube-controller-manager: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/

55
docs/release-channels.md
View File

@@ -1,55 +0,0 @@
# Release Channels and Upgrades
Microk8s is a snap deploying Kubernetes. Upstream Kubernetes ships a new release about every three months, while old releases get periodic updates. At the time of this writing the latest release series is `v1.12` with `v1.12.0` being the latest release. On the `v1.11` series, `v1.11.3` is the latest release. It is important to remember that upstream Kubernetes is committed to maintain backwards compatibility only within a release series. That means that your Kubernetes will not break due to API changes when you upgrade from `v1.11.x` to `v1.11.y` but may break if you upgrade from `v1.11.x` to `v1.12.z`.
## Choosing the Right Channel
When installing MicroK8s you can select your desired upstream Kubernetes series with the corresponding snap channel. For example, to install MicroK8s and let it follow the `v1.12` release series you:
```
snap install microk8s --classic --channel=1.12/stable
```
If you omit the `--channel` argument MicroK8s will follow the latest stable upstream Kubernetes. This means that your deployment will eventually upgrade to a new release series. At the time of this writing you will get `v1.12.0` with:
```
snap install microk8s --classic
```
Since no `--channel` is specified such deployment will eventually upgrade to `v1.13.0`.
Switching from one channel to another is done with [`snap refresh --channel=<new_channel>`](https://docs.snapcraft.io/channels/551). For example, switch MicroK8s to the v1.11 release series with:
```
snap refresh --channel=1.11/stable microk8s
```
## Availability of Releases and Channels
The `*/stable` channels serve the latest stable upstream Kubernetes release of the respective release series. Upstream releases are propagated to the MicroK8s snap in about a week. This means your MicroK8s will upgrade to the latest upstream release in your selected channel roughly one week after the upstream release.
The `*/candidate` and `*/beta` channels get updated within hours of an upstream release. Getting a MicroK8s deployment pointing to `1.12/beta` is as simple as:
```
snap install microk8s --classic --channel=1.12/beta
```
The `*/edge` channels get updated for each MicroK8s patch or upstream Kubernetes release.
Keep in mind that edge and beta are snap constructs and do not relate to Kubernetes release names.
## Summary
To always track the latest stable Kubernetes versions:
```
snap install microk8s --classic
```
To track the latest version in a specific release series:
```
snap install microk8s --classic --channel=<track>/stable
```

483
docs/working.md
View File

@@ -1,483 +0,0 @@
# Working with image registries
Kubernetes manages containerised applications based on images. These images can
be created locally, or more commonly are fetched from a remote image registry.
The following documentation explains how to use MicroK8s with local images, or
images fetched from public or private registries.
A familiarity with building, pushing and tagging container images will be
helpful. These examples use Docker but you can use your preferred
container tool chain.
To install Docker on Ubuntu 18.04:
```
sudo apt-get install docker.io
```
Add the user to the `docker` group:
```bash
sudo usermod -aG docker ${USER}
```
Open a new shell for the user, with updated group membership:
```bash
su - ${USER}
```
The Dockerfile we will be using is:
```
FROM nginx:alpine
```
To build the image tagged with `mynginx:local`, navigate to the directory where
`Dockerfile` is and run:
```bash
docker build . -t mynginx:local
```
This will generate a new local image tagged `mynginx:local`.
## Working with locally built images without a registry
When an image is built it is cached on the Docker daemon used during the build.
Having run the `docker build . -t mynginx:local` command, you can see the newly built image by
running:
```bash
docker images
```
This will list the images currently known to Docker, for example:
```no-highlight
REPOSITORY TAG IMAGE ID CREATED SIZE
mynginx local 1fe3d8f47868 30 minutes ago 16.1MB
```
The image we created is known to Docker. However, Kubernetes is not aware of
the newly built image. This is because your local Docker daemon is not part of
the MicroK8s Kubernetes cluster. We can export the built image from the local
Docker daemon and "inject" it into the MicroK8s image cache like this:
```bash
docker save mynginx > myimage.tar
microk8s.ctr -n k8s.io image import myimage.tar
```
Note that when we import the image to MicroK8s we do so under the `k8s.io` namespace
(the `-n k8s.io` argument).
Now we can list the images present in MicroK8s:
```bash
microk8s.ctr -n k8s.io images ls
```
At this point we are ready to `microk8s.kubectl apply -f` a deployment with this image:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: mynginx:local
ports:
- containerPort: 80
```
We reference the image with `image: mynginx:local`. Kubernetes will behave as though
there is an image in docker.io (the Dockerhub registry) for which it already has a cached
copy. This process can be repeated any time changes are made to the image. Note that
containerd will not cache images with the `latest` tag so make sure you avoid it.
## Working with public registries
After building an image with `docker build . -t mynginx:local`, it can be pushed to one of
the mainstream public registries. You will need to create an account and register a
username. For this example we created an account with [https://hub.docker.com/]() and
we log in as `kjackal`.
First we run the login command:
```bash
docker login
```
Docker will ask for a Docker ID and password to complete the login.
``` no-highlight
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: kjackal
Password: *******
```
Pushing to the registry requires that the image is tagged with
`your-hub-username/image-name:tag`. We can either add proper tagging during build:
```bash
docker build . -t kjackal/mynginx:public
```
Or tag an already existing image using the image ID. Obtain the ID by running:
```bash
docker images
```
The ID is listed in the output:
```no-highlight
REPOSITORY TAG IMAGE ID CREATED SIZE
mynginx local 1fe3d8f47868 2 hours ago 16.1MB
....
```
Then use the `tag` command:
```bash
docker tag 1fe3d8f47868 kjackal/mynginx:public
```
Now that the image is tagged correctly, it can be pushed to the registry:
```bash
docker push kjackal/mynginx
```
At this point we are ready to `microk8s.kubectl apply -f` a deployment with our image:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: kjackal/mynginx:public
ports:
- containerPort: 80
```
We refer to the image as `image: kjackal/mynginx:public`. Kubernetes will search for the
image in its default registry, `docker.io`.
## Working with MicroK8s' registry add-on
Having a private Docker registry can significantly improve your productivity by reducing
the time spent in uploading and downloading Docker images. The registry shipped with
MicroK8s is hosted within the Kubernetes cluster and is exposed as a NodePort service
on port `32000` of the `localhost`. Note that this is an insecure registry and you may
need to take extra steps to limit access to it.
You can install the registry with:
```bash
microk8s.enable registry
```
The add-on registry is backed up by a `20Gi` persistent volume is claimed for storing
images. To satisfy this claim the storage add-on is also enabled along with the registry.
The containerd daemon used by MicroK8s is configured to trust this insecure registry. To
upload images we have to tag them with `localhost:32000/your-image` before pushing
them:
We can either add proper tagging during build:
```bash
docker build . -t localhost:32000/mynginx:registry
```
Or tag an already existing image using the image ID. Obtain the ID by running:
```bash
docker images
```
The ID is listed in the output:
```no-highlight
REPOSITORY TAG IMAGE ID CREATED SIZE
mynginx local 1fe3d8f47868 2 hours ago 16.1MB
....
```
Then use the `tag` command:
```bash
docker tag 1fe3d8f47868 localhost:32000/mynginx:registry
```
Now that the image is tagged correctly, it can be pushed to the registry:
```bash
docker push localhost:32000/mynginx
```
Pushing to this insecure registry may fail in some versions of Docker unless the daemon
is explicitly configured to trust this registry. To address this we need to edit
`/etc/docker/daemon.json` and add:
```json
{
"insecure-registries" : ["localhost:32000"]
}
```
The new configuration should be loaded with a Docker daemon restart:
```bash
sudo systemctl restart docker
```
At this point we are ready to `microk8s.kubectl apply -f` a deployment with our image:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: localhost:32000/mynginx:registry
ports:
- containerPort: 80
```
### What if MicroK8s runs inside a VM?
Often MicroK8s is placed in a VM while the development process takes place on the host
machine. In this setup pushing container images to the in-VM registry requires some
extra configuration.
Let's assume the IP of the VM running MicroK8s is `10.141.241.175`. When we are on the
host the Docker registry is not on `localhost:32000` but on `10.141.241.175:32000`. As a
result the first thing we need to do is to tag the image we are building on the host with
the right registry endpoint:
```bash
docker build . -t 10.141.241.175:32000/mynginx:registry
```
If we immediately try to push the `mynginx` image we will fail because the local Docker
does not trust the in-VM registry. Here is what happens if we try a push:
```bash
docker push 10.141.241.175:32000/mynginx
```
```no-highlight
The push refers to repository [10.141.241.175:32000/mynginx]
Get https://10.141.241.175:32000/v2/: http: server gave HTTP response to HTTPS client
```
We need to be explicit and configure the Docker daemon running on the host to trust the
in-VM insecure registry. Add the registry endpoint in `/etc/docker/daemon.json`:
```json
{
"insecure-registries" : ["10.141.241.175:32000"]
}
```
Then restart the docker daemon on the host to load the new configuration:
```bash
sudo systemctl restart docker
```
We can now `docker push 10.141.241.175:32000/mynginx` and see the image getting
uploaded. During the push our Docker client instructs the in-host Docker daemon to
upload the newly built image to the `10.141.241.175:32000` endpoint as marked by the
tag on the image. The Docker daemon sees (on `/etc/docker/daemon.json`) that it trusts
the registry and proceeds with uploading the image.
Consuming the image from inside the VM involves no changes:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: localhost:32000/mynginx:registry
ports:
- containerPort: 80
```
Reference the image with `localhost:32000/mynginx:registry` since the registry runs
inside the VM so it is on `localhost:32000`.
## Working with a private registry
Often organisations have their own private registry to assist collaboration and accelerate
development. Kubernetes (and thus MicroK8s) need to be aware of the registry
endpoints before being able to pull container images.
### Insecure registry
Let's assume the private insecure registry is at `10.141.241.175` on port `32000`. The
images we build need to be tagged with the registry endpoint:
```bash
docker build . -t 10.141.241.175:32000/mynginx:registry
```
Pushing the `mynginx` image at this point will fail because the local Docker does not
rust the private insecure registry. The docker daemon used for building images should be
configured to trust the private insecure registry. This is done by marking the registry
endpoint in `/etc/docker/daemon.json`:
```json
{
"insecure-registries" : ["10.141.241.175:32000"]
}
```
Restart the Docker daemon on the host to load the new configuration:
```
sudo systemctl restart docker
```
Now running
```bash
docker push 10.141.241.175:32000/mynginx
```
...should succeed in uploading the image to the registry.
Attempting to pull an image in MicroK8s at this point will result in an error like this:
```no-highlight
Warning Failed 1s (x2 over 16s) kubelet, jackal-vgn-fz11m Failed to pull image "10.141.241.175:32000/mynginx:registry": rpc error: code = Unknown desc = failed to resolve image "10.141.241.175:32000/mynginx:registry": no available registry endpoint: failed to do request: Head https://10.141.241.175:32000/v2/mynginx/manifests/registry: http: server gave HTTP response to HTTPS client
```
We need to edit `/var/snap/microk8s/current/args/containerd-template.toml` and add
the following under `[plugins] -> [plugins.cri.registry] -> [plugins.cri.registry.mirrors]`:
```
[plugins.cri.registry.mirrors."10.141.241.175:32000"]
endpoint = ["http://10.141.241.175:32000"]
```
See the full file [here](/docs/containerd-template.toml).
Restart MicroK8s to have the new configuration loaded:
```bash
microk8s.stop
```
Allow a few seconds for the service to close fully before starting again:
```bash
microk8s.start
```
The image can now be deployed with:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: 10.141.241.175:32000/mynginx:registry
ports:
- containerPort: 80
```
Note that the image is referenced with `10.141.241.175:32000/mynginx:registry`.
### Secure registry
There are a lot of ways to setup a private secure registry that may slightly change the
way you interact with it. Instead of diving into the specifics of each setup we provide
here two pointers on how you can approach the integration with Kubernetes.
- In the [official Kubernetes documentation][kubernetes-docs] a method is described for
creating a secret from the Docker login credentials and using this to access the secure
registry. To achieve this, `imagePullSecrets` is used as part of the container spec.
- MicroK8s v1.14 and onwards uses **containerd**. [As described here](https://github.com/containerd/cri/blob/master/docs/registry.md)
to be aware of the secure registry and the credentials needed to access it.
As shown above, configuring containerd involves editing
`/var/snap/microk8s/current/args/containerd-template.toml` and reloading the
new configuration via a `microk8s.stop`, `microk8s.start` cycle.
# Further Reading
- [Test an insecure registry](https://docs.docker.com/registry/insecure/)
- [Configuring containerd](https://github.com/containerd/cri/blob/master/docs/registry.md)
- [Upstream Kubernetes documentation on pulling images from a private registry][kubernetes-docs]
[containerd-template]: /containerd-template.toml
[kubernetes-docs]: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/