alihan/microk8s
1
0
Fork 0
mirror of https://github.com/ubuntu/microk8s.git synced 2021-05-23 02:23:41 +03:00
Code Issues Projects Releases Wiki Activity

Switch to containerd

Browse Source
This commit is contained in:
Konstantinos Tsakalozos
2018-09-12 17:17:24 +03:00
parent 6d6e9093c2
commit 7c3311a6d2
38 changed files with 372 additions and 257 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
README.md
View File

@@ -86,9 +86,9 @@ With `microk8s.status` you can see the list of available addons and which ones a
- **dashboard**: Deploy kubernetes dashboard as well as grafana and influxdb. To access grafana point your browser to the url reported by `microk8s.kubectl cluster-info`.
- **storage**: Create a default storage class. This storage class makes use of the hostpath-provisioner pointing to a directory on the host. Persistent volumes are created under `${SNAP_COMMON}/default-storage`. Upon disabling this addon you will be asked if you want to delete the persistent volumes created.
- **ingress**: Create an ingress controller.
- **gpu**: Expose GPU(s) to MicroK8s by enabling the nvidia-docker runtime and nvidia-device-plugin-daemonset. Requires NVIDIA drivers to already be installed on the host system.
- **gpu**: Expose GPU(s) to MicroK8s by enabling the nvidia runtime and nvidia-device-plugin-daemonset. Requires NVIDIA drivers to already be installed on the host system.
- **istio**: Deploy the core [Istio](https://istio.io/) services. You can use the `microk8s.istioctl` command to manage your deployments.
- **registry**: Deploy a docker private registry and expose it on `localhost:32000`. The storage addon will be enabled as part of this addon. To [use the registry](docs/registry.md) you can use the `microk8s.docker` command.
- **registry**: Deploy an image private registry and expose it on `localhost:32000`. The storage addon will be enabled as part of this addon. See [the registry documentation](docs/registry.md) for more details.
- **metrics-server**: Deploy the [Metrics Server](https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/#metrics-server).
- **prometheus**: Deploy the [Prometheus Operator](https://github.com/coreos/prometheus-operator) v0.25.
- **fluentd**: Deploy [Elasticsearch-Kibana-Fluentd](https://kubernetes.io/docs/tasks/debug-application-cluster/logging-elasticsearch-kibana/) logging and monitoring solution.
@@ -126,22 +126,22 @@ The following systemd services will be running in your system:
- **snap.microk8s.daemon-scheduler**, is the [kube-scheduler](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/) daemon started using the arguments in `${SNAP_DATA}/args/kube-scheduler`
- **snap.microk8s.daemon-kubelet**, is the [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) daemon started using the arguments in `${SNAP_DATA}/args/kubelet`
- **snap.microk8s.daemon-proxy**, is the [kube-proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) daemon started using the arguments in `${SNAP_DATA}/args/kube-proxy`
- **snap.microk8s.daemon-docker**, is the [docker](https://docs.docker.com/engine/reference/commandline/dockerd/) daemon started using the arguments in `${SNAP_DATA}/args/dockerd`
- **snap.microk8s.daemon-containerd**, is the [containerd](https://containerd.io/) daemon started using the configuration in `${SNAP_DATA}/args/containerd` and `${SNAP_DATA}/args/containerd-template.toml`.
- **snap.microk8s.daemon-etcd**, is the [etcd](https://coreos.com/etcd/docs/latest/v2/configuration.html) daemon started using the arguments in `${SNAP_DATA}/args/etcd`
Normally, `${SNAP_DATA}` points to `/var/snap/microk8s/current`.
To reconfigure a service you will need to edit the corresponding file and then restart the respective daemon. For example:
```
echo '--config-file=/path-to-my/daemon.json' | sudo tee -a /var/snap/microk8s/current/args/dockerd
sudo systemctl restart snap.microk8s.daemon-docker.service
echo '-l=debug' | sudo tee -a /var/snap/microk8s/current/args/containerd
sudo systemctl restart snap.microk8s.daemon-containerd.service
```
### Deploy Behind a Proxy
To let MicroK8s use a proxy enter the proxy details in `${SNAP_DATA}/args/dockerd-env` and restart the docker daemon service with:
To let MicroK8s use a proxy enter the proxy details in `${SNAP_DATA}/args/containerd-env` and restart the containerd daemon service with:
```
sudo systemctl restart snap.microk8s.daemon-docker.service
sudo systemctl restart snap.microk8s.daemon-containerd.service
```
@@ -166,7 +166,7 @@ or, if using `ufw`:
`sudo ufw default allow routed`
The microk8s inspect command can be used to check the firewall configuration:
The MicroK8s inspect command can be used to check the firewall configuration:
`microk8s.inspect`
@@ -180,7 +180,7 @@ microk8s.start
```
### My log collector is not collecting any logs.
By default docker container logs are located in `/var/lib/docker/containers/{id}/{id}-json.log` but microk8s is packaged with snap and it uses it's own docker. So the logs are located in `/var/snap/microk8s/common/var/lib/docker/containers/{id}/{id}-json.log`. You have to mount this location in your log collector for that to work. Following is an example diff for [fluent-bit](https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds.yaml):
By default container logs are located in `/var/log/pods/{id}`. You have to mount this location in your log collector for that to work. Following is an example diff for [fluent-bit](https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds.yaml):
```diff
@@ -36,6 +36,9 @@
@@ -188,7 +188,7 @@ By default docker container logs are located in `/var/lib/docker/containers/{id}
mountPath: /var/lib/docker/containers
readOnly: true
+ - name: varlibdockercontainers
+ mountPath: /var/snap/microk8s/common/var/lib/docker/containers/
+ mountPath: /var/snap/microk8s/common/var/lib/containerd/
+ readOnly: true
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
@@ -198,7 +198,7 @@ By default docker container logs are located in `/var/lib/docker/containers/{id}
- name: varlibdockercontainers
hostPath:
- path: /var/lib/docker/containers
+ path: /var/snap/microk8s/common/var/lib/docker/containers/
+ path: /var/log/pods/
- name: fluent-bit-config
configMap:
name: fluent-bit-config
@@ -208,7 +208,7 @@ By default docker container logs are located in `/var/lib/docker/containers/{id}
Build the snap with:
```
snapcraft
snapcraft cleanbuild
```
### Building for specific versions
@@ -219,6 +219,8 @@ You can set the following environment variables prior to building:
- CNI_VERSION: version of CNI tools. Defaults to v0.7.1.
- KUBE_TRACK: kubernetes release series (e.g., 1.10) to package. Defaults to latest stable.
- ISTIO_VERSION: istio release. Defaults to v1.0.0.
- RUNC_COMMIT: the commit hash from which to build runc
- CONTAINERD_COMMIT: the commit hash from which to build containerd
For example:
```

8
build-scripts/prepare-env.sh
View File

@@ -14,6 +14,12 @@ export KUBE_ARCH
export ETCD_VERSION="${ETCD_VERSION:-v3.3.4}"
export CNI_VERSION="${CNI_VERSION:-v0.7.1}"
export ISTIO_VERSION="${ISTIO_VERSION:-v1.0.5}"
# RUNC commit matching the containerd release commit
# Tag 1.2.2
export CONTAINERD_COMMIT="${CONTAINERD_COMMIT:-9754871865f7fe2f4e74d43e2fc7ccd237edcbce}"
# Release v1.0.0~rc6
export RUNC_COMMIT="${RUNC_COMMIT:-ccb5efd37fb7c86364786e9137e22948751de7ed}"
export KUBE_TRACK="${KUBE_TRACK:-}"
export KUBE_SNAP_BINS="${KUBE_SNAP_BINS:-}"
@@ -36,3 +42,5 @@ echo "CNI_VERSION=${CNI_VERSION}"
echo "KUBE_ARCH=${KUBE_ARCH}"
echo "KUBE_SNAP_BINS=${KUBE_SNAP_BINS}"
echo "ISTIO_VERSION=${ISTIO_VERSION}"
echo "RUNC_COMMIT=${RUNC_COMMIT}"
echo "CONTAINERD_COMMIT=${CONTAINERD_COMMIT}"

64
docs/build_from_source.md
View File

@@ -1,64 +0,0 @@
# Building from source
Building a new version of MicroK8s from the source is straightforward.
1. If you don't already have it, install the latest version of the `snapcraft` tool:
```bash
sudo snap install snapcraft --classic
```
1. Clone the MicroK8s repository:
```bash
git clone https://github.com/ubuntu/microk8s.git
cd microk8s/
```
1. Ensure local package lists are up to date. On Ubuntu, run:
```bash
sudo apt update
```
1. Run snapcraft to build the snap package:
```bash
snapcraft
```
1. Once the snap is built it can be installed with:
```bash
snap install microk8s_v1.12.2_amd64.snap --classic --dangerous
```
(substitute the name of the version that was actually built as required).
For more information on managing snaps, see the [Snapcraft documentation](https://docs.snapcraft.io/getting-started/3876)
## Using different build options
You can set the following environment variables prior to building:
| Variable | Description | Default value |
|--------------------------|-----------------------------------------------------------|-------------------------------------|
| KUBE_VERSION | Kubernetes release to package | Latest stable version |
| ETCD_VERSION | Version of etcd | 3.3.4 |
| CNI_VERSION | Version of CNI tools | 0.7.1 |
| KUBE_TRACK | Kubernetes series (e.g., 1.10) to package | Latest stable |
| ISTIO_VERSION | istio release | v1.0.0 |
For example:
```bash
KUBE_VERSION=v1.9.6 snapcraft
```
## Faster builds
To speed-up the build process you can reuse the binaries already downloaded from a
previous build. Binaries are placed under `parts/microk8s/build/build/kube_bins`.
All you need to do is to make a copy of this directory and have the `KUBE_SNAP_BINS`
environment variable point to it.
After a standard build, run the following:
```bash
cp -r parts/microk8s/build/build/kube_bins .
export KUBE_SNAP_BINS=$PWD/kube_bins/v1.12.2/
snapcraft
```
...this build will be much faster and will reuse binaries in `KUBE_SNAP_BINS`

20
docs/dockerd.md
View File

@@ -1,20 +0,0 @@
# Dockerd in MicroK8s
The docker daemon used by MicroK8s is listening on `unix:///var/snap/microk8s/current/docker.sock`. You can access it with the `microk8s.docker` command. To skip the `microk8s` prefix we suggest you employ a snap alias:
```
sudo snap alias microk8s.docker docker
docker ps
```
Export `DOCKER_HOST` for other tools using docker daemon:
```
export DOCKER_HOST="unix:///var/snap/microk8s/current/docker.sock"
```
When AppArmor is enabled all docker daemons running in a system will apply the same `docker-default` profile on running containers. Each daemon makes sure that it is the only process managing the docker containers (e.g., sending start stop signals). Effectively this allowes only one dockerd running on any host. Therefore, you have to make sure no other dockerd is running on your sytem along with MicroK8s.
Restarting MicroK8s' dockerd (`sudo systemctl restart snap.microk8s.daemon-docker`) or calling the `microk8s.reset` command will ensure the correct AppArmor profile is loaded.
## References
- Issue describing the AppArmor profile limitation: https://forum.snapcraft.io/t/commands-and-aliases/3950

16
docs/registry.md
View File

@@ -12,17 +12,12 @@ microk8s.enable registry
As you can see in the applied [manifest](../microk8s-resources/actions/registry.yaml) a `20Gi` persistent volume is claimed for storing images. To satisfy this claim the storage addon is also enabled along with the registry.
The docker daemon used by MicroK8s is [configured to trust](../microk8s-resources/default-args/docker-daemon.json) this insecure registry. It is on this daemon we will have to talk to when we want to upload images. The easiest way to do so is by using the `microk8s.docker` client:
The containerd daemon used by MicroK8s is [configured to trust](../microk8s-resources/default-args/containerd-template.toml) this insecure registry. The easiest way to upload images to the registry is using the docker client:
```
microk8s.docker pull busybox
microk8s.docker tag busybox localhost:32000/my-busybox
microk8s.docker push localhost:32000/my-busybox
```
If you prefer to use an external docker client you should point it to the socket dockerd is listening on:
```
docker -H unix:///var/snap/microk8s/current/docker.sock ps
docker pull busybox
docker tag busybox localhost:32000/my-busybox
docker push localhost:32000/my-busybox
```
To consume an image from the local registry we need to reference it in our yaml manifests:
@@ -45,5 +40,4 @@ spec:
## References
- Insecure registry: https://docs.docker.com/registry/insecure/
- Test a registry: https://docs.docker.com/registry/deploying/#copy-an-image-from-docker-hub-to-your-registry
- Containerd registry: https://github.com/containerd/cri/blob/master/docs/registry.md

18
microk8s-resources/actions/common/utils.sh
View File

@@ -79,3 +79,21 @@ use_manifest() {
use_manifest_result="$?"
rm "${tmp_manifest}"
}
wait_for_service() {
# Wait for a service to start
# Return fail if the service did not start in 30 seconds
local service_name="$1"
local TRY_ATTEMPT=0
while ! (sudo systemctl is-active --quiet snap.${SNAP_NAME}.daemon-${service_name}) &&
! [ ${TRY_ATTEMPT} -eq 30 ]
do
TRY_ATTEMPT=$((TRY_ATTEMPT+1))
sleep 1
done
if [ ${TRY_ATTEMPT} -eq 30 ]
then
echo "fail"
fi
}

30
microk8s-resources/actions/disable.dns.sh
View File

@@ -6,15 +6,31 @@ source $SNAP/actions/common/utils.sh
echo "Disabling DNS"
echo "Reconfiguring kubelet"
KUBECTL="$SNAP/kubectl --kubeconfig=$SNAP/client.config"
# Delete the dns yaml
# We need to wait for the dns pods to terminate before we restart kubelet
echo "Removing DNS manifest"
use_manifest dns delete
sleep 15
timeout=30
start_timer="$(date +%s)"
while ($KUBECTL get po -n kube-system | grep -z " Terminating") &> /dev/null
do
now="$(date +%s)"
if [[ "$now" > "$(($start_timer + $timeout))" ]] ; then
break
fi
sleep 5
done
skip_opt_in_config "cluster-domain" kubelet
skip_opt_in_config "cluster-dns" kubelet
sudo systemctl restart snap.${SNAP_NAME}.daemon-kubelet
sleep 5
# Apply the dns yaml
# We do not need to see dns pods running at this point just give some slack
echo "Removing DNS manifest"
use_manifest dns delete
kubelet=$(wait_for_service kubelet)
if [[ $kubelet == fail ]]
then
echo "Kubelet did not start on time. Proceeding."
fi
sleep 15
echo "DNS is disabled"

15
microk8s-resources/actions/disable.gpu.sh
View File

@@ -5,7 +5,18 @@ set -e
source $SNAP/actions/common/utils.sh
echo "Disabling NVIDIA GPU support"
skip_opt_in_config "default-runtime" dockerd
sudo systemctl restart snap.${SNAP_NAME}.daemon-kubelet
use_manifest gpu delete
sudo sh -c "sed 's@\${SNAP}@'"${SNAP}"'@g;s@\${SNAP_DATA}@'"${SNAP_DATA}"'@g;s@\${RUNTIME}@runc@g' $SNAP_DATA/args/containerd-template.toml > $SNAP_DATA/args/containerd.toml"
sudo systemctl restart snap.${SNAP_NAME}.daemon-containerd
containerd_up=$(wait_for_service containerd)
if [[ $containerd_up == fail ]]
then
echo "Containerd did not start on time. Proceeding."
fi
# Allow for some seconds for containerd processes to start
sleep 10
echo "GPU support disabled"

1
microk8s-resources/actions/enable.dns.sh
View File

@@ -17,4 +17,5 @@ refresh_opt_in_config "cluster-domain" "cluster.local" kubelet
refresh_opt_in_config "cluster-dns" "10.152.183.10" kubelet
sudo systemctl restart snap.${SNAP_NAME}.daemon-kubelet
echo "DNS is enabled"

20
microk8s-resources/actions/enable.gpu.sh
View File

@@ -13,20 +13,16 @@ else
exit 1
fi
refresh_opt_in_config "default-runtime" "nvidia" dockerd
sudo systemctl restart snap.${SNAP_NAME}.daemon-docker
TRY_ATTEMPT=0
while (! (sudo systemctl is-active --quiet snap.${SNAP_NAME}.daemon-docker) ||
! (sudo "$SNAP/usr/bin/docker" "-H" "unix://${SNAP_DATA}/docker.sock" ps &> /dev/null)) &&
! [ ${TRY_ATTEMPT} -eq 30 ]
do
TRY_ATTEMPT=$((TRY_ATTEMPT+1))
sleep 1
done
if [ ${TRY_ATTEMPT} -eq 30 ]
sudo sh -c "sed 's@\${SNAP}@'"${SNAP}"'@g;s@\${SNAP_DATA}@'"${SNAP_DATA}"'@g;s@\${RUNTIME}@nvidia-container-runtime@g' $SNAP_DATA/args/containerd-template.toml > $SNAP_DATA/args/containerd.toml"
sudo systemctl restart snap.${SNAP_NAME}.daemon-containerd
containerd_up=$(wait_for_service containerd)
if [[ $containerd_up == fail ]]
then
echo "Snapped docker not responding after 30 seconds. Proceeding"
echo "Containerd did not start on time. Proceeding."
fi
# Allow for some seconds for containerd processes to start
sleep 10
"$SNAP/microk8s-enable.wrapper" dns

4
microk8s-resources/actions/fluentd/fluentd-es-ds.yaml
View File

@@ -91,7 +91,7 @@ spec:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/snap/microk8s/common/var/lib/docker/containers
mountPath: /var/lib/docker/containers
readOnly: true
- name: config-volume
mountPath: /etc/fluent/config.d
@@ -104,7 +104,7 @@ spec:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/snap/microk8s/common/var/lib/docker/containers
path: /var/snap/microk8s/common/var/lib/containerd
- name: config-volume
configMap:
name: fluentd-es-config-v0.1.5

7
microk8s-resources/docker-profile → microk8s-resources/containerd-profile
View File

@@ -3,7 +3,7 @@
#include <tunables/global>
profile docker-default flags=(attach_disconnected,mediate_deleted) {
profile cri-containerd.apparmor.d flags=(attach_disconnected,mediate_deleted) {
#include <abstractions/base>
@@ -35,7 +35,8 @@ profile docker-default flags=(attach_disconnected,mediate_deleted) {
# suppress ptrace denials when using 'docker ps' or using 'ps' inside a container
ptrace (trace,read) peer=docker-default,
ptrace (trace,read) peer=cri-containerd.apparmod.d,
signal (receive) peer=snap.microk8s.daemon-docker,
signal (receive) peer=snap.microk8s.daemon-kubelet,
signal (receive) peer=snap.microk8s.daemon-containerd,
}

19
microk8s-resources/default-args/cni-network/cni.conf Normal file
View File

@@ -0,0 +1,19 @@
{
"name": "k8s-pod-network",
"cniVersion": "0.3.0",
"type": "bridge",
"bridge": "cbr0",
"isGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.1.1.0/24",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"dataDir": "/run/ipam-state"
},
"dns": {
"nameservers": [ "8.8.8.8", "8.8.4.4" ]
}
}

4
microk8s-resources/default-args/containerd Normal file
View File

@@ -0,0 +1,4 @@
--config ${SNAP_DATA}/args/containerd.toml
--root ${SNAP_COMMON}/var/lib/containerd
--state ${SNAP_COMMON}/run/containerd
--address ${SNAP_COMMON}/run/containerd.sock

11
microk8s-resources/default-args/containerd-env Normal file
View File

@@ -0,0 +1,11 @@
# To start containerd behind a proxy you need to add an HTTPS_PROXY
# environment variable in this file. HTTPS_PROXY is of the following form:
# HTTPS_PROXY=http://username:password@proxy:port/
# where username: and password@ are optional. eg:
#
# HTTPS_PROXY=https://squid.internal:3128
#
#
# Remember to restart the containerd daemon after editing this file:
#
# sudo systemctl restart snap.microk8s.daemon-containerd.service

67
microk8s-resources/default-args/containerd-template.toml Normal file
View File

@@ -0,0 +1,67 @@
oom_score = 0
[grpc]
uid = 0
gid = 0
max_recv_message_size = 16777216
max_send_message_size = 16777216
[debug]
address = ""
uid = 0
gid = 0
[metrics]
address = "127.0.0.1:1338"
grpc_histogram = false
[cgroup]
path = ""
[plugins]
[plugins.cgroups]
no_prometheus = false
[plugins.cri]
stream_server_address = ""
stream_server_port = "10010"
enable_selinux = false
sandbox_image = "k8s.gcr.io/pause:3.1"
stats_collect_period = 10
systemd_cgroup = false
enable_tls_streaming = false
max_container_log_line_size = 16384
[plugins.cri.containerd]
snapshotter = "overlayfs"
no_pivot = false
[plugins.cri.containerd.default_runtime]
runtime_type = "io.containerd.runtime.v1.linux"
runtime_engine = ""
runtime_root = ""
[plugins.cri.containerd.untrusted_workload_runtime]
runtime_type = ""
runtime_engine = ""
runtime_root = ""
[plugins.cri.cni]
bin_dir = "${SNAP}/opt/cni/bin"
conf_dir = "${SNAP_DATA}/args/cni-network"
conf_template = ""
[plugins.cri.registry]
[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins.cri.registry.mirrors."local.insecure-registry.io"]
endpoint = ["http://localhost:32000"]
[plugins.diff-service]
default = ["walking"]
[plugins.linux]
shim = "containerd-shim"
runtime = "${RUNTIME}"
runtime_root = ""
no_shim = false
shim_debug = true
[plugins.scheduler]
pause_threshold = 0.02
deletion_threshold = 0
mutation_threshold = 100
schedule_delay = "0s"
startup_delay = "100ms"

1
microk8s-resources/default-args/ctr Normal file
View File

@@ -0,0 +1 @@
--address=${SNAP_COMMON}/run/containerd.sock

3
microk8s-resources/default-args/docker-daemon.json
View File

@@ -1,3 +0,0 @@
{
"insecure-registries" : ["localhost:32000"]
}

6
microk8s-resources/default-args/dockerd
View File

@@ -1,6 +0,0 @@
--add-runtime nvidia=${SNAP}/usr/bin/nvidia-container-runtime
-H unix://${SNAP_DATA}/docker.sock
--exec-root ${SNAP_COMMON}/var/run/docker
--graph ${SNAP_COMMON}/var/lib/docker
--pidfile ${SNAP_COMMON}/docker-pid
--config-file=${SNAP_DATA}/args/docker-daemon.json

11
microk8s-resources/default-args/dockerd-env
View File

@@ -1,11 +0,0 @@
# To start dockerd behind a proxy you need to add an HTTPS_PROXY
# environment variable in this file. HTTPS_PROXY is of the following form:
# HTTPS_PROXY=http://username:password@proxy:port/ dockerd.
# where username: and password@ are optional. eg:
#
# HTTPS_PROXY=https://squid.internal:3128
#
#
# Remember to restart the docker daemon after editing this file:
#
# sudo systemctl restart snap.microk8s.daemon-docker.service

5
microk8s-resources/default-args/kubelet
View File

@@ -4,13 +4,12 @@
--anonymous-auth=false
--network-plugin=kubenet
--root-dir=${SNAP_COMMON}/var/lib/kubelet
--docker-root=${SNAP_COMMON}/var/lib/docker
--fail-swap-on=false
--pod-cidr=10.1.1.0/24
--non-masquerade-cidr=10.152.183.0/24
--cni-bin-dir=${SNAP}/opt/cni/bin/
--docker unix://${SNAP_DATA}/docker.sock
--docker-endpoint unix://${SNAP_DATA}/docker.sock
--feature-gates=DevicePlugins=true
--eviction-hard="memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi"
--container-runtime=remote
--container-runtime-endpoint=${SNAP_COMMON}/run/containerd.sock
--node-labels="microk8s.io/cluster=true"

2
microk8s-resources/wrappers/apiservice-kicker
View File

@@ -3,6 +3,8 @@
set -eu
export PATH="$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$PATH"
export LD_LIBRARY_PATH="$SNAP/lib:$SNAP/usr/lib:$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu"
export LD_LIBRARY_PATH=$SNAP_LIBRARY_PATH:$LD_LIBRARY_PATH
restart_attempt=0

5
microk8s-resources/wrappers/microk8s-docker.wrapper → microk8s-resources/wrappers/microk8s-ctr.wrapper
View File

@@ -1,8 +1,9 @@
#!/bin/bash
#!/usr/bin/env bash
set -eu
export PATH="$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$PATH"
export LD_LIBRARY_PATH="$SNAP/lib:$SNAP/usr/lib:$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu"
sudo -E "${SNAP}/usr/bin/docker" "-H" "unix://${SNAP_DATA}/docker.sock" "$@"
declare -a args="($(cat $SNAP_DATA/args/ctr))"
sudo -E "${SNAP}/bin/ctr" "${args[@]}" "$@"

2
microk8s-resources/wrappers/microk8s-disable.wrapper
View File

@@ -1,4 +1,6 @@
#!/bin/bash
export PATH="$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$PATH"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$SNAP/lib:$SNAP/usr/lib:$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu"
export LD_LIBRARY_PATH=$SNAP_LIBRARY_PATH:$LD_LIBRARY_PATH

2
microk8s-resources/wrappers/microk8s-enable.wrapper
View File

@@ -1,4 +1,6 @@
#!/bin/bash
export PATH="$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$PATH"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$SNAP/lib:$SNAP/usr/lib:$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu"
export LD_LIBRARY_PATH=$SNAP_LIBRARY_PATH:$LD_LIBRARY_PATH

9
microk8s-resources/wrappers/microk8s-reset.wrapper
View File

@@ -33,6 +33,13 @@ clean_cluster() {
}
exit_if_stopped
sudo systemctl restart snap.${SNAP_NAME}.daemon-docker
if systemctl status snap.${SNAP_NAME}.daemon-docker >& /dev/null
then
sudo systemctl restart snap.${SNAP_NAME}.daemon-docker
elif systemctl status snap.${SNAP_NAME}.daemon-containerd >& /dev/null
then
sudo systemctl restart snap.${SNAP_NAME}.daemon-containerd
fi
clean_cluster
sudo systemctl restart snap.${SNAP_NAME}.daemon-apiserver

30
microk8s-resources/wrappers/run-containerd-with-args Executable file
View File

@@ -0,0 +1,30 @@
#!/bin/bash
set -e
export PATH="$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$PATH"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$SNAP/lib:$SNAP/usr/lib:$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu"
export LD_LIBRARY_PATH=$SNAP_LIBRARY_PATH:$LD_LIBRARY_PATH
export XDG_RUNTIME_DIR="${SNAP_COMMON}/run"
mkdir -p "${XDG_RUNTIME_DIR}"
if [ -d "/etc/apparmor.d" ]; then
echo "Using a default profile template"
cp ${SNAP}/containerd-profile /etc/apparmor.d/cri-containerd.apparmor.d
echo "Reloading AppArmor profiles"
if ! service apparmor reload
then
echo "AppArmor profiles loading failed. AppArmor may be unavailable on this host."
fi
fi
app=containerd
sed 's@${SNAP}@'"${SNAP}"'@g;s@${SNAP_DATA}@'"${SNAP_DATA}"'@g;s@${RUNTIME}@runc@g' $SNAP_DATA/args/containerd-template.toml > $SNAP_DATA/args/containerd.toml
# This is really the only way I could find to get the args passed in correctly. WTF
declare -a args="($(cat $SNAP_DATA/args/$app))"
set -a
. "${SNAP_DATA}/args/${app}-env"
set +a
exec "$SNAP/bin/$app" "${args[@]}"

39
microk8s-resources/wrappers/run-docker-with-args
View File

@@ -1,39 +0,0 @@
#!/bin/bash
set -e
export PATH="$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$PATH"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$SNAP/lib:$SNAP/usr/lib:$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu"
export LD_LIBRARY_PATH=$SNAP_LIBRARY_PATH:$LD_LIBRARY_PATH
#TODO(kjackal): Make sure this works on every distro
# Until we get a way to properly patch the docker-default profile
# https://github.com/moby/moby/issues/33060
if [ -d "/etc/apparmor.d" ]; then
# we have AppArmor
if [ -f /etc/apparmor.d/docker ]; then
# docker default profile exists
if ! $(grep -qE "snap.microk8s.daemon-docker" /etc/apparmor.d/docker); then
echo "Patching docker-default profile"
"$SNAP/bin/sed" 's/^}$/\ \ signal\ (receive)\ peer=snap.microk8s.daemon-docker,\n}/' -i /etc/apparmor.d/docker
else
echo "Docker default profile already patched"
fi
else
echo "Using a docker-default template"
cp ${SNAP}/docker-profile /etc/apparmor.d/docker
fi
echo "Reloading AppArmor profiles"
if ! service apparmor reload
then
echo "AppArmor profiles loading failed. AppArmor may be unavailable on this host."
fi
fi
app=dockerd
# This is really the only way I could find to get the args passed in correctly. WTF
declare -a args="($(cat $SNAP_DATA/args/$app))"
set -a
. "${SNAP_DATA}/args/${app}-env"
set +a
exec "$SNAP/usr/bin/$app" "${args[@]}"

17
microk8s-resources/wrappers/run-with-config-args
View File

@@ -8,20 +8,19 @@ app=$1
if [ "${app}" = "kubelet" ]
then
# wait for docker socket
if grep -e "-H " $SNAP_DATA/args/dockerd &> /dev/null
# wait for containerd socket
if grep -e "--address " $SNAP_DATA/args/containerd &> /dev/null
then
socket=$(grep -e "-H " $SNAP_DATA/args/dockerd | awk '{print $2}')
socket_file=${socket#"unix://"}
# socket_file may be of the form ${SNAP_DATA}/docker.sock
socket=$(grep -e "--address " $SNAP_DATA/args/containerd | awk '{print $2}')
# socket_file may be of the form ${SNAP_DATA}/containerd.sock
# we need to replace any variables
socket_file_expand=$(eval echo ${socket_file})
socket_file_expand=$(eval echo ${socket})
# wait up until 20 seconds for the docker socket to appear
n=0
until [ $n -ge 10 ]
do
test -S "${socket_file_expand}" && break
echo "Waiting for docker socket ${socket_file_expand} to appear. (attempt $n)"
echo "Waiting for containerd socket ${socket_file_expand} to appear. (attempt $n)"
n=$[$n+1]
sleep 2
done
@@ -57,10 +56,6 @@ then
# microk8s.stop and then snap start microk8s
rm -f ${SNAP_DATA}/var/lock/stopped.lock &> /dev/null
fi
else
# ensure docker dirs
mkdir -p $SNAP_COMMON/var/run/docker
mkdir -p $SNAP_COMMON/var/lib/docker
fi
# This is really the only way I could find to get the args passed in correctly. WTF

42
snap/hooks/configure vendored
View File

@@ -1,6 +1,8 @@
#!/usr/bin/env bash
set -eux
set -eu
source $SNAP/actions/common/utils.sh
# This is a one-off patch. It will allow us to refresh the beta snap without breaking the user's deployment.
# We make sure the certificates used by the deployment from beta do not change. We copy them to SNAP_DATA
@@ -37,16 +39,40 @@ then
"$SNAP/bin/sed" -i 's@requestheader-client-ca-file=/var/snap/microk8s/.../certs/ca.crt@requestheader-client-ca-file=\${SNAP_DATA}/certs/ca.crt@g' ${SNAP_DATA}/args/kube-apiserver
fi
if [ ! -e ${SNAP_DATA}/args/dockerd-env ]
then
echo "Making sure we have dockerd-env file"
cp ${SNAP}/default-args/dockerd-env ${SNAP_DATA}/args/dockerd-env
systemctl restart snap.${SNAP_NAME}.daemon-docker
fi
# Create the locks directory
mkdir -p ${SNAP_DATA}/var/lock/
# Upgrading to containerd
if [ ! -e ${SNAP_DATA}/args/containerd ]
then
echo "Making sure we have containerd file"
cp ${SNAP}/default-args/containerd ${SNAP_DATA}/args/containerd
cp ${SNAP}/default-args/containerd-template.toml ${SNAP_DATA}/args/containerd-template.toml
cp ${SNAP}/default-args/containerd-env ${SNAP_DATA}/args/containerd-env
cp -r ${SNAP}/default-args/cni-network ${SNAP_DATA}/args/
cp ${SNAP}/default-args/ctr ${SNAP_DATA}/args/ctr
refresh_opt_in_config container-runtime remote kubelet
refresh_opt_in_config container-runtime-endpoint \${SNAP_COMMON}/run/containerd.sock kubelet
skip_opt_in_config docker-root kubelet
skip_opt_in_config docker kubelet
skip_opt_in_config docker-endpoint kubelet
systemctl restart snap.${SNAP_NAME}.daemon-containerd
systemctl restart snap.${SNAP_NAME}.daemon-kubelet
if [ -e ${SNAP_DATA}/args/dockerd ] && grep -e "default-runtime=nvidia" ${SNAP_DATA}/args/dockerd
then
# Deployment used to run docker with nvidia enabled we need to enable nvidia on containerd
# Allow for kubelet and containerd to restart
sleep 10
${SNAP}/microk8s-enable.wrapper gpu
fi
fi
# This will allow us to refresh the snap to the more secure version.
# We need to make sure the client certificate used in microk8s.kubectl is available under $SNAP_DATA

18
snap/hooks/remove
View File

@@ -3,17 +3,17 @@ set -eu
snapctl stop ${SNAP_NAME}.daemon-kubelet 2>&1 || true
snapctl stop ${SNAP_NAME}.daemon-docker 2>&1 || true
snapctl stop ${SNAP_NAME}.daemon-containerd 2>&1 || true
# wait for containerd to stop its processes or we will be getting a umount error
# because the mount points are busy
sleep 10
(cat /proc/mounts | grep ${SNAP_COMMON}/var/lib/kubelet/pods | cut -d ' ' -f 2 | xargs umount) || true
# in case this is a pre root-dir fix deployment
(cat /proc/mounts | grep ${SNAP_COMMON}/pods | cut -d ' ' -f 2 | xargs umount) || true
(cat /proc/mounts | grep ${SNAP_COMMON}/var/lib/containerd | cut -d ' ' -f 2 | xargs umount) || true
(cat /proc/mounts | grep ${SNAP_COMMON}/run/containerd | cut -d ' ' -f 2 | xargs umount) || true
(cat /proc/mounts | grep ${SNAP_COMMON}/var/lib/docker | cut -d ' ' -f 2 | xargs umount) || true
(cat /proc/mounts | grep ${SNAP_COMMON}/var/run/docker | cut -d ' ' -f 2 | xargs umount) || true
#TODO(kjackal): Make sure this works everywhere we want
if [ -f /etc/apparmor.d/docker ]; then
echo "Updating docker-default profile"
"${SNAP}/bin/sed" -i '/ signal (receive) peer=snap.microk8s.daemon-docker,/d' /etc/apparmor.d/docker
echo "Reloading AppArmor profiles"
service apparmor reload
echo "AppArmor patched"
fi

82
snapcraft.yaml
View File

@@ -18,8 +18,8 @@ apps:
daemon-etcd:
command: run-etcd-with-args
daemon: simple
daemon-docker:
command: run-docker-with-args
daemon-containerd:
command: run-containerd-with-args
daemon: simple
daemon-apiserver:
command: run-with-config-args kube-apiserver
@@ -42,9 +42,8 @@ apps:
kubectl:
command: microk8s-kubectl.wrapper
completer: kubectl.bash
docker:
command: microk8s-docker.wrapper
completer: docker.bash
ctr:
command: microk8s-ctr.wrapper
inspect:
command: sudo SNAP_DATA=${SNAP_DATA} ${SNAP}/inspect.sh
enable:
@@ -88,27 +87,59 @@ parts:
- "--disable-shared"
- "--enable-static"
prime: [ -bin/iptables-xml ]
docker:
after: [iptables]
plugin: dump
stage-packages:
- conntrack
- docker.io
- aufs-tools
- gawk
- sed
go:
source-tag: go1.10.4
containerd:
after: [go, iptables]
source: .
plugin: go
go-importpath: github.com/containerd/containerd
build-packages:
- btrfs-tools
- libseccomp-dev
override-build: |
set -eu
. build-scripts/prepare-env.sh
go version
export GOPATH=$(realpath ../go)
mkdir -p $SNAPCRAFT_PART_INSTALL/bin
# Build containerd
go get github.com/containerd/containerd
(
cd $GOPATH/src/github.com/containerd/containerd
git checkout ${CONTAINERD_COMMIT}
# building the btrfs driver can be disabled via the
# build tag no_btrfs, removing this dependency
make
)
cp $GOPATH/src/github.com/containerd/containerd/bin/* $SNAPCRAFT_PART_INSTALL/bin/
rm $SNAPCRAFT_PART_INSTALL/bin/containerd-stress
# Build runc
go get github.com/opencontainers/runc
(
cd $GOPATH/src/github.com/opencontainers/runc
git checkout ${RUNC_COMMIT}
make BUILDTAGS='seccomp apparmor'
)
cp $GOPATH/src/github.com/opencontainers/runc/runc $SNAPCRAFT_PART_INSTALL/bin/
# Assemble the snap
# snapcraftctl build
organize:
containerd/install/bin/*: bin/
stage:
- -sbin/xtables-multi
- -sbin/iptables*
- -lib/xtables
override-build: "true"
microk8s:
after: [docker]
after: [containerd]
plugin: dump
build-attributes: [no-patchelf]
build-packages:
- curl
- openssl
- file
stage-packages:
- net-tools
@@ -117,6 +148,10 @@ parts:
- socat
- iproute2
- dpkg
- conntrack
- aufs-tools
- gawk
- sed
- libssl1.0.0
- coreutils
source: .
@@ -134,15 +169,15 @@ parts:
cp -r $KUBE_SNAP_ROOT/microk8s-resources/default-args .
echo "Building certs"
cp -r $KUBE_SNAP_ROOT/microk8s-resources/certs .
build-scripts/build-certs.sh
cp -r $KUBE_SNAP_ROOT/microk8s-resources/certs-beta .
echo "Preparing cni"
mkdir -p opt/cni/bin/
cp $KUBE_SNAP_BINS/cni/* opt/cni/bin/
echo "Preparing dockerd"
cp $KUBE_SNAP_ROOT/microk8s-resources/docker-profile .
echo "Preparing containerd"
cp $KUBE_SNAP_ROOT/microk8s-resources/containerd-profile .
echo "Preparing etcd"
cp $KUBE_SNAP_BINS/etcd/etcd .
@@ -150,9 +185,7 @@ parts:
echo "Preparing kube-apiserver"
cp $KUBE_SNAP_BINS/$KUBE_ARCH/kube-apiserver .
# Old versions will be pointing to these .csv files from inside their kube-apiserver config
# Keep them around for a couple of releases.
touch known_token.csv
cp $KUBE_SNAP_ROOT/microk8s-resources/known_token.csv .
cp $KUBE_SNAP_ROOT/microk8s-resources/basic_auth.csv .
echo "Preparing kube-controller-manager"
@@ -175,7 +208,6 @@ parts:
echo "Preparing user config"
cp $KUBE_SNAP_ROOT/microk8s-resources/client.config .
cp $KUBE_SNAP_ROOT/microk8s-resources/client.config.template .
echo "Creating commands and wrappers"
cp $KUBE_SNAP_ROOT/microk8s-resources/wrappers/* .
@@ -219,10 +251,6 @@ parts:
echo "Creating inspect hook"
cp $KUBE_SNAP_ROOT/scripts/inspect.sh .
# Add bash completion for microk8s.docker
cp $SNAPCRAFT_STAGE/usr/share/bash-completion/completions/docker docker.bash
sed -i "s/complete -F _docker docker docker.exe dockerd dockerd.exe/complete -F _docker microk8s.docker/g" docker.bash
# Add bash completion for microk8s.kubectl.
./kubectl completion bash | sed "s/complete -o default -F __start_kubectl kubectl/complete -o default -F __start_kubectl microk8s.kubectl/g" | sed "s/complete -o default -o nospace -F __start_kubectl kubectl/complete -o default -o nospace -F __start_kubectl kubectl/g" > kubectl.bash

2
tests/lxc/install-deps/ubuntu:16.04
View File

@@ -3,5 +3,5 @@
export $(grep -v '^#' /etc/environment | xargs)
apt-get update
apt-get install python3-pip -y
apt-get install python3-pip docker.io -y
pip3 install -U pytest requests pyyaml

2
tests/lxc/install-deps/ubuntu:18.04
View File

@@ -3,5 +3,5 @@
export $(grep -v '^#' /etc/environment | xargs)
apt-get update
apt-get install python3-pip -y
apt-get install python3-pip docker.io -y
pip3 install -U pytest requests pyyaml

6
tests/patch-dockerd.sh Executable file
View File

@@ -0,0 +1,6 @@
#!/bin/bash
echo "{
\"insecure-registries\" : [\"localhost:32000\"]
}" > /etc/docker/docker.json
/bin/systemctl restart docker

8
tests/test-addons.py
View File

@@ -14,7 +14,12 @@ from validators import (
validate_fluentd,
validate_jaeger,
)
from utils import microk8s_enable, wait_for_pod_state, microk8s_disable, microk8s_reset
from utils import (
microk8s_enable,
wait_for_pod_state,
microk8s_disable,
microk8s_reset
)
from subprocess import Popen, PIPE, STDOUT, CalledProcessError
@@ -83,6 +88,7 @@ class TestAddons(object):
gpu_enable_outcome = microk8s_enable("gpu")
except CalledProcessError:
# Failed to enable gpu. Skip the test.
print("Could not enable GPU support")
return
validate_gpu()
print("Disable gpu")

1
tests/test-distro.sh
View File

@@ -35,6 +35,7 @@ function create_machine() {
sleep 20
tar cf - ./tests | lxc exec $NAME -- tar xvf - -C /tmp
lxc exec $NAME -- /bin/bash "/tmp/tests/lxc/install-deps/$DISTRO"
lxc exec $NAME -- /tmp/tests/patch-dockerd.sh
}
set -uex

6
tests/utils.py
View File

@@ -1,3 +1,4 @@
import os.path
import datetime
import time
import yaml
@@ -56,7 +57,10 @@ def docker(cmd):
Returns: the docker response in a string
"""
cmd = '/snap/bin/microk8s.docker ' + cmd
docker_bin='/usr/bin/docker'
if os.path.isfile('/snap/bin/microk8s.docker'):
docker_bin='/snap/bin/microk8s.docker'
cmd = docker_bin + ' ' + cmd
return run_until_success(cmd)