Merge pull request #372 from fnproject/docs-k8s

update docs for running Fn on k8s
This commit is contained in:
Chad Arimura
2017-09-29 07:11:46 -07:00
committed by GitHub
12 changed files with 190 additions and 401 deletions

View File

@@ -1,133 +0,0 @@
# Docker Swarm and Oracle Functions
How to run Oracle Functions as a scheduler on top of Docker Standalone Swarm cluster.
## Quick installation
*Prerequisite 1: Make sure you have a working Docker 1.12+ Standalone Swarm cluster in place, you can build one by following the instructions at [Docker's website](https://docs.docker.com/swarm/).*
*Prerequisite 2: It assumes that your running environment is already configured to use Swarm's master scheduler.*
This is a step-by-step procedure to execute Oracle Functions on top of Docker Swarm cluster. It works by having Oracle Functions daemon started through Swarm's master, and there enqueueing tasks through Swarm API.
### Steps
1. Start Oracle Functions in the Swarm Master. It expects all basic Docker environment variables to be present (DOCKER_TLS_VERIFY, DOCKER_HOST, DOCKER_CERT_PATH, DOCKER_MACHINE_NAME). The important part is that the working Swarm master environment must be passed to Functions daemon:
```ShellSession
$ docker login # if you plan to use private images
$ docker volume create --name functions-datafiles
$ docker run -d --name functions \
-p 8080:8080 \
-e DOCKER_TLS_VERIFY \
-e DOCKER_HOST \
-e DOCKER_CERT_PATH="/docker-cert" \
-e DOCKER_MACHINE_NAME \
-v $DOCKER_CERT_PATH:/docker-cert \
-v functions-datafiles:/app/data \
treeder/functions
```
2. Once the daemon is started, check where it is listening for connections:
```ShellSession
# docker info
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5a0846e6a025 treeder/functions "/usr/local/bin/entry" 59 seconds ago Up 58 seconds 2375/tcp, 10.0.0.1:8080->8080/tcp swarm-agent-00/functions
````
Note `10.0.0.1:8080` in `PORTS` column, this is where the service is listening. Oracle Functions will use Docker Swarm scheduler to deliver tasks to all nodes present in the cluster.
3. Test the cluster:
```ShellSession
$ export FUNCTIONS=$(docker port functions | cut -d ' ' -f3)
$ curl -H "Content-Type: application/json" -X POST -d '{ "app": { "name":"myapp" } }' http://$FUNCTIONS/v1/apps
{"message":"App successfully created","app":{"name":"myapp","config":null}}
$ curl -H "Content-Type: application/json" -X POST -d '{ "route": { "type": "sync", "path":"/hello-sync", "image":"fnproject/hello" } }' http://$FUNCTIONS/v1/apps/myapp/routes
{"message":"Route successfully created","route":{"app_name":"myapp","path":"/hello-sync","image":"fnproject/hello","memory":128,"type":"sync","config":null}}
$ curl -H "Content-Type: application/json" -X POST -d '{ "name":"Johnny" }' http://$FUNCTIONS/r/myapp/hello-sync
Hello Johnny!
```
## Production installation
*Prerequisite 1: Make sure you have a working Docker Standalone Swarm cluster with multi-node network mode in place, you can build one by following the instructions at [Docker's website](https://docs.docker.com/swarm/). The instructions to build a multi-host network can be found at [Docker's engine manual](https://docs.docker.com/engine/userguide/networking/get-started-overlay/#overlay-networking-with-an-external-key-value-store).*
*Prerequisite 2: It assumes that your running environment is already configured to use Swarm's master scheduler.*
This is a step-by-step procedure to execute Oracle Functions on top of Docker Swarm cluster. It works by having Oracle Functions daemon started through Swarm's master, however the tasks are executed on each host locally. In production, database and message queue must be external to Oracle Functions execution, this guarantees robustness of the service against failures.
We strongly recommend you deploy your own HA Redis and PostgreSQL clusters. Otherwise, you can follow the instructions below and have them set in single nodes.
### Groundwork
Although we're assuming you already have your Docker Swarm installed and configured, these `docker-machine` calls are instructive regarding some configuration details:
```bash
#!/bin/bash
# Note how every host points to an external etcd both for swarm discovery (--swarm-discovery) as much as network configuration (--engine-opt=cluster-store=)
docker-machine create -d virtualbox --swarm --swarm-master --swarm-discovery etcd://$ETCD_HOST:2379/swarm --engine-opt="cluster-store=etcd://$ETCD_HOST:2379/network" --engine-opt="cluster-advertise=eth1:2376" swarm-manager;
# Set aside one host for DB activities
docker-machine create -d virtualbox --engine-label use=db --swarm --swarm-discovery etcd://$ETCD_HOST:2379/swarm --engine-opt="cluster-store=etcd://$ETCD_HOST:2379/network" --engine-opt="cluster-advertise=eth1:2376" swarm-db;
# The rest is a horizontally scalable set of hosts for Oracle Functions
docker-machine create -d virtualbox --engine-label use=worker --swarm --swarm-discovery etcd://$ETCD_HOST:2379/swarm --engine-opt="cluster-store=etcd://$ETCD_HOST:2379/network" --engine-opt="cluster-advertise=eth1:2376" swarm-worker-00;
docker-machine create -d virtualbox --engine-label use=worker --swarm --swarm-discovery etcd://$ETCD_HOST:2379/swarm --engine-opt="cluster-store=etcd://$ETCD_HOST:2379/network" --engine-opt="cluster-advertise=eth1:2376" swarm-worker-01
```
### Steps
If you using externally deployed Redis and PostgreSQL cluster, you may skip to step 4.
1. Build a multi-host network for Oracle Functions:
```ShellSession
$ docker network create --driver overlay --subnet=10.0.9.0/24 functions-network
````
2. Setup Redis as message queue service:
```ShellSession
$ docker create -e constraint:use==db --network=functions-network -v /data --name redis-data redis /bin/true;
$ docker run -d -e constraint:use==db --network=functions-network --volumes-from redis-data --name functions-redis redis;
````
3. Setup PostgreSQL as datastore:
```ShellSession
$ docker create -e constraint:use==db --network=functions-network -v /var/lib/postgresql/data --name postgresql-data postgres /bin/true;
$ docker run -d -e constraint:use==db --network=functions-network --volumes-from postgresql-data --name functions-postgres -e POSTGRES_PASSWORD=mysecretpassword postgres
```
4. Start Oracle Functions:
```ShellSession
$ docker run -d --name functions-00 \
-l functions \
-e constraint:use==worker \
--network=functions-network \
-p 8080:8080 \
-v /var/run/docker.sock:/var/run/docker.sock \
-e 'MQ_URL=redis://functions-redis' \
-e 'DB_URL=postgres://postgres:mysecretpassword@functions-postgres/?sslmode=disable' \
treeder/functions
```
5. Load Balancer:
```ShellSession
$ export BACKENDS=$(docker ps --filter label=functions --format="{{ .ID }}" | xargs docker inspect | jq -r '.[].NetworkSettings.Ports["8080/tcp"][] | .HostIp + ":" + .HostPort' | paste -d, -s -)
$ docker run -d --name functions-lb -p 80:80 -e BACKENDS noqcks/haproxy
$ export FUNCTIONS=$(docker port functions-lb | cut -d ' ' -f3)
$ curl -H "Content-Type: application/json" -X POST -d '{ "app": { "name":"myapp" } }' http://$FUNCTIONS/v1/apps
{"message":"App successfully created","app":{"name":"myapp","config":null}}
$ curl -H "Content-Type: application/json" -X POST -d '{ "route": { "type": "sync", "path":"/hello-sync", "image":"fnproject/hello" } }' http://$FUNCTIONS/v1/apps/myapp/routes
{"message":"Route successfully created","route":{"app_name":"myapp","path":"/hello-sync","image":"fnproject/hello","memory":128,"type":"sync","config":null}}
$ curl -H "Content-Type: application/json" -X POST -d '{ "name":"Johnny" }' http://$FUNCTIONS/r/myapp/hello-sync
Hello Johnny!
```

View File

@@ -1,121 +1,66 @@
# HOWTO run Oracle Functions in Kubernetes at AWS
*Prerequisite 1: it assumes you have a working Kubernetes, and a locally configured kubectl.*
*Prerequisite 2: It assumes you are using Kubernetes 1.4 or newer.*
# How to run Fn on Kubernetes
*Prerequisite 1: working Kubernetes cluster (v1.7+), and a locally configured kubectl.*
## Quickstart
### Steps
1. Start Oracle Functions in the Kubernetes cluster:
```ShellSession
$ cd docs/
$ kubectl create -f kubernetes-quick
1. Deploy Fn to the Kubernetes cluster:
```bash
$ cd docs/operating/
$ kubectl create -f fn-service.yaml
```
2. Once the daemon is started, check where it is listening for connections:
```ShellSession
# kubectl describe svc functions
Name: functions
Namespace: default
Labels: app=functions
Selector: app=functions
Type: LoadBalancer
IP: 10.0.116.122
LoadBalancer Ingress: a23122e39900111e681ba0e29b70bb46-630391493.us-east-1.elb.amazonaws.com
Port: <unset> 8080/TCP
NodePort: <unset> 30802/TCP
Endpoints: 10.244.1.12:8080
Session Affinity: None
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
22m 22m 1 {service-controller } Normal CreatingLoadBalancer Creating load balancer
22m 22m 1 {service-controller } Normal CreatedLoadBalancer Created load balancer
2. Once the Pods have started, check the service for the load balanacer IP:
```bash
$ kubectl get svc --watch
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
fn-mysql-master 10.96.57.185 <none> 3306/TCP 10m
fn-redis-master 10.96.127.51 <none> 6379/TCP 10m
fn-service 10.96.245.95 <pending> 8080:30768/TCP,80:31921/TCP 10m
kubernetes 10.96.0.1 <none> 443/TCP 15d
```
Note `a23122e39900111e681ba0e29b70bb46-630391493.us-east-1.elb.amazonaws.com` in `LoadBalancer Ingress` line, this is where the service is listening.
Note that `fn-service` is initially pending on allocating an external IP. The `kubectl get svc --watch` command will update this once an IP has been assigned.
3. Test the cluster:
If you are using a Kubernetes setup that can expose a public loadbalancer run:
```ShellSession
$ export FUNCTIONS=$(kubectl get -o json svc functions | jq -r '.status.loadBalancer.ingress[0].hostname'):8080
If you are using a Kubernetes setup that can expose a public load balancer, run:
```bash
$ export FUNCTIONS=$(kubectl get -o json svc fn-service | jq -r '.status.loadBalancer.ingress[0].ip'):8080
```
If you are using a Kubernetes setup like minikube run
```ShellSession
$ export ns=default ; export label='app=functions'; kubectl -n $ns get pod -l $label -o jsonpath='{.items[0].metadata.name}' | xargs -I{} kubectl -n $ns port-forward {} 8080:8080
$ export FUNCTIONS=localhost:8080
If you are using a Kubernetes setup like minikube, run
```bash
$ echo $(minikube ip):$(kubectl get svc fn-service -o json | jq -r '.spec.ports[0].nodePort')
192.168.99.100:30966
$ export API_URL=http://192.168.99.100:30966
```
Now setup the functions:
Now, test by creating a function via curl:
```ShellSession
$ curl -H "Content-Type: application/json" -X POST -d '{ "app": { "name":"myapp" } }' http://$FUNCTIONS/v1/apps
```bash
$ curl -H "Content-Type: application/json" -X POST -d '{ "app": { "name":"myapp" } }' http://$API_URL/v1/apps
{"message":"App successfully created","app":{"name":"myapp","config":null}}
$ curl -H "Content-Type: application/json" -X POST -d '{ "route": { "type": "sync", "path":"/hello-sync", "image":"fnproject/hello" } }' http://$FUNCTIONS/v1/apps/myapp/routes
{"message":"Route successfully created","route":{"app_name":"myapp","path":"/hello-sync","image":"fnproject/hello","memory":128,"type":"sync","config":null}}
$ curl -H "Content-Type: application/json" -X POST -d '{ "route": { "type": "sync", "path":"/hello-sync", "image":"fnproject/hello" } }' http://$API_URL/v1/apps/myapp/routes
{"message":"Route successfully created","route":{"app_name":"myapp","path":"/hello-sync","image":"fnproject/hello","memory":128,"headers":{},"type":"sync","format":"default","timeout":30,"idle_timeout":30,"config":{}}}
$ curl -H "Content-Type: application/json" -X POST -d '{ "name":"Johnny" }' http://$FUNCTIONS/r/myapp/hello-sync
$ curl -H "Content-Type: application/json" -X POST -d '{ "name":"Johnny" }' http://$API_URL/r/myapp/hello-sync
Hello Johnny!
```
## Production
You can also use the [Fn CLI](https://github.com/fnproject/cli):
### Steps
1. Start Oracle Functions and its dependencies:
```ShellSession
$ cd docs/
$ kubectl create -f kubernetes-production
```
*Optionally, you might have both Redis and PostgreSQL started somewhere else, in this case, remember to update kubernetes-production/functions-config.yaml with the appropriate configuration.*
2. Once the daemon is started, check where it is listening for connections:
```ShellSession
# kubectl describe svc functions
Name: functions
Namespace: default
Labels: app=functions
Selector: app=functions
Type: LoadBalancer
IP: 10.0.116.122
LoadBalancer Ingress: a23122e39900111e681ba0e29b70bb46-630391493.us-east-1.elb.amazonaws.com
Port: <unset> 8080/TCP
NodePort: <unset> 30802/TCP
Endpoints: 10.244.1.12:8080
Session Affinity: None
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
22m 22m 1 {service-controller } Normal CreatingLoadBalancer Creating load balancer
22m 22m 1 {service-controller } Normal CreatedLoadBalancer Created load balancer
```
Note `a23122e39900111e681ba0e29b70bb46-630391493.us-east-1.elb.amazonaws.com` in `LoadBalancer Ingress` line, this is where the service is listening.
3. Test the cluster:
```ShellSession
$ export FUNCTIONS=$(kubectl get -o json svc functions | jq -r '.status.loadBalancer.ingress[0].hostname'):8080
$ curl -H "Content-Type: application/json" -X POST -d '{ "app": { "name":"myapp" } }' http://$FUNCTIONS/v1/apps
{"message":"App successfully created","app":{"name":"myapp","config":null}}
$ curl -H "Content-Type: application/json" -X POST -d '{ "route": { "type": "sync", "path":"/hello-sync", "image":"fnproject/hello" } }' http://$FUNCTIONS/v1/apps/myapp/routes
{"message":"Route successfully created","route":{"app_name":"myapp","path":"/hello-sync","image":"fnproject/hello","memory":128,"type":"sync","config":null}}
$ curl -H "Content-Type: application/json" -X POST -d '{ "name":"Johnny" }' http://$FUNCTIONS/r/myapp/hello-sync
Hello Johnny!
```
```bash
$ export API_URL=http://192.168.99.100:30966
$ fn apps list
myapp
$ fn routes list myapp
path image endpoint
/hello-sync fnproject/hello 192.168.99.100:30966/r/myapp/hello-sync
```

View File

@@ -0,0 +1,150 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: fn-service-config
namespace: default
data:
MQ_URL: redis://fn-redis-master.default
DB_URL: mysql://root:fnsecretpassword@tcp(fn-mysql-master:3306)/fn
API_URL: http://fn-service:8080
---
apiVersion: v1
kind: Service
metadata:
name: fn-service
labels:
app: fn
role: fn-service
spec:
type: LoadBalancer
ports:
- name: fn-service
port: 8080
targetPort: 8080
- name: fn-ui
port: 80
targetPort: 80
selector:
app: fn
role: fn-service
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: fn-service
spec:
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
minReadySeconds: 30
template:
metadata:
labels:
app: fn
role: fn-service
spec:
containers:
- name: fn-service
image: fnproject/functions:latest
securityContext:
privileged: true
ports:
- containerPort: 8080
env:
- name: MQ_URL
valueFrom:
configMapKeyRef:
name: fn-service-config
key: MQ_URL
- name: DB_URL
valueFrom:
configMapKeyRef:
name: fn-service-config
key: DB_URL
- name: fn-ui
image: fnproject/ui:latest
ports:
- containerPort: 80
env:
- name: PORT
value: "80"
- name: API_URL
valueFrom:
configMapKeyRef:
name: fn-service-config
key: API_URL
---
apiVersion: v1
kind: Service
metadata:
name: fn-mysql-master
labels:
app: mysql
role: datastore
spec:
ports:
- port: 3306
targetPort: 3306
selector:
app: mysql
role: datastore
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: fn-mysql-master
spec:
replicas: 1
template:
metadata:
labels:
app: mysql
role: datastore
spec:
containers:
- name: fn-mysql
image: mysql:5.7
args:
- "--max-connections=500"
- "--wait-timeout=300"
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: fnsecretpassword
- name: MYSQL_DATABASE
value: fn
---
apiVersion: v1
kind: Service
metadata:
name: fn-redis-master
labels:
app: redis
role: mq
spec:
ports:
- port: 6379
targetPort: 6379
selector:
app: redis
role: mq
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: fn-redis-master
spec:
replicas: 1
template:
metadata:
labels:
app: redis
role: mq
spec:
containers:
- name: fn-redis
image: redis:4.0
ports:
- containerPort: 6379

View File

@@ -1,7 +0,0 @@
kind: ConfigMap
apiVersion: v1
metadata:
name: functions-config
data:
MQ_URL: redis://redis-master.default
DB_URL: postgres://postgres:mysecretpassword@postgresql-master.default/?sslmode=disable

View File

@@ -1,43 +0,0 @@
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: functions
labels:
app: functions
spec:
replicas: 1
template:
metadata:
labels:
app: functions
spec:
containers:
- name: functions
image: treeder/functions
securityContext:
privileged: true
env:
- name: DOCKER_HOST
value: unix:///var/run/docker.sock
- name: MQ_URL
valueFrom:
configMapKeyRef:
name: functions-config
key: MQ_URL
- name: DB_URL
valueFrom:
configMapKeyRef:
name: functions-config
key: DB_URL
volumeMounts:
- mountPath: "/var/run/docker.sock"
name: docker-socket
readOnly: false
ports:
- name: http-server
containerPort: 8080
volumes:
- name: docker-socket
hostPath:
path: "/var/run/docker.sock"

View File

@@ -1,14 +0,0 @@
---
kind: Service
apiVersion: v1
metadata:
name: functions
labels:
app: functions
spec:
ports:
- port: 8080
targetPort: http-server
selector:
app: functions
type: LoadBalancer

View File

@@ -1,17 +0,0 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: postgresql-master
spec:
replicas: 1
template:
metadata:
labels:
app: postgresql
role: datastore
spec:
containers:
- name: functions-postgresql
image: postgres
ports:
- containerPort: 5432

View File

@@ -1,14 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: postgresql-master
labels:
app: postgresql
role: datastore
spec:
ports:
- port: 5432
targetPort: 5432
selector:
app: postgresql
role: datastore

View File

@@ -1,17 +0,0 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: redis-master
spec:
replicas: 1
template:
metadata:
labels:
app: redis
role: mq
spec:
containers:
- name: functions-redis
image: redis
ports:
- containerPort: 6379

View File

@@ -1,14 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: redis-master
labels:
app: redis
role: mq
spec:
ports:
- port: 6379
targetPort: 6379
selector:
app: redis
role: mq

View File

@@ -1,33 +0,0 @@
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: functions
labels:
app: functions
spec:
replicas: 1
template:
metadata:
labels:
app: functions
spec:
containers:
- name: functions
image: treeder/functions
securityContext:
privileged: true
env:
- name: DOCKER_HOST
value: unix:///var/run/docker.sock
volumeMounts:
- mountPath: "/var/run/docker.sock"
name: docker-socket
readOnly: false
ports:
- name: http-server
containerPort: 8080
volumes:
- name: docker-socket
hostPath:
path: "/var/run/docker.sock"

View File

@@ -1,14 +0,0 @@
---
kind: Service
apiVersion: v1
metadata:
name: functions
labels:
app: functions
spec:
ports:
- port: 8080
targetPort: http-server
selector:
app: functions
type: LoadBalancer