Added kubernetes and docker swarm to main TOC. (#497)

docs/operating/databases/README.md

@@ -0,0 +1,30 @@
+
+# Databases
+
+We currently support the following databases and they are passed in via the `DB_URL` environment variable. For example:
+
+```sh
+docker run -e "DB_URL=postgres://user:pass@localhost:6212/mydb" ...
+```
+
+## [Bolt](https://github.com/boltdb/bolt) (default)
+
+URL: `bolt:///functions/data/functions.db`
+
+Bolt is an embedded database which stores to disk. If you want to use this, be sure you don't lose the data directory by mounting
+the directory on your host. eg: `docker run -v $PWD/data:/functions/data -e DB_URL=bolt:///functions/data/bolt.db ...`
+
+[More on BoltDB](boltdb.md)
+
+## [PostgreSQL](http://www.postgresql.org/)
+
+URL: `postgres://user123:pass456@ec2-117-21-174-214.compute-1.amazonaws.com:6212/db982398`
+
+Use a PostgreSQL database. If you're using IronFunctions in production, you should probably start here.
+
+[More on Postgres](postgres.md)
+
+## What about database X?
+
+We're happy to add more and we love pull requests, so feel free to add one! Copy one of the implementations above as a starting point. 
+

docs/operating/databases/boltdb.md

@@ -0,0 +1,11 @@
+# IronFunctions using BoltDB
+
+BoltDB is the default database, you just need to run the API.
+
+## Persistent
+
+To keep it persistent, add a volume flag to the command:
+
+```
+docker run --rm -it --privileged -v $PWD/bolt.db:/app/bolt.db -p 8080:8080 iron/functions
+```

docs/operating/databases/postgres.md

@@ -0,0 +1,34 @@
+# IronFunctions using Postgres
+
+Let's presuppose you don't have even a postgres DB ready.
+
+### 1. Let's start a postgres instance:
+
+```
+docker run --name iron-postgres \
+        -e POSTGRES_PASSWORD=ironfunctions -d postgres
+``` 
+
+### 2. Now let's create a new database to IronFunctions
+
+Creating database:
+
+```
+docker run -it --rm --link iron-postgres:postgres postgres \
+    psql -h postgres -U postgres -c "CREATE DATABASE funcs;"
+```
+
+Granting access to postgres user
+
+```
+docker run -it --rm --link iron-postgres:postgres postgres \
+    psql -h postgres -U postgres -c 'GRANT ALL PRIVILEGES ON DATABASE funcs TO postgres;'
+```
+
+### 3. Now let's start IronFunctions connecting to our new postgres instance
+
+```
+docker run --rm --privileged --link "iron-postgres:postgres" \
+    -e "DB_URL=postgres://postgres:ironfunctions@postgres/funcs?sslmode=disable" \
+    -it -p 8080:8080 iron/functions
+```

docs/operating/docker-swarm/README.md

@@ -0,0 +1,133 @@
+# Docker Swarm and IronFunctions
+
+How to run IronFunction as a scheduler on top of Docker Standalone Swarm cluster.
+
+## Quick installation
+
+*Prerequisite 1: Make sure you have a working Docker 1.12+ Standalone Swarm cluster in place, you can build one by following the instructions at [Docker's website](https://docs.docker.com/swarm/).*
+
+*Prerequisite 2: It assumes that your running environment is already configured to use Swarm's master scheduler.*
+
+This is a step-by-step procedure to execute IronFunction on top of Docker Swarm cluster. It works by having IronFunction daemon started through Swarm's master, and there enqueueing tasks through Swarm API.
+
+### Steps
+
+1. Start IronFunction in the Swarm Master. It expects all basic Docker environment variables to be present (DOCKER_TLS_VERIFY, DOCKER_HOST, DOCKER_CERT_PATH, DOCKER_MACHINE_NAME). The important part is that the working Swarm master environment must be passed to Functions daemon:
+```ShellSession
+$ docker login # if you plan to use private images
+$ docker volume create --name functions-datafiles
+$ docker run -d --name functions \
+        -p 8080:8080 \
+        -e DOCKER_TLS_VERIFY \
+        -e DOCKER_HOST \
+        -e DOCKER_CERT_PATH="/docker-cert" \
+        -e DOCKER_MACHINE_NAME \
+        -v $DOCKER_CERT_PATH:/docker-cert \
+        -v functions-datafiles:/app/data \
+        iron/functions
+```
+
+2. Once the daemon is started, check where it is listening for connections:
+
+```ShellSession
+# docker info
+CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                                     NAMES
+5a0846e6a025        iron/functions       "/usr/local/bin/entry"   59 seconds ago      Up 58 seconds       2375/tcp, 10.0.0.1:8080->8080/tcp   swarm-agent-00/functions
+````
+
+Note `10.0.0.1:8080` in `PORTS` column, this is where the service is listening. IronFunction will use Docker Swarm scheduler to deliver tasks to all nodes present in the cluster.
+
+3. Test the cluster:
+
+```ShellSession
+$ export IRON_FUNCTION=$(docker port functions | cut -d ' ' -f3)
+
+$ curl -H "Content-Type: application/json" -X POST -d '{ "app": { "name":"myapp" } }' http://$IRON_FUNCTION/v1/apps
+{"message":"App successfully created","app":{"name":"myapp","config":null}}
+
+$ curl -H "Content-Type: application/json" -X POST -d '{ "route": { "type": "sync", "path":"/hello-sync", "image":"iron/hello" } }' http://$IRON_FUNCTION/v1/apps/myapp/routes
+{"message":"Route successfully created","route":{"app_name":"myapp","path":"/hello-sync","image":"iron/hello","memory":128,"type":"sync","config":null}}
+
+$ curl -H "Content-Type: application/json" -X POST -d '{ "name":"Johnny" }' http://$IRON_FUNCTION/r/myapp/hello-sync
+Hello Johnny!
+```
+
+## Production installation
+
+*Prerequisite 1: Make sure you have a working Docker Standalone Swarm cluster with multi-node network mode in place, you can build one by following the instructions at [Docker's website](https://docs.docker.com/swarm/). The instructions to build a multi-host network can be found at [Docker's engine manual](https://docs.docker.com/engine/userguide/networking/get-started-overlay/#overlay-networking-with-an-external-key-value-store).*
+
+*Prerequisite 2: It assumes that your running environment is already configured to use Swarm's master scheduler.*
+
+This is a step-by-step procedure to execute IronFunction on top of Docker Swarm cluster. It works by having IronFunction daemon started through Swarm's master, however the tasks are executed on each host locally. In production, database and message queue must be external to IronFunction execution, this guarantees robustness of the service against failures.
+
+We strongly recommend you deploy your own HA Redis and PostgreSQL clusters. Otherwise, you can follow the instructions below and have them set in single nodes.
+
+### Groundwork
+
+Although we're assuming you already have your Docker Swarm installed and configured, these `docker-machine` calls are instructive regarding some configuration details:
+```bash
+#!/bin/bash
+
+# Note how every host points to an external etcd both for swarm discovery (--swarm-discovery) as much as network configuration (--engine-opt=cluster-store=)
+docker-machine create -d virtualbox --swarm --swarm-master --swarm-discovery etcd://$ETCD_HOST:2379/swarm --engine-opt="cluster-store=etcd://$ETCD_HOST:2379/network" --engine-opt="cluster-advertise=eth1:2376" swarm-manager;
+
+# Set aside one host for DB activities
+docker-machine create -d virtualbox --engine-label use=db --swarm --swarm-discovery etcd://$ETCD_HOST:2379/swarm --engine-opt="cluster-store=etcd://$ETCD_HOST:2379/network" --engine-opt="cluster-advertise=eth1:2376" swarm-db;
+
+# The rest is a horizontally scalable set of hosts for IronFunction
+docker-machine create -d virtualbox --engine-label use=worker --swarm --swarm-discovery etcd://$ETCD_HOST:2379/swarm --engine-opt="cluster-store=etcd://$ETCD_HOST:2379/network" --engine-opt="cluster-advertise=eth1:2376" swarm-worker-00;
+docker-machine create -d virtualbox --engine-label use=worker --swarm --swarm-discovery etcd://$ETCD_HOST:2379/swarm --engine-opt="cluster-store=etcd://$ETCD_HOST:2379/network" --engine-opt="cluster-advertise=eth1:2376" swarm-worker-01
+```
+
+### Steps
+
+If you using externally deployed Redis and PostgreSQL cluster, you may skip to step 4.
+
+1. Build a multi-host network for IronFunction:
+```ShellSession
+$ docker network create --driver overlay --subnet=10.0.9.0/24 functions-network
+````
+
+2. Setup Redis as message queue service:
+```ShellSession
+$ docker create -e constraint:use==db --network=functions-network -v /data --name redis-data redis /bin/true;
+$ docker run -d -e constraint:use==db --network=functions-network --volumes-from redis-data --name functions-redis redis;
+````
+
+3. Setup PostgreSQL as datastore:
+```ShellSession
+$ docker create -e constraint:use==db --network=functions-network -v /var/lib/postgresql/data --name postgresql-data postgres /bin/true;
+$ docker run -d -e constraint:use==db --network=functions-network --volumes-from postgresql-data --name functions-postgres -e POSTGRES_PASSWORD=mysecretpassword postgres
+```
+
+4. Start IronFunctions:
+```ShellSession
+$ docker run -d --name functions-00 \
+        -l functions \
+        -e constraint:use==worker \
+        --network=functions-network \
+        -p 8080:8080 \
+        -v /var/run/docker.sock:/var/run/docker.sock \
+        -e 'MQ_URL=redis://functions-redis' \
+        -e 'DB_URL=postgres://postgres:mysecretpassword@functions-postgres/?sslmode=disable' \
+        iron/functions
+```
+
+5. Load Balancer:
+
+```ShellSession
+$ export BACKENDS=$(docker ps --filter label=functions --format="{{ .ID }}" | xargs docker inspect | jq -r '.[].NetworkSettings.Ports["8080/tcp"][] | .HostIp + ":" + .HostPort' | paste -d, -s -)
+
+$ docker run -d --name functions-lb -p 80:80 -e BACKENDS noqcks/haproxy
+
+$ export IRON_FUNCTION=$(docker port functions-lb | cut -d ' ' -f3)
+
+$ curl -H "Content-Type: application/json" -X POST -d '{ "app": { "name":"myapp" } }' http://$IRON_FUNCTION/v1/apps
+{"message":"App successfully created","app":{"name":"myapp","config":null}}
+
+$ curl -H "Content-Type: application/json" -X POST -d '{ "route": { "type": "sync", "path":"/hello-sync", "image":"iron/hello" } }' http://$IRON_FUNCTION/v1/apps/myapp/routes
+{"message":"Route successfully created","route":{"app_name":"myapp","path":"/hello-sync","image":"iron/hello","memory":128,"type":"sync","config":null}}
+
+$ curl -H "Content-Type: application/json" -X POST -d '{ "name":"Johnny" }' http://$IRON_FUNCTION/r/myapp/hello-sync
+Hello Johnny!
+```

docs/operating/kubernetes/README.md

@@ -0,0 +1,110 @@
+# HOWTO run IronFunction in Kubernetes at AWS
+
+*Prerequisite 1: it assumes you have a working Kubernetes, and a locally configured kubectl.*
+
+*Prerequisite 2: It assumes you are using Kubernetes 1.4 or newer.*
+
+
+## Quickstart
+
+### Steps
+
+1. Start IronFunction in the Kubernetes cluster:
+```ShellSession
+$ cd docs/
+$ kubectl create -f kubernetes-quick
+```
+
+2. Once the daemon is started, check where it is listening for connections:
+
+```ShellSession
+# kubectl describe svc functions
+
+Name:			functions
+Namespace:		default
+Labels:			app=functions
+Selector:		app=functions
+Type:			LoadBalancer
+IP:			10.0.116.122
+LoadBalancer Ingress:	a23122e39900111e681ba0e29b70bb46-630391493.us-east-1.elb.amazonaws.com
+Port:			<unset>	8080/TCP
+NodePort:		<unset>	30802/TCP
+Endpoints:		10.244.1.12:8080
+Session Affinity:	None
+Events:
+  FirstSeen	LastSeen	Count	From			SubobjectPath	Type		Reason			Message
+  ---------	--------	-----	----			-------------	--------	------			-------
+  22m		22m		1	{service-controller }			Normal		CreatingLoadBalancer	Creating load balancer
+  22m		22m		1	{service-controller }			Normal		CreatedLoadBalancer	Created load balancer
+
+```
+
+Note `a23122e39900111e681ba0e29b70bb46-630391493.us-east-1.elb.amazonaws.com` in `LoadBalancer Ingress` line, this is where the service is listening.
+
+3. Test the cluster:
+
+```ShellSession
+$ export IRON_FUNCTION=$(kubectl get -o json svc functions | jq -r '.status.loadBalancer.ingress[0].hostname'):8080
+
+$ curl -H "Content-Type: application/json" -X POST -d '{ "app": { "name":"myapp" } }' http://$IRON_FUNCTION/v1/apps
+{"message":"App successfully created","app":{"name":"myapp","config":null}}
+
+$ curl -H "Content-Type: application/json" -X POST -d '{ "route": { "type": "sync", "path":"/hello-sync", "image":"iron/hello" } }' http://$IRON_FUNCTION/v1/apps/myapp/routes
+{"message":"Route successfully created","route":{"app_name":"myapp","path":"/hello-sync","image":"iron/hello","memory":128,"type":"sync","config":null}}
+
+$ curl -H "Content-Type: application/json" -X POST -d '{ "name":"Johnny" }' http://$IRON_FUNCTION/r/myapp/hello-sync
+Hello Johnny!
+```
+
+## Production
+
+### Steps
+
+1. Start IronFunction and its dependencies:
+```ShellSession
+$ cd docs/
+$ kubectl create -f kubernetes-production
+```
+
+*Optionally, you might have both Redis and PostgreSQL started somewhere else, in this case, remember to update kubernetes-production/functions-config.yaml with the appropriate configuration.*
+
+2. Once the daemon is started, check where it is listening for connections:
+
+```ShellSession
+# kubectl describe svc functions
+
+Name:			functions
+Namespace:		default
+Labels:			app=functions
+Selector:		app=functions
+Type:			LoadBalancer
+IP:			10.0.116.122
+LoadBalancer Ingress:	a23122e39900111e681ba0e29b70bb46-630391493.us-east-1.elb.amazonaws.com
+Port:			<unset>	8080/TCP
+NodePort:		<unset>	30802/TCP
+Endpoints:		10.244.1.12:8080
+Session Affinity:	None
+Events:
+  FirstSeen	LastSeen	Count	From			SubobjectPath	Type		Reason			Message
+  ---------	--------	-----	----			-------------	--------	------			-------
+  22m		22m		1	{service-controller }			Normal		CreatingLoadBalancer	Creating load balancer
+  22m		22m		1	{service-controller }			Normal		CreatedLoadBalancer	Created load balancer
+
+```
+
+Note `a23122e39900111e681ba0e29b70bb46-630391493.us-east-1.elb.amazonaws.com` in `LoadBalancer Ingress` line, this is where the service is listening.
+
+3. Test the cluster:
+
+```ShellSession
+$ export IRON_FUNCTION=$(kubectl get -o json svc functions | jq -r '.status.loadBalancer.ingress[0].hostname'):8080
+
+$ curl -H "Content-Type: application/json" -X POST -d '{ "app": { "name":"myapp" } }' http://$IRON_FUNCTION/v1/apps
+{"message":"App successfully created","app":{"name":"myapp","config":null}}
+
+$ curl -H "Content-Type: application/json" -X POST -d '{ "route": { "type": "sync", "path":"/hello-sync", "image":"iron/hello" } }' http://$IRON_FUNCTION/v1/apps/myapp/routes
+{"message":"Route successfully created","route":{"app_name":"myapp","path":"/hello-sync","image":"iron/hello","memory":128,"type":"sync","config":null}}
+
+$ curl -H "Content-Type: application/json" -X POST -d '{ "name":"Johnny" }' http://$IRON_FUNCTION/r/myapp/hello-sync
+Hello Johnny!
+```

docs/operating/kubernetes/kubernetes-production/functions-config.yaml

@@ -0,0 +1,7 @@
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: functions-config
+data:
+  MQ_URL: redis://redis-master.default
+  DB_URL: postgres://postgres:mysecretpassword@postgresql-master.default/?sslmode=disable

docs/operating/kubernetes/kubernetes-production/functions-deployment.yaml

@@ -0,0 +1,43 @@
+---
+kind: Deployment
+apiVersion: extensions/v1beta1
+metadata:
+  name: functions
+  labels:
+    app: functions
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        app: functions
+    spec:
+      containers:
+      - name: functions
+        image: iron/functions
+        securityContext:
+          privileged: true
+        env:
+        - name: DOCKER_HOST
+          value: unix:///var/run/docker.sock
+        - name: MQ_URL
+          valueFrom:
+            configMapKeyRef:
+              name: functions-config
+              key: MQ_URL
+        - name: DB_URL
+          valueFrom:
+            configMapKeyRef:
+              name: functions-config
+              key: DB_URL
+        volumeMounts:
+        - mountPath: "/var/run/docker.sock"
+          name: docker-socket
+          readOnly: false
+        ports:
+        - name: http-server
+          containerPort: 8080
+      volumes:
+      - name: docker-socket
+        hostPath:
+          path: "/var/run/docker.sock"

docs/operating/kubernetes/kubernetes-production/functions-service.yaml

@@ -0,0 +1,14 @@
+---
+kind: Service
+apiVersion: v1
+metadata:
+  name: functions
+  labels:
+    app: functions
+spec:
+  ports:
+  - port: 8080
+    targetPort: http-server
+  selector:
+    app: functions
+  type: LoadBalancer

docs/operating/kubernetes/kubernetes-production/postgresql-deployment.yaml

@@ -0,0 +1,17 @@
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: postgresql-master
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        app: postgresql
+        role: datastore
+    spec:
+      containers:
+      - name: functions-postgresql
+        image: postgres
+        ports:
+        - containerPort: 5432

docs/operating/kubernetes/kubernetes-production/postgresql-service.yaml

@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: postgresql-master
+  labels:
+    app: postgresql
+    role: datastore
+spec:
+  ports:
+  - port: 5432
+    targetPort: 5432
+  selector:
+    app: postgresql
+    role: datastore

docs/operating/kubernetes/kubernetes-production/redis-deployment.yaml

@@ -0,0 +1,17 @@
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: redis-master
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        app: redis
+        role: mq
+    spec:
+      containers:
+      - name: functions-redis
+        image: redis
+        ports:
+        - containerPort: 6379

docs/operating/kubernetes/kubernetes-production/redis-service.yaml

@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis-master
+  labels:
+    app: redis
+    role: mq
+spec:
+  ports:
+  - port: 6379
+    targetPort: 6379
+  selector:
+    app: redis
+    role: mq

docs/operating/kubernetes/kubernetes-quick/deployment.yaml

@@ -0,0 +1,33 @@
+---
+kind: Deployment
+apiVersion: extensions/v1beta1
+metadata:
+  name: functions
+  labels:
+    app: functions
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        app: functions
+    spec:
+      containers:
+      - name: functions
+        image: iron/functions
+        securityContext:
+          privileged: true
+        env:
+        - name: DOCKER_HOST
+          value: unix:///var/run/docker.sock
+        volumeMounts:
+        - mountPath: "/var/run/docker.sock"
+          name: docker-socket
+          readOnly: false
+        ports:
+        - name: http-server
+          containerPort: 8080
+      volumes:
+      - name: docker-socket
+        hostPath:
+          path: "/var/run/docker.sock"

docs/operating/kubernetes/kubernetes-quick/service.yaml

@@ -0,0 +1,14 @@
+---
+kind: Service
+apiVersion: v1
+metadata:
+  name: functions
+  labels:
+    app: functions
+spec:
+  ports:
+  - port: 8080
+    targetPort: http-server
+  selector:
+    app: functions
+  type: LoadBalancer