mirror of
https://github.com/fnproject/fn.git
synced 2022-10-28 21:29:17 +03:00
Moves out node pool manager behind an extension using runner pool abstraction (Part 2) (#862)
* Move out node-pool manager and replace it with RunnerPool extension * adds extension points for runner pools in load-balanced mode * adds error to return values in RunnerPool and Runner interfaces * Implements runner pool contract with context-aware shutdown * fixes issue with range * fixes tests to use runner abstraction * adds empty test file as a workaround for build requiring go source files in top-level package * removes flappy timeout test * update docs to reflect runner pool setup * refactors system tests to use runner abstraction * removes poolmanager * moves runner interfaces from models to api/runnerpool package * Adds a second runner to pool docs example * explicitly check for request spillover to second runner in test * moves runner pool package name for system tests * renames runner pool pointer variable for consistency * pass model json to runner * automatically cast to http.ResponseWriter in load-balanced call case * allow overriding of server RunnerPool via a programmatic ServerOption * fixes return type of ResponseWriter in test * move Placer interface to runnerpool package * moves hash-based placer out of open source project * removes siphash from Gopkg.lock
This commit is contained in:
Gerardo Viedma
@@ -1,252 +0,0 @@
|
||||
# Running fn in Multitenant Compute Mode
|
||||
|
||||
## Motivation
|
||||
|
||||
By running Fn in multitenant mode, you can define independent pools of compute resources available to functions in the platform. By associating a function with a particular _load balancing group_, its invocations are guaranteed to execute on the compute resources assigned to that specific group. The pluggable _node pool manager_ abstraction provides a mechanism to scale compute resources dynamically, based on capacity requirements advertised by the load-balancing layer. Together with load balancer groups, it allows you to implement independent capacity and scaling policies for different sets of users or tenants.
|
||||
|
||||
## Create certificates
|
||||
|
||||
This is a useful article to read for quickly generating mutual TLS certs:
|
||||
|
||||
http://www.levigross.com/2015/11/21/mutual-tls-authentication-in-go/
|
||||
|
||||
tl;dr: Get this https://github.com/levigross/go-mutual-tls/blob/master/generate\_client\_cert.go
|
||||
|
||||
add IP `127.0.0.1` to the cert by adding the line
|
||||
|
||||
```golang
|
||||
template.IPAddresses = append(template.IPAddresses, net.ParseIP("127.0.0.1"))
|
||||
```
|
||||
|
||||
somewhere around line 124,
|
||||
add the "net" package to the list of import packages and run it with
|
||||
|
||||
```bash
|
||||
go run generate_client_cert.go --email-address a@a.com
|
||||
```
|
||||
|
||||
Tada! Certs.
|
||||
|
||||
## Starting the components (as regular processes)
|
||||
|
||||
### API server
|
||||
|
||||
```bash
|
||||
FN_NODE_TYPE=api ./fnserver
|
||||
```
|
||||
|
||||
### Runner
|
||||
|
||||
```bash
|
||||
mkdir /tmp/runnerdata
|
||||
FN_NODE_TYPE=pure-runner FN_PORT=8082 FN_NODE_CERT=cert.pem FN_NODE_CERT_AUTHORITY=cert.pem FN_NODE_CERT_KEY=key.pem ./fnserver
|
||||
```
|
||||
|
||||
### LB
|
||||
|
||||
```bash
|
||||
mkdir /tmp/lbdata
|
||||
FN_NODE_TYPE=lb FN_PORT=8081 FN_RUNNER_API_URL=http://localhost:8080 FN_NODE_CERT=cert.pem FN_NODE_CERT_AUTHORITY=cert.pem FN_NPM_ADDRESS=localhost:8083 FN_NODE_CERT_KEY=key.pem FN_LOG_LEVEL=DEBUG ./fnserver
|
||||
```
|
||||
|
||||
### Node Pool Manager (NPM)
|
||||
|
||||
Currently the NPM uses a fixed, single-node instance of the Runner to simulate its "pool". The runner answers on port 8082 in this example,
|
||||
but the GRPC port is 9190.
|
||||
Grap the runner address and put in as value for the `FN_RUNNER_ADDRESSES` env variable.
|
||||
|
||||
```bash
|
||||
go build -buildmode=plugin -o noop.so poolmanager/server/controlplane/plugin/noop.go
|
||||
go build -o fnnpm poolmanager/server/main.go
|
||||
|
||||
FN_LOG_LEVEL=DEBUG \
|
||||
FN_NODE_CERT=cert.pem \
|
||||
FN_NODE_CERT_KEY=key.pem \
|
||||
FN_NODE_CERT_AUTHORITY=cert.pem \
|
||||
FN_PORT=8083 \
|
||||
FN_RUNNER_ADDRESSES=<RUNNER_ADDRESS_HERE>:9190 \
|
||||
CONTROL_PLANE_SO=noop.so \
|
||||
./fnnpm
|
||||
```
|
||||
|
||||
### Directing a request to a specific LB Group
|
||||
|
||||
Until a generic metadata mechanism is available in fn, an application or route can be [configured][docs/developers/configs.md] so that incoming requests are forwarded to runners in the specified LB group. In the absence of this configuration, requests will map to the _default_ LB group.
|
||||
|
||||
For example, to set an app's LB group:
|
||||
|
||||
```bash
|
||||
fn apps config set myapp FN_LB_GROUP_ID my-app-pool
|
||||
```
|
||||
|
||||
Note that the value of _FN\_LB\_GROUP\_ID_ above will then be visible to the function as an environment variable.
|
||||
|
||||
## Starting the components (in Docker containers)
|
||||
|
||||
### Build the images
|
||||
|
||||
The images don't yet exist in a registry, so they need building first.
|
||||
|
||||
```bash
|
||||
docker build -f images/fnnpm/Dockerfile -t fnproject/fnnpm:latest .
|
||||
docker build -f images/lb/Dockerfile -t fnproject/lb:latest .
|
||||
docker build -f images/api/Dockerfile -t fnproject/api:latest .
|
||||
docker build -f images/runner/Dockerfile -t fnproject/runner:latest .
|
||||
```
|
||||
|
||||
### Start the containers
|
||||
|
||||
### API
|
||||
|
||||
This *shouldn't* need to talk to the Docker daemon, but it still tries to *for now*. So mount the socket.
|
||||
|
||||
```
|
||||
docker run -d \
|
||||
--name api \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-p 8080:8080 \
|
||||
fnproject/api:latest
|
||||
```
|
||||
|
||||
|
||||
#### First runner
|
||||
```bash
|
||||
docker run -d \
|
||||
--name runner \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-p 9190:9190 \
|
||||
-e FN_GRPC_PORT=9190 \
|
||||
-p 8095:8080 \
|
||||
-v $(pwd)/cert.pem:/certs/cert.pem \
|
||||
-v $(pwd)/key.pem:/certs/key.pem \
|
||||
-e FN_NODE_CERT=/certs/cert.pem \
|
||||
-e FN_NODE_CERT_KEY=/certs/key.pem \
|
||||
-e FN_NODE_CERT_AUTHORITY=/certs/cert.pem \
|
||||
fnproject/runner:latest
|
||||
```
|
||||
|
||||
#### Second runner
|
||||
```bash
|
||||
docker run -d \
|
||||
--name runner-2 \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-p 9191:9191 \
|
||||
-e FN_GRPC_PORT=9191 \
|
||||
-p 8096:8080 \
|
||||
-v $(pwd)/cert.pem:/certs/cert.pem \
|
||||
-v $(pwd)/key.pem:/certs/key.pem \
|
||||
-e FN_NODE_CERT=/certs/cert.pem \
|
||||
-e FN_NODE_CERT_KEY=/certs/key.pem \
|
||||
-e FN_NODE_CERT_AUTHORITY=/certs/cert.pem \
|
||||
fnproject/runner:latest
|
||||
```
|
||||
|
||||
|
||||
### Node Pool Manager (NPM)
|
||||
Retrieve the IP addresses for the runners:
|
||||
|
||||
```bash
|
||||
export RUNNER1=`docker inspect --format '{{ .NetworkSettings.IPAddress }}' runner`
|
||||
export RUNNER2=`docker inspect --format '{{ .NetworkSettings.IPAddress }}' runner-2`
|
||||
|
||||
```
|
||||
|
||||
|
||||
```
|
||||
docker run -d \
|
||||
--name fnnpm \
|
||||
-e FN_RUNNER_ADDRESSES=$RUNNER1:9190,$RUNNER2:9191 \
|
||||
-p 8083:8080 \
|
||||
-v $(pwd)/cert.pem:/certs/cert.pem \
|
||||
-v $(pwd)/key.pem:/certs/key.pem \
|
||||
-e FN_NODE_CERT=/certs/cert.pem \
|
||||
-e FN_NODE_CERT_KEY=/certs/key.pem \
|
||||
-e FN_NODE_CERT_AUTHORITY=/certs/cert.pem \
|
||||
-e FN_LOG_LEVEL=INFO \
|
||||
-e FN_PORT=8083 \
|
||||
fnproject/fnnpm:latest
|
||||
```
|
||||
|
||||
### LB
|
||||
|
||||
Again, this *shouldn't* need to talk to the Docker daemon, but it still tries to *for now*. So mount the socket.
|
||||
|
||||
Retrieve the IP address for API and NPM:
|
||||
|
||||
```bash
|
||||
export API=`docker inspect --format '{{ .NetworkSettings.IPAddress }}' api`
|
||||
export NPM=`docker inspect --format '{{ .NetworkSettings.IPAddress }}' fnnpm`
|
||||
```
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name lb \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-p 8081:8080 \
|
||||
-v $(pwd)/cert.pem:/certs/cert.pem \
|
||||
-v $(pwd)/key.pem:/certs/key.pem \
|
||||
-e FN_NODE_TYPE=lb \
|
||||
-e FN_RUNNER_API_URL=http://$API:8080 \
|
||||
-e FN_NPM_ADDRESS=$NPM:8083 \
|
||||
-e FN_NODE_CERT=/certs/cert.pem \
|
||||
-e FN_NODE_CERT_KEY=/certs/key.pem \
|
||||
-e FN_NODE_CERT_AUTHORITY=/certs/cert.pem \
|
||||
fnproject/lb:latest
|
||||
```
|
||||
## Running without the Node Pool Manager
|
||||
This mode assumes that LB is started with a static set of runners in a single global pool. Note that this configuration does not support runner certificates and is that the communication between LB and runners is unencrypted.
|
||||
|
||||
### API
|
||||
|
||||
```
|
||||
docker run -d \
|
||||
--name api \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-p 8080:8080 \
|
||||
fnproject/api:latest
|
||||
```
|
||||
|
||||
#### First runner
|
||||
```bash
|
||||
docker run -d \
|
||||
--name runner \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-p 9190:9190 \
|
||||
-e FN_GRPC_PORT=9190 \
|
||||
-p 8095:8080 \
|
||||
fnproject/runner:latest
|
||||
```
|
||||
|
||||
#### Second runner
|
||||
```bash
|
||||
docker run -d \
|
||||
--name runner-2 \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-p 9191:9191 \
|
||||
-e FN_GRPC_PORT=9191 \
|
||||
-p 8096:8080 \
|
||||
fnproject/runner:latest
|
||||
```
|
||||
|
||||
### LB
|
||||
|
||||
Retrieve the IP addresses for the runners and the API:
|
||||
|
||||
```bash
|
||||
export RUNNER1=`docker inspect --format '{{ .NetworkSettings.IPAddress }}' runner`
|
||||
export RUNNER2=`docker inspect --format '{{ .NetworkSettings.IPAddress }}' runner-2`
|
||||
export API=`docker inspect --format '{{ .NetworkSettings.IPAddress }}' api`
|
||||
|
||||
```
|
||||
|
||||
Pass in the static set of runners to _FN\_RUNNER\_ADDRESSES_:
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name lb \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-p 8081:8080 \
|
||||
-e FN_RUNNER_API_URL=http://$API:8080 \
|
||||
-e FN_RUNNER_ADDRESSES=$RUNNER1:9190,$RUNNER2:9191 \
|
||||
fnproject/lb:latest
|
||||
```
|
||||
101
docs/operating/runner_pools.md
Normal file
101
docs/operating/runner_pools.md
Normal file
@@ -0,0 +1,101 @@
|
||||
# Running load-balanced fn against a pool of runners
|
||||
|
||||
## Motivation
|
||||
|
||||
You can run a load-balanced setup for fn to route requests to a group of one or more runners.
|
||||
|
||||
## Starting the components (as regular processes)
|
||||
|
||||
### API server
|
||||
|
||||
```bash
|
||||
FN_NODE_TYPE=api ./fnserver
|
||||
```
|
||||
|
||||
### Runners
|
||||
|
||||
```bash
|
||||
mkdir /tmp/runnerdata
|
||||
# first runner
|
||||
FN_NODE_TYPE=pure-runner FN_PORT=8082 FN_GRPC_PORT=9190 ./fnserver
|
||||
# on another terminal, start a second runner
|
||||
FN_NODE_TYPE=pure-runner FN_PORT=8083 FN_GRPC_PORT=9191 ./fnserver
|
||||
```
|
||||
|
||||
### LB
|
||||
|
||||
```bash
|
||||
mkdir /tmp/lbdata
|
||||
FN_NODE_TYPE=lb FN_PORT=8081 FN_RUNNER_API_URL=http://localhost:8080 FN_RUNNER_ADDRESSES=localhost:9190,localhost:9191 FN_LOG_LEVEL=DEBUG ./fnserver
|
||||
```
|
||||
|
||||
## Starting the components (in Docker containers)
|
||||
|
||||
### Build the images
|
||||
|
||||
The images don't yet exist in a registry, so they need building first.
|
||||
|
||||
```bash
|
||||
docker build -f images/lb/Dockerfile -t fnproject/lb:latest .
|
||||
docker build -f images/api/Dockerfile -t fnproject/api:latest .
|
||||
docker build -f images/runner/Dockerfile -t fnproject/runner:latest .
|
||||
```
|
||||
|
||||
### Start the containers
|
||||
|
||||
This mode assumes that LB is started with a static set of runners in a single global pool. Note that this configuration does not support runner certificates and is that the communication between LB and runners is unencrypted.
|
||||
|
||||
### API
|
||||
|
||||
```
|
||||
docker run -d \
|
||||
--name api \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-p 8080:8080 \
|
||||
fnproject/api:latest
|
||||
```
|
||||
|
||||
#### First runner
|
||||
```bash
|
||||
docker run -d \
|
||||
--name runner \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-p 9190:9190 \
|
||||
-e FN_GRPC_PORT=9190 \
|
||||
-p 8095:8080 \
|
||||
fnproject/runner:latest
|
||||
```
|
||||
|
||||
#### Second runner
|
||||
```bash
|
||||
docker run -d \
|
||||
--name runner-2 \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-p 9191:9191 \
|
||||
-e FN_GRPC_PORT=9191 \
|
||||
-p 8096:8080 \
|
||||
fnproject/runner:latest
|
||||
```
|
||||
|
||||
### LB
|
||||
|
||||
Retrieve the IP addresses for the runners and the API:
|
||||
|
||||
```bash
|
||||
export RUNNER1=`docker inspect --format '{{ .NetworkSettings.IPAddress }}' runner`
|
||||
export RUNNER2=`docker inspect --format '{{ .NetworkSettings.IPAddress }}' runner-2`
|
||||
export API=`docker inspect --format '{{ .NetworkSettings.IPAddress }}' api`
|
||||
|
||||
```
|
||||
|
||||
Pass in the static set of runners to _FN\_RUNNER\_ADDRESSES_:
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name lb \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
-p 8081:8080 \
|
||||
-e FN_RUNNER_API_URL=http://$API:8080 \
|
||||
-e FN_RUNNER_ADDRESSES=$RUNNER1:9190,$RUNNER2:9191 \
|
||||
fnproject/lb:latest
|
||||
```
|
||||
Reference in New Issue
Block a user