2341456334
this adds `FN_` in front of env vars that we are injecting into calls, for namespacing reasons. this will break code relying on the current variables but if we want to do this, the chance is now really. alternatively, we could maintain both the old and new for a short period of time to ease the adjustment (speak now...). updated the docs, as well. this also adds tests for the notoriously finicky configuration of the env vars and headers when setting up a call. this won't test the container / request for the call is actually receiving them, but it's a decent start and will yell loudly enough upon formatting breakage. added back FXLB_WAIT to a couple places so the lb can ride again one thing for feedback: headers are a bit confusing at the moment (not from this change, but that behavior is kept here for now), we've a chance to fix them. currently, headers in the request __are not__ prefixed with `FN_HEADER_`, i.e. 'hot'+sync containers will receive `Content-Length` in the http request headers, yet a 'cold' container from the same request would receive `FN_HEADER_Content-Length` in its environment. This is additionally confusing because if this function were hot+async, it would receive `FN_HEADER_Content-Length` in the headers, where just changing it to sync goes back to `Content-Length`. If that was confusing, then point made ;) I propose to remove the `FN_HEADER_` prefix for request headers in the environment, so that the request headers and env will match, as request headers already are of this format (not prefixed). please lmk thoughts here Would be fine with going back to the 'plain' vars too, then this patch will mostly just be adding tests and changing `FN_FORMAT` to `FORMAT`. obviously, from the examples, it's a bit ingrained now. anyway, entirely up to y'all.
fnlb-test-harness
Test harness that exercises the fnlb load balancer in order to verify that it works properly.
How it works
This is a test harness that makes calls to an Oracle Functions route through the fnlb load balancer, which routes traffic to multiple Oracle Functions nodes. The test harness keeps track of which node each request was routed to so we can assess how the requests are being distributed across the nodes. The functionality of fnlb is to normally route traffic to the same small number of nodes so that efficiences can be achieved and to support reuse of hot functions.
Primes function
The test harness utilizes the "primes" function, which calculates prime numbers as an excuse for consuming CPU resources. The function is invoked as follows:
curl http://host:8080/r/primesapp/primes?max=1000000&loops=1
where:
- max: calculate all primes <= max (increasing max will increase memory usage, due to the Sieve of Eratosthenes algorithm)
- loops: number of times to calculate the primes (repeating the count consumes additional CPU without consuming additional memory)
How to use it
The test harness requires running one or more Oracle Functions nodes and one instance of fnlb. The list of nodes must be provided both to fnlb and to the test harness because the test harness must call each node directly one time in order to discover the node's container id.
After it has run, examine the results to see how the requests were distributed across the nodes.
How to run it locally
Each of the Oracle Functions nodes needs to connect to the same database.
STEP 1: Create a route for the primes function. Example:
fn apps create primesapp
fn routes create primesapp /primes jconning/primes:0.0.1
STEP 2: Run five Oracle Functions nodes locally. Example (runs five nodes in the background using Docker):
sudo docker run -d -it --name functions-8082 --privileged -v ${HOME}/data-8082:/app/data -p 8082:8080 -e "DB_URL=postgres://dbUser:dbPassword@dbHost:5432/dbName" treeder/functions
sudo docker run -d -it --name functions-8083 --privileged -v ${HOME}/data-8083:/app/data -p 8083:8080 -e "DB_URL=postgres://dbUser:dbPassword@dbHost:5432/dbName" treeder/functions
sudo docker run -d -it --name functions-8084 --privileged -v ${HOME}/data-8084:/app/data -p 8084:8080 -e "DB_URL=postgres://dbUser:dbPassword@dbHost:5432/dbName" treeder/functions
sudo docker run -d -it --name functions-8085 --privileged -v ${HOME}/data-8085:/app/data -p 8085:8080 -e "DB_URL=postgres://dbUser:dbPassword@dbHost:5432/dbName" treeder/functions
sudo docker run -d -it --name functions-8086 --privileged -v ${HOME}/data-8086:/app/data -p 8086:8080 -e "DB_URL=postgres://dbUser:dbPassword@dbHost:5432/dbName" treeder/functions
STEP 3: Run fnlb locally. Example (runs fnlb on the default port 8081):
fnlb -nodes localhost:8082,localhost:8083,localhost:8084,localhost:8085,localhost:8086
STEP 4: Run the test harness. Note that the 'nodes' parameter should be the same that was used with fnlb. Example:
cd functions/test/fnlb-test-harness
go run main.go -nodes localhost:8082,localhost:8083,localhost:8084,localhost:8085,localhost:8086 -calls 10 -v
STEP 5: Examine the output to determine how many times fnlb called each node. Assess whether it is working properly.
Usage
go run main.go -help
Command line parameters:
- -calls: number of times to call the route (default 100)
- -lb: host and port of load balancer (default "localhost:8081")
- -loops: number of times to execute the primes calculation (ex: '-loops 2' means run the primes calculation twice) (default 1)
- -max: maximum number to search for primes (higher number consumes more memory) (default 1000000)
- -nodes: comma-delimited list of nodes (host:port) balanced by the load balancer (needed to discover container id of each) (default "localhost:8080")
- -route: path representing the route to the primes function (default "/r/primesapp/primes")
- -v: flag indicating verbose output
Examples: quick vs long running
Quick function:: calculate primes up to 1000
go run main.go -nodes localhost:8082,localhost:8083,localhost:8084,localhost:8085,localhost:8086 -max 1000 -v
where -max is default of 1M, -calls is default of 100, -route is default of "/r/primesapp/primes", -lb is default localhost:8081
Normal function: calculate primes up to 1M
go run main.go -nodes localhost:8082,localhost:8083,localhost:8084,localhost:8085,localhost:8086 -v
where -max is default of 1M, -calls is default of 100, -route is default of "/r/primesapp/primes", -lb is default localhost:8081
Longer running function: calculate primes up to 1M and perform the calculation ten times
go run main.go -nodes localhost:8082,localhost:8083,localhost:8084,localhost:8085,localhost:8086 -loops 10 -v
where -max is default of 1M, -calls is default of 100, -route is default of "/r/primesapp/primes", -lb is default localhost:8081
1000 calls to the route: send 1000 requests through the load balancer
go run main.go -nodes localhost:8082,localhost:8083,localhost:8084,localhost:8085,localhost:8086 -calls 1000 -v
where -max is default of 1M, -calls is default of 100, -route is default of "/r/primesapp/primes", -lb is default localhost:8081
Planned Enhancements
- Create 1000 routes and distribute calls amongst them.
- Use concurrent programming to enable the test harness to call multiple routes at the same time.