alihan/fn-serverless
1
0
Fork 0
mirror of https://github.com/fnproject/fn.git synced 2022-10-28 21:29:17 +03:00
Code Issues Packages Projects Releases Wiki Activity

Updated docs, cleaned things up, DRY's up function format stuff. (#688)

Browse Source 
* Updated docs, cleaned things up, DRY's up function format stuff.

* deleted files

* updated bad link
This commit is contained in:
Travis Reeder
2018-01-16 14:45:44 -08:00
committed by GitHub
parent e3a573d11a
commit 5a2602d42e
17 changed files with 164 additions and 352 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
docs/developers/async.md Normal file
View File

@@ -0,0 +1,18 @@
# Asynchronous Functions
Asynchronous (async) functions will run your function at some point in the future. The default mode is synchronous which means
a function is executed and the caller blocks while waiting for the response. Asynchronous on the other, puts the request into a
message queue and responds immediately to the caller. The function will then be executed at some point in the future, upon resource availability giving priority
to synchronous calls. Also, since it is using a message queue, you can safely queue up millions of function calls without worrying about
capacity.
Async will return immediately with a `call_id`, for example:
```json
{"call_id": "abc123"}
```
The `call_id` can then be used to retrieve the status at a later time.
Asynchronous function calls are great for tasks that are CPU heavy or take more than a few seconds to complete.
For instance, image processing, video processing, data processing, ETL, etc.

13
docs/developers/clients.md
View File

@@ -1,15 +1,4 @@
# Client Libraries
## Function Developer Kits - FDKs
Language wrappers to help write functions:
* [Go](https://github.com/fnproject/fdk-go)
* [Java](https://github.com/fnproject/fdk-java)
* [Python](https://github.com/fnproject/fdk-python)
* [Ruby](https://github.com/fnproject/fdk-ruby)
## Client libraries for using the API
# Client Libraries for the API
* [Elixir](https://github.com/fnproject/fn_elixir)
* [Go](https://github.com/fnproject/fn_go)

2
docs/developers/configs.md
View File

@@ -11,7 +11,7 @@ fn apps config set myapp LOG_LEVEL debug
## 2. Function configuration from func.yaml
See [Function file](../function-file.md) for more info.
See [Function file](function-file.md) for more info.
## 3. Route level configuration

8
docs/developers/fdks.md Normal file
View File

@@ -0,0 +1,8 @@
# Function Developer Kits - FDKs
FDKs helps make writing functions easier.
* [Go](https://github.com/fnproject/fdk-go)
* [Java](https://github.com/fnproject/fdk-java)
* [Python](https://github.com/fnproject/fdk-python)
* [Ruby](https://github.com/fnproject/fdk-ruby)

78
docs/developers/function-file.md Normal file
View File

@@ -0,0 +1,78 @@
# Function files
Functions files are used to assist fn to help you when creating functions.
An example of a function file:
```yaml
name: fnproject/hello
version: 0.0.1
type: sync
memory: 128
config:
key: value
key2: value2
keyN: valueN
headers:
Content-Type: text/plain
build:
- make
- make test
expects:
config:
- name: SECRET_1
required: true
- name: SECRET_2
required: false
```
`name` is the name and tag to which this function will be pushed to and the
route updated to use it.
`path` (optional) allows you to overwrite the calculated route from the path
position. You may use it to override the calculated route. If you plan to use
`fn test --remote=""`, this is mandatory.
`version` represents current version of the function. When deploying, it is
appended to the image as a tag.
`runtime` represents programming language runtime, for example,
'go', 'python', 'java', etc. The runtime 'docker' will use the existing Dockerfile if one exists.
`build` (optional) is an array of local shell calls which are used to help
building the function. TODO: Deprecate this?
`type` (optional) allows you to set the type of the route. `sync`, for functions
whose response are sent back to the requester; or `async`, for functions that
are started and return a call ID to customer while it executes in background.
Default: `sync`.
`memory` (optional) allows you to set a maximum memory threshold for this
function. If this function exceeds this limit during execution, it is stopped
and error message is logged. Default: `128`.
`cpus` (optional) is the amount of available CPU cores for this function. For example, `100m` or `0.1`
+will allow the function to consume at most 1/10 of a CPU core on the running machine. It
+expects to be a string in MilliCPUs format ('100m') or floating-point number ('0.5').
+Default: unlimited.
`timeout` (optional) is the maximum time a function will be allowed to run. Default is 30 seconds.
`headers` (optional) is a set of HTTP headers to be returned in the response of
this function calls.
`config` (optional) is a set of configuration variables to be passed onto the function as environment variables.
These configuration options shall override application configuration during functions execution. See [Configuration](configs.md)
for more information.
`expects` (optional) a list of config/env vars that are required to run this function. These vars will be used when running/testing locally,
if found in your local environment. If these vars are not found, local testing will fail.
## Hot functions
hot functions support also adds two extra options to this configuration file.
`format` (optional) is one of the streaming formats covered at [function-format.md](function-format.md).
`idle_timeout` (optional) is the time in seconds a container will remain alive without receiving any new requests;
hot functions will stay alive as long as they receive a request in this interval. Default: `30`.

222
docs/developers/function-format.md Normal file
View File

@@ -0,0 +1,222 @@
# Open Function Format
This document will describe the details of how a function works, inputs/outputs, etc.
## I/O Formats
### STDIN and Environment Variables
While wanting to keep things simple, flexible and expandable, we decided to go back to the basics, using Unix input and output. Standard in is easy to use in any language and doesn't require anything extra. It also allows streaming input so we can do things like keeping a container running some time and stream requests into the container.
Configuration values, environment information and other things will be passed in through environment variables.
The goals of the I/O formats are the following:
* Very easy to use and parse
* Supports hot for increasing performance (more than one call per container execution)
* Ability to build higher level abstractions on top (ie: Lambda syntax compatible)
The format is still up for discussion and in order to move forward and remain flexible, it's likely we will just allow different I/O formats and the function creator can decide what they want, on a per function basis. Default being the simplest format to use.
The way that input data is supplied to functions depends on the input format (as specified in `func.yml`) that your function is using.
#### Environment Variables used by All Formats
Your function has access to a set of environment variables, independent of
the function's format:
* `FN_APP_NAME` - the name of the application that matched this route, eg: `myapp`
* `FN_PATH` - the matched route, eg: `/hello`
* `FN_METHOD` - the HTTP method for the request, eg: `GET` or `POST`
* `FN_FORMAT` - a string representing one of the [function formats](function-format.md), currently either `default` or `http`. Default is `default`.
* `FN_TYPE` - the type for this call, currently 'sync' or 'async'
* `FN_MEMORY` - a number representing the amount of memory available to the call, in MB
* `FN_CPUS` - a string representing the amount of CPU available to the call, in MilliCPUs or floating-point number, eg. `100m` or `0.1`. Header is present only if `cpus` is set for the route.
### Default I/O Format
The default format is simply the request body itself plus some environment variables. For instance, if someone were to post a JSON body, the unmodified body would be sent in via STDIN. The result comes via STDOUT. When task is done, pipes are closed and the container running the function is terminated.
#### Default Format Env Vars
For `default` format, the following environment variables will be available:
* `FN_DEADLINE` - RFC3339 time stamp of the expiration (deadline) date of function execution.
* `FN_REQUEST_URL` - the full URL for the request ([parsing example](https://github.com/fnproject/fn/tree/master/examples/tutorial/params))
* `FN_CALL_ID` - a unique ID for each function execution.
* `FN_METHOD` - http method used to invoke this function
* `FN_HEADER_$X` - the HTTP headers that were set for this request. Replace $X with the upper cased name of the header and replace dashes in the header with underscores.
* `$X` - $X is the header that came in the http request that invoked this function.
#### Pros/Cons
Pros:
* Very simple to use
Cons:
* Not very efficient resource utilization - one new container execution per event.
### JSON I/O Format
`format: json`
The JSON format is a nice hot format as it is easy to parse in most languages.
If a request comes in with the following body:
```json
{
"some": "input"
}
```
then, the input will be:
#### Input
Internally functions receive data in the example format below:
```json
{
"call_id": "123",
"content_type": "application/json",
"body": "{\"some\":\"input\"}",
"protocol": {
"type": "http",
"request_url": "http://localhost:8080/r/myapp/myfunc?q=hi",
"headers": {
"Content-Type": ["application/json"],
"Other-Header": ["something"]
}
}
}
BLANK LINE
{
NEXT INPUT OBJECT
}
```
* call_id - the unique ID for the call.
* content_type - format of the `body` parameter.
* protocol - arbitrary map of protocol specific data. The above example shows what the HTTP protocol handler passes in. Subject to change and reduces reusability of your functions. **USE AT YOUR OWN RISK**.
TODO: Add config map
Under `protocol`, `headers` contains all of the HTTP headers exactly as defined in the incoming request.
Each request will be separated by a blank line.
#### Output
Function's output format should have the following format:
```json
{
"body": "{\"some\":\"output\"}",
"content_type": "application/json",
"protocol": {
"status_code": 200,
"headers": {
"Other-Header": ["something"]
}
}
}
BLANK LINE
{
NEXT OUTPUT OBJECT
}
```
* body - required - the response body.
* content_type - optional - format of `body`. Default is application/json.
* protocol - optional - protocol specific response options. Entirely optional. Contents defined by each protocol.
#### Pros/Cons of JSON format
Pros:
* Supports hot format
* Easy to parse
Cons:
* Not streamable
### HTTP I/O Format
`format: http`
HTTP format could be a good option as it is in very common use obviously, most languages have some semi-easy way to parse it, and it supports hot format. The response will look like a HTTP response. The communication is still done via stdin/stdout, but these pipes are never closed unless the container is explicitly terminated.
#### Request
```text
GET / HTTP/1.1
Content-Length: 5
world
```
#### Input
The input to the function will be in standard HTTP format, similar to the incoming request, but with the
following additional headers:
* `Fn_deadline` - RFC3339 time stamp of the expiration (deadline) date of function execution.
* `Fn_request_url` - the full URL for the request ([parsing example](https://github.com/fnproject/fn/tree/master/examples/tutorial/params))
* `Fn_call_id` - a unique ID for each function execution.
* `Fn_method` - the HTTP method used to invoke
* `$X` - the HTTP headers that were set for this request, exactly as they were sent in the request.
#### Output
Your function should output the exact response in HTTP format you'd like to be returned to the client:
```text
HTTP/1.1 200 OK
Content-Length: 11
hello world
```
#### Pros/Cons of HTTP Format
Pros:
* Supports streaming
* Common format
Cons:
* Requires a parsing library or fair amount of code to parse headers properly
* Double parsing - headers + body (if body is to be parsed, such as json)
## Logging
Standard out is where you should write response data for synchronous functions. Standard error
is where you should write for logging, as [it was intended](http://www.jstorimer.com/blogs/workingwithcode/7766119-when-to-use-stderr-instead-of-stdout).
And if you use a log collector like logspout, you can collect those logs in a central location. See [logging](../operating/logging.md).
So to write output to logs, simply log to STDERR. Here are some examples in a few languages.
In Go, simply use the [log](https://golang.org/pkg/log/) package, it writes to STDERR by default.
```go
log.Println("hi")
```
In Node.js:
```node
console.error("hi");
```
[More details for Node.js here](http://stackoverflow.com/a/27576486/105562).
In Ruby:
```ruby
STDERR.puts("hi")
```

58
docs/developers/function-timeouts.md Normal file
View File

@@ -0,0 +1,58 @@
# Function timeouts
Within Function API, each functions supplied with 2 timeouts parameters, both optional, see [swagger.yaml](swagger.yml) for more details.
So, what are those timeouts and what are they used for?
## Function call timeout
This time of timeouts defines for how long function execution may happen before it'll be terminated along with notifying caller that function terminated with error - timed out.
```json
{
"route":{
...
"timeout": 30,
...
}
}
```
This timeout parameter used with both types of functions: async and sync.
It starts at the beginning of function call.
## Hot function idle timeout
This type of timeout defines for how long should hot function hang around before its termination in case if there are no incoming requests.
```json
{
"route":{
...
"idle_timeout": 30,
...
}
}
```
This timeout parameter is valid for hot functions, see what [hot functions](developers/hot-functions.md) is. By default this parameter equals to 30 seconds.
It starts after last request being processed by hot function.
## Correlation between idle and regular timeout
This two timeouts are independent. The order of timeouts for hot functions:
0. start hot function be sending first timeout-bound request to it
1. make request to function with `timeout`
2. if call finished (no matter successful or not) check for more requests to dispatch
3. if none - start idle timeout
4. if new request appears - stop idle timeout and serve request
5. if none - terminate hot function
## Hot function idle timeout edge cases
Having both timeouts may cause confusion while configuring hot function.
So, there are certain limitations for `idle_timeout` as well as for regular `timeout`:
* Idle timeout might be equal to zero. Such case may lead to satiation when function would be terminated immediately after last request processing, i.e. no idle timeout at all.
* Idle timeout can't be negative.
* Idle timeout can't be changed while hot function is running. Idle timeout is permanent within hot function execution lifecycle. It means that idle timeout should be considered for changing once functions is not running.

15
docs/developers/hot-functions.md Normal file
View File

@@ -0,0 +1,15 @@
# Hot functions
By default, Fn uses "cold functions" where every request starts up a new container, feeds it with the payload then sends the
answer back to the caller. You can expect an average start time of [300ms per execution]((https://medium.com/travis-on-docker/the-overhead-of-docker-run-f2f06d47c9f3#.96tj75ugb)) to start the function/container.
Hot functions improve performance by starting a function then keeping it alive to handle additional requests. This
makes it a bit trickier to use because you'll have to parse a stream of requests, but if you use an [FDK](fdks.md) it's
all taken care of for you, so you should probably use an FDK in most cases.
## Making a hot function
In your `func.yaml`, add `format: json`.
From there, we recommend using one of our [FDKs](fdks.md) to handle all the parsing and formatting. But if you'd like to learn
about the nitty gritty details, [check here](function-format.md).

44
docs/developers/packaging.md Normal file
View File

@@ -0,0 +1,44 @@
# Packaging your Function
## Option 1 (recommended): Use the `fn` cli tool
We recommend using the [fn cli tool](../fn/README.md) which will handle all of this for you. But if you'd like to dig in
and customize your images, look at Option 2.
## Option 2: Build your own images
Packaging a function has two parts:
* Create a Docker image for your function with an ENTRYPOINT
* Push your Docker image to a registry (Docker Hub by default)
Once it's pushed to a registry, you can use the image location when adding a route.
### Creating an image
The basic Dockerfile for most languages is along these lines:
```
# Choose base image
FROM node:alpine
# Set the working directory
WORKDIR /function
# Add your binary or code to the working directory
ADD . /function/
# Set what will run when a container is started for this image
ENTRYPOINT ["node func.js"]
```
Then build your function image:
```sh
fn build
```
### Push your image
```sh
fn push
```
Now you can use that image when creating or updating routes.

79
docs/developers/testing.md Normal file
View File

@@ -0,0 +1,79 @@
# Testing Functions
`fn` has testing built in that allows you to create inputs and expected outputs and verify the expected output with actual output.
## Write a Test File
Create a file called `test.json` in your functions directory (beside your `func.yaml` file). Here's a simple example:
```json
{
"tests": [
{
"input": {
"body": {
"name": "Johnny"
}
},
"output": {
"body": {
"message": "Hello Johnny"
}
}
},
{
"input": {
"body": ""
},
"output": {
"body": {
"message": "Hello World"
}
}
}
]
}
```
The example above has two tests, one with the following input:
```json
{
"name": "Johnny"
}
```
and a second one with no input.
The first one is expected to return a json response with the following:
```json
{
"message": "Hello Johnny"
}
```
And the second should return:
```json
{
"message": "Hello World"
}
```
## Run Tests
This is simply running:
```sh
fn test
```
in your function directory.
You can also test against a remote `fn` server by using the `--remote` flag. eg:
```sh
fn test --remote myapp
```

128
docs/developers/usage.md Normal file
View File

@@ -0,0 +1,128 @@
# Detailed Usage
This is a more detailed explanation of the main commands you'll use in Fn as a developer.
### Create an Application
An application is essentially a grouping of functions, that put together, form an API. Here's how to create an app.
```sh
fn apps create myapp
```
Or using a cURL:
```sh
curl -H "Content-Type: application/json" -X POST -d '{
"app": { "name":"myapp" }
}' http://localhost:8080/v1/apps
```
[More on apps](apps.md).
Now that we have an app, we can route endpoints to functions.
### Add a Route
A route is a way to define a path in your application that maps to a function. In this example, we'll map
`/hello` to a simple `Hello World!` function called `fnproject/hello` which is a function we already made that you
can use -- yes, you can share functions! The source code for this function is in the [examples directory](/examples/hello/go).
```sh
fn routes create myapp /hello -i fnproject/hello
```
Or using cURL:
```sh
curl -H "Content-Type: application/json" -X POST -d '{
"route": {
"path":"/hello",
"image":"fnproject/hello"
}
}' http://localhost:8080/v1/apps/myapp/routes
```
[More on routes](../operating/routes.md).
### Calling your Function
Calling your function is as simple as requesting a URL. Each app has its own namespace and each route mapped to the app.
The app `myapp` that we created above along with the `/hello` route we added would be called via the following
URL: http://localhost:8080/r/myapp/hello
Either surf to it in your browser or use `fn`:
```sh
fn call myapp /hello
```
Or using a cURL:
```sh
curl http://localhost:8080/r/myapp/hello
```
### Passing data into a function
Your function will get the body of the HTTP request via STDIN, and the headers of the request will be passed in
as env vars. You can test a function with the CLI tool:
```sh
echo '{"name":"Johnny"}' | fn call myapp /hello
```
Or using cURL:
```sh
curl -H "Content-Type: application/json" -X POST -d '{
"name":"Johnny"
}' http://localhost:8080/r/myapp/hello
```
You should see it say `Hello Johnny!` now instead of `Hello World!`.
### Add an asynchronous function
FN supports synchronous function calls like we just tried above, and asynchronous for background processing.
[Asynchronous functions](async.md) are great for tasks that are CPU heavy or take more than a few seconds to complete.
For instance, image processing, video processing, data processing, ETL, etc.
Architecturally, the main difference between synchronous and asynchronous is that requests
to asynchronous functions are put in a queue and executed on upon resource availability so that they do not interfere with the fast synchronous responses required for an API.
Also, since it uses a message queue, you can queue up millions of function calls without worrying about capacity as requests will
just be queued up and run at some point in the future.
To add an asynchronous function, create another route with the `"type":"async"`, for example:
```sh
curl -H "Content-Type: application/json" -X POST -d '{
"route": {
"type": "async",
"path":"/hello-async",
"image":"fnproject/hello"
}
}' http://localhost:8080/v1/apps/myapp/routes
```
or set `type: async` in your `func.yaml`.
Now if you request this route:
```sh
curl -H "Content-Type: application/json" -X POST -d '{
"name":"Johnny"
}' http://localhost:8080/r/myapp/hello-async
```
You will get a `call_id` in the response:
```json
{"call_id":"572415fd-e26e-542b-846f-f1f5870034f2"}
```
If you watch the logs, you will see the function actually runs in the background:
![async log](/docs/assets/async-log.png)
Read more on [logging](../operating/logging.md).