Add lambda documentation (#215)

* Add import documentation * minor fix * Add lambda introductions in the docs * Add lambda test-function docs * Add getting-started * Added lambda docs for create-function * fixes suggested by @ccirello * Add lamba environment documentation * quick fixes * use 3 ticks and language for docs * use 3 ticks and language * Add missing lambda README.md * link to iron/node in the envirtonment.md * fix getting-started.md * notate bash commands * minor cleanup * minor cleanup * minor cleanup * minor cleanup * minor cleanup * fnctl/docs: fix formatting in create.md
2022-10-28 21:29:17 +03:00 · 2016-11-06 18:13:57 +01:00
parent a61c4dab78
commit a396a26410
6 changed files with 484 additions and 0 deletions
--- a/fnctl/docs/lambda/README.md
+++ b/fnctl/docs/lambda/README.md
@@ -0,0 +1,51 @@
+# Lambda everywhere.
+
+AWS Lambda introduced server-less computing to the masses. Wouldn't it be nice
+if you could run the same Lambda functions on any platform, in any cloud?
+Iron.io is proud to release a set of tools that allow just this. Package your
+Lambda function in a Docker container and run it anywhere with an environment
+similar to AWS Lambda.
+
+Using a job scheduler such as Iron Functions, you can connect these functions to
+webhooks and run them on-demand, at scale. You can also use a container
+management system paired with a task queue to run these functions in
+a self-contained, platform-independent manner.
+
+## Use cases
+
+Lambda functions are great for writing "worker" processes that perform some
+simple, parallelizable task like image processing, ETL transformations,
+asynchronous operations driven by Web APIs, or large batch processing.
+
+All the benefits that containerization brings apply here. Our tools make it
+easy to write containerized applications that will run anywhere without having
+to fiddle with Docker and get the various runtimes set up. Instead you can just
+write a simple function and have an "executable" ready to go.
+
+## How does it work?
+
+We provide base Docker images for the various runtimes that AWS Lambda
+supports. The `fnclt` tool helps package up your Lambda function into
+a Docker image layered on the base image. We provide a bootstrap script and
+utilities that provide a AWS Lambda environment to your code. You can then run
+the Docker image on any platform that supports Docker. This allows you to
+easily move Lambda functions to any cloud provider, or host it yourself.
+
+The Docker container has to be run with a certain configuration, described
+[here](./docker-configuration.md)
+
+## Next steps
+
+Write, package and run your Lambda functions with our [Getting started
+guide](./getting-started.md). [Here is the environment](./environment.md) that
+Lambda provides. `fnclt lambda` lists the commands to work with Lambda
+functions locally.
+
+You can [import](./import.md) existing Lambda functions hosted on Amazon!
+The Docker environment required to run Lambda functions is described
+[here](./docker.md).
+
+Non-AWS Lambda functions can continue to interact with AWS services. [Working
+with AWS](./aws.md) describes how to access AWS credentials, interact with
+services like S3 and how to launch a Lambda function due a notification from
+SNS.
--- a/fnctl/docs/lambda/create.md
+++ b/fnctl/docs/lambda/create.md
@@ -0,0 +1,22 @@
+# Creating Docker images out of Lambda functions
+Docker images created by running the `create-function` subcommand on a Lambda function are ready to execute. 
+
+You can convert any Lambda function of type nodejs 0.10, python 2.7 and Java 8 into an
+IronFunction compatible Docker Image as follows:
+```bash
+fnctl lambda create-function <name> <runtime> <handler> <files...>
+```
+
+* name: the name of the created docker image which should have the format `<username>/<image-name>`
+* runtime: any of the following `nodejs`, `python2.7` or `java8`
+* handler: a handler takes a different form per runtime
+    * java8: `<namespace>.<class>::<handler>`
+    * python2.7:  `<filename>.<handler>`
+    * nodejs: `<filename>.<handler>`
+* file: the files to be converted, however for java8 only one file of type `jar` is allowed.
+
+e.g:
+```bash
+fnctl lambda create-function irontest/node-exec:1 nodejs node_exec.handler node_exec.js
+```
+
--- a/fnctl/docs/lambda/environment.md
+++ b/fnctl/docs/lambda/environment.md
@@ -0,0 +1,170 @@
+# Environment
+
+The base images strive to provide the same environment that AWS provides to
+Lambda functions. This page describes it and any incompatibilities between AWS
+Lambda and Dockerized Lambda.
+
+## Request/Response
+
+IronFunctions has sync/async communication with Request/Response workflows.
+* sync: returns the result as the body of the response
+* async: returns the task id in the body of the response as a json
+
+The `context.succeed()` will not do anything with result
+on node.js, nor will returning anything from a Python function.
+
+## Paths
+
+We do not make any compatibility efforts towards running your lambda function
+in the same working directory as it would run on AWS. If your function makes
+such assumptions, please rewrite it.
+
+## nodejs
+
+* node.js version [0.10.42][iron/node]
+* ImageMagick version [6.9.3][magickv] and nodejs [wrapper 6.9.3][magickwrapperv]
+* aws-sdk version [2.2.12][awsnodev]
+
+[iron/node]: https://github.com/iron-io/dockers/blob/master/node/Dockerfile
+[magickv]: https://pkgs.alpinelinux.org/package/main/x86_64/imagemagick
+[magickwrapperv]: https://www.npmjs.com/package/imagemagick
+[awsnodev]: https://aws.amazon.com/sdk-for-node-js/
+
+### Event
+
+Payloads MUST be a valid JSON object literal.
+
+### Context object
+
+* context.fail() does not currently truncate error logs.
+* `context.functionName` is of the form of a docker image, for example
+  `iron/test-function`.
+* `context.functionVersion` is always the string `"$LATEST"`.
+* `context.invokedFunctionArn` is not supported. Value is empty string.
+* `context.memoryLimitInMB` does not reflect reality. Value is always `256`.
+* `context.awsRequestId` reflects the environment variable `TASK_ID`. On local
+  runs from `ironcli` this is a UUID. On IronFunctions this is the task ID.
+* `logGroupName` and `logStreamName` are empty strings.
+* `identity` and `clientContext` are always `null`.
+
+### Exceptions
+
+If your handler throws an exception, we only log the error message. There is no
+`v8::CallSite` compatible stack trace yet.
+
+## Python 2.7
+
+* CPython [2.7.11][pythonv]
+* boto3 (Python AWS SDK) [1.2.3][botov].
+
+[pythonv]: https://hub.docker.com/r/iron/python/tags/
+[botov]: https://github.com/boto/boto3/releases/tag/1.2.3
+
+### Event
+
+Event is always a `__dict__` and the payload MUST be a valid JSON object
+literal.
+
+### Context object
+
+* `context.functionName` is of the form of a docker image, for example
+  `iron/test-function`.
+* `context.functionVersion` is always the string `"$LATEST"`.
+* `context.invokedFunctionArn` is `None`.
+* `context.awsRequestId` reflects the environment variable `TASK_ID` which is
+  set to the task ID on IronFunctions. If TASK_ID is empty, a new UUID is used.
+* `logGroupName`, `logStreamName`, `identity` and `clientContext` are `None`.
+
+### Exceptions
+
+If your Lambda function throws an Exception, it will not currently be logged as
+a JSON object with trace information.
+
+## Java 8
+
+* OpenJDK Java Runtime [1.8.0][javav]
+
+[javav]: https://hub.docker.com/r/iron/java/tags/
+
+The Java8 runtime is significantly lacking at this piont and we **do not
+recommend** using it.
+
+### Handler types
+
+There are some restrictions on the handler types supported.
+
+#### Only a void return type is allowed
+
+Since Lambda does not support request/response invocation, we explicitly
+prohibit a non-void return type on the handler.
+
+#### JSON parse error stack differences
+
+AWS uses the Jackson parser, this project uses the GSON parser. So JSON parse
+errors will have different traces.
+
+#### Single item vs. List
+
+Given a list handler like:
+
+```java
+public static void myHandler(List<Double> l) {
+    // ...
+}
+```
+
+If the payload is a single number, AWS Lambda will succeed and pass the handler
+a list with a single item. This project will raise an exception.
+
+#### Collections of POJOs
+
+This project cannot currently deserialize a List or Map containing POJOs. For
+example:
+
+```java
+public class Handler {
+  public static MyPOJO {
+    private String attribute;
+    public void setAttribute(String a) {
+      attribute = a;
+    }
+
+    public String getAttribute() {
+      return attribute;
+    }
+  }
+
+  public static void myHandler(List<MyPOJO> l) {
+    // ...
+  }
+}
+```
+
+This handler invoked with the below event will fail!
+
+```js
+[{ "attribute": "value 1"}, { "attribute": "value 2" }]
+```
+
+#### Leveraging predefined types is not supported
+
+Using the types in `aws-lambda-java-core` to [implement handlers][predef] is
+untested and unsupported right now. While the package is available in your
+function, we have not tried it out.
+
+[predef]: http://docs.aws.amazon.com/lambda/latest/dg/java-handler-using-predefined-interfaces.html
+
+### Logging
+
+The [log4j and LambdaLogger
+styles](http://docs.aws.amazon.com/lambda/latest/dg/java-logging.html) that log
+to CloudWatch are not supported.
+
+### Context object
+
+* `context.getFunctionName()` returns a String of the form of a docker image,
+  for example `iron/test-function`.
+* `context.getFunctionVersion()` is always the string `"$LATEST"`.
+* `context.getAwsRequestId()` reflects the environment variable `TASK_ID` which is
+  set to the task ID on IronFunctions. If TASK_ID is empty, a new UUID is used.
+* `getInvokedFunctionArn()`, `getLogGroupName()`, `getLogStreamName()`, `getIdentity()`, `getClientContext()`, `getLogger()` return `null`.
--- a/fnctl/docs/lambda/getting-started.md
+++ b/fnctl/docs/lambda/getting-started.md
@@ -0,0 +1,128 @@
+# Introduction
+
+This guide will walk you through creating and testing a simple Lambda function.
+
+We need the the `fnctl` tool for the rest of this guide. You can install it
+by following [these instructions](https://github.com/iron-io/function/fnctl).
+
+*For this getting started we are assuming you already have working lambda function code available, if not head to the [import instructions] (import.md) and skip the next section.*
+
+## Creating the function
+
+Let's convert the `hello_world` AWS Lambda example to Docker.
+
+```python
+def my_handler(event, context):
+    message = 'Hello {} {}!'.format(event['first_name'], 
+                                    event['last_name'])  
+    return { 
+        'message' : message
+    }  
+```
+
+Create an empty directory for your project and save this code in a file called
+`hello_world.py`.
+
+Now let's use `fnctl`'s Lambda functionality to create a Docker image. We can
+then run the Docker image with a payload to execute the Lambda function.
+
+```sh
+$ fnctl lambda create-function irontest/hello_world:1 python2.7 hello_world.my_handler hello_world.py
+Creating directory: irontest/hello_world:1 ... OK
+Creating Dockerfile: irontest/hello_world:1/Dockerfile ... OK
+Copying file: irontest/hello_world/hello_world:1.py ... OK
+Creating function.yaml ... OK
+```
+
+As you can see, this is very similar to creating a Lambda function using the
+`aws` CLI tool. We name the function as we would name other Docker images. The
+`1` indicates the version. You can use any string. This way you can configure
+your deployment environment to use different versions. The handler is
+the name of the function to run, in the form that python expects
+(`module.function`). Where you would package the files into a `.zip` to upload
+to Lambda, we just pass the list of files to `fnctl`.
+
+## Publishing the function to IronFunctions
+
+Next we want to publish the function to our IronFunctions
+```sh
+    $ fnctl publish -v -f -d ./irontest
+    publishing irontest/hello_world:1/function.yaml
+    Sending build context to Docker daemon 4.096 kB
+    Step 1 : FROM iron/lambda-python2.7
+    latest: Pulling from iron/lambda-python2.7
+    c52e3ed763ff: Pull complete 
+    789cf808491a: Pull complete 
+    d1b635efed57: Pull complete 
+    fe23c3dbcfa8: Pull complete 
+    63c874a9687e: Pull complete 
+    a6d462dae1df: Pull complete 
+    Digest: sha256:c5dde3bf3be776c0f6b909d4ad87255a0af9b6696831fbe17c5f659655a0494a
+    Status: Downloaded newer image for iron/lambda-python2.7:latest
+    ---> 66d3adf47835
+    Step 2 : ADD hello_world.py ./hello_world:1.py
+    ---> 91a592e0dfa9
+    Removing intermediate container 1a1ef40ff0dd
+    Step 3 : CMD hello_world.my_handler
+    ---> Running in 318da1bba060
+    ---> db9b9644168e
+    Removing intermediate container 318da1bba060
+    Successfully built db9b9644168e
+    The push refers to a repository [docker.io/irontest/hello_world:1]
+    5d9d142e21b2: Pushed 
+    11d8145d6038: Layer already exists 
+    23885f85dbd0: Layer already exists 
+    6a350a8d14ee: Layer already exists 
+    e67f7ef625c5: Layer already exists 
+    321db514ef85: Layer already exists 
+    6102f0d2ad33: Layer already exists 
+    latest: digest: sha256:5926ff413f134fa353e4b42f2d4a0d2d4f5b3a39489cfdf6dd5b4a63c4e40dee size: 1784
+    updating API with appName: irontest route: /hello_world:1 image: irontest/hello_world:1 
+    path                                    result
+    irontest/hello_world:1/function.yaml     done
+```
+
+This will publish the generated function under the app `irontest` with `hello_world` as a route, e.g:
+`http://<hostname>/r/irontest/hello_world:1`,
+
+You should also now see the generated Docker image.
+
+```sh
+    $ docker images
+    REPOSITORY                TAG         IMAGE ID            CREATED              VIRTUAL SIZE
+    irontest/hello_world:1      latest      db9b9644168e        About a minute ago   108.4 MB
+    ...
+```
+
+## Testing the function
+
+The `test-function` subcommand can launch the Dockerized function with the
+right parameters.
+
+```sh
+    $ fnctl lambda test-function irontest/hello_world:1 --payload '{ "first_name": "Jon", "last_name": "Snow" }'
+    {"message": "Hello Jon Snow!"}
+```
+
+You should see the output. 
+
+## Calling the function from IronFunctions
+
+The `fnctl call` command can call the published version with a given payload.
+
+```sh
+    $ echo  '{ "first_name": "Jon", "last_name": "Snow" }' | ./fnctl call irontest /hello_world:1
+    {"message": "Hello Jon Snow!"}
+```
+
+You should see the output. 
+
+
+## Commands documentation
+* [create-function](create.md)
+* [test-function](test.md)
+* [aws-import](import.md)
+
+## More documentation
+* [env](environment.md)
+* [aws](aws.md)
--- a/fnctl/docs/lambda/import.md
+++ b/fnctl/docs/lambda/import.md
@@ -0,0 +1,48 @@
+Import existing AWS Lambda functions
+====================================
+
+The [fnctl](https://github.com/iron-io/functions/fnctl/) tool includes a set of
+commands to act on Lambda functions. Most of these are described in
+[getting-started](./getting-started.md). One more subcommand is `aws-import`.
+
+If you have an existing AWS Lambda function, you can use this command to
+automatically convert it to a Docker image that is ready to be deployed on
+other platforms.
+
+### Credentials
+
+To use this, either have your AWS access key and secret key set in config
+files, or in environment variables. In addition, you'll want to set a default
+region. You can use the `aws` tool to set this up. Full instructions are in the
+[AWS documentation][awscli].
+
+[awscli]: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files
+
+### Importing
+
+The aws-import command is constructed as follows:
+
+```bash
+fnclt lambda aws-import <arn> <region> <image>
+```
+
+* arn: describes the ARN formats which uniquely identify the AWS lambda resource
+* region: region on which the lambda is hosted
+* image: the name of the created docker image which should have the format <username>/<image-name>
+
+Assuming you have a lambda with the following arn `arn:aws:lambda:us-west-2:123141564251:function:my-function`, the following command:
+```bash
+fnclt lambda aws-import arn:aws:lambda:us-west-2:123141564251:function:my-function us-east-1 user/my-function
+```
+will import the function code from the region `us-east-1` to a directory called `./my-function`. It will
+then create a docker image called `my-function`.
+
+Using Lambda with Docker Hub and IronFunctions requires that the Docker image be
+named `<Docker Hub username>/<image name>`. This is used to uniquely identify
+images on Docker Hub. Please use the `<Docker Hub username>/<image
+name>` as the image name with `aws-import` to create a correctly named image.
+
+If you only want to download the code, pass the `--download-only` flag. The
+ `--profile` flag is available similar to the `aws` tool to help
+you tweak the settings on a command level. Finally, you can import a different version of your lambda function than the latest one
+by passing `--version <version>.`
--- a/fnctl/docs/lambda/test.md
+++ b/fnctl/docs/lambda/test.md
@@ -0,0 +1,65 @@
+# Testing the Lambda Docker images
+
+The `test-function` subcommand can pass the correct parameters to `docker run`
+to run those images with the payload and environment variables set up
+correctly. If you would like more control, like mounting volumes, or adding
+more environment variables this guide describes how to directly run these
+images using:
+```sh
+docker run
+```
+
+An example of a valid `test-function` command would look as follows:
+```
+fnctl lambda test-function user/my-function --payload='{"firstName":"John", "lastName":"Yo" }'
+```
+
+## Payload
+
+The payload is passed via stdin.
+It is also possible to pass the payload by using the `payload` argument. Using it the payload is written to a random, opaque directory on the host.
+The file itself is called `payload.json`. This directory is mapped to the
+`/mnt` volume in the container, so that the payload is available in
+`/mnt/payload.json`. This is not REQUIRED, since the actual runtimes use the
+`PAYLOAD_FILE` environment variable to discover the payload location.
+
+
+## Environment variables
+
+The `TASK_ID` variable maps to the AWS Request ID. This should be set to
+something unique (a UUID, or an incrementing number).
+
+`test-function` runs a container with 300MB memory allocated to it. This same
+information is available inside the container in the `TASK_MAXRAM` variable.
+This value can be a number in bytes, or a number suffixed by `b`, `k`, `m`, `g`
+for bytes, kilobytes, megabytes and gigabytes respectively. These are
+case-insensitive.
+
+The following variables are set for AWS compatibility:
+* `AWS_LAMBDA_FUNCTION_NAME` - The name of the docker image.
+* `AWS_LAMBDA_FUNCTION_VERSION` - The default is `$LATEST`, but any string is
+  allowed.
+* `AWS_ACCESS_KEY_ID` - Set this to the Access Key to allow the Lambda function
+  to use AWS APIs.
+* `AWS_SECRET_ACCESS_KEY` - Set this to the Secret Key to allow the Lambda
+  function to use AWS APIs.
+
+## Running the container
+
+The default `test-function` can then be approximated as the following `docker
+run` command:
+
+```sh
+mkdir /tmp/payload_dir
+echo "<payload>" |
+docker run -v /tmp/payload_dir:/mnt \
+           -m 1G \
+           -e TASK_ID=$RANDOM \
+           -e TASK_MAXRAM=1G \
+           -e AWS_LAMBDA_FUNCTION_NAME=user/fancyfunction \
+           -e AWS_LAMBDA_FUNCTION_VERSION=1.0 \
+           -e AWS_ACCESS_KEY_ID=<access key> \
+           -e AWS_SECRET_ACCESS_KEY=<secret key> \
+           --rm -it
+           user/fancyfunction
+```