diff --git a/fnctl/docs/lambda/README.md b/fnctl/docs/lambda/README.md new file mode 100644 index 000000000..66ee21605 --- /dev/null +++ b/fnctl/docs/lambda/README.md @@ -0,0 +1,51 @@ +# Lambda everywhere. + +AWS Lambda introduced server-less computing to the masses. Wouldn't it be nice +if you could run the same Lambda functions on any platform, in any cloud? +Iron.io is proud to release a set of tools that allow just this. Package your +Lambda function in a Docker container and run it anywhere with an environment +similar to AWS Lambda. + +Using a job scheduler such as Iron Functions, you can connect these functions to +webhooks and run them on-demand, at scale. You can also use a container +management system paired with a task queue to run these functions in +a self-contained, platform-independent manner. + +## Use cases + +Lambda functions are great for writing "worker" processes that perform some +simple, parallelizable task like image processing, ETL transformations, +asynchronous operations driven by Web APIs, or large batch processing. + +All the benefits that containerization brings apply here. Our tools make it +easy to write containerized applications that will run anywhere without having +to fiddle with Docker and get the various runtimes set up. Instead you can just +write a simple function and have an "executable" ready to go. + +## How does it work? + +We provide base Docker images for the various runtimes that AWS Lambda +supports. The `fnclt` tool helps package up your Lambda function into +a Docker image layered on the base image. We provide a bootstrap script and +utilities that provide a AWS Lambda environment to your code. You can then run +the Docker image on any platform that supports Docker. This allows you to +easily move Lambda functions to any cloud provider, or host it yourself. + +The Docker container has to be run with a certain configuration, described +[here](./docker-configuration.md) + +## Next steps + +Write, package and run your Lambda functions with our [Getting started +guide](./getting-started.md). [Here is the environment](./environment.md) that +Lambda provides. `fnclt lambda` lists the commands to work with Lambda +functions locally. + +You can [import](./import.md) existing Lambda functions hosted on Amazon! +The Docker environment required to run Lambda functions is described +[here](./docker.md). + +Non-AWS Lambda functions can continue to interact with AWS services. [Working +with AWS](./aws.md) describes how to access AWS credentials, interact with +services like S3 and how to launch a Lambda function due a notification from +SNS. diff --git a/fnctl/docs/lambda/create.md b/fnctl/docs/lambda/create.md new file mode 100644 index 000000000..6a26aa9e1 --- /dev/null +++ b/fnctl/docs/lambda/create.md @@ -0,0 +1,22 @@ +# Creating Docker images out of Lambda functions +Docker images created by running the `create-function` subcommand on a Lambda function are ready to execute. + +You can convert any Lambda function of type nodejs 0.10, python 2.7 and Java 8 into an +IronFunction compatible Docker Image as follows: +```bash +fnctl lambda create-function +``` + +* name: the name of the created docker image which should have the format `/` +* runtime: any of the following `nodejs`, `python2.7` or `java8` +* handler: a handler takes a different form per runtime + * java8: `.::` + * python2.7: `.` + * nodejs: `.` +* file: the files to be converted, however for java8 only one file of type `jar` is allowed. + +e.g: +```bash +fnctl lambda create-function irontest/node-exec:1 nodejs node_exec.handler node_exec.js +``` + diff --git a/fnctl/docs/lambda/environment.md b/fnctl/docs/lambda/environment.md new file mode 100644 index 000000000..e5ca1c24b --- /dev/null +++ b/fnctl/docs/lambda/environment.md @@ -0,0 +1,170 @@ +# Environment + +The base images strive to provide the same environment that AWS provides to +Lambda functions. This page describes it and any incompatibilities between AWS +Lambda and Dockerized Lambda. + +## Request/Response + +IronFunctions has sync/async communication with Request/Response workflows. +* sync: returns the result as the body of the response +* async: returns the task id in the body of the response as a json + +The `context.succeed()` will not do anything with result +on node.js, nor will returning anything from a Python function. + +## Paths + +We do not make any compatibility efforts towards running your lambda function +in the same working directory as it would run on AWS. If your function makes +such assumptions, please rewrite it. + +## nodejs + +* node.js version [0.10.42][iron/node] +* ImageMagick version [6.9.3][magickv] and nodejs [wrapper 6.9.3][magickwrapperv] +* aws-sdk version [2.2.12][awsnodev] + +[iron/node]: https://github.com/iron-io/dockers/blob/master/node/Dockerfile +[magickv]: https://pkgs.alpinelinux.org/package/main/x86_64/imagemagick +[magickwrapperv]: https://www.npmjs.com/package/imagemagick +[awsnodev]: https://aws.amazon.com/sdk-for-node-js/ + +### Event + +Payloads MUST be a valid JSON object literal. + +### Context object + +* context.fail() does not currently truncate error logs. +* `context.functionName` is of the form of a docker image, for example + `iron/test-function`. +* `context.functionVersion` is always the string `"$LATEST"`. +* `context.invokedFunctionArn` is not supported. Value is empty string. +* `context.memoryLimitInMB` does not reflect reality. Value is always `256`. +* `context.awsRequestId` reflects the environment variable `TASK_ID`. On local + runs from `ironcli` this is a UUID. On IronFunctions this is the task ID. +* `logGroupName` and `logStreamName` are empty strings. +* `identity` and `clientContext` are always `null`. + +### Exceptions + +If your handler throws an exception, we only log the error message. There is no +`v8::CallSite` compatible stack trace yet. + +## Python 2.7 + +* CPython [2.7.11][pythonv] +* boto3 (Python AWS SDK) [1.2.3][botov]. + +[pythonv]: https://hub.docker.com/r/iron/python/tags/ +[botov]: https://github.com/boto/boto3/releases/tag/1.2.3 + +### Event + +Event is always a `__dict__` and the payload MUST be a valid JSON object +literal. + +### Context object + +* `context.functionName` is of the form of a docker image, for example + `iron/test-function`. +* `context.functionVersion` is always the string `"$LATEST"`. +* `context.invokedFunctionArn` is `None`. +* `context.awsRequestId` reflects the environment variable `TASK_ID` which is + set to the task ID on IronFunctions. If TASK_ID is empty, a new UUID is used. +* `logGroupName`, `logStreamName`, `identity` and `clientContext` are `None`. + +### Exceptions + +If your Lambda function throws an Exception, it will not currently be logged as +a JSON object with trace information. + +## Java 8 + +* OpenJDK Java Runtime [1.8.0][javav] + +[javav]: https://hub.docker.com/r/iron/java/tags/ + +The Java8 runtime is significantly lacking at this piont and we **do not +recommend** using it. + +### Handler types + +There are some restrictions on the handler types supported. + +#### Only a void return type is allowed + +Since Lambda does not support request/response invocation, we explicitly +prohibit a non-void return type on the handler. + +#### JSON parse error stack differences + +AWS uses the Jackson parser, this project uses the GSON parser. So JSON parse +errors will have different traces. + +#### Single item vs. List + +Given a list handler like: + +```java +public static void myHandler(List l) { + // ... +} +``` + +If the payload is a single number, AWS Lambda will succeed and pass the handler +a list with a single item. This project will raise an exception. + +#### Collections of POJOs + +This project cannot currently deserialize a List or Map containing POJOs. For +example: + +```java +public class Handler { + public static MyPOJO { + private String attribute; + public void setAttribute(String a) { + attribute = a; + } + + public String getAttribute() { + return attribute; + } + } + + public static void myHandler(List l) { + // ... + } +} +``` + +This handler invoked with the below event will fail! + +```js +[{ "attribute": "value 1"}, { "attribute": "value 2" }] +``` + +#### Leveraging predefined types is not supported + +Using the types in `aws-lambda-java-core` to [implement handlers][predef] is +untested and unsupported right now. While the package is available in your +function, we have not tried it out. + +[predef]: http://docs.aws.amazon.com/lambda/latest/dg/java-handler-using-predefined-interfaces.html + +### Logging + +The [log4j and LambdaLogger +styles](http://docs.aws.amazon.com/lambda/latest/dg/java-logging.html) that log +to CloudWatch are not supported. + +### Context object + +* `context.getFunctionName()` returns a String of the form of a docker image, + for example `iron/test-function`. +* `context.getFunctionVersion()` is always the string `"$LATEST"`. +* `context.getAwsRequestId()` reflects the environment variable `TASK_ID` which is + set to the task ID on IronFunctions. If TASK_ID is empty, a new UUID is used. +* `getInvokedFunctionArn()`, `getLogGroupName()`, `getLogStreamName()`, `getIdentity()`, `getClientContext()`, `getLogger()` return `null`. diff --git a/fnctl/docs/lambda/getting-started.md b/fnctl/docs/lambda/getting-started.md new file mode 100644 index 000000000..71b2e54cf --- /dev/null +++ b/fnctl/docs/lambda/getting-started.md @@ -0,0 +1,128 @@ +# Introduction + +This guide will walk you through creating and testing a simple Lambda function. + +We need the the `fnctl` tool for the rest of this guide. You can install it +by following [these instructions](https://github.com/iron-io/function/fnctl). + +*For this getting started we are assuming you already have working lambda function code available, if not head to the [import instructions] (import.md) and skip the next section.* + +## Creating the function + +Let's convert the `hello_world` AWS Lambda example to Docker. + +```python +def my_handler(event, context): + message = 'Hello {} {}!'.format(event['first_name'], + event['last_name']) + return { + 'message' : message + } +``` + +Create an empty directory for your project and save this code in a file called +`hello_world.py`. + +Now let's use `fnctl`'s Lambda functionality to create a Docker image. We can +then run the Docker image with a payload to execute the Lambda function. + +```sh +$ fnctl lambda create-function irontest/hello_world:1 python2.7 hello_world.my_handler hello_world.py +Creating directory: irontest/hello_world:1 ... OK +Creating Dockerfile: irontest/hello_world:1/Dockerfile ... OK +Copying file: irontest/hello_world/hello_world:1.py ... OK +Creating function.yaml ... OK +``` + +As you can see, this is very similar to creating a Lambda function using the +`aws` CLI tool. We name the function as we would name other Docker images. The +`1` indicates the version. You can use any string. This way you can configure +your deployment environment to use different versions. The handler is +the name of the function to run, in the form that python expects +(`module.function`). Where you would package the files into a `.zip` to upload +to Lambda, we just pass the list of files to `fnctl`. + +## Publishing the function to IronFunctions + +Next we want to publish the function to our IronFunctions +```sh + $ fnctl publish -v -f -d ./irontest + publishing irontest/hello_world:1/function.yaml + Sending build context to Docker daemon 4.096 kB + Step 1 : FROM iron/lambda-python2.7 + latest: Pulling from iron/lambda-python2.7 + c52e3ed763ff: Pull complete + 789cf808491a: Pull complete + d1b635efed57: Pull complete + fe23c3dbcfa8: Pull complete + 63c874a9687e: Pull complete + a6d462dae1df: Pull complete + Digest: sha256:c5dde3bf3be776c0f6b909d4ad87255a0af9b6696831fbe17c5f659655a0494a + Status: Downloaded newer image for iron/lambda-python2.7:latest + ---> 66d3adf47835 + Step 2 : ADD hello_world.py ./hello_world:1.py + ---> 91a592e0dfa9 + Removing intermediate container 1a1ef40ff0dd + Step 3 : CMD hello_world.my_handler + ---> Running in 318da1bba060 + ---> db9b9644168e + Removing intermediate container 318da1bba060 + Successfully built db9b9644168e + The push refers to a repository [docker.io/irontest/hello_world:1] + 5d9d142e21b2: Pushed + 11d8145d6038: Layer already exists + 23885f85dbd0: Layer already exists + 6a350a8d14ee: Layer already exists + e67f7ef625c5: Layer already exists + 321db514ef85: Layer already exists + 6102f0d2ad33: Layer already exists + latest: digest: sha256:5926ff413f134fa353e4b42f2d4a0d2d4f5b3a39489cfdf6dd5b4a63c4e40dee size: 1784 + updating API with appName: irontest route: /hello_world:1 image: irontest/hello_world:1 + path result + irontest/hello_world:1/function.yaml done +``` + +This will publish the generated function under the app `irontest` with `hello_world` as a route, e.g: +`http:///r/irontest/hello_world:1`, + +You should also now see the generated Docker image. + +```sh + $ docker images + REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE + irontest/hello_world:1 latest db9b9644168e About a minute ago 108.4 MB + ... +``` + +## Testing the function + +The `test-function` subcommand can launch the Dockerized function with the +right parameters. + +```sh + $ fnctl lambda test-function irontest/hello_world:1 --payload '{ "first_name": "Jon", "last_name": "Snow" }' + {"message": "Hello Jon Snow!"} +``` + +You should see the output. + +## Calling the function from IronFunctions + +The `fnctl call` command can call the published version with a given payload. + +```sh + $ echo '{ "first_name": "Jon", "last_name": "Snow" }' | ./fnctl call irontest /hello_world:1 + {"message": "Hello Jon Snow!"} +``` + +You should see the output. + + +## Commands documentation +* [create-function](create.md) +* [test-function](test.md) +* [aws-import](import.md) + +## More documentation +* [env](environment.md) +* [aws](aws.md) \ No newline at end of file diff --git a/fnctl/docs/lambda/import.md b/fnctl/docs/lambda/import.md new file mode 100644 index 000000000..7a61317ca --- /dev/null +++ b/fnctl/docs/lambda/import.md @@ -0,0 +1,48 @@ +Import existing AWS Lambda functions +==================================== + +The [fnctl](https://github.com/iron-io/functions/fnctl/) tool includes a set of +commands to act on Lambda functions. Most of these are described in +[getting-started](./getting-started.md). One more subcommand is `aws-import`. + +If you have an existing AWS Lambda function, you can use this command to +automatically convert it to a Docker image that is ready to be deployed on +other platforms. + +### Credentials + +To use this, either have your AWS access key and secret key set in config +files, or in environment variables. In addition, you'll want to set a default +region. You can use the `aws` tool to set this up. Full instructions are in the +[AWS documentation][awscli]. + +[awscli]: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files + +### Importing + +The aws-import command is constructed as follows: + +```bash +fnclt lambda aws-import +``` + +* arn: describes the ARN formats which uniquely identify the AWS lambda resource +* region: region on which the lambda is hosted +* image: the name of the created docker image which should have the format / + +Assuming you have a lambda with the following arn `arn:aws:lambda:us-west-2:123141564251:function:my-function`, the following command: +```bash +fnclt lambda aws-import arn:aws:lambda:us-west-2:123141564251:function:my-function us-east-1 user/my-function +``` +will import the function code from the region `us-east-1` to a directory called `./my-function`. It will +then create a docker image called `my-function`. + +Using Lambda with Docker Hub and IronFunctions requires that the Docker image be +named `/`. This is used to uniquely identify +images on Docker Hub. Please use the `/` as the image name with `aws-import` to create a correctly named image. + +If you only want to download the code, pass the `--download-only` flag. The + `--profile` flag is available similar to the `aws` tool to help +you tweak the settings on a command level. Finally, you can import a different version of your lambda function than the latest one +by passing `--version .` \ No newline at end of file diff --git a/fnctl/docs/lambda/test.md b/fnctl/docs/lambda/test.md new file mode 100644 index 000000000..50591c7d4 --- /dev/null +++ b/fnctl/docs/lambda/test.md @@ -0,0 +1,65 @@ +# Testing the Lambda Docker images + +The `test-function` subcommand can pass the correct parameters to `docker run` +to run those images with the payload and environment variables set up +correctly. If you would like more control, like mounting volumes, or adding +more environment variables this guide describes how to directly run these +images using: +```sh +docker run +``` + +An example of a valid `test-function` command would look as follows: +``` +fnctl lambda test-function user/my-function --payload='{"firstName":"John", "lastName":"Yo" }' +``` + +## Payload + +The payload is passed via stdin. +It is also possible to pass the payload by using the `payload` argument. Using it the payload is written to a random, opaque directory on the host. +The file itself is called `payload.json`. This directory is mapped to the +`/mnt` volume in the container, so that the payload is available in +`/mnt/payload.json`. This is not REQUIRED, since the actual runtimes use the +`PAYLOAD_FILE` environment variable to discover the payload location. + + +## Environment variables + +The `TASK_ID` variable maps to the AWS Request ID. This should be set to +something unique (a UUID, or an incrementing number). + +`test-function` runs a container with 300MB memory allocated to it. This same +information is available inside the container in the `TASK_MAXRAM` variable. +This value can be a number in bytes, or a number suffixed by `b`, `k`, `m`, `g` +for bytes, kilobytes, megabytes and gigabytes respectively. These are +case-insensitive. + +The following variables are set for AWS compatibility: +* `AWS_LAMBDA_FUNCTION_NAME` - The name of the docker image. +* `AWS_LAMBDA_FUNCTION_VERSION` - The default is `$LATEST`, but any string is + allowed. +* `AWS_ACCESS_KEY_ID` - Set this to the Access Key to allow the Lambda function + to use AWS APIs. +* `AWS_SECRET_ACCESS_KEY` - Set this to the Secret Key to allow the Lambda + function to use AWS APIs. + +## Running the container + +The default `test-function` can then be approximated as the following `docker +run` command: + +```sh +mkdir /tmp/payload_dir +echo "" | +docker run -v /tmp/payload_dir:/mnt \ + -m 1G \ + -e TASK_ID=$RANDOM \ + -e TASK_MAXRAM=1G \ + -e AWS_LAMBDA_FUNCTION_NAME=user/fancyfunction \ + -e AWS_LAMBDA_FUNCTION_VERSION=1.0 \ + -e AWS_ACCESS_KEY_ID= \ + -e AWS_SECRET_ACCESS_KEY= \ + --rm -it + user/fancyfunction +```