mirror of
https://github.com/fnproject/fn.git
synced 2022-10-28 21:29:17 +03:00
56a2861748
* move calls to logstore, implement s3 closes #482 the basic motivation is that logs and calls will be stored with a very high write rate, while apps and routes will be relatively infrequently updated; it follows that we should likely split up their storage location, to back them with appropriate storage facilities. s3 is a good candidate for ingesting higher write rate data than a sql database, and will make it easier to manage that data set. can read #482 for more detailed justification. summary: * calls api moved from datastore to logstore * logstore used in front-end to serve calls endpoints * agent now throws calls into logstore instead of datastore * s3 implementation of calls api for logstore * s3 logs key changed (nobody using / nbd?) * removed UpdateCall api (not in use) * moved call tests from datastore to logstore tests * mock logstore now tested (prev. sqlite3 only) * logstore tests run against every datastore (mysql, pg; prev. only sqlite3) * simplify NewMock in tests commentary: brunt of the work is implementing the listing of calls in GetCalls for the s3 logstore implementation. the GetCalls API requires returning items in the newest to oldest order, and the s3 api lists items in lexicographic order based on created_at. An easy thing to do here seemed to be to reverse the encoding of our id format to return a lexicographically descending order, since ids are time based, reasonably encoded to be lexicographically sortable, and de-duped (unlike created_at). This seems to work pretty well, it's not perfect around the boundaries of to_time and from_time and a tiny amount of results may be omitted, but to me this doesn't seem like a deal breaker to get 6999 results instead of 7000 when trying to get calls between 3:00pm and 4:00pm Monday 3 weeks ago. Of course, without to_time and from_time, there are no issues in listing results. We could use created at and encode it, but it would be an additional marker for point lookup (GetCall) since we would have to search for a created_at stamp, search for ids around that until we find the matching one, just to do a point lookup. So, the tradeoff here seems worth it. There is additional optimization around to_time to seek over newer results (since we have descending order). The other complication in GetCalls is returning a list of calls for a given path. Since the keys to do point lookups are only app_id + call_id, and we need listing across an app as well, this leads us to the 'marker' collection which is sorted by app_id + path + call_id, to allow quick listing by path. All in all, it should be pretty straightforward to follow the implementation and I tried to be lavish with the comments, please let me know if anything needs further clarification in the code. The implementation itself has some glaring inefficiencies, but they're relatively minute: json encoding is kinda lazy, but workable; s3 doesn't offer batch retrieval, so we point look up each call one by one in get call; not re-using buffers -- but the seeking around the keys should all be relatively fast, not too worried about performance really and this isn't a hot path for reads (need to make a cut point and turn this in!). Interestingly, in testing, minio performs significantly worse than pg for storing both logs and calls (or just logs, I tested that too). minio seems to have really high cpu consumption, but in any event, we won't be using minio, we'll be using a cloud object store that implements the s3 api. Anyway, mostly a knock on using minio for high performance, not really anything to do with this, just thought it was interesting. I think it's safe to remove UpdateCall, admittedly this made implementing the s3 api a lot easier. This operation may also be something we never need, it was unused at present and was only in the cards for a previous hybrid implementation, which we've now abandoned. If we need, we can always resurrect from git. Also not worried about changing the log key, we need to put a prefix on this thing anyway, but I don't think anybody is using this anyway. in any event, it simply means old logs won't show up through the API, but aside from nobody using this yet, that doesn't seem a big deal breaker really -- new logs will appear fine. future: TODO make logstore implementation optional for datastore, check in front-end at runtime and offer a nil logstore that errors appropriately TODO low hanging fruit optimizations of json encoding, re-using buffers for download, get multiple calls at a time, id reverse encoding could be optimized like normal encoding to not be n^2 TODO api for range removal of logs and calls * address review comments * push id to_time magic into id package * add note about s3 key sizes * fix validation check
Quickstart | Tutorials | Docs | API | Operating | Flow | UI
Welcome
Fn is an event-driven, open source, Functions-as-a-Service (FaaS) compute platform that you can run anywhere. Some of its key features:
- Open Source
- Native Docker: use any Docker container as your Function
- Supports all languages
- Run anywhere
- Public, private and hybrid cloud
- Import Lambda functions and run them anywhere
- Easy to use for developers
- Easy to manage for operators
- Written in Go
- Simple yet powerful extensibility
The fastest way to experience Fn is to follow the quickstart below, or you can jump right to our full documentation, API Docs, or hit us up in our Slack Community!
Quickstart
Pre-requisites
- Docker 17.10.0-ce or later installed and running
- A Docker Hub account (Docker Hub) (or other Docker-compliant registry)
- Log Docker into your Docker Hub account:
docker login
Install CLI tool
The command line tool isn't required, but it sure makes things a lot easier. There are a few options to install it:
1. Homebrew - MacOS
If you're on a Mac and use Homebrew, this one is for you:
brew install fn
2. Shell script - Linux and MacOS
This one works on Linux and MacOS (partially on Windows):
curl -LSs https://raw.githubusercontent.com/fnproject/cli/master/install | sh
This will download a shell script and execute it. If the script asks for a password, that is because it invokes sudo.
3. Download the bin - Linux, MacOS and Windows
Head over to our releases and download it.
Run Fn Server
Now fire up an Fn server:
fn start
This will start Fn in single server mode, using an embedded database and message queue. You can find all the configuration options here. If you are on Windows, check here. If you are on a Linux system where the SELinux security policy is set to "Enforcing", such as Oracle Linux 7, check here.
Your First Function
Functions are small but powerful blocks of code that generally do one simple thing. Forget about monoliths when using functions, just focus on the task that you want the function to perform. Our CLI tool will help you get started super quickly.
Create hello world function:
fn init --runtime go hello
This will create a simple function in the directory hello, so let's cd into it:
cd hello
Feel free to check out the files it created or just keep going and look at it later.
# Set your Docker Hub username
export FN_REGISTRY=<DOCKERHUB_USERNAME>
# Run your function locally
fn run
# Deploy your functions to your local Fn server
fn deploy --app myapp --local
Now you can call your function:
curl http://localhost:8080/r/myapp/hello
# or:
fn call myapp /hello
Or in a browser: http://localhost:8080/r/myapp/hello
That's it! You just deployed your first function and called it. Try updating the function code in func.go then deploy it again to see the change.
Learn More
- With our Fn Getting Started Series, quickly create Fn Hello World applications in multiple languages. This is a great Fn place to start!
- Visit Fn tutorials for step by step guides to creating apps with Fn . These tutorials range from introductory to more advanced.
- See our full documentation
- View all of our examples
- View our YouTube Channel
- View our API Docs
- Check out our sub-projects: Flow, UI, FnLB
- For a full presentation with lots of content you can use in your own presentations, see The Fn Project Presentation Master
Get Help
- Ask your question on StackOverflow and tag it with
fn - Join our Slack Community
Get Involved
- Join our Slack Community
- Learn how to contribute
- See issues for issues you can help with