* POC code for inotify UDS-io-socket
* http-stream format
introducing the `http-stream` format support in fn. there are many details for
this, none of which can be linked from github :( -- docs are coming (I could
even try to add some here?). this is kinda MVP-ish level, but does not
implement the remaining spec, ie 'headers' fixing up / invoke fixing up. the
thinking being we can land this to test fdks / cli with and start splitting
work up on top of this. all other formats work the same as previous (no
breakage, only new stuff)
with the cli you can set `format: http-stream` and deploy, and then invoke a
function via the `http-stream` format. this uses unix domain socket (uds) on
the container instead of previous stdin/stdout, and fdks will have to support
this in a new fashion (will see about getting docs on here). fdk-go works,
which is here: https://github.com/fnproject/fdk-go/pull/30 . the output looks
the same as an http format function when invoking a function. wahoo.
there's some amount of stuff we can clean up here, enumerated:
* the cleanup of the sock files is iffy, high pri here
* permissions are a pain in the ass and i punted on dealing with them. you can
run `sudo ./fnserver` if running locally, it may/may not work in dind(?) ootb
* no pipe usage at all (yay), still could reduce buffer usage around the pipe
behavior, we could clean this up potentially before removal (and tests)
* my brain can’t figure out if dispatchOldFormats changes pipe behavior, but
tests work
* i marked XXX to do some clean up which will follow soon… need this to test fdk
tho so meh, any thoughts on those marked would be appreciated however (1 less
decision for me). mostly happy w/ general shape/plumbing tho
* there are no tests atm, this is a tricky dance indeed. attempts were made.
need to futz with the permission stuff before committing to adding any tests
here, which I don't like either. also, need to get the fdk-go based test image
updated according to the fdk-go, and there's a dance there too. rumba time..
* delaying the big big cleanup until we have good enough fdk support to kill
all the other formats.
open to ideas on how to maneuver landing stuff...
* fix unmount
* see if the tests work on ci...
* add call id header
* fix up makefile
* add configurable iofs opts
* add format file describing http-stream contract
* rm some cruft
* default iofs to /tmp, remove mounting
out of the box fn we can't mount. /tmp will provide a memory backed fs for us
on most systems, this will be fine for local developing and this can be
configured to be wherever for anyone that wants to make things more difficult
for themselves.
also removes the mounting, this has to be done as root. we can't do this in
the oss fn (short of requesting root, but no). in the future, we may want to
have a knob here to have a function that can be configured in fn that allows
further configuration here. since we don't know what we need in this dept
really, not doing that yet (it may be the case that it could be done
operationally outside of fn, eg, but not if each directory needs to be
configured itself, which seems likely, anyway...)
* add WIP note just in case...
* fn: paused and evicted container stats
With this change, now stats reports paused state
as well as incidents of container exit due to evictions.
* fn: update/document state transitions in state tracker
There's no case of a transition moving from done to waiting. This
must be deprecated behavior.
agent/lb-agent/runner roles execute call.End() in the background
in some cases to reduce latency. With this change, we simplify this
and switch to non-background execution of call.End(). This fixes
hard to detect issues such as non-deterministic calculation of
call.CompletedAt or incomplete Call.Stats in runners.
Downstream projects if impacted by the now blocking call.End()
latency should take steps to handle this according to their requirements.
Copies the log endpoints up to the V2 endpoints, in a similar way to
the call endpoints.
The main change is to when logs are inserted into S3. The signature of
the function has been changed to take the whole call object, rather
than just the app and call id's. This allows the function to switch
between calls for Routes and those for Fns. Obviously this switching
can be removed when v1 is removed.
In the sql implementation it inserts with both appID and fnID, this
allows the two get's to work, and the down grade of the
migration. When the v1 logs are removed, the appId can be dropped.
The log fetch test and error messages have been changed to be FnID specific.
* fn: check context timeout when waiting for non-blocking attach
With this change, we no longer allow docker client AttachToContainerNonBlocking
to block on Success channel more than our context deadline/timeout.
* fn: move nbio chan handling in attach to docker from docker-client
* fn: add httpç to Server
This allows to time limit slow/malicious clients when
reading HTTP headers. In GetBody() buffering, same timeout
can be used to time limit to give consistent I/O wait
limits for the service in addition to per handler
imposed limits we already have.
* fn: generic http Server settings for services
Currently, when the calls endpoints are disabled a 501 is
returned. While this is technically correct, it's not hard to see this
causing trouble when people tend to create 5xx roll up alerting
metrics.
This changes it to a 410, Gone, response, which is close enough and
should allow clients to know what's going on.
/fns/{fnID}/calls
/fns/{fnID}/calls/{callID}
The S3 implementation forces our hand as we if we want to list Calls
under a Fn, we have to use the FnID as a prefix on the object names,
which mean we need it to look up any Call. It also makes sense in
terms of resource hierarchy.
These endpoints can optionally be disabled (as other endpoints), if a
service provider needs to provide this functionality via other means.
The 'calls' test has been fully migrated to fn calls. This has been
done to reduce the copy pasta a bit, and on balance is ok as the
routes calls will be removed soon.
The config equality test doesn't check that keys exist between config
maps. Meaning that when two maps, both with a single, yet differing
key, with the lefts value being "", the default string, are compared they are
considered equal.
This change uses the two value assignment version of the map get,
allowing it to test for key existence.
fixes#1101
additional context:
* this was introduced in docker 1.13 (1/2017), we require docker 17.10
(10/2017), this should not have any issues dependency-wise, as `docker-init`
is in the docker install from that point in time. unless explicitly removed,
it should be in the dind container we use as well...
* the PR that introduced this to docker is
https://github.com/moby/moby/pull/26061 for additional context
* it may be wise to put this through some paces, if anybody has any...
interesting... function containers. the tests seem to work fine, however, and
this shouldn't be something users have to think about (?) at all, just
something that we are doing. this isn't the default in docker for
compatibility reasons, which is maybe a yellow flag but I am not sure tbh
* fn: introducing docker-syslog driver as default logger
With this change, fn-agent prefers RFC2454 docker-syslog driver
for logging stdout/stderr from containers. The advantage
of this is to offload it to docker itself instead of
streaming stderr along with stdout, which gets multiplexed
through single connection via docker-API.
The change will need support from FDKs in order to log
correct call-id and supress '\n' that splits syslog lines.
* update fsouza to v1.2.0
* unwind timeouts on docker
previously, we were setting our own transport on the docker client, but this
does not work anymore as fsouza now needs to call this:
https://github.com/fsouza/go-dockerclient/blob/master/client_unix.go
which makes a platform dependent client. fsouza now also appears to make a
transport that modifies the default http client with some saner values for
things like max idle conns per host (they get reaped if idle 90s):
https://github.com/fsouza/go-dockerclient/blob/master/client.go#L1059
-- these settings are sane and were why we were doing this to begin with.
additionally, have removed our setting of timeout on the docker client for 2
minutes. this is a leftover relic of a bygone era from a time when we relied
on these timeouts to timeout higher level things, which now we're properly
timing out in the enclosing methods. so, they gone, this makes the docker
client a little less whacky now.
* Insist trigger sources are prefixed
All trigger sources must have a '/' prefix to be allowed into the datastore.
* Adding condition to novelValue for gen tests
NovelValue was failing to detect same Config values correctly. This
adds a specific check for Config, like the one for Annotation, to
ensure a novel value is indeed generated.
Creates a test that aims to assert that the Equals and Clone functions
for our three entity structs actually work.
The bulk of the code is spent creating gopter generators for the entities. See information of generative or property based testing for
explainations on that topic, but basically it's an object that is
capable of creating a stream of unique instances of the given struct.
With the generator we then make three assertions:
1) Entities are always equal to themselves.
2) A .Clone() of an entity is Equal to the original entity.
3) A .Clone() of an entity that has a field modified is not equal to the
orignal.
The third property is the worse for implementation, as it does not
generate the field to modify, it simply loops all fields for each generated
entity, and checks Equals always breaks.
Break testing shows that this would have caught earlier bugs in Equals
due to field addition. It will add to the work to add further fields,
generators have to be manually specified for each field, but that
seems a worthy cost.
* Add annotations for creation of triggers and fns along with the test for them fixes#1178
* Log errors and still return created resource for annotation failures
Clone of the trigger work to inject invoke urls into the annotations
on a fn when it is returned from the server.
Small changes to trigges code following code review of the fn code.
