the mqs are storing a models.Task, which was not incorporating all the fields
that are in a task.Config. I would very much like to merge these two things,
but expect to do this in a future restructuring as both are used widely and
not cordoned off properly (Config has a channel, stdin, stdout, stderr -- and
isn't just a 'config', so to speak, as Task is).
Since a task.Config is what is used to actually run a container, the result of
the aforementioned deficiency was #193 where tasks are improperly configured
and ran (namely, memory wrong).
async tasks can still not be hot, they will be reverted to default format.
would also like to fix this (also part of restructuring). I actually started
doing this, hence the changes to those files (the surface area of the change
is small and discourages improper future use, so I've left what I've done).
this will:
closes#193closes#195closes#154
removes many unused fields in models.Task, since we have not implemented
retries. priority & delay are left, even though they are not used either,
the main goal of this is to resolve#193 and both these fields are strongly
plumbed into all the mqs, so punting on those two.
this changes the behavior of hot containers:
1) we are no longer populating a hot container with all of the env vars from the
first request to start up that hot container. this will only populate the
container with any vars that are defined on the app or route.
2) when env vars are changed on the route or app, we will now start up a new
hot container that contains those changes.
3) fixes a bug where we could have a collision if the image and path name
created one, e.g. `/yo/foo` & `oze/yo:latest` collides with `/yo/fo` and
`ooze/yo:latest` (if all other fields are held constant), since we're
name spacing with app name in theory it would happen to the same user (though
we were relying on a comma delimiter there, not great). now we use NULL bytes
which should be hard to get in through a json api ;) i added a sha1 to keep
the size of the (soon to be very large) map down, i don't expect collisions
but, well, it's a hash function.
a small note that we could add a few things to the hot container that will not
change on a request basis, such as `app_name`, `format` and `route` but it's a
bit pedantic. ultimately, it's confusing imo that we have a different set of
vars in the env and in the request itself for hot, which is unavoidable unless
we choose to omit setting env vars entirely, but it seems to be what the
people want (lmk, people, if otherwise).
this works by having every request from the functions server kick back a
FXLB-WAIT header on every request with the wait time for that function to
start. the lb then keeps track on a per node+function basis an ewma of the
last 10 request's wait times (to reduce jitter). now that we don't have max
concurrency it's actually pretty challenging to get the wait time stuff to
tick. i expect in the near future we will be throttling functions on a given
node in order to induce this, but that is for another day as that code needs a
lot of reworking. i tested this by introducing some arbitrary throttling (not
checked in) and load spreads over nodes correctly (see images). we will also
need to play with the intervals we want to use, as if you have a func with
50ms run time then basically 10 of those will rev up another node (this was
before removing max_c, with max_c=1) but in any event this wires in the basic
plumbing.
* make docs great again. renamed lb dir to fnlb
* added wait time to dashboard
* wires in a ready channel to await the first pull for hot images to count in
the wait time (should be otherwise useful)
future:
TODO rework lb code api to be pluggable + wire in data store
TODO toss out first data point containing pull to not jump onto another node
immediately (maybe this is actually a good thing?)
this patch gets rid of max concurrency for functions altogether, as discussed,
since it will be challenging to support across functions nodes. as a result of
doing so, the previous version of functions would fall over when offered 1000
functions, so there was some work needed in order to push this through.
further work is necessary as docker basically falls over when trying to start
enough containers at the same time, and with this patch essentially every
function can scale infinitely. it seems like we could add some kind of
adaptive restrictions based on task run length and configured wait time so
that fast running functions will line up to run in a hot container instead of
them all creating new hot containers.
this patch takes a first cut at whacking out some of the insanity that was the
previous concurrency model, which was problematic in that it limited
concurrency significantly across all functions since every task went through
the same unbuffered channel, which could create blocking issues for all
functions if the channel is not picked off fast enough (it's not apparent that
this was impossible in the previous implementation). in any event, each
request has a goroutine already, there's no reason not to use it. not too hard
to wrap a map in a lock, not sure what the benefits were (added insanity?) in effect
this is marginally easier to understand and less insane (marginally). after
getting rid of max c this adds a blocking mechanism for the first invocation
of any function so that all other hot functions will wait on the first one to
finish to avoid a herd issue (was making docker die...) -- this could be
slightly improved, but works in a pinch. reduced some memory usage by having
redundant maps of htfnsvr's and task.Requests (by a factor of 2!). cleaned up
some of the protocol stuff, need to clean this up further. anyway, it's a
first cut. have another patch that rewrites all of it but was getting into
rabbit hole territory, would be happy to oblige if anybody else has problems
understanding this rat's nest of channels. there is a good bit of work left to
make this prod ready (regardless of removing max c).
a warning that this will break the db schemas, didn't put the effort in to add
migration stuff since this isn't deployed anywhere in prod...
TODO need to clean out the htfnmgr bucket with LRU
TODO need to clean up runner interface
TODO need to unify the task running paths across protocols
TODO need to move the ram checking stuff into worker for noted reasons
TODO need better elasticity of hot f(x) containers
* functions: modify datastore to accomodate hot containers support
* functions: protocol between functions and hot containers
* functions: add hot containers clockwork
* fn: add hot containers support
* functions: add bounded concurrency
* functions: plug runners to sync and async interfaces
* functions: update documentation about the new env var
* functions: fix test flakiness
* functions: the runner is self-regulated, no need to set a number of runners
* functions: push the execution to the background on incoming requests
* functions: ensure async tasks are always on
* functions: add prioritization to tasks consumption
Ensure that Sync tasks are consumed before Async tasks. Also, fixes
termination races problems for free.
* functions: remove stale comments
* functions: improve mem availability calculation
* functions: parallel run for async tasks
* functions: check for memory availability before pulling async task
* functions: comment about rnr.hasAvailableMemory and sync.Cond
* functions: implement memory check for async runners using Cond vars
* functions: code grooming
- remove unnecessary goroutines
- fix stale docs
- reorganize import group
* Revert "functions: implement memory check for async runners using Cond vars"
This reverts commit 922e64032201a177c03ce6a46240925e3d35430d.
* Revert "functions: comment about rnr.hasAvailableMemory and sync.Cond"
This reverts commit 49ad7d52d341f12da9603b1a1df9d145871f0e0a.
* functions: set a minimum memory availability for sync
* functions: simplify the implementation by removing the priority queue
* functions: code grooming
- code deduplication
- review waitgroups Waits
