alihan/fn-serverless
1
0
Fork 0
mirror of https://github.com/fnproject/fn.git synced 2022-10-28 21:29:17 +03:00
Code Issues Packages Projects Releases Wiki Activity

hang the runner, agent=new sheriff (#270)

Browse Source 
* fix docker build

this is trivially incorrect since glide doesn't actually provide reproducible
builds. the idea is to build with the deps that we have checked into git, so
that we actually know what code is executing so that we might debug it...

all for multi stage build instead of what we had, but adding the glide step is
wrong. i added a loud warning so as to discourage this behavior in the future.

* hang the runner, agent=new sheriff

tl;dr agent is now runner, with a hopefully saner api

the general idea is get rid of all the various 'task' structs now, change our
terminology to only be 'calls' now, push a lot of the http construction of a
call into the agent, allow calls to mutate their state around their execution
easily and to simplify the number of code paths, channels and context timeouts
in something [hopefully] easy to understand.

this introduces the idea of 'slots' which are either hot or cold and are
separate from reserving memory (memory is denominated in 'tokens' now).
a 'slot' is essentially a container that is ready for execution of a call, be
it hot or cold (it just means different things based on hotness). taking a
look into Submit should make these relatively easy to grok.

sorry, things were pretty broken especially wrt timings. I tried to keep good
notes (maybe too good), to highlight stuff so that we don't make the same
mistakes again (history repeating itself blah blah quote). even now, there is
lots of work to do :)

I encourage just reading the agent.go code, Submit is really simple and
there's a description of how the whole thing works at the head of the file
(after TODOs). call.go contains code for constructing calls, as well as Start
/ End (small atm). I did some amount of code massaging to try to make things
simple / straightforward / fit reasonable mental model, but as always am open
to critique (the more negative the better) as I'm just one guy and wth do i
know...

-----------------------------------------------------------------------------

below enumerates a number of changes as briefly as possible (heh..):

models.Call all the things

removes models.Task as models.Call is now what it previously was.
models.FnCall is now rid of in favor of models.Call, despite the datastore
only storing a few fields of it [for now]. we should probably store entire
calls in the db, since app & route configurations can change at any given
moment, it would be nice to see the parameters of each call (costs db space,
obviously).

this removes the endpoints for getting & deleting messages, we were just
looping back to localhost to call the MQ (wtf? this was for iron integration i
think) and just calls the MQ.

changes the name of the FnLog to LogStore, confusing cause there's also a
`FuncLogger` which uses the Logstore (punting). removes other `Fn` prefixed
structs (redundant naming convention).

removes some unused and/or weird structs (IDStatus, CompleteTime)

updates the swagger

makes the db methods consistent to use 'Call' nomenclature.

remove runner nuisances:

* push down registry stuff to docker driver
* remove Environment / Stats stuff of yore
* remove unused writers (now in FuncLogger)
* remove 2 of the task types, old hot stuff, runner, etc

fixes ram available calculation on startup to not always be 300GB (helps a lot
on a laptop!)

format for DOCKER_AUTH env now is not a list but a map (there are no docs,
would prefer to get rid of this altogether anyway). the ~/.docker/cfg expected
format is unchanged.

removes arbitrary task queue, if a machine is out of ram we can probably just
time out without queueing... (can open separate discussion) in any case the
old one didn't really account well for hot tasks, it just lined everyone up in
the task queue if there wasn't a place to run hot and then timed them out
[even if a slot became free].

removes HEADER_ prefixing on any headers in the request to a invoke a call.
(this was inconsistent with cli for test anyway)

removes TASK_ID header sent in to hot only (this is a dupe of FN_CALL_ID,
which has not been removed)

now user functions can reply directly to the client. this means that for
cold containers if they write to stdout it will send a 200 + headers. for
hot containers, the user can reply directly to the client from the container,
i.e. with its preferred status code / headers (vs. always getting a 200).
the dispatch itself is a little http specific atm, i think we can add an
interchange format but the current version is easily extended to add json for
now, separate discussion. this eliminates a lot of the request/response
rewriting and buffering we were doing (yey). now Dispatch ONLY does input and
output, vs. managing the call timeout and having access to a call's fields.

cache is pushed down into agent now instead of in the front end, would like to
push it down to the datastore actually but it's here for now anyway. cache
delete functions removed (b/c fn is distributed anyway?). added app caching,
should help with latency.

in general, a lot of server/runner.go got pushed down into the agent. i think
it will be useful in testing to be able to construct calls without having to
invoke http handlers + async also needs to construct calls without a handler.

safe shutdown actually works now for everything (leaked / didn't wait on
certain things before)

now we're waiting for hot slots to open up while we're attempting to get ram
to launch a container if we didn't find any hot slots to run the call in
immediately. we can change this policy really easily now (no more channel
jungle; still some channels). also looking for somewhere else to go while the
container is launching now. slots now get sent _out_ of a container, vs.
a container receiving calls, which makes this kind of policy easier to
implement. this fixes a number of bugs around things like trying to execute
calls against containers that have not and may never start and trying to
launch a bazillion containers when there are no free containers. the driver api
underwent some changes to make this possible (relatively minimal, added Wait).
the easiest way to think about this is that allocating ram has moved 'up'
instead of just wrapping launching containers, so that we can select on a
channel trying to find ram.

not dispatching hot calls to containers that died anymore either...

the timeout is now started at the beginning of Submit, rather than Dispatch or
the container itself having to manage the call timeout, which was an
inaccurate way of doing things since finding a slot / allocating ram / pulling
image can all take a non-trivial (timeout amount, even!) amount of time. this
makes for much more reasonable response times from fn under load, there's
still a little TODO about handling cold+timeout container removal response
times but it's much improved.

if call.Start is called with < call.timeout/2 time left, then the call will
not be executed and return a timeout. we can discuss. this makes async play
_a lot_ nicer, specifically. for large timeouts / 2 makes less sense.

env is no longer getting upper cased (admittedly, this can look a little weird
now). our whole route.Config/app.Config/env/headers stuff probably deserves a
whole discussion...

sync output no longer has the call id in json if there's an error / timeout.
we could add this back to signify that it's _us_ writing these but this was
out of place. FN_CALL_ID is still shipped out to get the id for sync calls,
and async [server] output remains unchanged.

async logs are now an entire raw http request (so that a user can write a 400
or something from their hot async container)

async hot now 'just works'

cold sync calls can now reply to the client before container removal, which
shaves a lot of latency off of those (still eat start). still need to figure
out async removal if timeout or something.

-----------------------------------------------------------------------------

i've located a number of bugs that were generally inherited, and also added
a number of TODOs in the head of the agent.go file according to robustness we
probably need to add. this is at least at parity with the previous
implementation, to my knowledge (hopefully/likely a good bit ahead). I can
memorialize these to github quickly enough, not that anybody searches before
adding bugs anyway (sigh).

the big thing to work on next imo is async being a lot more robust,
specifically to survive fn server failures / network issues.

thanks for review (gulp)
This commit is contained in:
Reed Allman
2017-09-05 10:32:51 -07:00
committed by Denis Makogon
parent 1b1b64436f
commit 71a88a991c
100 changed files with 2151 additions and 4121 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
api/models/app.go
View File

@@ -1,9 +1,5 @@
package models
type Apps []*App
type Tasks []*Task
type FnCalls []*FnCall
type App struct {
Name string `json:"name"`
Routes Routes `json:"routes,omitempty"`

134
api/models/call.go Normal file
View File

@@ -0,0 +1,134 @@
package models
import (
strfmt "github.com/go-openapi/strfmt"
)
const (
// TypeNone ...
TypeNone = ""
// TypeSync ...
TypeSync = "sync"
// TypeAsync ...
TypeAsync = "async"
)
const (
// FormatDefault ...
FormatDefault = "default"
// FormatHTTP ...
FormatHTTP = "http"
)
var possibleStatuses = [...]string{"delayed", "queued", "running", "success", "error", "cancelled"}
type CallLog struct {
CallID string `json:"call_id"`
Log string `json:"log"`
}
// Call is a representation of a specific invocation of a route.
type Call struct {
// Unique identifier representing a specific call.
ID string `json:"id"`
// NOTE: this is stale, retries are not implemented atm, but this is nice, so leaving
// States and valid transitions.
//
// +---------+
// +---------> delayed <----------------+
// +----+----+ |
// | |
// | |
// +----v----+ |
// +---------> queued <----------------+
// +----+----+ *
// | *
// | retry * creates new call
// +----v----+ *
// | running | *
// +--+-+-+--+ |
// +---------|-|-|-----+-------------+
// +---|---------+ | +-----|---------+ |
// | | | | | |
// +-----v---^-+ +--v-------^+ +--v---^-+
// | success | | cancelled | | error |
// +-----------+ +-----------+ +--------+
//
// * delayed - has a delay.
// * queued - Ready to be consumed when it's turn comes.
// * running - Currently consumed by a runner which will attempt to process it.
// * success - (or complete? success/error is common javascript terminology)
// * error - Something went wrong. In this case more information can be obtained
// by inspecting the "reason" field.
// - timeout
// - killed - forcibly killed by worker due to resource restrictions or access
// violations.
// - bad_exit - exited with non-zero status due to program termination/crash.
// * cancelled - cancelled via API. More information in the reason field.
// - client_request - Request was cancelled by a client.
Status string `json:"status"`
// App this call belongs to.
AppName string `json:"app_name"`
// Path of the route that is responsible for this call
Path string `json:"path"`
// Name of Docker image to use.
Image string `json:"image"`
// Number of seconds to wait before queueing the call for consumption for the
// first time. Must be a positive integer. Calls with a delay start in state
// "delayed" and transition to "running" after delay seconds.
Delay int32 `json:"delay,omitempty"`
// Type indicates whether a task is to be run synchronously or asynchronously.
Type string `json:"type,omitempty"`
// Format is the format to pass input into the function.
Format string `json:"format,omitempty"`
// Payload for the call. This is only used by async calls, to store their input.
// TODO should we copy it into here too for debugging sync?
Payload string `json:"payload,omitempty"`
// Full request url that spawned this invocation.
URL string `json:"url,omitempty"`
// Method of the http request used to make this call.
Method string `json:"method,omitempty"`
// Priority of the call. Higher has more priority. 3 levels from 0-2. Calls
// at same priority are processed in FIFO order.
Priority *int32 `json:"priority"`
// Maximum runtime in seconds.
Timeout int32 `json:"timeout,omitempty"`
// Hot function idle timeout in seconds before termination.
IdleTimeout int32 `json:"idle_timeout,omitempty"`
// Memory is the amount of RAM this call is allocated.
Memory uint64 `json:"memory,omitempty"`
// BaseEnv are the env vars for hot containers, not request specific.
BaseEnv map[string]string `json:"base_env,omitempty"`
// Env vars for the call. Comes from the ones set on the Route.
EnvVars map[string]string `json:"env_vars,omitempty"`
// Time when call completed, whether it was successul or failed. Always in UTC.
CompletedAt strfmt.DateTime `json:"completed_at,omitempty"`
// Time when call was submitted. Always in UTC.
CreatedAt strfmt.DateTime `json:"created_at,omitempty"`
// Time when call started execution. Always in UTC.
StartedAt strfmt.DateTime `json:"started_at,omitempty"`
}
type CallFilter struct {
Path string
AppName string
}

27
api/models/complete.go
View File

@@ -1,27 +0,0 @@
package models
import "github.com/go-openapi/strfmt"
/*Complete complete
swagger:model Complete
*/
type Complete struct {
/* Time when task was completed. Always in UTC.
*/
CompletedAt strfmt.DateTime `json:"completed_at,omitempty"`
/* Error message, if status=error. Only used by the /error endpoint.
*/
Error string `json:"error,omitempty"`
/* Machine readable reason failure, if status=error. Only used by the /error endpoint.
*/
Reason string `json:"reason,omitempty"`
}
// Validate validates this complete
func (m *Complete) Validate(formats strfmt.Registry) error {
return nil
}

22
api/models/datastore.go
View File

@@ -38,11 +38,11 @@ type Datastore interface {
GetRoute(ctx context.Context, appName, routePath string) (*Route, error)
// GetRoutes gets a slice of Routes, optionally filtered by filter.
GetRoutes(ctx context.Context, filter *RouteFilter) (routes []*Route, err error)
GetRoutes(ctx context.Context, filter *RouteFilter) ([]*Route, error)
// GetRoutesByApp gets a slice of routes for a appName, optionally filtering on filter (filter.AppName is ignored).
// Returns ErrDatastoreEmptyAppName if appName is empty.
GetRoutesByApp(ctx context.Context, appName string, filter *RouteFilter) (routes []*Route, err error)
GetRoutesByApp(ctx context.Context, appName string, filter *RouteFilter) ([]*Route, error)
// InsertRoute inserts a route. Returns ErrDatastoreEmptyRoute when route is nil, and ErrDatastoreEmptyAppName
// or ErrDatastoreEmptyRoutePath for empty AppName or Path.
@@ -57,13 +57,19 @@ type Datastore interface {
// ErrDatastoreEmptyRoutePath when routePath is empty. Returns ErrRoutesNotFound when no route exists.
RemoveRoute(ctx context.Context, appName, routePath string) error
// InsertTask inserts a task
InsertTask(ctx context.Context, task *Task) error
GetTask(ctx context.Context, callID string) (*FnCall, error)
GetTasks(ctx context.Context, filter *CallFilter) (FnCalls, error)
// InsertCall inserts a call into the datastore, it will error if the call already
// exists.
InsertCall(ctx context.Context, call *Call) error
// Implement FnLog methods for convenience
FnLog
// GetCall returns a call at a certain id and app name.
GetCall(ctx context.Context, appName, callID string) (*Call, error)
// GetCalls returns a list of calls that satisfy the given CallFilter. If no
// calls exist, an empty list and a nil error are returned.
GetCalls(ctx context.Context, filter *CallFilter) ([]*Call, error)
// Implement LogStore methods for convenience
LogStore
// GetDatabase returns the underlying sqlx database implementation
GetDatabase() *sqlx.DB

6
api/models/error.go
View File

@@ -16,7 +16,7 @@ var (
code: http.StatusBadRequest,
error: errors.New("Invalid JSON"),
}
ErrRunnerTimeout = err{
ErrCallTimeout = err{
code: http.StatusGatewayTimeout,
error: errors.New("Timed out"),
}
@@ -72,9 +72,9 @@ var (
code: http.StatusBadRequest,
error: errors.New("Missing key"),
}
ErrDatastoreEmptyTaskID = err{
ErrDatastoreEmptyCallID = err{
code: http.StatusBadRequest,
error: errors.New("Missing task ID"),
error: errors.New("Missing call ID"),
}
ErrInvalidPayload = err{
code: http.StatusBadRequest,

102
api/models/id_status.go
View File

@@ -1,102 +0,0 @@
package models
import (
"encoding/json"
strfmt "github.com/go-openapi/strfmt"
"github.com/go-openapi/swag"
"github.com/go-openapi/validate"
)
// TODO get rid of this. id and status are not more coupled than anything else?
// burn it at the stake
/*IDStatus Id status
swagger:model IdStatus
*/
type IDStatus struct {
/* Unique identifier representing a specific task.
Read Only: true
*/
ID string `json:"id,omitempty"`
/* States and valid transitions.
+---------+
+---------> delayed <----------------+
+----+----+ |
| |
| |
+----v----+ |
+---------> queued <----------------+
+----+----+ *
| *
| retry * creates new task
+----v----+ *
| running | *
+--+-+-+--+ |
+---------|-|-|-----+-------------+
+---|---------+ | +-----|---------+ |
| | | | | |
+-----v---^-+ +--v-------^+ +--v---^-+
| success | | cancelled | | error |
+-----------+ +-----------+ +--------+
* delayed - has a delay.
* queued - Ready to be consumed when it's turn comes.
* running - Currently consumed by a runner which will attempt to process it.
* success - (or complete? success/error is common javascript terminology)
* error - Something went wrong. In this case more information can be obtained
by inspecting the "reason" field.
- timeout
- killed - forcibly killed by worker due to resource restrictions or access
violations.
- bad_exit - exited with non-zero status due to program termination/crash.
* cancelled - cancelled via API. More information in the reason field.
- client_request - Request was cancelled by a client.
Read Only: true
*/
Status string `json:"status,omitempty"`
}
// Validate validates this Id status
func (m *IDStatus) Validate(formats strfmt.Registry) error { return m.validateStatus(formats) }
var idStatusTypeStatusPropEnum []interface{}
// prop value enum
func (m *IDStatus) validateStatusEnum(path, location string, value string) error {
if idStatusTypeStatusPropEnum == nil {
var res []string
if err := json.Unmarshal([]byte(`["delayed","queued","running","success","error","cancelled"]`), &res); err != nil {
return err
}
for _, v := range res {
idStatusTypeStatusPropEnum = append(idStatusTypeStatusPropEnum, v)
}
}
if err := validate.Enum(path, location, value, idStatusTypeStatusPropEnum); err != nil {
return err
}
return nil
}
func (m *IDStatus) validateStatus(formats strfmt.Registry) error {
if swag.IsZero(m.Status) { // not required
return nil
}
// value enum
if err := m.validateStatusEnum("status", "body", m.Status); err != nil {
return err
}
return nil
}

8
api/models/logs.go
View File

@@ -4,14 +4,18 @@ import (
"context"
)
type FnLog interface {
type LogStore interface {
// TODO TODO TODO BAD BUG BUG BUG WILL ROBINSON
// TODO these need to take an app name or users can provide ids for
// other users calls with their own app name and access their logs.
// InsertLog will insert the log at callID, overwriting if it previously
// existed.
InsertLog(ctx context.Context, callID string, callLog string) error
// GetLog will return the log at callID, an error will be returned if the log
// cannot be found.
GetLog(ctx context.Context, callID string) (*FnCallLog, error)
GetLog(ctx context.Context, callID string) (*CallLog, error)
// DeleteLog will remove the log at callID, it will not return an error if
// the log does not exist before removal.

20
api/models/mq.go
View File

@@ -3,18 +3,18 @@ package models
import "context"
// Message Queue is used to impose a total ordering on jobs that it will
// execute in order. Tasks are added to the queue via the Push() interface. The
// execute in order. calls are added to the queue via the Push() interface. The
// MQ must support a reserve-delete 2 step dequeue to allow implementing
// timeouts and retries.
//
// The Reserve() operation must return a job based on this total ordering
// (described below). At this point, the MQ backend must start a timeout on the
// job. If Delete() is not called on the Task within the timeout, the Task should
// job. If Delete() is not called on the call within the timeout, the call should
// be restored to the queue.
//
// Total ordering: The queue should maintain an ordering based on priority and
// logical time. Priorities are currently 0-2 and available in the Task's
// priority field. Tasks with higher priority always get pulled off the queue
// logical time. Priorities are currently 0-2 and available in the call's
// priority field. call with higher priority always get pulled off the queue
// first. Within the same priority, jobs should be available in FIFO order.
// When a job is required to be restored to the queue, it should maintain it's
@@ -22,12 +22,12 @@ import "context"
// the head of the queue:
// Reserve() leads to A being passed to a consumer, and timeout started.
// Next Reserve() leads to B being dequeued. This consumer finishes running the
// task, leading to Delete() being called. B is now permanently erased from the
// call, leading to Delete() being called. B is now permanently erased from the
// queue.
// A's timeout occurs before the job is finished. At this point the ordering
// should be [A, C] and not [C, A].
type MessageQueue interface {
// Push a Task onto the queue. If any error is returned, the Task SHOULD not be
// Push a call onto the queue. If any error is returned, the call SHOULD not be
// queued. Note that this does not completely avoid double queueing, that is
// OK, a check against the datastore will be performed after a dequeue.
//
@@ -37,18 +37,16 @@ type MessageQueue interface {
// delays. That is, if jobs {A, C} are queued at t seconds, both with Delay
// = 5 seconds, and the same priority, then they may be available on the
// queue as [C, A] or [A, C].
Push(context.Context, *Task) (*Task, error)
Push(context.Context, *Call) (*Call, error)
// Remove a job from the front of the queue, reserve it for a timeout and
// return it. MQ implementations MUST NOT lose jobs in case of errors. That
// is, in case of reservation failure, it should be possible to retrieve the
// job on a future reservation.
Reserve(context.Context) (*Task, error)
Reserve(context.Context) (*Call, error)
// If a reservation is pending, consider it acknowledged and delete it. If
// the job does not have an outstanding reservation, error. If a job did not
// exist, succeed.
Delete(context.Context, *Task) error
Delete(context.Context, *Call) error
}
type Enqueue func(context.Context, MessageQueue, *Task) (*Task, error)

8
api/models/route.go
View File

@@ -8,8 +8,8 @@ import (
)
const (
defaultRouteTimeout = 30 // seconds
htfnScaleDownTimeout = 30 // seconds
DefaultRouteTimeout = 30 // seconds
DefaultIdleTimeout = 30 // seconds
)
type Routes []*Route
@@ -50,11 +50,11 @@ func (r *Route) SetDefaults() {
}
if r.Timeout == 0 {
r.Timeout = defaultRouteTimeout
r.Timeout = DefaultRouteTimeout
}
if r.IdleTimeout == 0 {
r.IdleTimeout = htfnScaleDownTimeout
r.IdleTimeout = DefaultIdleTimeout
}
}

19
api/models/start.go
View File

@@ -1,19 +0,0 @@
package models
import "github.com/go-openapi/strfmt"
/*Start start
swagger:model Start
*/
type Start struct {
/* Time when task started execution. Always in UTC.
*/
StartedAt strfmt.DateTime `json:"started_at,omitempty"`
}
// Validate validates this start
func (m *Start) Validate(formats strfmt.Registry) error {
return nil
}

106
api/models/task.go
View File

@@ -1,106 +0,0 @@
package models
import (
strfmt "github.com/go-openapi/strfmt"
)
const (
// TypeNone ...
TypeNone = ""
// TypeSync ...
TypeSync = "sync"
// TypeAsync ...
TypeAsync = "async"
)
const (
// FormatDefault ...
FormatDefault = "default"
// FormatHTTP ...
FormatHTTP = "http"
)
// TODO this should either be Task, or should be removed in favor of Task
type FnCall struct {
IDStatus
CompletedAt strfmt.DateTime `json:"completed_at,omitempty"`
CreatedAt strfmt.DateTime `json:"created_at,omitempty"`
StartedAt strfmt.DateTime `json:"started_at,omitempty"`
AppName string `json:"app_name,omitempty"`
Path string `json:"path"`
}
type FnCallLog struct {
CallID string `json:"call_id"`
Log string `json:"log"`
}
func (fnCall *FnCall) FromTask(task *Task) *FnCall {
return &FnCall{
CreatedAt: task.CreatedAt,
StartedAt: task.StartedAt,
CompletedAt: task.CompletedAt,
AppName: task.AppName,
Path: task.Path,
IDStatus: IDStatus{
ID: task.ID,
Status: task.Status,
},
}
}
// Task is a representation of a specific invocation of a route.
type Task struct {
IDStatus
// App this task belongs to.
AppName string `json:"app_name"`
// Path of the route that is responsible for this task
Path string `json:"path"`
// Name of Docker image to use.
Image string `json:"image"`
// Number of seconds to wait before queueing the task for consumption for the first time. Must be a positive integer. Tasks with a delay start in state "delayed" and transition to "running" after delay seconds.
Delay int32 `json:"delay,omitempty"`
// Payload for the task. This is only used by async tasks, to store their input.
Payload string `json:"payload,omitempty"`
// Priority of the task. Higher has more priority. 3 levels from 0-2. Tasks at same priority are processed in FIFO order.
Priority *int32 `json:"priority"`
// Maximum runtime in seconds.
Timeout int32 `json:"timeout,omitempty"`
// Hot function idle timeout in seconds before termination.
IdleTimeout int32 `json:"idle_timeout,omitempty"`
// Memory is the amount of RAM this task is allocated.
Memory uint64 `json:"memory,omitempty"`
// BaseEnv are the env vars for hot containers, not request specific.
BaseEnv map[string]string `json:"base_env,omitempty"`
// Env vars for the task. Comes from the ones set on the Route.
EnvVars map[string]string `json:"env_vars,omitempty"`
// Format is the format to pass input into the function.
// TODO plumb this in async land
// Format string `json:"format,omitempty"`
// Time when task completed, whether it was successul or failed. Always in UTC.
CompletedAt strfmt.DateTime `json:"completed_at,omitempty"`
// Time when task was submitted. Always in UTC.
CreatedAt strfmt.DateTime `json:"created_at,omitempty"`
// Time when task started execution. Always in UTC.
StartedAt strfmt.DateTime `json:"started_at,omitempty"`
}
type CallFilter struct {
Path string
AppName string
}