Commit Graph

17 Commits

Author SHA1 Message Date
Kyle Corbitt
54369dba54 Fix seeds and update eval field names 2023-07-17 14:14:20 -07:00
Kyle Corbitt
3b99b7bd2b remove the default value for PromptVariant.model
We should be explicit about setting the appropriate model so it always matches the constructFn.
2023-07-14 17:43:52 -07:00
arcticfly
b98eb9b729 Trigger llm output retrieval on server (#39)
* Rename tables, add graphile workers, update types

* Add dev:worker command

* Update pnpm-lock.yaml

* Remove sentry config import from worker.ts

* Stop generating new cells in cell router get query

* Generate new cells for new scenarios, variants, and experiments

* Remove most error throwing from queryLLM.task.ts

* Remove promptVariantId and testScenarioId from ModelOutput

* Remove duplicate index from ModelOutput

* Move inputHash from cell to output

* Add TODO

* Add todo

* Show cost and time for each cell

* Always show output stats if there is output

* Trigger LLM outputs when scenario variables are updated

* Add newlines to ends of files

* Add another newline

* Cascade ModelOutput deletion

* Fix linting and prettier

* Return instead of throwing for non-pending cell

* Remove pnpm dev:worker from pnpm:dev

* Update pnpm-lock.yaml
2023-07-14 16:38:46 -06:00
Kyle Corbitt
a5378b106b store model and use to calculate completion costs 2023-07-14 11:06:07 -07:00
Kyle Corbitt
4770ea34a8 Use javascript functions for prompt completions instead of templated json 2023-07-13 18:01:07 -07:00
Kyle Corbitt
f728027ef6 add evaluations 2023-07-06 13:44:03 -07:00
arcticfly
1ae5612d55 Add promptTokens and completionTokens to model output (#11)
* Default to streaming in config

* Add tokens to database

* Add NEXT_PUBLIC_SOCKET_URL to .env.example

* Disable streaming for functions

* Add newline to types
2023-07-06 13:12:59 -07:00
arcticfly
6389bd54de Save and display timeToComplete on model outputs (#6)
* Calculate and save timeToComplete on model outputs

* Add output stats to function call as well

* Record timeToComplete before parsing response json

* Add default value for timeToComplete
2023-06-30 21:29:28 -07:00
Kyle Corbitt
ab32995eb9 slightly better error handling 2023-06-27 10:48:09 -07:00
Kyle Corbitt
0646fc0ba3 lots of changes on making new experiments work 2023-06-26 18:03:26 -07:00
Kyle Corbitt
ce783f6279 do some completion caching 2023-06-26 14:44:47 -07:00
Kyle Corbitt
2b0c2ad603 editing scenarios is kinda working 2023-06-23 16:18:28 -07:00
Kyle Corbitt
bf41069442 more work on getting the editor to cache and update properly 2023-06-23 13:23:35 -07:00
Kyle Corbitt
a31c112745 we're actually calling openai 2023-06-22 17:57:21 -07:00
Kyle Corbitt
3be022786a basic db seeding 2023-06-21 15:20:33 -07:00
Kyle Corbitt
3d85984f98 more schema work 2023-06-21 13:26:51 -07:00
Kyle Corbitt
451e7b1d90 initial commit 2023-06-19 16:15:35 -07:00