Commit Graph

31 Commits

Author SHA1 Message Date
arcticfly
58892d8b63 Remove unused fields, refine model translation (#62)
* Remove unused ScenarioVariantCell fields

* Refine deriveNewConstructFn

* Fix prettier
2023-07-19 13:59:11 -07:00
Kyle Corbitt
1dcdba04a6 User accounts
Allows for the creation of user accounts. A few notes on the specifics:

 - Experiments are the main access control objects. If you can view an experiment, you can view all its prompts/scenarios/evals. If you can edit it, you can edit or delete all of those as well.
 - Experiments are owned by Organizations in the database. Organizations can have multiple members and members can have roles of ADMIN, MEMBER or VIEWER.
 - Organizations can either be "personal" or general. Each user has a "personal" organization created as soon as they try to create an experiment. There's currently no UI support for creating general orgs or adding users to them; they're just in the database to future-proof all the ACL logic.
 - You can require that a user is signed-in to see a route using the `protectedProcedure` helper. When you use `protectedProcedure`, you also have to call `ctx.markAccessControlRun()` (or delegate to a function that does it for you; see accessControl.ts). This is to remind us to actually check for access control when we define a new endpoint.
2023-07-18 21:19:03 -07:00
Kyle Corbitt
7d41e94ca2 cache eval outputs and add gpt4 eval 2023-07-17 17:55:36 -07:00
Kyle Corbitt
011b12abb9 cache output evals 2023-07-17 17:52:30 -07:00
Kyle Corbitt
54369dba54 Fix seeds and update eval field names 2023-07-17 14:14:20 -07:00
Kyle Corbitt
3b99b7bd2b remove the default value for PromptVariant.model
We should be explicit about setting the appropriate model so it always matches the constructFn.
2023-07-14 17:43:52 -07:00
arcticfly
b98eb9b729 Trigger llm output retrieval on server (#39)
* Rename tables, add graphile workers, update types

* Add dev:worker command

* Update pnpm-lock.yaml

* Remove sentry config import from worker.ts

* Stop generating new cells in cell router get query

* Generate new cells for new scenarios, variants, and experiments

* Remove most error throwing from queryLLM.task.ts

* Remove promptVariantId and testScenarioId from ModelOutput

* Remove duplicate index from ModelOutput

* Move inputHash from cell to output

* Add TODO

* Add todo

* Show cost and time for each cell

* Always show output stats if there is output

* Trigger LLM outputs when scenario variables are updated

* Add newlines to ends of files

* Add another newline

* Cascade ModelOutput deletion

* Fix linting and prettier

* Return instead of throwing for non-pending cell

* Remove pnpm dev:worker from pnpm:dev

* Update pnpm-lock.yaml
2023-07-14 16:38:46 -06:00
Kyle Corbitt
a5378b106b store model and use to calculate completion costs 2023-07-14 11:06:07 -07:00
Kyle Corbitt
4770ea34a8 Use javascript functions for prompt completions instead of templated json 2023-07-13 18:01:07 -07:00
Kyle Corbitt
46344d8fc4 small bugfixes 2023-07-07 12:22:27 -07:00
Kyle Corbitt
878cf0f68d limit seeded data 2023-07-06 16:56:26 -07:00
Kyle Corbitt
0844fb0da7 fewer reddit stories for now 2023-07-06 15:20:48 -07:00
Kyle Corbitt
f728027ef6 add evaluations 2023-07-06 13:44:03 -07:00
arcticfly
1ae5612d55 Add promptTokens and completionTokens to model output (#11)
* Default to streaming in config

* Add tokens to database

* Add NEXT_PUBLIC_SOCKET_URL to .env.example

* Disable streaming for functions

* Add newline to types
2023-07-06 13:12:59 -07:00
arcticfly
e1f05a7b59 Add seed file for reddit, restrict max height of variable value editor (#10)
* Add reddit seeds

* Restrict scenario variable value editor maxH to 60

* Add seed file for reddit experiments

* Further decrease max height

* Make score a string
2023-07-05 18:51:34 -07:00
arcticfly
6389bd54de Save and display timeToComplete on model outputs (#6)
* Calculate and save timeToComplete on model outputs

* Add output stats to function call as well

* Record timeToComplete before parsing response json

* Add default value for timeToComplete
2023-06-30 21:29:28 -07:00
Kyle Corbitt
625d34a9e0 start using migrations 2023-06-30 19:00:32 -07:00
Kyle Corbitt
7cb5f4a957 tweak default prompt wording 2023-06-28 14:23:43 -07:00
Kyle Corbitt
74d2493a1b README updates and minor tweaks 2023-06-28 11:55:25 -07:00
Kyle Corbitt
ab32995eb9 slightly better error handling 2023-06-27 10:48:09 -07:00
Kyle Corbitt
4742d9bd02 layout 2023-06-26 23:29:04 -07:00
Kyle Corbitt
0646fc0ba3 lots of changes on making new experiments work 2023-06-26 18:03:26 -07:00
Kyle Corbitt
ce783f6279 do some completion caching 2023-06-26 14:44:47 -07:00
Kyle Corbitt
5fcefad461 rip out mantine table 2023-06-23 17:03:26 -07:00
Kyle Corbitt
2b0c2ad603 editing scenarios is kinda working 2023-06-23 16:18:28 -07:00
Kyle Corbitt
bf41069442 more work on getting the editor to cache and update properly 2023-06-23 13:23:35 -07:00
Kyle Corbitt
ca78406ad1 saving prompt configs works 2023-06-23 12:18:09 -07:00
Kyle Corbitt
a31c112745 we're actually calling openai 2023-06-22 17:57:21 -07:00
Kyle Corbitt
3be022786a basic db seeding 2023-06-21 15:20:33 -07:00
Kyle Corbitt
3d85984f98 more schema work 2023-06-21 13:26:51 -07:00
Kyle Corbitt
451e7b1d90 initial commit 2023-06-19 16:15:35 -07:00