* Make CompareFunctions more configurable
* Change RefinePromptModal styles
* Accept newModel in getModifiedPromptFn
* Show prompt comparison in SelectModelModal
* Pass variant to SelectModelModal
* Update instructions
* Properly use isDisabled
Adds a `modelProvider` field to `promptVariants`, currently just set to "openai/ChatCompletion" for all variants for now.
Adds a `modelProviders/` directory where we can define and store pluggable model providers. Currently just OpenAI. Not everything is pluggable yet -- notably the code to actually generate completions hasn't been migrated to this setup yet.
Does a lot of work to get the types working. Prompts are now defined with a function `definePrompt(modelProvider, config)` instead of `prompt = config`. Added a script to migrate old prompt definitions.
This is still partial work, but the diff is large enough that I want to get it in. I don't think anything is broken but I haven't tested thoroughly.
Storing the model on promptVariant is problematic because it isn't always in sync with the actual prompt definition. I'm removing it for now to see if we can get away with that -- might have to add it back in later if this causes trouble.
Added `cost` to modelOutput as well so we can cache that, which is important given that the cost calculations won't be the same between different API providers.
Allows for the creation of user accounts. A few notes on the specifics:
- Experiments are the main access control objects. If you can view an experiment, you can view all its prompts/scenarios/evals. If you can edit it, you can edit or delete all of those as well.
- Experiments are owned by Organizations in the database. Organizations can have multiple members and members can have roles of ADMIN, MEMBER or VIEWER.
- Organizations can either be "personal" or general. Each user has a "personal" organization created as soon as they try to create an experiment. There's currently no UI support for creating general orgs or adding users to them; they're just in the database to future-proof all the ACL logic.
- You can require that a user is signed-in to see a route using the `protectedProcedure` helper. When you use `protectedProcedure`, you also have to call `ctx.markAccessControlRun()` (or delegate to a function that does it for you; see accessControl.ts). This is to remind us to actually check for access control when we define a new endpoint.
* Add dropdown header for model switching
* Allow variant duplication
* Fix prettier
* Use env variable to restrict prisma logs
* Fix env.mjs
* Remove unnecessary scroll bar from function call output
* Properly record when 404 error occurs in queryLLM task
* Add SelectedModelInfo in SelectModelModal
* Add react-select
* Calculate new prompt after switching model
* Send newly selected model with creation request
* Get new prompt construction function back from GPT-4
* Fix prettier
* Fix prettier
* Fix ScenariosHeader stickiness
* Move meta tag from _app.tsx to _document.tsx
* Show spinner when saving variant
* Escape quotes and regex in evaluations
* Prevent zoom in on iOS
* Expand function return code background to fill cell
* Keep OutputStats on far right of cells
* Continue polling prompt stats while cells are retrieving from LLM
* Add comment to _document.tsx
* Fix prettier
Use a block comment to explain the expected prompt formatting instead of function chrome. The advantage here is that once a user builds a mental model of how OpenPipe works they can just delete the comment, instead of the function chrome sitting around and taking up space in the UI forever.
This will (1) make sure that anything we push directly passes CI, and also (2) cache the pnpm store on the main branch, which will make it available to PR runs as well and hopefully speed up CI a bit (see https://stackoverflow.com/a/75250061).``
