OpenPipe-llm/README.md

🧪 Prompt Lab

Prompt Lab is a flexible playground for comparing and optimizing LLM prompts. It lets you quickly generate, test and compare candidate prompts with realistic sample data.

[]

Currently there's a public playground available at https://promptlab.corbt.com/, but the recommended approach is to run locally.

High-Level Features

Configure Multiple Prompts

Set up multiple prompt configurations and compare their output side-by-side. Each configuration can be configured independently.

Visualize Responses

Inspect prompt completions side-by-side.

Test Many Inputs

Prompt Lab lets you template a prompt. Use the templating feature to run the prompts you're testing against many potential inputs for broader coverage of your problem space than you'd get with manual testing.

🪄 Auto-generate Test Scenarios

Prompt Lab includes a tool to generate new test scenarios based on your existing prompts and scenarios. Just click "Autogenerate Scenario" to try it out!

Prompt Validation and Typeahead

We use OpenAI's OpenAPI spec to automatically provide typeahead and validate prompts.

[]

Function Call Support

Natively supports OpenAI function calls on supported models.

[]

Supported Models

Prompt Lab currently supports GPT-3.5 and GPT-4. Wider model support is planned.

Running Locally

1. Install Postgresql.
2. Install NodeJS 20 (earlier versions will very likely work but aren't tested).
3. Install pnpm: npm i -g pnpm
4. Clone this repository: git clone https://github.com/prompt-lab/prompt-lab
5. Install the dependencies: cd prompt-lab && pnpm install
6. Create a .env file (cp .env.example .env) and enter your OPENAI_API_KEY.
7. Start the app: pnpm dev
8. Navigate to http://localhost:3000