OpenPipe-llm/README.md

OpenPipe

OpenPipe is a flexible playground for comparing and optimizing LLM prompts. It lets you quickly generate, test and compare candidate prompts with realistic sample data.

Sample Experiments

These are simple experiments users have created that show how OpenPipe works.

• Country Capitals
• Reddit User Needs
• OpenAI Function Calls
• Activity Classification

You can use our hosted version of OpenPipe at [https://openpipe.ai]. You can also clone this repository and run it locally.

High-Level Features

Configure Multiple Prompts
Set up multiple prompt configurations and compare their output side-by-side. Each configuration can be configured independently.

Visualize Responses
Inspect prompt completions side-by-side.

Test Many Inputs
OpenPipe lets you template a prompt. Use the templating feature to run the prompts you're testing against many potential inputs for broader coverage of your problem space than you'd get with manual testing.

🪄 Auto-generate Test Scenarios
OpenPipe includes a tool to generate new test scenarios based on your existing prompts and scenarios. Just click "Autogenerate Scenario" to try it out!

Prompt Validation and Typeahead
We use OpenAI's OpenAPI spec to automatically provide typeahead and validate prompts.

Function Call Support
Natively supports OpenAI function calls on supported models.

Supported Models

• All models available through the OpenAI chat completion API
• Llama2 7b chat, 13b chat, 70b chat.

Running Locally

1. Install Postgresql.
2. Install NodeJS 20 (earlier versions will very likely work but aren't tested).
3. Install pnpm: npm i -g pnpm
4. Clone this repository: git clone https://github.com/openpipe/openpipe
5. Install the dependencies: cd openpipe && pnpm install
6. Create a .env file (cp .env.example .env) and enter your OPENAI_API_KEY.
7. Update DATABASE_URL if necessary to point to your Postgres instance and run pnpm prisma db push to create the database.
8. Create a GitHub OAuth App and update the GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET values. (Note: a PR to make auth optional when running locally would be a great contribution!)
9. Start the app: pnpm dev.
10. Navigate to http://localhost:3000