alihan/12-factor-agents
1
0
Fork 0
mirror of https://github.com/humanlayer/12-factor-agents.git synced 2025-08-20 18:59:53 +03:00
Code Issues Packages Projects Releases Wiki Activity

wip

Browse Source
This commit is contained in:
dexhorthy
2025-07-15 19:21:20 -07:00
parent be6395c40f
commit f8c5b1608a
4 changed files with 1127 additions and 655 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
workshops/2025-07-16/hack/scratch.md Normal file
View File

@@ -0,0 +1,33 @@
okay but we have complications - colab can’t show baml files, i have hacked a few workaround to make this work, i need you to distill out all the changes that would need to happen in the logic flow from walkthrough.yaml to translate the steps so far into baml notebook - off the top of my head:
1) for each ts file, translate to python and drop in the walkthrough folder (claude will do this)
1b) update walkthrough.yaml to point to the python instead of ts file, lead the baml files unchanged
2) we won’t build section-by-section, we’ll build just one big notebook file (similar to the one-big-walkthrough-md target provided by the typescript library)
3) for each text section, add a markdown cell
4) for each code cell, add the full python file from the walkthrough, so that running the code cell will refresh and update any function definitions
5) rather than separate python files, we'll just update the function definitions in the notebook as we go - you might have to get creative / clever in just redefining what we used, and rather than commands to run each cell, you'll want two cells: 1 to update the function, and 1 to re-run the main() function (note: THIS is the part i'm the most unsure of and we might need to adjust the approach as we go!)
6) for each baml file, follow the example in the notebook, fetching the file from the public github url and printing it with cat
6b) if you need to update a baml file for some reason, i will need to push it to the public github repo
7) note that there may be some hackery where we need to re-import the baml_client after changing the baml sources!
other information - i have an example in the hack/ folder of a notebook that has everything working end to end for a subset of chapter 1, including setting secrets, installing/generating baml in a clean reusable way, fetching and displaying baml files
note that the implementation plan will need ways to run/verify the notebook after ever implementation change, so the implementation flow will be
1) make a change to the walkthrough.yaml OR make a change to hack/walkthroughgen_py.py
2) run the walkthroughgen_py.py script to generate the notebook
3) run the notebook to test that it works
3) read the notebook file and check for errors and that the outputs are expected
you will evolve each thing in parallel, targeting finishing a complete chapter in both the walkthrough.yaml and the walkthroughgen_py.py script before proceeding the the next chapter
### important notes
- There is a reference walkthrough from the typescript version in walkthrough-reference.yaml, which you can use to convert one chapter at a time
- `file: ` objects in the python / ipynb target will not have a dest, just a src
### before you start researching
first - review that plan, does it make sense? as you are researching, know that there will be things that I missed and that need to be adjusted in the plan
Ask any questions you have now before you start please.

412
workshops/2025-07-16/hack/working_notebook_1.ipynb Normal file
View File

@@ -0,0 +1,412 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "37c4763c",
"metadata": {
"id": "37c4763c"
},
"source": [
"# Workshop Notebook - July 16, 2025\n",
"\n",
"Welcome to today's workshop! This notebook contains some basic examples to get started.\n",
"\n",
"## Overview\n",
"- Basic Python operations\n",
"- Simple calculations\n",
"- String manipulation"
]
},
{
"cell_type": "code",
"source": [
"def get_baml_client():\n",
" \"\"\"\n",
" a bunch of fun jank to work around the google colab import cache\n",
" \"\"\"\n",
"\n",
" from google.colab import userdata\n",
" import os\n",
" os.environ['OPENAI_API_KEY'] = userdata.get('OPENAI_API_KEY')\n",
"\n",
" import importlib\n",
" import baml_client\n",
" importlib.reload(baml_client)\n",
" return baml_client.sync_client.b\n",
"\n"
],
"metadata": {
"id": "Wu1JAPuEv4AH"
},
"id": "Wu1JAPuEv4AH",
"execution_count": 8,
"outputs": []
},
{
"cell_type": "code",
"execution_count": 1,
"id": "57d39979",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "57d39979",
"outputId": "5d8ed700-6f7f-42b0-bc09-49b84f1fca9f"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Requirement already satisfied: baml-py in /usr/local/lib/python3.11/dist-packages (0.201.0)\n"
]
}
],
"source": [
"!pip install baml-py\n"
]
},
{
"cell_type": "code",
"source": [
"!baml-cli init\n"
],
"metadata": {
"id": "RKpwjadRtQ4E"
},
"id": "RKpwjadRtQ4E",
"execution_count": 2,
"outputs": []
},
{
"cell_type": "code",
"source": [
"!baml-cli generate"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "uKU4rfG0tVuQ",
"outputId": "478ce0f8-41b5-4f69-e88a-491a769a3ce2"
},
"id": "uKU4rfG0tVuQ",
"execution_count": 3,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"2025-07-16T01:14:07.315 [BAML \u001b[92mINFO\u001b[0m] Wrote 13 files to baml_client\n",
"2025-07-16T01:14:07.315 [BAML \u001b[92mINFO\u001b[0m] Generated 1 baml_client: ../baml_client\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"!baml-cli test"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "rsLwuS_cuUyO",
"outputId": "de29b395-b0a8-4e37-bd2e-76ab34f522eb"
},
"id": "rsLwuS_cuUyO",
"execution_count": 6,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Summary: 0 failures, 0 passes, 0 running, 0 pending, 0 done \r\u001b[2KSummary: 0/2 tests \r\u001b[2KSummary: 0/2 tests \r\u001b[2KSummary: 0/2 tests \n",
"\u001b[32m⠁\u001b[0m Running DetermineNextStep::HelloWorld \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1ASummary: 0/2 tests \n",
"\u001b[32m⠁\u001b[0m Running DetermineNextStep::HelloWorld \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1ASummary: 0/2 tests \n",
"\u001b[32m⠁\u001b[0m Running DetermineNextStep::HelloWorld \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1ASummary: 0/2 tests \n",
"\u001b[32m⠁\u001b[0m Running DetermineNextStep::HelloWorld\n",
"\u001b[32m⠁\u001b[0m Running ExtractResume::vaibhav_resume \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2ASummary: 0/2 tests \n",
"\u001b[32m⠁\u001b[0m Running DetermineNextStep::HelloWorld\n",
"\u001b[32m⠁\u001b[0m Running ExtractResume::vaibhav_resume \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2ASummary: 0/2 tests \n",
"\u001b[32m⠁\u001b[0m Running DetermineNextStep::HelloWorld\n",
"\u001b[32m⠁\u001b[0m Running ExtractResume::vaibhav_resume 2025-07-16T01:14:25.520 [BAML \u001b[33mWARN\u001b[0m] \u001b[35mFunction DetermineNextStep\u001b[0m:\n",
" \u001b[33mClient: Qwen3 (<unknown>) - 0ms\u001b[0m\n",
" \u001b[34m---PROMPT---\u001b[0m\n",
" \u001b[2m\u001b[43msystem: \u001b[0m\u001b[2m/nothink \n",
" \n",
" You are a helpful assistant that can help with tasks.\n",
" \u001b[43muser: \u001b[0m\u001b[2mYou are working on the following thread:\n",
" \n",
" {\n",
" \"type\": \"user_input\",\n",
" \"data\": \"hello!\"\n",
" }\n",
" \n",
" What should the next step be?\n",
" \n",
" Answer in JSON using this schema:\n",
" {\n",
" intent: \"done_for_now\",\n",
" message: string,\n",
" }\n",
" \u001b[0m\n",
" \u001b[34m---REQUEST OPTIONS---\u001b[0m\n",
" \u001b[31m---ERROR (Unspecified error code: 2)---\u001b[0m\n",
" \u001b[31mFailed to build request: reqwest::Error {\n",
" kind: Builder,\n",
" source: RelativeUrlWithoutBase,\n",
" }\u001b[0m\n",
"\u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2ASummary: 1/2 tests - 1 🛑\n",
"\u001b[32m⠁\u001b[0m Running DetermineNextStep::HelloWorld\n",
"\u001b[32m⠁\u001b[0m Running ExtractResume::vaibhav_resume \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2ASummary: 1/2 tests - 1 🛑\n",
"\u001b[32m⠁\u001b[0m Running DetermineNextStep::HelloWorld\n",
"\u001b[32m⠁\u001b[0m Running ExtractResume::vaibhav_resume \u001b[2A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[2ASummary: 1/2 tests - 1 🛑\n",
"\u001b[32m⠁\u001b[0m Running ExtractResume::vaibhav_resume \u001b[1A\r\u001b[2K\u001b[1B\r\u001b[2K\u001b[1A0.04s \u001b[91mERROR \u001b[0m DetermineNextStep::HelloWorld\n",
" \u001b[2;31mUnspecified error code: 2 Failed to build request: reqwest::Error {\u001b[0m\n",
" \u001b[2;31m kind: Builder,\u001b[0m\n",
" \u001b[2;31m source: RelativeUrlWithoutBase,\u001b[0m\n",
" \u001b[2;31m}\u001b[0m\n",
" \u001b[2;31m\u001b[0m\n",
" \u001b[2;31mRequest options: {}\u001b[0m\n",
"Summary: 1/2 tests - 1 🛑\n",
"\u001b[2K\u001b[1ASummary: 1/2 tests - 1 🛑\n",
"\u001b[2K\u001b[1ASummary: 1/2 tests - 1 🛑\n",
"\u001b[2K\u001b[1ASummary: 1/2 tests - 1 🛑\n",
"\u001b[2K\u001b[1ASummary: 1/2 tests - 1 🛑\n",
"\u001b[2K\u001b[1ASummary: 1/2 tests - 1 🛑\n",
"\u001b[2K\u001b[1ASummary: 1/2 tests - 1 🛑\n",
"\u001b[2K\u001b[1ASummary: 1/2 tests - 1 🛑\n",
"\u001b[2K\u001b[1ASummary: 1/2 tests - 1 🛑\n",
"\u001b[2K\u001b[1ASummary: 1/2 tests - 1 🛑\n",
"\u001b[2K\u001b[1ASummary: 1/2 tests - 1 🛑\n",
"\u001b[32m⠤\u001b[0m Running ExtractResume::vaibhav_resume 2025-07-16T01:14:26.555 [BAML \u001b[92mINFO\u001b[0m] \u001b[35mFunction ExtractResume\u001b[0m:\n",
" \u001b[33mClient: openai/gpt-4o (gpt-4o-2024-08-06) - 1038ms. StopReason: stop. Tokens(in/out): 81/68\u001b[0m\n",
" \u001b[34m---PROMPT---\u001b[0m\n",
" \u001b[2m\u001b[43msystem: \u001b[0m\u001b[2mExtract from this content:\n",
" Vaibhav Gupta\n",
" vbv@boundaryml.com\n",
" \n",
" Experience:\n",
" - Founder at BoundaryML\n",
" - CV Engineer at Google\n",
" - CV Engineer at Microsoft\n",
" \n",
" Skills:\n",
" - Rust\n",
" - C++\n",
" \n",
" Answer in JSON using this schema:\n",
" {\n",
" name: string,\n",
" email: string,\n",
" experience: string[],\n",
" skills: string[],\n",
" }\n",
" \u001b[0m\n",
" \u001b[34m---LLM REPLY---\u001b[0m\n",
" \u001b[2m{\n",
" \"name\": \"Vaibhav Gupta\",\n",
" \"email\": \"vbv@boundaryml.com\",\n",
" \"experience\": [\n",
" \"Founder at BoundaryML\",\n",
" \"CV Engineer at Google\",\n",
" \"CV Engineer at Microsoft\"\n",
" ],\n",
" \"skills\": [\n",
" \"Rust\",\n",
" \"C++\"\n",
" ]\n",
" }\u001b[0m\n",
" \u001b[34m---Parsed Response (class Resume)---\u001b[0m\n",
" {\n",
" \"name\": \"Vaibhav Gupta\",\n",
" \"email\": \"vbv@boundaryml.com\",\n",
" \"experience\": [\n",
" \"Founder at BoundaryML\",\n",
" \"CV Engineer at Google\",\n",
" \"CV Engineer at Microsoft\"\n",
" ],\n",
" \"skills\": [\n",
" \"Rust\",\n",
" \"C++\"\n",
" ]\n",
" }\n",
"\u001b[2K\u001b[1ASummary: 2/2 tests - 1 ✅, 1 🛑\n",
"\u001b[2K\n",
"INFO: Test results:\n",
"---------------------------------------------------------\n",
"\u001b[1;34mfunction\u001b[0m \u001b[1;34mDetermineNextStep\u001b[0m\n",
"1 tests (1 🛑)\n",
" 0.04s \u001b[91mERROR \u001b[0m DetermineNextStep::HelloWorld\n",
" \u001b[2m ./baml_src/agent.baml:40\u001b[0m\n",
" \u001b[2;31mUnspecified error code: 2 Failed to build request: reqwest::Error {\u001b[0m\n",
" \u001b[2;31m kind: Builder,\u001b[0m\n",
" \u001b[2;31m source: RelativeUrlWithoutBase,\u001b[0m\n",
" \u001b[2;31m}\u001b[0m\n",
" \u001b[2;31m\u001b[0m\n",
" \u001b[2;31mRequest options: {}\u001b[0m\n",
"\u001b[1;34mfunction\u001b[0m \u001b[1;34mExtractResume\u001b[0m\n",
"1 tests (1 ✅)\n",
" 1.08s \u001b[32mPASSED \u001b[0m ExtractResume::vaibhav_resume\n",
" \u001b[2m ./baml_src/resume.baml:25\u001b[0m\n",
"---------------------------------------------------------\n",
"INFO: Test run completed, 2 tests (1 ✅, 1 🛑)\n",
"\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"!rm baml_src/resume.baml"
],
"metadata": {
"id": "eJgErOv8zCR1"
},
"id": "eJgErOv8zCR1",
"execution_count": 7,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"now lets download the agent"
],
"metadata": {
"id": "MVqkHHOFzFNz"
},
"id": "MVqkHHOFzFNz"
},
{
"cell_type": "code",
"source": [
"!curl -fsSL -o baml_src/agent.baml https://raw.githubusercontent.com/humanlayer/12-factor-agents/refs/heads/main/workshops/2025-07-16/walkthrough/01-agent.baml && cat baml_src/agent.baml"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "wV0WipZytZNt",
"outputId": "1adecefd-2c8f-441f-da9a-84e975eebef6"
},
"id": "wV0WipZytZNt",
"execution_count": 13,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"class DoneForNow {\n",
" intent \"done_for_now\"\n",
" message string \n",
"}\n",
"\n",
"function DetermineNextStep(\n",
" thread: string \n",
") -> DoneForNow {\n",
" client \"openai/gpt-4o\"\n",
"\n",
" // use /nothink for now because the thinking tokens (or streaming thereof) screw with baml (i think (no pun intended))\n",
" prompt #\"\n",
" {{ _.role(\"system\") }}\n",
"\n",
" You are a helpful assistant that can help with tasks.\n",
"\n",
" {{ _.role(\"user\") }}\n",
"\n",
" You are working on the following thread:\n",
"\n",
" {{ thread }}\n",
"\n",
" What should the next step be?\n",
"\n",
" {{ ctx.output_format }}\n",
" \"#\n",
"}\n",
"\n",
"test HelloWorld {\n",
" functions [DetermineNextStep]\n",
" args {\n",
" thread #\"\n",
" {\n",
" \"type\": \"user_input\",\n",
" \"data\": \"hello!\"\n",
" }\n",
" \"#\n",
" }\n",
"}"
]
}
]
},
{
"cell_type": "code",
"source": [
"!baml-cli generate"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "JAfVRgy3wK4v",
"outputId": "d87fc300-cc8f-43ad-fa95-ab194db365a0"
},
"id": "JAfVRgy3wK4v",
"execution_count": 14,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"2025-07-16T01:22:21.660 [BAML \u001b[92mINFO\u001b[0m] Wrote 13 files to baml_client\n",
"2025-07-16T01:22:21.660 [BAML \u001b[92mINFO\u001b[0m] Generated 1 baml_client: ../baml_client\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"b = get_baml_client()\n",
"\n",
"step = b.DetermineNextStep(\"hi\")\n",
"print(step)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "gyIrbt-ZuXrK",
"outputId": "28a83364-248f-4c36-afb3-faaf271aa485"
},
"id": "gyIrbt-ZuXrK",
"execution_count": 9,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"intent='done_for_now' message='Hello! Please let me know if there is anything specific you need assistance with.'\n"
]
}
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"language_info": {
"name": "python"
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

664
workshops/2025-07-16/walkthrough-reference.yaml Normal file
View File

@@ -0,0 +1,664 @@
title: "Building the 12-factor agent template from scratch in python"
text: "Steps to start from a bare python repo and build up a 12-factor agent. This walkthrough will guide you through creating a python agent that follows the 12-factor methodology with baml."
targets:
- ipynb: "./build/workshop-2025-07-16.ipynb"
sections:
- name: hello-world
title: "Chapter 0 - Hello World"
text: "Let's start with a basic TypeScript setup and a hello world program."
steps:
- text: |
This guide is written in TypeScript (yes, a python version is coming soon)
There are many checkpoints between the every file edit in theworkshop steps,
so even if you aren't super familiar with typescript,
you should be able to keep up and run each example.
To run this guide, you'll need a relatively recent version of nodejs and npm installed
You can use whatever nodejs version manager you want, [homebrew](https://formulae.brew.sh/formula/node) is fine
command:
brew install node@20
results:
- text: "You should see the node version"
code: |
node --version
- text: "Copy initial package.json"
file: {src: ./walkthrough/00-package.json, dest: package.json}
- text: "Install dependencies"
command: |
npm install
incremental: true
- text: "Copy tsconfig.json"
file: {src: ./walkthrough/00-tsconfig.json, dest: tsconfig.json}
- text: "add .gitignore"
file: {src: ./walkthrough/00-.gitignore, dest: .gitignore}
- text: "Create src folder"
dir: {create: true, path: src}
- text: "Add a simple hello world index.ts"
file: {src: ./walkthrough/00-index.ts, dest: src/index.ts}
- text: "Run it to verify"
command: |
npx tsx src/index.ts
results:
- text: "You should see:"
code: |
hello, world!
- name: cli-and-agent
title: "Chapter 1 - CLI and Agent Loop"
text: "Now let's add BAML and create our first agent with a CLI interface."
steps:
- text: |
First, we'll need to install [BAML](https://github.com/boundaryml/baml)
which is a tool for prompting and structured outputs.
command: |
npm install @boundaryml/baml
incremental: true
- text: "Initialize BAML"
command: |
npx baml-cli init
incremental: true
- text: "Remove default resume.baml"
command: |
rm baml_src/resume.baml
incremental: true
- text: "Add our starter agent, a single baml prompt that we'll build on"
file: {src: ./walkthrough/01-agent.baml, dest: baml_src/agent.baml}
- text: "Generate BAML client code"
command: |
npx baml-cli generate
incremental: true
- text: "Enable BAML logging for this section"
command: |
export BAML_LOG=debug
- text: "Add the CLI interface"
file: {src: ./walkthrough/01-cli.ts, dest: src/cli.ts}
- text: "Update index.ts to use the CLI"
file: {src: ./walkthrough/01-index.ts, dest: src/index.ts}
- text: "Add the agent implementation"
file: {src: ./walkthrough/01-agent.ts, dest: src/agent.ts}
- text: |
The the BAML code is configured to use BASETEN_API_KEY by default
To get a Baseten API key and URL, create an account at [baseten.co](https://baseten.co),
and then deploy [Qwen3 32B from the model library](https://www.baseten.co/library/qwen-3-32b/).
```rust
function DetermineNextStep(thread: string) -> DoneForNow {
client Qwen3
// ...
```
If you want to run the example with no changes, you can set the BASETEN_API_KEY env var to any valid baseten key.
If you want to try swapping out the model, you can change the `client` line.
[Docs on baml clients can be found here](https://docs.boundaryml.com/guide/baml-basics/switching-llms)
For example, you can configure [gemini](https://docs.boundaryml.com/ref/llm-client-providers/google-ai-gemini)
or [anthropic](https://docs.boundaryml.com/ref/llm-client-providers/anthropic) as your model provider.
For example, to use openai with an OPENAI_API_KEY, you can do:
client "openai/gpt-4o"
- text: Set your env vars
command: |
export BASETEN_API_KEY=...
export BASETEN_BASE_URL=...
- text: "Try it out"
command: |
npx tsx src/index.ts hello
results:
- text: you should see a familiar response from the model
code: |
{
intent: 'done_for_now',
message: 'Hello! How can I assist you today?'
}
- name: calculator-tools
title: "Chapter 2 - Add Calculator Tools"
text: "Let's add some calculator tools to our agent."
steps:
- text: |
Let's start by adding a tool definition for the calculator
These are simpile structured outputs that we'll ask the model to
return as a "next step" in the agentic loop.
file: {src: ./walkthrough/02-tool_calculator.baml, dest: baml_src/tool_calculator.baml}
- text: |
Now, let's update the agent's DetermineNextStep method to
expose the calculator tools as potential next steps
file: {src: ./walkthrough/02-agent.baml, dest: baml_src/agent.baml}
- text: "Generate updated BAML client"
command: |
npx baml-cli generate
incremental: true
- text: "Try out the calculator"
command: |
npx tsx src/index.ts 'can you add 3 and 4'
results:
- text: "You should see a tool call to the calculator"
code: |
{
intent: 'add',
a: 3,
b: 4
}
- name: tool-loop
title: "Chapter 3 - Process Tool Calls in a Loop"
text: "Now let's add a real agentic loop that can run the tools and get a final answer from the LLM."
steps:
- text: |
First, lets update the agent to handle the tool call
file: {src: ./walkthrough/03-agent.ts, dest: src/agent.ts}
- text: |
Now, lets try it out
command: |
npx tsx src/index.ts 'can you add 3 and 4'
results:
- text: you should see the agent call the tool and then return the result
code: |
{
intent: 'done_for_now',
message: 'The sum of 3 and 4 is 7.'
}
- text: "For the next step, we'll do a more complex calculation, let's turn off the baml logs for more concise output"
command: |
export BAML_LOG=off
- text: "Try a multi-step calculation"
command: |
npx tsx src/index.ts 'can you add 3 and 4, then add 6 to that result'
- text: "you'll notice that tools like multiply and divide are not available"
command: |
npx tsx src/index.ts 'can you multiply 3 and 4'
- text: |
next, let's add handlers for the rest of the calculator tools
file: {src: ./walkthrough/03b-agent.ts, dest: src/agent.ts}
- text: "Test subtraction"
command: |
npx tsx src/index.ts 'can you subtract 3 from 4'
- text: |
now, let's test the multiplication tool
command: |
npx tsx src/index.ts 'can you multiply 3 and 4'
- text: |
finally, let's test a more complex calculation with multiple operations
command: |
npx tsx src/index.ts 'can you multiply 3 and 4, then divide the result by 2 and then add 12 to that result'
- text: |
congratulations, you've taking your first step into hand-rolling an agent loop.
from here, we're going to start incorporating some more intermediate and advanced
concepts for 12-factor agents.
- name: baml-tests
title: "Chapter 4 - Add Tests to agent.baml"
text: "Let's add some tests to our BAML agent."
steps:
- text: to start, leave the baml logs enabled
command: |
export BAML_LOG=debug
- text: |
next, let's add some tests to the agent
We'll start with a simple test that checks the agent's ability to handle
a basic calculation.
file: {src: ./walkthrough/04-agent.baml, dest: baml_src/agent.baml}
- text: "Run the tests"
command: |
npx baml-cli test
- text: |
now, let's improve the test with assertions!
Assertions are a great way to make sure the agent is working as expected,
and can easily be extended to check for more complex behavior.
file: {src: ./walkthrough/04b-agent.baml, dest: baml_src/agent.baml}
- text: "Run the tests"
command: |
npx baml-cli test
- text: |
as you add more tests, you can disable the logs to keep the output clean.
You may want to turn them on as you iterate on specific tests.
command: |
export BAML_LOG=off
- text: |
now, let's add some more complex test cases,
where we resume from in the middle of an in-progress
agentic context window
file: {src: ./walkthrough/04c-agent.baml, dest: baml_src/agent.baml}
- text: |
let's try to run it
command: |
npx baml-cli test
- name: human-tools
title: "Chapter 5 - Multiple Human Tools"
text: |
In this section, we'll add support for multiple tools that serve to
contact humans.
steps:
- text: "for this section, we'll disable the baml logs. You can optionally enable them if you want to see more details."
command: |
export BAML_LOG=off
- text: |
first, let's add a tool that can request clarification from a human
this will be different from the "done_for_now" tool,
and can be used to more flexibly handle different types of human interactions
in your agent.
file: {src: ./walkthrough/05-agent.baml, dest: baml_src/agent.baml}
- text: |
next, let's re-generate the client code
NOTE - if you're using the VSCode extension for BAML,
the client will be regenerated automatically when you save the file
in your editor.
command: |
npx baml-cli generate
incremental: true
- text: |
now, let's update the agent to use the new tool
file: {src: ./walkthrough/05-agent.ts, dest: src/agent.ts}
- text: |
next, let's update the CLI to handle clarification requests
by requesting input from the user on the CLI
file: {src: ./walkthrough/05-cli.ts, dest: src/cli.ts}
- text: |
let's try it out
command: |
npx tsx src/index.ts 'can you multiply 3 and FD*(#F&& '
- text: |
next, let's add a test that checks the agent's ability to handle
a clarification request
file: {src: ./walkthrough/05b-agent.baml, dest: baml_src/agent.baml}
- text: |
and now we can run the tests again
command: |
npx baml-cli test
- text: |
you'll notice the new test passes, but the hello world test fails
This is because the agent's default behavior is to return "done_for_now"
file: {src: ./walkthrough/05c-agent.baml, dest: baml_src/agent.baml}
- text: "Verify tests pass"
command: |
npx baml-cli test
- name: customize-prompt
title: "Chapter 6 - Customize Your Prompt with Reasoning"
text: |
In this section, we'll explore how to customize the prompt of the agent
with reasoning steps.
this is core to [factor 2 - own your prompts](https://github.com/humanlayer/12-factor-agents/blob/main/content/factor-2-own-your-prompts.md)
there's a deep dive on reasoning on AI That Works [reasoning models versus reasoning steps](https://github.com/hellovai/ai-that-works/tree/main/2025-04-07-reasoning-models-vs-prompts)
steps:
- text: "for this section, it will be helpful to leave the baml logs enabled"
command: |
export BAML_LOG=debug
- text: |
update the agent prompt to include a reasoning step
file: {src: ./walkthrough/06-agent.baml, dest: baml_src/agent.baml}
- text: generate the updated client
command: |
npx baml-cli generate
incremental: true
- text: |
now, you can try it out with a simple prompt
command: |
npx tsx src/index.ts 'can you multiply 3 and 4'
results:
- text: you should see output from the baml logs showing the reasoning steps
- text: |
#### optional challenge
add a field to your tool output format that includes the reasoning steps in the output!
- name: context-window
title: "Chapter 7 - Customize Your Context Window"
text: |
In this section, we'll explore how to customize the context window
of the agent.
this is core to [factor 3 - own your context window](https://github.com/humanlayer/12-factor-agents/blob/main/content/factor-3-own-your-context-window.md)
steps:
- text: |
update the agent to pretty-print the Context window for the model
file: {src: ./walkthrough/07-agent.ts, dest: src/agent.ts}
- text: "Test the formatting"
command: |
BAML_LOG=info npx tsx src/index.ts 'can you multiply 3 and 4, then divide the result by 2 and then add 12 to that result'
- text: |
next, let's update the agent to use XML formatting instead
this is a very popular format for passing data to a model,
among other things, because of the token efficiency of XML.
file: {src: ./walkthrough/07b-agent.ts, dest: src/agent.ts}
- text: |
let's try it out
command: |
BAML_LOG=info npx tsx src/index.ts 'can you multiply 3 and 4, then divide the result by 2 and then add 12 to that result'
- text: |
lets update our tests to match the new output format
file: {src: ./walkthrough/07c-agent.baml, dest: baml_src/agent.baml}
- text: |
check out the updated tests
command: |
npx baml-cli test
- name: api-endpoints
title: "Chapter 8 - Adding API Endpoints"
text: "Add an Express server to expose the agent via HTTP."
steps:
- text: "for this section, we'll disable the baml logs. You can optionally enable them if you want to see more details."
command: |
export BAML_LOG=off
- text: "Install Express and types"
command: |
npm install express && npm install --save-dev @types/express supertest
incremental: true
- text: "Add the server implementation"
file: {src: ./walkthrough/08-server.ts, dest: src/server.ts}
- text: "Start the server"
command: |
npx tsx src/server.ts
- text: "Test with curl (in another terminal)"
command: |
curl -X POST http://localhost:3000/thread \
-H "Content-Type: application/json" \
-d '{"message":"can you add 3 and 4"}'
results:
- text: |
You should get an answer from the agent which includes the
agentic trace, ending in a message like:
code: |
{"intent":"done_for_now","message":"The sum of 3 and 4 is 7."}
- name: state-management
title: "Chapter 9 - In-Memory State and Async Clarification"
text: "Add state management and async clarification support."
steps:
- text: "for this section, we'll disable the baml logs. You can optionally enable them if you want to see more details."
command: |
export BAML_LOG=off
- text: "Add some simple in-memory state management for threads"
file: {src: ./walkthrough/09-state.ts, dest: src/state.ts}
- text: |
update the server to use the state management
* Add thread state management using `ThreadStore`
* return thread IDs and response URLs from the /thread endpoint
* implement GET /thread/:id
* implement POST /thread/:id/response
file: {src: ./walkthrough/09-server.ts, dest: src/server.ts}
- text: "Start the server"
command: |
npx tsx src/server.ts
- text: "Test clarification flow"
command: |
curl -X POST http://localhost:3000/thread \
-H "Content-Type: application/json" \
-d '{"message":"can you multiply 3 and xyz"}'
- name: human-approval
title: "Chapter 10 - Adding Human Approval"
text: "Add support for human approval of operations."
steps:
- text: "for this section, we'll disable the baml logs. You can optionally enable them if you want to see more details."
command: |
export BAML_LOG=off
- text: |
update the server to handle human approvals
* Import `handleNextStep` to execute approved actions
* Add two payload types to distinguish approvals from responses
* Handle responses and approvals differently in the endpoint
* Show better error messages when things go wrongs
file: {src: ./walkthrough/10-server.ts, dest: src/server.ts}
- text: "Add a few methods to the agent to handle approvals and responses"
file: {src: ./walkthrough/10-agent.ts, dest: src/agent.ts}
- text: "Start the server"
command: |
npx tsx src/server.ts
- text: "Test division with approval"
command: |
curl -X POST http://localhost:3000/thread \
-H "Content-Type: application/json" \
-d '{"message":"can you divide 3 by 4"}'
results:
- text: "You should see:"
code: |
{
"thread_id": "2b243b66-215a-4f37-8bc6-9ace3849043b",
"events": [
{
"type": "user_input",
"data": "can you divide 3 by 4"
},
{
"type": "tool_call",
"data": {
"intent": "divide",
"a": 3,
"b": 4,
"response_url": "/thread/2b243b66-215a-4f37-8bc6-9ace3849043b/response"
}
}
]
}
- text: "reject the request with another curl call, changing the thread ID"
command: |
curl -X POST 'http://localhost:3000/thread/{thread_id}/response' \
-H "Content-Type: application/json" \
-d '{"type": "approval", "approved": false, "comment": "I dont think thats right, use 5 instead of 4"}'
results:
- text: 'You should see: the last tool call is now `"intent":"divide","a":3,"b":5`'
code: |
{
"events": [
{
"type": "user_input",
"data": "can you divide 3 by 4"
},
{
"type": "tool_call",
"data": {
"intent": "divide",
"a": 3,
"b": 4,
"response_url": "/thread/2b243b66-215a-4f37-8bc6-9ace3849043b/response"
}
},
{
"type": "tool_response",
"data": "user denied the operation with feedback: \"I dont think thats right, use 5 instead of 4\""
},
{
"type": "tool_call",
"data": {
"intent": "divide",
"a": 3,
"b": 5,
"response_url": "/thread/1f1f5ff5-20d7-4114-97b4-3fc52d5e0816/response"
}
}
]
}
- text: "now you can approve the operation"
command: |
curl -X POST 'http://localhost:3000/thread/{thread_id}/response' \
-H "Content-Type: application/json" \
-d '{"type": "approval", "approved": true}'
results:
- text: "you should see the final message includes the tool response and final result!"
code: |
...
{
"type": "tool_response",
"data": 0.5
},
{
"type": "done_for_now",
"message": "I divided 3 by 6 and the result is 0.5. If you have any more operations or queries, feel free to ask!",
"response_url": "/thread/2b469403-c497-4797-b253-043aae830209/response"
}
- name: humanlayer-approval
title: "Chapter 11 - Human Approvals over email"
text: |
in this section, we'll add support for human approvals over email.
This will start a little bit contrived, just to get the concepts down -
We'll start by invoking the workflow from the CLI but approvals for `divide`
and `request_more_information` will be handled over email,
then the final `done_for_now` answer will be printed back to the CLI
While contrived, this is a great example of the flexibility you get from
[factor 7 - contact humans with tools](https://github.com/humanlayer/12-factor-agents/blob/main/content/factor-7-contact-humans-with-tools.md)
steps:
- text: "for this section, we'll disable the baml logs. You can optionally enable them if you want to see more details."
command: |
export BAML_LOG=off
- text: "Install HumanLayer"
command: |
npm install humanlayer
incremental: true
- text: "Update CLI to send `divide` and `request_more_information` to a human via email"
file: {src: ./walkthrough/11-cli.ts, dest: src/cli.ts}
- text: "Run the CLI"
command: |
npx tsx src/index.ts 'can you divide 4 by 5'
results:
- text: "The last line of your program should mention human review step"
code: |
nextStep { intent: 'divide', a: 4, b: 5 }
HumanLayer: Requested human approval from HumanLayer cloud
- text: |
go ahead and respond to the email with some feedback:
![reject-email](https://github.com/humanlayer/12-factor-agents/blob/main/workshops/2025-05/walkthrough/11-email-reject.png?raw=true)
- text: |
you should get another email with an updated attempt based on your feedback!
You can go ahead and approve this one:
![approve-email](https://github.com/humanlayer/12-factor-agents/blob/main/workshops/2025-05/walkthrough/11-email-approve.png?raw=true)
results:
- text: and your final output will look like
code: |
nextStep {
intent: 'done_for_now',
message: 'The division of 4 by 5 is 0.8. If you have any other calculations or questions, feel free to ask!'
}
The division of 4 by 5 is 0.8. If you have any other calculations or questions, feel free to ask!
- text: |
lets implement the `request_more_information` flow as well
file: {src: ./walkthrough/11b-cli.ts, dest: src/cli.ts}
- text: |
lets test the require_approval flow as by asking for a calculation
with garbled input:
command: |
npx tsx src/index.ts 'can you multiply 4 and xyz'
- text: "You should get an email with a request for clarification"
command: |
Can you clarify what 'xyz' represents in this context? Is it a specific number, variable, or something else?
- text: you can response with something like
command: |
use 8 instead of xyz
results:
- text: you should see a final result on the CLI like
code: |
I have multiplied 4 and xyz, using the value 8 for xyz, resulting in 32.
- text: |
as a final step, lets explore using a custom html template for the email
file: {src: ./walkthrough/11c-cli.ts, dest: src/cli.ts}
- text: |
first try with divide:
command: |
npx tsx src/index.ts 'can you divide 4 by 5'
results:
- text: |
you should see a slightly different email with the custom template
![custom-template-email](https://github.com/humanlayer/12-factor-agents/blob/main/workshops/2025-05/walkthrough/11-email-custom.png?raw=true)
feel free to run with the flow and then you can try updating the template to your liking
(if you're using cursor, something as simple as highlighting the template and asking to "make it better"
should do the trick)
try triggering "request_more_information" as well!
- text: |
thats it - in the next chapter, we'll build a fully email-driven
workflow agent that uses webhooks for human approval
- name: humanlayer-webhook
title: "Chapter XX - HumanLayer Webhook Integration"
text: |
the previous sections used the humanlayer SDK in "synchronous mode" - that
means every time we wait for human approval, we sit in a loop
polling until the human response if received.
That's obviously not ideal, especially for production workloads,
so in this section we'll implement [factor 6 - launch / pause / resume with simple APIs](https://github.com/humanlayer/12-factor-agents/blob/main/content/factor-6-launch-pause-resume.md)
by updating the server to end processing after contacting a human, and use webhooks to receive the results.
steps:
- text: |
add code to initialize humanlayer in the server
file: {src: ./walkthrough/12-1-server-init.ts, dest: src/server.ts}
- text: |
next, lets update the /thread endpoint to
1. handle requests asynchronously, returning immediately
2. create a human contact on request_more_information and done_for_now calls
# file: {src: }
- text: |
Update the server to be able to handle request_clarification responses
- remove the old /response endpoint and types
- update the /thread endpoint to run processing asynchronously, return immediately
- send a state.threadId when requesting human responses
- add a handleHumanResponse function to process the human response
- add a /webhook endpoint to handle the webhook response
file: {src: ./walkthrough/12a-server.ts, dest: src/server.ts}
- text: "Start the server in another terminal"
command: |
npx tsx src/server.ts
- text: |
now that the server is running, send a payload to the '/thread' endpoint
- text: __ do the response step
- text: __ now handle approvals for divide
- text: __ now also handle done_for_now

673
workshops/2025-07-16/walkthrough.yaml
View File

@@ -1,74 +1,18 @@
title: "Building the 12-factor agent template from scratch"
text: "Steps to start from a bare TS repo and build up a 12-factor agent. This walkthrough will guide you through creating a TypeScript agent that follows the 12-factor methodology."
title: "Building the 12-factor agent template from scratch in python"
text: "Steps to start from a bare python repo and build up a 12-factor agent. This walkthrough will guide you through creating a python agent that follows the 12-factor methodology with baml."
targets:
- markdown: "./build/walkthrough.md"
onChange:
diff: true
cp: true
newFiles:
cat: false
cp: true
- folders:
path: "./build/sections"
skip:
- "cleanup"
final:
dirName: "final"
- ipynb: "./build/workshop-2025-07-16.ipynb"
sections:
- name: cleanup
title: "Cleanup"
text: "Make sure you're starting from a clean slate"
steps:
- text: "Clean up existing files"
command: |
rm -rf baml_src/ && rm -rf src/
- name: hello-world
title: "Chapter 0 - Hello World"
text: "Let's start with a basic TypeScript setup and a hello world program."
text: "Let's start with a basic python setup and a hello world program."
steps:
- text: |
This guide is written in TypeScript (yes, a python version is coming soon)
There are many checkpoints between the every file edit in theworkshop steps,
so even if you aren't super familiar with typescript,
you should be able to keep up and run each example.
To run this guide, you'll need a relatively recent version of nodejs and npm installed
You can use whatever nodejs version manager you want, [homebrew](https://formulae.brew.sh/formula/node) is fine
command:
brew install node@20
results:
- text: "You should see the node version"
code: |
node --version
- text: "Copy initial package.json"
file: {src: ./walkthrough/00-package.json, dest: package.json}
- text: "Install dependencies"
command: |
npm install
incremental: true
- text: "Copy tsconfig.json"
file: {src: ./walkthrough/00-tsconfig.json, dest: tsconfig.json}
- text: "add .gitignore"
file: {src: ./walkthrough/00-.gitignore, dest: .gitignore}
- text: "Create src folder"
dir: {create: true, path: src}
- text: "Add a simple hello world index.ts"
file: {src: ./walkthrough/00-index.ts, dest: src/index.ts}
- text: "Run it to verify"
command: |
npx tsx src/index.ts
results:
- text: "You should see:"
code: |
hello, world!
- text: "Let's start with a basic python setup and a hello world program."
file: {src: ./walkthrough/00-main.py}
- text: "lets run it"
command: <something to tell our script to run the main function>
- name: cli-and-agent
title: "Chapter 1 - CLI and Agent Loop"
text: "Now let's add BAML and create our first agent with a CLI interface."
@@ -77,608 +21,27 @@ sections:
First, we'll need to install [BAML](https://github.com/boundaryml/baml)
which is a tool for prompting and structured outputs.
command: |
npm install @boundaryml/baml
incremental: true
!pip install baml-py
- text: "Initialize BAML"
command: |
npx baml-cli init
incremental: true
!baml-cli init
- text: "Remove default resume.baml"
command: |
rm baml_src/resume.baml
incremental: true
!rm baml_src/resume.baml
- text: "Add our starter agent, a single baml prompt that we'll build on"
file: {src: ./walkthrough/01-agent.baml, dest: baml_src/agent.baml}
- text: "Generate BAML client code"
command: |
npx baml-cli generate
incremental: true
!baml-cli generate
- text: "Enable BAML logging for this section"
command: |
export BAML_LOG=debug
- text: "Add the CLI interface"
file: {src: ./walkthrough/01-cli.ts, dest: src/cli.ts}
- text: "Update index.ts to use the CLI"
file: {src: ./walkthrough/01-index.ts, dest: src/index.ts}
file: {src: ./walkthrough/01-cli.py}
- text: "Add the agent implementation"
file: {src: ./walkthrough/01-agent.ts, dest: src/agent.ts}
file: {src: ./walkthrough/01-agent.py}
- text: "update our main.py to use the CLI"
file: {src: ./walkthrough/01-main.py}
- text: |
The the BAML code is configured to use BASETEN_API_KEY by default
To get a Baseten API key and URL, create an account at [baseten.co](https://baseten.co),
and then deploy [Qwen3 32B from the model library](https://www.baseten.co/library/qwen-3-32b/).
```rust
function DetermineNextStep(thread: string) -> DoneForNow {
client Qwen3
// ...
```
If you want to run the example with no changes, you can set the BASETEN_API_KEY env var to any valid baseten key.
If you want to try swapping out the model, you can change the `client` line.
[Docs on baml clients can be found here](https://docs.boundaryml.com/guide/baml-basics/switching-llms)
For example, you can configure [gemini](https://docs.boundaryml.com/ref/llm-client-providers/google-ai-gemini)
or [anthropic](https://docs.boundaryml.com/ref/llm-client-providers/anthropic) as your model provider.
For example, to use openai with an OPENAI_API_KEY, you can do:
client "openai/gpt-4o"
- text: Set your env vars
command: |
export BASETEN_API_KEY=...
export BASETEN_BASE_URL=...
- text: "Try it out"
command: |
npx tsx src/index.ts hello
results:
- text: you should see a familiar response from the model
code: |
{
intent: 'done_for_now',
message: 'Hello! How can I assist you today?'
}
- name: calculator-tools
title: "Chapter 2 - Add Calculator Tools"
text: "Let's add some calculator tools to our agent."
steps:
- text: |
Let's start by adding a tool definition for the calculator
These are simpile structured outputs that we'll ask the model to
return as a "next step" in the agentic loop.
file: {src: ./walkthrough/02-tool_calculator.baml, dest: baml_src/tool_calculator.baml}
- text: |
Now, let's update the agent's DetermineNextStep method to
expose the calculator tools as potential next steps
file: {src: ./walkthrough/02-agent.baml, dest: baml_src/agent.baml}
- text: "Generate updated BAML client"
command: |
npx baml-cli generate
incremental: true
- text: "Try out the calculator"
command: |
npx tsx src/index.ts 'can you add 3 and 4'
results:
- text: "You should see a tool call to the calculator"
code: |
{
intent: 'add',
a: 3,
b: 4
}
- name: tool-loop
title: "Chapter 3 - Process Tool Calls in a Loop"
text: "Now let's add a real agentic loop that can run the tools and get a final answer from the LLM."
steps:
- text: |
First, lets update the agent to handle the tool call
file: {src: ./walkthrough/03-agent.ts, dest: src/agent.ts}
- text: |
Now, lets try it out
command: |
npx tsx src/index.ts 'can you add 3 and 4'
results:
- text: you should see the agent call the tool and then return the result
code: |
{
intent: 'done_for_now',
message: 'The sum of 3 and 4 is 7.'
}
- text: "For the next step, we'll do a more complex calculation, let's turn off the baml logs for more concise output"
command: |
export BAML_LOG=off
- text: "Try a multi-step calculation"
command: |
npx tsx src/index.ts 'can you add 3 and 4, then add 6 to that result'
- text: "you'll notice that tools like multiply and divide are not available"
command: |
npx tsx src/index.ts 'can you multiply 3 and 4'
- text: |
next, let's add handlers for the rest of the calculator tools
file: {src: ./walkthrough/03b-agent.ts, dest: src/agent.ts}
- text: "Test subtraction"
command: |
npx tsx src/index.ts 'can you subtract 3 from 4'
- text: |
now, let's test the multiplication tool
command: |
npx tsx src/index.ts 'can you multiply 3 and 4'
- text: |
finally, let's test a more complex calculation with multiple operations
command: |
npx tsx src/index.ts 'can you multiply 3 and 4, then divide the result by 2 and then add 12 to that result'
- text: |
congratulations, you've taking your first step into hand-rolling an agent loop.
from here, we're going to start incorporating some more intermediate and advanced
concepts for 12-factor agents.
- name: baml-tests
title: "Chapter 4 - Add Tests to agent.baml"
text: "Let's add some tests to our BAML agent."
steps:
- text: to start, leave the baml logs enabled
command: |
export BAML_LOG=debug
- text: |
next, let's add some tests to the agent
We'll start with a simple test that checks the agent's ability to handle
a basic calculation.
file: {src: ./walkthrough/04-agent.baml, dest: baml_src/agent.baml}
- text: "Run the tests"
command: |
npx baml-cli test
- text: |
now, let's improve the test with assertions!
Assertions are a great way to make sure the agent is working as expected,
and can easily be extended to check for more complex behavior.
file: {src: ./walkthrough/04b-agent.baml, dest: baml_src/agent.baml}
- text: "Run the tests"
command: |
npx baml-cli test
- text: |
as you add more tests, you can disable the logs to keep the output clean.
You may want to turn them on as you iterate on specific tests.
command: |
export BAML_LOG=off
- text: |
now, let's add some more complex test cases,
where we resume from in the middle of an in-progress
agentic context window
file: {src: ./walkthrough/04c-agent.baml, dest: baml_src/agent.baml}
- text: |
let's try to run it
command: |
npx baml-cli test
- name: human-tools
title: "Chapter 5 - Multiple Human Tools"
text: |
In this section, we'll add support for multiple tools that serve to
contact humans.
steps:
- text: "for this section, we'll disable the baml logs. You can optionally enable them if you want to see more details."
command: |
export BAML_LOG=off
- text: |
first, let's add a tool that can request clarification from a human
this will be different from the "done_for_now" tool,
and can be used to more flexibly handle different types of human interactions
in your agent.
file: {src: ./walkthrough/05-agent.baml, dest: baml_src/agent.baml}
- text: |
next, let's re-generate the client code
NOTE - if you're using the VSCode extension for BAML,
the client will be regenerated automatically when you save the file
in your editor.
command: |
npx baml-cli generate
incremental: true
- text: |
now, let's update the agent to use the new tool
file: {src: ./walkthrough/05-agent.ts, dest: src/agent.ts}
- text: |
next, let's update the CLI to handle clarification requests
by requesting input from the user on the CLI
file: {src: ./walkthrough/05-cli.ts, dest: src/cli.ts}
- text: |
let's try it out
command: |
npx tsx src/index.ts 'can you multiply 3 and FD*(#F&& '
- text: |
next, let's add a test that checks the agent's ability to handle
a clarification request
file: {src: ./walkthrough/05b-agent.baml, dest: baml_src/agent.baml}
- text: |
and now we can run the tests again
command: |
npx baml-cli test
- text: |
you'll notice the new test passes, but the hello world test fails
This is because the agent's default behavior is to return "done_for_now"
file: {src: ./walkthrough/05c-agent.baml, dest: baml_src/agent.baml}
- text: "Verify tests pass"
command: |
npx baml-cli test
- name: customize-prompt
title: "Chapter 6 - Customize Your Prompt with Reasoning"
text: |
In this section, we'll explore how to customize the prompt of the agent
with reasoning steps.
this is core to [factor 2 - own your prompts](https://github.com/humanlayer/12-factor-agents/blob/main/content/factor-2-own-your-prompts.md)
there's a deep dive on reasoning on AI That Works [reasoning models versus reasoning steps](https://github.com/hellovai/ai-that-works/tree/main/2025-04-07-reasoning-models-vs-prompts)
steps:
- text: "for this section, it will be helpful to leave the baml logs enabled"
command: |
export BAML_LOG=debug
- text: |
update the agent prompt to include a reasoning step
file: {src: ./walkthrough/06-agent.baml, dest: baml_src/agent.baml}
- text: generate the updated client
command: |
npx baml-cli generate
incremental: true
- text: |
now, you can try it out with a simple prompt
command: |
npx tsx src/index.ts 'can you multiply 3 and 4'
results:
- text: you should see output from the baml logs showing the reasoning steps
- text: |
#### optional challenge
add a field to your tool output format that includes the reasoning steps in the output!
- name: context-window
title: "Chapter 7 - Customize Your Context Window"
text: |
In this section, we'll explore how to customize the context window
of the agent.
this is core to [factor 3 - own your context window](https://github.com/humanlayer/12-factor-agents/blob/main/content/factor-3-own-your-context-window.md)
steps:
- text: |
update the agent to pretty-print the Context window for the model
file: {src: ./walkthrough/07-agent.ts, dest: src/agent.ts}
- text: "Test the formatting"
command: |
BAML_LOG=info npx tsx src/index.ts 'can you multiply 3 and 4, then divide the result by 2 and then add 12 to that result'
- text: |
next, let's update the agent to use XML formatting instead
this is a very popular format for passing data to a model,
among other things, because of the token efficiency of XML.
file: {src: ./walkthrough/07b-agent.ts, dest: src/agent.ts}
- text: |
let's try it out
command: |
BAML_LOG=info npx tsx src/index.ts 'can you multiply 3 and 4, then divide the result by 2 and then add 12 to that result'
- text: |
lets update our tests to match the new output format
file: {src: ./walkthrough/07c-agent.baml, dest: baml_src/agent.baml}
- text: |
check out the updated tests
command: |
npx baml-cli test
- name: api-endpoints
title: "Chapter 8 - Adding API Endpoints"
text: "Add an Express server to expose the agent via HTTP."
steps:
- text: "for this section, we'll disable the baml logs. You can optionally enable them if you want to see more details."
command: |
export BAML_LOG=off
- text: "Install Express and types"
command: |
npm install express && npm install --save-dev @types/express supertest
incremental: true
- text: "Add the server implementation"
file: {src: ./walkthrough/08-server.ts, dest: src/server.ts}
- text: "Start the server"
command: |
npx tsx src/server.ts
- text: "Test with curl (in another terminal)"
command: |
curl -X POST http://localhost:3000/thread \
-H "Content-Type: application/json" \
-d '{"message":"can you add 3 and 4"}'
results:
- text: |
You should get an answer from the agent which includes the
agentic trace, ending in a message like:
code: |
{"intent":"done_for_now","message":"The sum of 3 and 4 is 7."}
- name: state-management
title: "Chapter 9 - In-Memory State and Async Clarification"
text: "Add state management and async clarification support."
steps:
- text: "for this section, we'll disable the baml logs. You can optionally enable them if you want to see more details."
command: |
export BAML_LOG=off
- text: "Add some simple in-memory state management for threads"
file: {src: ./walkthrough/09-state.ts, dest: src/state.ts}
- text: |
update the server to use the state management
* Add thread state management using `ThreadStore`
* return thread IDs and response URLs from the /thread endpoint
* implement GET /thread/:id
* implement POST /thread/:id/response
file: {src: ./walkthrough/09-server.ts, dest: src/server.ts}
- text: "Start the server"
command: |
npx tsx src/server.ts
- text: "Test clarification flow"
command: |
curl -X POST http://localhost:3000/thread \
-H "Content-Type: application/json" \
-d '{"message":"can you multiply 3 and xyz"}'
- name: human-approval
title: "Chapter 10 - Adding Human Approval"
text: "Add support for human approval of operations."
steps:
- text: "for this section, we'll disable the baml logs. You can optionally enable them if you want to see more details."
command: |
export BAML_LOG=off
- text: |
update the server to handle human approvals
* Import `handleNextStep` to execute approved actions
* Add two payload types to distinguish approvals from responses
* Handle responses and approvals differently in the endpoint
* Show better error messages when things go wrongs
file: {src: ./walkthrough/10-server.ts, dest: src/server.ts}
- text: "Add a few methods to the agent to handle approvals and responses"
file: {src: ./walkthrough/10-agent.ts, dest: src/agent.ts}
- text: "Start the server"
command: |
npx tsx src/server.ts
- text: "Test division with approval"
command: |
curl -X POST http://localhost:3000/thread \
-H "Content-Type: application/json" \
-d '{"message":"can you divide 3 by 4"}'
results:
- text: "You should see:"
code: |
{
"thread_id": "2b243b66-215a-4f37-8bc6-9ace3849043b",
"events": [
{
"type": "user_input",
"data": "can you divide 3 by 4"
},
{
"type": "tool_call",
"data": {
"intent": "divide",
"a": 3,
"b": 4,
"response_url": "/thread/2b243b66-215a-4f37-8bc6-9ace3849043b/response"
}
}
]
}
- text: "reject the request with another curl call, changing the thread ID"
command: |
curl -X POST 'http://localhost:3000/thread/{thread_id}/response' \
-H "Content-Type: application/json" \
-d '{"type": "approval", "approved": false, "comment": "I dont think thats right, use 5 instead of 4"}'
results:
- text: 'You should see: the last tool call is now `"intent":"divide","a":3,"b":5`'
code: |
{
"events": [
{
"type": "user_input",
"data": "can you divide 3 by 4"
},
{
"type": "tool_call",
"data": {
"intent": "divide",
"a": 3,
"b": 4,
"response_url": "/thread/2b243b66-215a-4f37-8bc6-9ace3849043b/response"
}
},
{
"type": "tool_response",
"data": "user denied the operation with feedback: \"I dont think thats right, use 5 instead of 4\""
},
{
"type": "tool_call",
"data": {
"intent": "divide",
"a": 3,
"b": 5,
"response_url": "/thread/1f1f5ff5-20d7-4114-97b4-3fc52d5e0816/response"
}
}
]
}
- text: "now you can approve the operation"
command: |
curl -X POST 'http://localhost:3000/thread/{thread_id}/response' \
-H "Content-Type: application/json" \
-d '{"type": "approval", "approved": true}'
results:
- text: "you should see the final message includes the tool response and final result!"
code: |
...
{
"type": "tool_response",
"data": 0.5
},
{
"type": "done_for_now",
"message": "I divided 3 by 6 and the result is 0.5. If you have any more operations or queries, feel free to ask!",
"response_url": "/thread/2b469403-c497-4797-b253-043aae830209/response"
}
- name: humanlayer-approval
title: "Chapter 11 - Human Approvals over email"
text: |
in this section, we'll add support for human approvals over email.
This will start a little bit contrived, just to get the concepts down -
We'll start by invoking the workflow from the CLI but approvals for `divide`
and `request_more_information` will be handled over email,
then the final `done_for_now` answer will be printed back to the CLI
While contrived, this is a great example of the flexibility you get from
[factor 7 - contact humans with tools](https://github.com/humanlayer/12-factor-agents/blob/main/content/factor-7-contact-humans-with-tools.md)
steps:
- text: "for this section, we'll disable the baml logs. You can optionally enable them if you want to see more details."
command: |
export BAML_LOG=off
- text: "Install HumanLayer"
command: |
npm install humanlayer
incremental: true
- text: "Update CLI to send `divide` and `request_more_information` to a human via email"
file: {src: ./walkthrough/11-cli.ts, dest: src/cli.ts}
- text: "Run the CLI"
command: |
npx tsx src/index.ts 'can you divide 4 by 5'
results:
- text: "The last line of your program should mention human review step"
code: |
nextStep { intent: 'divide', a: 4, b: 5 }
HumanLayer: Requested human approval from HumanLayer cloud
- text: |
go ahead and respond to the email with some feedback:
![reject-email](https://github.com/humanlayer/12-factor-agents/blob/main/workshops/2025-05/walkthrough/11-email-reject.png?raw=true)
- text: |
you should get another email with an updated attempt based on your feedback!
You can go ahead and approve this one:
![approve-email](https://github.com/humanlayer/12-factor-agents/blob/main/workshops/2025-05/walkthrough/11-email-approve.png?raw=true)
results:
- text: and your final output will look like
code: |
nextStep {
intent: 'done_for_now',
message: 'The division of 4 by 5 is 0.8. If you have any other calculations or questions, feel free to ask!'
}
The division of 4 by 5 is 0.8. If you have any other calculations or questions, feel free to ask!
- text: |
lets implement the `request_more_information` flow as well
file: {src: ./walkthrough/11b-cli.ts, dest: src/cli.ts}
- text: |
lets test the require_approval flow as by asking for a calculation
with garbled input:
command: |
npx tsx src/index.ts 'can you multiply 4 and xyz'
- text: "You should get an email with a request for clarification"
command: |
Can you clarify what 'xyz' represents in this context? Is it a specific number, variable, or something else?
- text: you can response with something like
command: |
use 8 instead of xyz
results:
- text: you should see a final result on the CLI like
code: |
I have multiplied 4 and xyz, using the value 8 for xyz, resulting in 32.
- text: |
as a final step, lets explore using a custom html template for the email
file: {src: ./walkthrough/11c-cli.ts, dest: src/cli.ts}
- text: |
first try with divide:
command: |
npx tsx src/index.ts 'can you divide 4 by 5'
results:
- text: |
you should see a slightly different email with the custom template
![custom-template-email](https://github.com/humanlayer/12-factor-agents/blob/main/workshops/2025-05/walkthrough/11-email-custom.png?raw=true)
feel free to run with the flow and then you can try updating the template to your liking
(if you're using cursor, something as simple as highlighting the template and asking to "make it better"
should do the trick)
try triggering "request_more_information" as well!
- text: |
thats it - in the next chapter, we'll build a fully email-driven
workflow agent that uses webhooks for human approval
- name: humanlayer-webhook
title: "Chapter XX - HumanLayer Webhook Integration"
text: |
the previous sections used the humanlayer SDK in "synchronous mode" - that
means every time we wait for human approval, we sit in a loop
polling until the human response if received.
That's obviously not ideal, especially for production workloads,
so in this section we'll implement [factor 6 - launch / pause / resume with simple APIs](https://github.com/humanlayer/12-factor-agents/blob/main/content/factor-6-launch-pause-resume.md)
by updating the server to end processing after contacting a human, and use webhooks to receive the results.
steps:
- text: |
add code to initialize humanlayer in the server
file: {src: ./walkthrough/12-1-server-init.ts, dest: src/server.ts}
- text: |
next, lets update the /thread endpoint to
1. handle requests asynchronously, returning immediately
2. create a human contact on request_more_information and done_for_now calls
# file: {src: }
- text: |
Update the server to be able to handle request_clarification responses
- remove the old /response endpoint and types
- update the /thread endpoint to run processing asynchronously, return immediately
- send a state.threadId when requesting human responses
- add a handleHumanResponse function to process the human response
- add a /webhook endpoint to handle the webhook response
file: {src: ./walkthrough/12a-server.ts, dest: src/server.ts}
- text: "Start the server in another terminal"
command: |
npx tsx src/server.ts
- text: |
now that the server is running, send a payload to the '/thread' endpoint
- text: __ do the response step
- text: __ now handle approvals for divide
- text: __ now also handle done_for_now
try it out
command: <something to tell our script to run the main.py>