WIP Text to SQL - checkpoint

skills/text_to_sql/data/results.csv

@@ -1,4 +1,4 @@
-user_query,[3.5 Sonnet] prompts.py:generate_prompt,[3.5 Sonnet] prompts.py:generate_prompt_with_examples,[3.5 Sonnet] prompts.py:generate_prompt_with_cot,[3.5 Sonnet] prompts.py:generate_prompt_with_rag,[3 Haiku] prompts.py:generate_prompt,[3 Haiku] prompts.py:generate_prompt_with_examples,[3 Haiku] prompts.py:generate_prompt_with_cot,[3 Haiku] prompts.py:generate_prompt_with_rag
+user_query,[3 Haiku] prompts.py:generate_prompt,[3 Haiku] prompts.py:generate_prompt_with_examples,[3 Haiku] prompts.py:generate_prompt_with_cot,[3 Haiku] prompts.py:generate_prompt_with_rag,[3 Haiku] prompts.py:generate_prompt_with_self_improvement,[3.5 Sonnet] prompts.py:generate_prompt,[3.5 Sonnet] prompts.py:generate_prompt_with_examples,[3.5 Sonnet] prompts.py:generate_prompt_with_cot,[3.5 Sonnet] prompts.py:generate_prompt_with_rag,[3.5 Sonnet] prompts.py:generate_prompt_with_self_improvement
 What are the names of all employees in the Engineering department?,"[PASS] (1.00)
 
 <sql>
@@ -19,6 +19,82 @@ WHERE d.name = 'Engineering';
 
 Pass Reason: All assertions passed","[PASS] (1.00)
 
+<thought_process>
+1. We need to join the employees and departments tables to get the department information for each employee.
+2. We'll match employees.department_id with departments.id to connect the two tables.
+3. We'll filter for the 'Engineering' department name.
+4. We only need to return the employee names.
+</thought_process>
+
+<sql>
+SELECT e.name
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering';
+</sql>
+
+Pass Reason: All assertions passed","[PASS] (1.00)
+
+<thought_process>
+To solve this query, I will need to join the `employees` and `departments` tables on the `department_id` and `id` columns, and then select the `name` column from the `employees` table where the `name` column in the `departments` table is 'Engineering'.
+
+The key steps are:
+1. Join the `employees` and `departments` tables on the `department_id` and `id` columns.
+2. Filter the results to only include rows where the `name` column in the `departments` table is 'Engineering'.
+3. Select the `name` column from the `employees` table.
+</thought_process>
+
+<sql>
+SELECT e.name
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering';
+</sql>
+
+Pass Reason: All assertions passed","[PASS] (1.00)
+
+<attempt1>
+<thought_process>
+To get the names of all employees in the Engineering department, we need to join the `employees` and `departments` tables on the `department_id` and `id` columns, and then filter the results to only include rows where the department name is 'Engineering'.
+</thought_process>
+<sql>
+SELECT e.name
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering';
+</sql>
+<error>
+This query should work as expected and return the names of all employees in the Engineering department.
+</error>
+</attempt1>
+
+<final_sql>
+SELECT e.name
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering';
+</final_sql>
+
+Pass Reason: All assertions passed","[PASS] (1.00)
+
+<sql>
+SELECT employees.name
+FROM employees
+JOIN departments ON employees.department_id = departments.id
+WHERE departments.name = 'Engineering';
+</sql>
+
+Pass Reason: All assertions passed","[PASS] (1.00)
+
+<sql>
+SELECT e.name
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering';
+</sql>
+
+Pass Reason: All assertions passed","[PASS] (1.00)
+
 <thought_process>
 1. We need to join the employees and departments tables to get the department information for each employee.
 2. We'll match employees.department_id with departments.id for the join.
@@ -59,18 +135,57 @@ WHERE departments.name = 'Engineering';
 
 Pass Reason: All assertions passed","[PASS] (1.00)
 
+<attempt1>
+<thought_process>
+To find the names of all employees in the Engineering department, we need to join the employees and departments tables. We'll use the department_id from the employees table to match with the id from the departments table. Then we'll filter for the Engineering department and select the employee names.
+</thought_process>
 <sql>
 SELECT employees.name
 FROM employees
 JOIN departments ON employees.department_id = departments.id
 WHERE departments.name = 'Engineering';
 </sql>
+<error>
+This query should work correctly, but there's a small chance it might fail if the 'Engineering' department name is case-sensitive or if there are leading/trailing spaces in the department name in the database.
+</error>
+</attempt1>
 
-Pass Reason: All assertions passed","[PASS] (1.00)
+<attempt2>
+<thought_process>
+To address the potential case-sensitivity issue and possible leading/trailing spaces, we can modify the WHERE clause to use LOWER() function and TRIM() function. This will ensure that the comparison is case-insensitive and ignores any extra spaces.
+</thought_process>
+<sql>
+SELECT employees.name
+FROM employees
+JOIN departments ON employees.department_id = departments.id
+WHERE LOWER(TRIM(departments.name)) = LOWER('Engineering');
+</sql>
+</attempt2>
+
+<final_sql>
+SELECT employees.name
+FROM employees
+JOIN departments ON employees.department_id = departments.id
+WHERE LOWER(TRIM(departments.name)) = LOWER('Engineering');
+</final_sql>
+
+Pass Reason: All assertions passed"
+How many employees are in the Engineering department?,"[PASS] (1.00)
+
+    <sql>
+SELECT COUNT(*) AS num_employees
+FROM employees
+WHERE department_id = (
+SELECT id
+FROM departments
+WHERE name = 'Engineering'
+);
+</sql>
+
+    Pass Reason: All assertions passed","[PASS] (1.00)
 
 <sql>
-SELECT e.name
-FROM employees e
+SELECT COUNT(*) FROM employees e
 JOIN departments d ON e.department_id = d.id
 WHERE d.name = 'Engineering';
 </sql>
@@ -81,12 +196,11 @@ Pass Reason: All assertions passed","[PASS] (1.00)
 1. We need to join the employees and departments tables to get the department information for each employee.
 2. We'll match employees.department_id with departments.id to connect the two tables.
 3. We'll filter for the 'Engineering' department name.
-4. We only need to return the employee names.
+4. We'll use the COUNT function to count the number of rows that match the filter.
 </thought_process>
 
 <sql>
-SELECT e.name
-FROM employees e
+SELECT COUNT(*) FROM employees e
 JOIN departments d ON e.department_id = d.id
 WHERE d.name = 'Engineering';
 </sql>
@@ -94,23 +208,46 @@ WHERE d.name = 'Engineering';
 Pass Reason: All assertions passed","[PASS] (1.00)
 
 <thought_process>
-To solve this query, I will need to join the `employees` and `departments` tables on the `department_id` and `id` columns, and then select the `name` column from the `employees` table where the `name` column in the `departments` table is 'Engineering'.
+To solve this query, I will need to join the `employees` and `departments` tables on the `department_id` column in the `employees` table and the `id` column in the `departments` table. This will allow me to access the department name and count the number of employees in the ""Engineering"" department.
 
 The key steps are:
-1. Join the `employees` and `departments` tables on the `department_id` and `id` columns.
-2. Filter the results to only include rows where the `name` column in the `departments` table is 'Engineering'.
-3. Select the `name` column from the `employees` table.
+1. Join the `employees` and `departments` tables on the appropriate columns.
+2. Filter the results to only include rows where the department name is ""Engineering"".
+3. Count the number of rows in the filtered result to get the total number of employees in the Engineering department.
 </thought_process>
 
 <sql>
-SELECT e.name
+SELECT COUNT(*) AS num_employees
 FROM employees e
 JOIN departments d ON e.department_id = d.id
 WHERE d.name = 'Engineering';
 </sql>
 
-Pass Reason: All assertions passed"
-How many employees are in the Engineering department?,"[PASS] (1.00)
+Pass Reason: All assertions passed","[PASS] (1.00)
+
+<attempt1>
+<thought_process>
+To get the number of employees in the Engineering department, I need to join the `employees` and `departments` tables on the `department_id` and `id` columns, and then filter for the department with the name 'Engineering'.
+</thought_process>
+<sql>
+SELECT COUNT(*) 
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering';
+</sql>
+<error>
+This query should work as expected and return the number of employees in the Engineering department.
+</error>
+</attempt1>
+
+<final_sql>
+SELECT COUNT(*) 
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering';
+</final_sql>
+
+Pass Reason: All assertions passed","[PASS] (1.00)
 
 <sql>
 SELECT COUNT(*) 
@@ -168,59 +305,203 @@ WHERE d.name = 'Engineering';
 
 Pass Reason: All assertions passed","[PASS] (1.00)
 
-    <sql>
-SELECT COUNT(*) AS num_employees
-FROM employees
-WHERE department_id = (
-SELECT id
-FROM departments
-WHERE name = 'Engineering'
-);
-</sql>
-
-    Pass Reason: All assertions passed","[PASS] (1.00)
-
+    <attempt1>
+<thought_process>
+To count the number of employees in the Engineering department, we need to join the employees and departments tables, filter for the Engineering department, and then count the number of employees. We'll use a LEFT JOIN to ensure we get all employees, even if there's no matching department.
+</thought_process>
 <sql>
-SELECT COUNT(*) FROM employees e
-JOIN departments d ON e.department_id = d.id
+SELECT COUNT(*) as employee_count
+FROM employees e
+LEFT JOIN departments d ON e.department_id = d.id
 WHERE d.name = 'Engineering';
 </sql>
+<error>
+This query might fail if the 'Engineering' department doesn't exist in the departments table. It would return 0 instead of an error, which might not be the desired behavior if we want to catch such cases.
+</error>
+</attempt1>
 
-Pass Reason: All assertions passed","[PASS] (1.00)
-
+<attempt2>
 <thought_process>
-1. We need to join the employees and departments tables to get the department information for each employee.
-2. We'll match employees.department_id with departments.id to connect the two tables.
-3. We'll filter for the 'Engineering' department name.
-4. We'll use the COUNT function to count the number of rows that match the filter.
+To address the potential issue of the 'Engineering' department not existing, we can first check if the department exists, and then count the employees if it does. We'll use a subquery to achieve this.
 </thought_process>
-
 <sql>
-SELECT COUNT(*) FROM employees e
-JOIN departments d ON e.department_id = d.id
+SELECT 
+CASE 
+  WHEN EXISTS (SELECT 1 FROM departments WHERE name = 'Engineering')
+  THEN (
+    SELECT COUNT(*) 
+    FROM employees e
+    JOIN departments d ON e.department_id = d.id
+    WHERE d.name = 'Engineering'
+  )
+  ELSE NULL
+END as employee_count;
+</sql>
+<error>
+This query is more robust, but it might be overly complex for the given task. It also doesn't provide a clear indication if the department doesn't exist, returning NULL instead.
+</error>
+</attempt2>
+
+<attempt3>
+<thought_process>
+Let's simplify the query while still ensuring we get a meaningful result. We'll use an INNER JOIN instead of a LEFT JOIN, which will naturally return 0 if the department doesn't exist or has no employees.
+</thought_process>
+<sql>
+SELECT COUNT(e.id) as employee_count
+FROM employees e
+INNER JOIN departments d ON e.department_id = d.id
 WHERE d.name = 'Engineering';
 </sql>
+</attempt3>
 
-Pass Reason: All assertions passed","[PASS] (1.00)
+<final_sql>
+SELECT COUNT(e.id) as employee_count
+FROM employees e
+INNER JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering';
+</final_sql>
 
-<thought_process>
-To solve this query, I will need to join the `employees` and `departments` tables on the `department_id` column in the `employees` table and the `id` column in the `departments` table. This will allow me to access the department name and count the number of employees in the ""Engineering"" department.
+    Pass Reason: All assertions passed"
+"Give me the name, age, and salary of the oldest employee in the Engineering department.","[PASS] (1.00)
 
-The key steps are:
-1. Join the `employees` and `departments` tables on the appropriate columns.
-2. Filter the results to only include rows where the department name is ""Engineering"".
-3. Count the number of rows in the filtered result to get the total number of employees in the Engineering department.
-</thought_process>
+Here is the SQL query to get the name, age, and salary of the oldest employee in the Engineering department:
 
 <sql>
-SELECT COUNT(*) AS num_employees
+SELECT e.name, e.age, e.salary
 FROM employees e
 JOIN departments d ON e.department_id = d.id
-WHERE d.name = 'Engineering';
+WHERE d.name = 'Engineering'
+ORDER BY e.age DESC
+LIMIT 1;
 </sql>
 
-Pass Reason: All assertions passed"
-"Give me the name, age, and salary of the oldest employee in the Engineering department.","[PASS] (1.00)
+Pass Reason: All assertions passed","[PASS] (1.00)
+
+<sql>
+SELECT e.name, e.age, e.salary
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering'
+ORDER BY e.age DESC
+LIMIT 1;
+</sql>
+
+Pass Reason: All assertions passed","[PASS] (1.00)
+
+<thought_process>
+1. We need to work with the employees and departments tables to get the information for the employees in the Engineering department.
+2. We'll join the two tables on the department_id foreign key.
+3. We'll filter for the Engineering department by checking the name column in the departments table.
+4. To get the oldest employee, we'll order the results by the age column in descending order and take the first result.
+5. We'll select the name, age, and salary columns from the employees table.
+</thought_process>
+
+<sql>
+SELECT e.name, e.age, e.salary
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering'
+ORDER BY e.age DESC
+LIMIT 1;
+</sql>
+
+Pass Reason: All assertions passed","[PASS] (1.00)
+
+   <thought_process>
+To solve this query, I will need to follow these steps:
+1. Identify the relevant tables and columns needed to answer the query.
+- The relevant tables are 'employees' and 'departments'.
+- The relevant columns are 'name', 'age', 'salary', and 'department_id' from the 'employees' table, and 'name' from the 'departments' table.
+2. Filter the 'employees' table to only include employees from the 'Engineering' department.
+3. Order the filtered results by the 'age' column in descending order to get the oldest employee.
+4. Select the 'name', 'age', and 'salary' columns from the filtered and ordered results.
+</thought_process>
+
+<sql>
+SELECT e.name, e.age, e.salary
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering'
+ORDER BY e.age DESC
+LIMIT 1;
+</sql>
+
+   Pass Reason: All assertions passed","[PASS] (1.00)
+
+    <attempt1>
+<thought_process>
+To get the name, age, and salary of the oldest employee in the Engineering department, I will first need to find the department ID for the Engineering department. I can do this by querying the departments table and filtering for the name 'Engineering'. Then, I can use that department ID to filter the employees table and select the oldest employee's name, age, and salary.
+</thought_process>
+<sql>
+SELECT e.name, e.age, e.salary
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering'
+ORDER BY e.age DESC
+LIMIT 1;
+</sql>
+<error>
+This query should work, but it assumes that the 'Engineering' department name is unique. If there are multiple departments with the name 'Engineering', this query will only return the oldest employee from the first matching department.
+</error>
+</attempt1>
+
+<attempt2>
+<thought_process>
+To ensure we get the oldest employee from the Engineering department, even if there are multiple departments with that name, we can modify the query to first find the department ID for the Engineering department with the maximum employee age, and then use that to filter the employees table.
+</thought_process>
+<sql>
+SELECT e.name, e.age, e.salary
+FROM employees e
+WHERE e.department_id = (
+SELECT d.id
+FROM departments d
+WHERE d.name = 'Engineering'
+ORDER BY (
+  SELECT MAX(e2.age)
+  FROM employees e2
+  WHERE e2.department_id = d.id
+) DESC
+LIMIT 1
+)
+ORDER BY e.age DESC
+LIMIT 1;
+</sql>
+<error>
+This query should work, but it may be less efficient than the previous attempt, as it requires two subqueries to find the department ID and the maximum age within that department.
+</error>
+</attempt2>
+
+<attempt3>
+<thought_process>
+Instead of using subqueries, we can try to achieve the same result with a single query by joining the employees and departments tables, and then using window functions to find the maximum age within each department.
+</thought_process>
+<sql>
+SELECT e.name, e.age, e.salary
+FROM (
+SELECT e.*, RANK() OVER (PARTITION BY e.department_id ORDER BY e.age DESC) AS rk
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering'
+) t
+WHERE t.rk = 1;
+</sql>
+<error>
+This query should work and is more efficient than the previous attempts, as it only requires a single join and a window function to find the oldest employee in the Engineering department.
+</error>
+</attempt3>
+
+<final_sql>
+SELECT e.name, e.age, e.salary
+FROM (
+SELECT e.*, RANK() OVER (PARTITION BY e.department_id ORDER BY e.age DESC) AS rk
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering'
+) t
+WHERE t.rk = 1;
+</final_sql>
+
+    Pass Reason: All assertions passed","[PASS] (1.00)
 
 <sql>
 SELECT e.name, e.age, e.salary
@@ -289,66 +570,80 @@ LIMIT 1;
 
   Pass Reason: All assertions passed","[PASS] (1.00)
 
-Here is the SQL query to get the name, age, and salary of the oldest employee in the Engineering department:
-
-<sql>
-SELECT e.name, e.age, e.salary
-FROM employees e
-JOIN departments d ON e.department_id = d.id
-WHERE d.name = 'Engineering'
-ORDER BY e.age DESC
-LIMIT 1;
-</sql>
-
-Pass Reason: All assertions passed","[PASS] (1.00)
-
-<sql>
-SELECT e.name, e.age, e.salary
-FROM employees e
-JOIN departments d ON e.department_id = d.id
-WHERE d.name = 'Engineering'
-ORDER BY e.age DESC
-LIMIT 1;
-</sql>
-
-Pass Reason: All assertions passed","[PASS] (1.00)
-
+  <attempt1>
 <thought_process>
-1. We need to work with the employees and departments tables to get the information for the employees in the Engineering department.
-2. We'll join the two tables on the department_id foreign key.
-3. We'll filter for the Engineering department by checking the name column in the departments table.
-4. To get the oldest employee, we'll order the results by the age column in descending order and take the first result.
-5. We'll select the name, age, and salary columns from the employees table.
+To find the oldest employee in the Engineering department, we need to join the employees and departments tables, filter for the Engineering department, and then select the employee with the highest age. We'll use MAX(age) to find the oldest employee and then join this result back to get the name, age, and salary.
 </thought_process>
-
 <sql>
 SELECT e.name, e.age, e.salary
 FROM employees e
 JOIN departments d ON e.department_id = d.id
 WHERE d.name = 'Engineering'
-ORDER BY e.age DESC
+AND e.age = (
+SELECT MAX(age)
+FROM employees e2
+JOIN departments d2 ON e2.department_id = d2.id
+WHERE d2.name = 'Engineering'
+)
 LIMIT 1;
 </sql>
+<error>
+This query might fail if there are multiple employees with the same maximum age in the Engineering department. The LIMIT 1 will arbitrarily select one of them, which may not be the desired behavior.
+</error>
+</attempt1>
 
-Pass Reason: All assertions passed","[PASS] (1.00)
-
-   <thought_process>
-To solve this query, I will need to follow these steps:
-1. Identify the relevant tables and columns needed to answer the query.
-- The relevant tables are 'employees' and 'departments'.
-- The relevant columns are 'name', 'age', 'salary', and 'department_id' from the 'employees' table, and 'name' from the 'departments' table.
-2. Filter the 'employees' table to only include employees from the 'Engineering' department.
-3. Order the filtered results by the 'age' column in descending order to get the oldest employee.
-4. Select the 'name', 'age', and 'salary' columns from the filtered and ordered results.
+<attempt2>
+<thought_process>
+To address the potential issue of multiple employees with the same maximum age, we can modify the query to order by hire_date as well. This way, if there are multiple oldest employees, we'll get the one who was hired first.
 </thought_process>
-
 <sql>
 SELECT e.name, e.age, e.salary
 FROM employees e
 JOIN departments d ON e.department_id = d.id
 WHERE d.name = 'Engineering'
-ORDER BY e.age DESC
+AND e.age = (
+SELECT MAX(age)
+FROM employees e2
+JOIN departments d2 ON e2.department_id = d2.id
+WHERE d2.name = 'Engineering'
+)
+ORDER BY e.hire_date ASC
 LIMIT 1;
 </sql>
+<error>
+This query should work correctly, but it might be more efficient to use a window function instead of a subquery.
+</error>
+</attempt2>
 
-   Pass Reason: All assertions passed"
+<attempt3>
+<thought_process>
+Let's optimize the query by using a window function (ROW_NUMBER()) to rank employees by age and hire date. This approach is more efficient and easier to read.
+</thought_process>
+<sql>
+WITH ranked_employees AS (
+SELECT e.name, e.age, e.salary,
+       ROW_NUMBER() OVER (ORDER BY e.age DESC, e.hire_date ASC) as rn
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering'
+)
+SELECT name, age, salary
+FROM ranked_employees
+WHERE rn = 1;
+</sql>
+</attempt3>
+
+<final_sql>
+WITH ranked_employees AS (
+SELECT e.name, e.age, e.salary,
+       ROW_NUMBER() OVER (ORDER BY e.age DESC, e.hire_date ASC) as rn
+FROM employees e
+JOIN departments d ON e.department_id = d.id
+WHERE d.name = 'Engineering'
+)
+SELECT name, age, salary
+FROM ranked_employees
+WHERE rn = 1;
+</final_sql>
+
+  Pass Reason: All assertions passed"

skills/text_to_sql/evaluation/promptfooconfig.yaml

@@ -3,13 +3,13 @@
 python_path: /opt/homebrew/bin/python3
 
 providers:
-  - id: anthropic:messages:claude-3-5-sonnet-20240620
-    label: "3.5 Sonnet"
+  - id: anthropic:messages:claude-3-haiku-20240307
+    label: "3 Haiku"
     config:
       max_tokens: 4096
       temperature: 0
-  - id: anthropic:messages:claude-3-haiku-20240307
-    label: "3 Haiku"
+  - id: anthropic:messages:claude-3-5-sonnet-20240620
+    label: "3.5 Sonnet"
     config:
       max_tokens: 4096
       temperature: 0
@@ -19,6 +19,7 @@ prompts:
   - prompts.py:generate_prompt_with_examples
   - prompts.py:generate_prompt_with_cot
   - prompts.py:generate_prompt_with_rag
+  - prompts.py:generate_prompt_with_self_improvement
 
 tests:
   - description: "Check syntax of simple query"

skills/text_to_sql/evaluation/prompts.py

@@ -168,3 +168,77 @@ def generate_prompt_with_rag(context):
 
     Ensure your SQL query is compatible with SQLite syntax.
     """
+
+
+def generate_prompt_with_self_improvement(context):
+    from vectordb import VectorDB
+
+    # Load the vector database
+    vectordb = VectorDB()
+    vectordb.load_db()
+
+    user_query = context['vars']['user_query']
+
+    if not vectordb.embeddings:
+        with sqlite3.connect(DATABASE_PATH) as conn:
+            cursor = conn.cursor()
+            cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
+            schema_data = [
+                {"text": f"Table: {table[0]}, Column: {col[1]}, Type: {col[2]}", 
+                "metadata": {"table": table[0], "column": col[1], "type": col[2]}}
+                for table in cursor.fetchall()
+                for col in cursor.execute(f"PRAGMA table_info({table[0]})").fetchall()
+            ]
+        vectordb.load_data(schema_data)
+
+    relevant_schema = vectordb.search(user_query, k=10, similarity_threshold=0.3)
+    schema_info = "\n".join([f"Table: {item['metadata']['table']}, Column: {item['metadata']['column']}, Type: {item['metadata']['type']}" 
+                             for item in relevant_schema])
+    
+    return f"""You are an AI assistant that converts natural language queries into SQL.
+    Given the following relevant columns from the SQL database schema:
+
+    <schema>
+    {schema_info}
+    </schema>
+
+    Convert the following natural language query into SQL:
+
+    <query>
+    {user_query}
+    </query>
+
+    Please provide up to three attempts to generate the correct SQL query. For each attempt:
+    1. Explain your thought process in <thought_process> tags.
+    2. Provide the SQL query in <sql> tags.
+    3. Imagine you are executing this query. If you think it might fail, explain why in <error> tags.
+    4. If you think the query might fail, provide an improved version in the next attempt.
+
+    After your attempts, provide your final, best SQL query in <final_sql> tags.
+
+    Format your response like this:
+
+    <attempt1>
+    <thought_process>
+    Your thought process here
+    </thought_process>
+    <sql>
+    Your SQL query here
+    </sql>
+    <error>
+    Potential error message here (if any)
+    </error>
+    </attempt1>
+
+    <attempt2>
+    ...
+    </attempt2>
+
+    <attempt3>
+    ...
+    </attempt3>
+
+    <final_sql>
+    Your final, best SQL query here
+    </final_sql>
+    """