alihan/reasoning-gym
1
0
Fork 0
mirror of https://github.com/open-thought/reasoning-gym.git synced 2025-10-09 13:40:09 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
reasoning-gym/notebooks/gsm_symbolic.ipynb
abdulhakeem 715102c277 Remove .DS_Store
2025-02-01 20:39:37 -06:00

391 lines
28 KiB
Plaintext
Raw Permalink Blame History

{
"cells": [
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"prompt_template = \"\"\"You need to generate python code for a synthetic procedural dataset. The dataset is similar to OpenAI's GSM8K which contains grade-school level math questions in natural language.\n",
"\n",
"Here is a the SOURCE item which you should translate into a python generator:\n",
"\n",
"```json\n",
"{0}\n",
"```\n",
"\n",
"As you can see we have already `question_annotated` and `answer_annotated`, but they are not really native python functions. Variable assignments are on the `question_annotated` field just after `#init:` \n",
"Some variables have `$` prefix, Some do not. These variables have some random values assigned to them through the use of `sample()` and `range()`. In addition, Since these are mathematical questions, there are also some conditions in the `question_annotatated` field marked by `#conditions:`. Your job is to do calculations making sure that these conditions are adhered to.\n",
"\n",
"Could you generate python code which would generate synthetic questions and answers, Essentially, you need to make sense of the question and adhere to the variable assignments and conditions in `question_annotated` field. Beside the question and answer I also need some metadata e.g. in a dict about the variables used(Some of the variables don't have intelligible name. e.g x, g, y etc. try to give make them intelligible name).\n",
"\n",
"I would like to use the generator function later to generate many different variants of questions and answers based on the same template while ensuring mathematical accuracy and consistency and inline with the conditions specified in the `question_annotated` field.\n",
"To control the difficulty I want to provide a floating point `difficulty` factor which could be used to scale the numeric ranges .. but please ensure the values integers (e.g cast back to int). If there are variables for which no values are provided like male_names, objects, names, fraction_alnum etc. please generate a list of values to sample from that fits in based on the context of the question.\n",
"\n",
"1. To make it modular and testable let's split the generator into one function called `generate_from_variables()` which gets the input variables and generates the question and answer texts. It should calculate the answer value from the inputs and the main randomized generator `generate_example()` (see below). \n",
"\n",
"2. The generator function should have a signature like`def generate_example(rng: Random, difficulty: float = 1.0) -> dict`.\n",
"\n",
"The output dict should contain:\n",
"{{\n",
" 'question': '<the generated question>',\n",
" 'answer': '<the_final_answer>', # here only the final answer, e.g. the number\n",
" 'metadata': {{\n",
" 'difficulty': difficulty,\n",
" 'answer_value:': <numeric_answer_value>,\n",
" 'answer_cot': '<full_long_form_answer>' # chain of thought, similar to 'answer' in the SOURCE\n",
" 'variables': {{\n",
" ... # the variable used\n",
" }}\n",
" }}\n",
"}}\n",
"\n",
"3. Write a simple `original_example()` function which calls `generate_from_variables()` and passes the original input values from SOURCE use in the json example above (in order to compare the output).\n",
"\n",
"Your task:\n",
"\n",
"- Generate reasonable random values for all the variables in line with the conditions in the `question_annotated` field\n",
"- Ensure mathematical consistency (results of divisions need to be integers)\n",
"- Create natural language question and answer texts\n",
"- Include metadata about the variables and solution\n",
"\n",
"\n",
"Here are useful examples of json input and python output:\n",
"\n",
"{1}\n",
"\n",
"Just generate the three python functions for the SOURCE dataset item - no additional explanation.\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"def write_python_code_with_prefix_and_number(code_string: str, num: int) -> str:\n",
" \"\"\"Writes Python code to (prefix + num) file\"\"\"\n",
" \n",
" file_path = Path(f\"../reasoning_gym/arithmetic/gsm_symbolic/generator_{num}.py\")\n",
" \n",
" try:\n",
" with open(file_path, 'w', encoding='utf-8') as f:\n",
" f.write(code_string)\n",
" print(f\"Python code written to {file_path.absolute()}\")\n",
" return file_path\n",
" except Exception as e:\n",
" print(f\"Error writing to file: {e}\")\n",
" return None"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"def read_file_to_string(file_path: str) -> str:\n",
" \"\"\"\n",
" Reads file content and returns as string\n",
" \n",
" Args:\n",
" file_path: Path to the file to read\n",
" \n",
" Returns:\n",
" str: File contents as string\n",
" \n",
" Raises:\n",
" FileNotFoundError: If file doesn't exist\n",
" IOError: If reading fails\n",
" \"\"\"\n",
" try:\n",
" with open(file_path, 'r', encoding='utf-8') as file:\n",
" return file.read()\n",
" except FileNotFoundError:\n",
" raise FileNotFoundError(f\"File not found: {file_path}\")\n",
" except IOError as e:\n",
" raise IOError(f\"Error reading file {file_path}: {str(e)}\")"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The dotenv extension is already loaded. To reload it, use:\n",
" %reload_ext dotenv\n"
]
}
],
"source": [
"# create open-router client, place your OPENROUTER_API_KEY in .env file\n",
"# .env contents:\n",
"# OPENROUTER_API_KEY=sk-or-v1- ...\n",
"\n",
"%load_ext dotenv\n",
"%dotenv\n",
"import os\n",
"import re\n",
"from pathlib import Path\n",
"from typing import Any, Iterable, Optional\n",
"import json\n",
"from openai import OpenAI\n",
"from openai.types.chat import ChatCompletion, ChatCompletionMessageParam\n",
"import time\n",
"\n",
"def llm_generate(\n",
" client: OpenAI,\n",
" messages: Iterable[ChatCompletionMessageParam],\n",
" sampling_params: dict[str, Any],\n",
") -> ChatCompletion:\n",
" max_retry = 3\n",
" for trial in range(max_retry):\n",
" try:\n",
" return client.chat.completions.create(\n",
" messages=messages,\n",
" **sampling_params,\n",
" )\n",
" except Exception as e:\n",
" print(\"failure response:\", e)\n",
" time.sleep(trial * trial) # quadratic backoff\n",
" if trial == max_retry - 1:\n",
" raise\n",
"\n",
"open_router_client = OpenAI(\n",
" base_url=\"https://openrouter.ai/api/v1\",\n",
" api_key=os.getenv(\"OPENROUTER_API_KEY\"),\n",
" timeout=90.0,\n",
")\n",
"\n",
"sampling_params = {\n",
" \"model\": \"anthropic/claude-3.5-sonnet\",\n",
" \"max_tokens\": 4096,\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Reading templates from path: /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/data/gsm_data/symbolic\n",
"Number of files: 100\n",
"Generating python code for GSM symbolic templates\n",
"\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_0.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_1.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_2.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_3.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_4.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_5.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_6.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_7.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_8.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_9.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_10.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_11.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_12.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_13.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_14.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_15.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_16.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_17.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_18.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_19.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_20.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_21.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_22.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_23.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_24.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_25.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_26.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_27.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_28.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_29.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_30.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_31.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_32.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_33.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_34.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_35.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_36.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_37.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_38.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_39.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_40.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_41.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_42.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_43.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_44.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_45.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_46.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_47.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_48.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_49.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_50.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_51.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_52.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_53.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_54.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_55.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_56.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_57.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_58.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_59.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_60.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_61.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_62.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_63.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_64.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_65.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_66.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_67.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_68.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_69.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_70.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_71.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_72.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_73.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_74.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_75.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_76.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_77.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_78.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_79.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_80.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_81.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_82.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_83.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_84.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_85.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_86.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_87.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_88.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_89.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_90.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_91.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_92.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_93.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_94.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_95.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_96.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_97.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_98.py\n",
"Python code written to /Users/abdulhakeemadefioye/Desktop/deep-learning/reasoning-gym/notebooks/../reasoning_gym/arithmetic/gsm_symbolic/generator_99.py\n",
"Python code generation for templates completed!\n",
"\n"
]
}
],
"source": [
"def generate_simple_request(user_prompt: str, developer_prompt: Optional[str] = None) -> list[dict]:\n",
" prompt = []\n",
" if developer_prompt is not None:\n",
" prompt.append( { \"role\": \"system\", \"content\": developer_prompt } )\n",
" \n",
" prompt.append( { \"role\": \"user\", \"content\": user_prompt })\n",
" return prompt\n",
" \n",
"\n",
"def eval_prompt_template(input: str, cot: str):\n",
" \n",
" user_request = prompt_template.format(input, cot)\n",
" input_messages = generate_simple_request(user_prompt=user_request)\n",
" output = llm_generate(open_router_client, input_messages, sampling_params)\n",
"\n",
" response = output.choices[0].message.content\n",
"\n",
" return response\n",
" \n",
"\n",
"# clone the gsm-symbolic from apple somewhere and set the path here, `git clone https://github.com/apple/ml-gsm-symbolic.git``\n",
"path_to_gsmsym = Path(\"../reasoning_gym/data/gsm_data/symbolic/\")\n",
"print(\"Reading templates from path: \", path_to_gsmsym.absolute())\n",
"\n",
"template_files = list(path_to_gsmsym.glob(\"*.json\"))\n",
"print(\"Number of files: \", len(template_files))\n",
"\n",
"gsm_symbolic_cot_file_path = Path(\"./gsm-symbolic-cot.txt\")\n",
"gsm_symbolic_cot = read_file_to_string(gsm_symbolic_cot_file_path)\n",
"\n",
"# Time to generate python code for all the gsm-symbolic templates\n",
"print(\"Generating python code for GSM symbolic templates\\n\")\n",
"for i, file in enumerate(template_files):\n",
" response_text = eval_prompt_template(file.read_text(), gsm_symbolic_cot)\n",
" # extract python source section\n",
" result_match = re.search(r\"^```.*\\n((.*\\n)+)```\", response_text, flags=re.MULTILINE)\n",
" python_source = result_match.group(1)\n",
" write_python_code_with_prefix_and_number(python_source, i)\n",
"\n",
"print(\"Python code generation for templates completed!\\n\")"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"John and Jack have 30 minutes to walk to school together. It takes them 6 minutes to get to the corner where the library is. It takes them another 13 minutes to get to the fire station. How much longer do they have to get to school without being late?\n",
"John and Jack have 30 minutes to walk to school together. It takes them 6 minutes to get to the corner where the library is. It takes them another 13 minutes to get to the fire station. How much longer do they have to get to school without being late?\n"
]
}
],
"source": [
"# WARNING: We are now executing the llm response without sandbox environment!\n",
"\n",
"scope = {} # eval generated python code here\n",
"\n",
"try:\n",
" exec(python_source, scope, scope)\n",
"except Exception as err:\n",
" raise\n",
"\n",
"\n",
"exec(\"output = original_example()\", scope, scope)\n",
"generated_data = scope[\"output\"]\n",
"print(generated_data['question'])\n",
"\n",
"\n",
"original_data = json.loads(template_files[54].read_text())\n",
"print(original_data['question'])\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "reasoning_gym",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.1"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Reference in New Issue View Git Blame Copy Permalink