more work
This commit is contained in:
Kyle Corbitt
@@ -9,7 +9,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
@@ -88,12 +88,12 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Remember I got a \"test.jsonl\" file from OpenPipe back in [./prepare.ipynb](./prepare.ipynb)? Since that is data formatted the same way as our training data but that we didn't use for training, we can use it to check our model's performance."
|
||||
"Remember I got a \"test.jsonl\" file from OpenPipe back in [./prepare.ipynb](./prepare.ipynb)? That's data from our dataset that we didn't use in training, so we can use it to check our model's performance."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
@@ -111,7 +111,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
@@ -121,7 +121,7 @@
|
||||
"Sample prompt:\n",
|
||||
"--------------\n",
|
||||
"### Instruction:\n",
|
||||
"[{\"role\":\"system\",\"content\":\"Your goal is to classify a recipe along several dimensions. You should \"},{\"role\":\"user\",\"content\":\"Strawberry Sorbet\\n\\nIngredients:\\n- 2 cups chopped strawberries Safeway 1 lb For $3.99 thru 02/09\\n- 1 cup cold water\\n- 2 cups boiling water\\n- 1 pkg. (4-serving size) JELL-O Strawberry Flavor Gelatin\\n- 1/2 cup sugar\\n\\nDirections:\\n- Place strawberries and cold water in blender container; cover.\\n- Blend on high speed until smooth.\\n- Stir boiling water into combined dry gelatin mix and sugar in medium bowl at least 2 minutes until completely dissolved.\\n- Add strawberry mixture; mix well.\\n- Pour into 9-inch square pan.\\n- Freeze 1 to 1-1/2 hours or until ice crystals form 1 inch around edges of pan.\\n- Spoon half of the gelatin mixture into blender container; cover.\\n- Blend on high speed about 30 seconds or until smooth; pour into bowl.\\n- Repeat with remaining gelatin mixture.\\n- Add to blended gelatin mixture in bowl; mix well.\\n- Return to pan.\\n- Freeze 6 hours or overnight until firm.\\n- Scoop into dessert dishes to serve.\\n- Store leftover sorbet in freezer.\"}]\n",
|
||||
"[{\"role\":\"system\",\"content\":\"Your goal is to classify a recipe along several dimensions.Pay attention to the instructions.\"},{\"role\":\"user\",\"content\":\"Pan Gravy\\n\\nIngredients:\\n- 1/3 cup all purpose flour\\n- 1/3 cup turkey drippings\\n- 3 cup water or broth\\n- 1/8 to 1/4 teaspoon salt\\n- 1/8 tsp pepper\\n\\nDirections:\\n- In a skillet or roasting pan, add flour to drippings; blend well.\\n- Cook over medium heat 2 to 3 minutes until smooth and light brown, stirring constantly.\\n- Add water; cook until mixture boils and thickens, stirring constantly.\\n- Stir in salt and pepper.\\n- *Flour and drippings can be decreased to 1/4 cup each for thinner gravy.\\n- *\"}]\n",
|
||||
"\n",
|
||||
"### Response:\n",
|
||||
"\n"
|
||||
@@ -152,22 +152,22 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"INFO 08-24 18:58:03 llm_engine.py:70] Initializing an LLM engine with config: model='./models/run1/merged', tokenizer='./models/run1/merged', tokenizer_mode=auto, trust_remote_code=False, dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)\n",
|
||||
"INFO 08-24 18:59:18 llm_engine.py:196] # GPU blocks: 3419, # CPU blocks: 512\n"
|
||||
"INFO 08-24 22:01:58 llm_engine.py:70] Initializing an LLM engine with config: model='./models/run1/merged', tokenizer='./models/run1/merged', tokenizer_mode=auto, trust_remote_code=False, dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)\n",
|
||||
"INFO 08-24 22:02:46 llm_engine.py:196] # GPU blocks: 3419, # CPU blocks: 512\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Processed prompts: 100%|██████████| 201/201 [00:16<00:00, 12.18it/s]"
|
||||
"Processed prompts: 100%|██████████| 500/500 [00:37<00:00, 13.44it/s]"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -176,7 +176,7 @@
|
||||
"text": [
|
||||
"Sample output:\n",
|
||||
"--------------\n",
|
||||
"{\"role\":\"assistant\",\"content\":null,\"function_call\":{\"name\":\"classify\",\"arguments\":\"{\\n\\\"has_non_fish_meat\\\": false,\\n\\\"requires_oven\\\": false,\\n\\\"requires_stove\\\": true,\\n\\\"cook_time_over_30_mins\\\": false,\\n\\\"main_course\\\": false\\n}\"}}\n"
|
||||
"{\"role\":\"assistant\",\"content\":null,\"function_call\":{\"name\":\"classify\",\"arguments\":\"{\\n\\\"has_non_fish_meat\\\": true,\\n\\\"requires_oven\\\": false,\\n\\\"requires_stove\\\": true,\\n\\\"cook_time_over_30_mins\\\": false,\\n\\\"main_dish\\\": false\\n}\"}}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -211,19 +211,19 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ok, we have our outputs! Since there are 5 categories we classify each recipe on, a natural metric would be for each recipe and each category, what percentage of the time our model's output matches GPT-4's. I'll write a quick eval function to check that."
|
||||
"Ok, we have our outputs! There are 5 categories we classify each recipe on, so let's check what percentage of the time our model's output matches GPT-4's. I'll write a quick eval function for that:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"execution_count": 26,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Overall accuracy: 0.91\n"
|
||||
"Overall accuracy: 0.95\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
@@ -231,7 +231,7 @@
|
||||
"import json\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def parse_fn_call_args(str):\n",
|
||||
"def parse_fn_call(str):\n",
|
||||
" \"\"\"Parse the function call arguments from the response\"\"\"\n",
|
||||
" response_dict = json.loads(str)\n",
|
||||
" args_dict = json.loads(response_dict[\"function_call\"][\"arguments\"])\n",
|
||||
@@ -241,12 +241,12 @@
|
||||
"\n",
|
||||
"def calculate_accuracy(row):\n",
|
||||
" \"\"\"Calculate the fraction of my model's outputs that match the reference outputs\"\"\"\n",
|
||||
" true_outputs = parse_fn_call_args(row[\"output\"])\n",
|
||||
" my_outputs = parse_fn_call_args(row[\"my_outputs\"])\n",
|
||||
" true_outputs = parse_fn_call(row[\"output\"])\n",
|
||||
" my_outputs = parse_fn_call(row[\"my_outputs\"])\n",
|
||||
"\n",
|
||||
" num_matching_outputs = 0\n",
|
||||
" for key in true_outputs.keys():\n",
|
||||
" if true_outputs[key] == my_outputs[key]:\n",
|
||||
" if key in my_outputs and true_outputs[key] == my_outputs[key]:\n",
|
||||
" num_matching_outputs += 1\n",
|
||||
"\n",
|
||||
" return num_matching_outputs / len(true_outputs)\n",
|
||||
@@ -261,9 +261,371 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Not bad! Of course, the next obvious step is to look at where Llama 2 is \"wrong\" and evaluate the types of errors it makes. I've exported a Google Sheet where I did exactly that with an earlier version of this model trained on the same dataset. You can see that [here](https://docs.google.com/spreadsheets/d/1vn-nA0CRQwz-BvEYvxUcO1-EP80ZbPhcxDoCTttvsmI/edit?usp=sharing).\n",
|
||||
"Not bad! However, there are still a few rows where the model outputs don't match. Let's take a closer look."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 35,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Alligator Sauce Piquant\n",
|
||||
"\n",
|
||||
"Ingredients:\n",
|
||||
"- 2 lb. alligator, boneless and cubed *\n",
|
||||
"- 4 onions, diced\n",
|
||||
"- 1 c. parsley, chopped\n",
|
||||
"- 4 stalks celery, chopped\n",
|
||||
"- 1 bell pepper, diced\n",
|
||||
"- 1 c. catsup\n",
|
||||
"- 2 Tbsp. Heinz steak sauce\n",
|
||||
"- 2 Tbsp. soy sauce\n",
|
||||
"- 2 Tbsp. Louisiana hot sauce\n",
|
||||
"- 2 Tbsp. cornstarch\n",
|
||||
"- 1 tsp. salt\n",
|
||||
"- 2 tsp. red pepper (ground)\n",
|
||||
"- 1/4 c. cooking oil\n",
|
||||
"\n",
|
||||
"Directions:\n",
|
||||
"- *Alligator must be free of all fat; also dark meat is the best (leg and body meat), boneless.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>GPT-4</th>\n",
|
||||
" <th>My model</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>cook_time_over_30_mins</th>\n",
|
||||
" <td>True</td>\n",
|
||||
" <td>False</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>main_dish</th>\n",
|
||||
" <td>True</td>\n",
|
||||
" <td>False</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" GPT-4 My model\n",
|
||||
"cook_time_over_30_mins True False\n",
|
||||
"main_dish True False"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Veggie Casserole\n",
|
||||
"\n",
|
||||
"Ingredients:\n",
|
||||
"- 1 (8 oz.) bag mixed veggies (corn, peas, carrots, green beans), steamed\n",
|
||||
"- 1 c. celery\n",
|
||||
"- 1 c. onions\n",
|
||||
"- 1 c. Cheddar cheese\n",
|
||||
"- 1 c. mayonnaise\n",
|
||||
"\n",
|
||||
"Directions:\n",
|
||||
"- Mix above ingredients.\n",
|
||||
"- Bake at 350° for 30 minutes, until bubbly.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>GPT-4</th>\n",
|
||||
" <th>My model</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>main_dish</th>\n",
|
||||
" <td>False</td>\n",
|
||||
" <td>True</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" GPT-4 My model\n",
|
||||
"main_dish False True"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Rhonda'S Butter Chess Pie\n",
|
||||
"\n",
|
||||
"Ingredients:\n",
|
||||
"- 5 eggs\n",
|
||||
"- 1 stick melted butter\n",
|
||||
"- 2 c. sugar\n",
|
||||
"- 1 tsp. vanilla\n",
|
||||
"- 1 Tbsp. cornstarch\n",
|
||||
"- 1/2 c. buttermilk\n",
|
||||
"- unbaked 9-inch deep dish pie shell\n",
|
||||
"\n",
|
||||
"Directions:\n",
|
||||
"- Mix eggs with sugar and cornstarch until smooth.\n",
|
||||
"- Add melted butter, vanilla and buttermilk.\n",
|
||||
"- Bake at 350° for 30 minutes or until done.\n",
|
||||
"- Let cool and chill.\n",
|
||||
"- Similar to Furr's Butter Chess Pie.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>GPT-4</th>\n",
|
||||
" <th>My model</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>cook_time_over_30_mins</th>\n",
|
||||
" <td>False</td>\n",
|
||||
" <td>True</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" GPT-4 My model\n",
|
||||
"cook_time_over_30_mins False True"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Broccoli Gorgonzola Cream Soup\n",
|
||||
"\n",
|
||||
"Ingredients:\n",
|
||||
"- 2 heads Broccoli\n",
|
||||
"- 700 milliliters Water\n",
|
||||
"- 1 Onion, Peeled And Cut Into Chunks\n",
|
||||
"- 1 pinch Salt\n",
|
||||
"- 1 teaspoon Oregano\n",
|
||||
"- 1 Potato, Peeled And Cut Into Chunks\n",
|
||||
"- 200 grams Crumbled Gorgonzola\n",
|
||||
"- 1 Tablespoon Finely Grated Parmesan\n",
|
||||
"\n",
|
||||
"Directions:\n",
|
||||
"- Cut off the hard trunks of the broccoli and cut it into small pieces. Prepare a pot with water, add broccoli, onion, salt and oregano and boil for about 30 minutes.\n",
|
||||
"- Add the peeled potato and boil for another 20 minutes. When vegetables are cooked, strain and save the stock.\n",
|
||||
"- Using a hand blender, puree vegetables, adding as much stock as desired. Bring soup back to heat over low heat, and sir in gorgonzola. Remove from heat and add Parmesan.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>GPT-4</th>\n",
|
||||
" <th>My model</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>main_dish</th>\n",
|
||||
" <td>False</td>\n",
|
||||
" <td>True</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" GPT-4 My model\n",
|
||||
"main_dish False True"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Wild Rice With Cucumber And Feta\n",
|
||||
"\n",
|
||||
"Ingredients:\n",
|
||||
"- 1 (8.5-ounce) package precooked wild rice (such as Archer Farms)\n",
|
||||
"- 1 cup diced English cucumber\n",
|
||||
"- 1 1/2 tablespoons olive oil\n",
|
||||
"- 1 tablespoon fresh lemon juice\n",
|
||||
"- 2 ounces crumbled feta cheese\n",
|
||||
"- 1/2 teaspoon pepper\n",
|
||||
"- 1/4 teaspoon salt\n",
|
||||
"\n",
|
||||
"Directions:\n",
|
||||
"- Prepare rice according to the package directions.\n",
|
||||
"- Combine cooked rice, cucumber, olive oil, lemon juice, and crumbled feta cheese in a medium bowl; toss to coat. Stir in pepper and salt.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>GPT-4</th>\n",
|
||||
" <th>My model</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>main_dish</th>\n",
|
||||
" <td>True</td>\n",
|
||||
" <td>False</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" GPT-4 My model\n",
|
||||
"main_dish True False"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"The main takeaway: generally places where GPT-4 and Llama 2 disagreed were genuinely ambiguous cases, where either answer was acceptable (eg. a dish that takes about 30 mins to cook might be classified as over 30 minutes by one, and under 30 minutes by the other).\n",
|
||||
"np.random.seed(42)\n",
|
||||
"\n",
|
||||
"for row in test_data[test_data.accuracy < 1].sample(5).itertuples():\n",
|
||||
" print(json.loads(row.instruction)[1][\"content\"])\n",
|
||||
"\n",
|
||||
" gpt4_output = parse_fn_call(row.output)\n",
|
||||
" my_output = parse_fn_call(row.my_outputs)\n",
|
||||
"\n",
|
||||
" table = pd.DataFrame(\n",
|
||||
" {\n",
|
||||
" \"GPT-4\": gpt4_output,\n",
|
||||
" \"My model\": my_output,\n",
|
||||
" }\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" table = table[table[\"GPT-4\"] != table[\"My model\"]]\n",
|
||||
" display(table)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Looking at the outputs, it's clear that our model still makes some mistakes. But at the same time, there are plenty of examples like \"Rhonda's Butter Chess Pie\" where our model gets it right, even though GPT-4 got it wrong! And there are also cases like the \"Veggie Casserole\", where the \"right\" answer is truly ambiguous and really both answers are defensible.\n",
|
||||
"\n",
|
||||
"Interested in cost/latency benchmarking? You can check out [./benchmarking.ipynb](./benchmarking.ipynb) for an overview of my findings!"
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user