mirror of
https://github.com/All-Hands-AI/OpenHands.git
synced 2024-08-29 01:18:33 +03:00
f9088766e8
* Add runtime container image setting * Fix typo in test * Fix sandbox base container image * Update variables * Update to base_container_image * Update tests/unit/test_config.py Co-authored-by: Xingyao Wang <xingyao6@illinois.edu> * Fixed eval * Fixed container_image * Fix typo --------- Co-authored-by: Xingyao Wang <xingyao6@illinois.edu>
Logic Reasoning Evaluation
This folder contains evaluation harness for evaluating agents on the logic reasoning benchmark ProntoQA and ProofWriter.
Setup Environment and LLM Configuration
Please follow instruction here to setup your local development environment and LLM.
Run Inference on logic_reasoning
The following code will run inference on the first example of the ProofWriter dataset,
./evaluation/logic_reasoning/scripts/run_infer.sh eval_gpt4_1106_preview_llm ProofWriter