alihan/OmniParser
1
0
Fork 0
mirror of https://github.com/microsoft/OmniParser.git synced 2025-02-18 03:18:33 +03:00
Code Issues Packages Projects Releases Wiki Activity

minor readme

Browse Source
This commit is contained in:
yadonglu
2025-01-27 13:50:38 -08:00
parent 5d49a5aca6
commit dcca70c364
3 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -12,7 +12,7 @@
**OmniParser** is a comprehensive method for parsing user interface screenshots into structured and easy-to-understand elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface.
## News
- [2025/1] V2 is coming. We achieve new state of the art results (39.5%) on [Screen Spot Pro](https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding/tree/main) with OmniParser v2 (will be released soon)! Read more details [here](https://github.com/microsoft/OmniParser/tree/master/docs).
- [2025/1] V2 is coming. We achieve new state of the art results 39.5% on the new grounding benchmark [Screen Spot Pro](https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding/tree/main) with OmniParser v2 (will be released soon)! Read more details [here](https://github.com/microsoft/OmniParser/tree/master/docs/Evaluation.md).
- [2024/11] We release an updated version, OmniParser V1.5 which features 1) more fine grained/small icon detection, 2) prediction of whether each screen element is interactable or not. Examples in the demo.ipynb.
- [2024/10] OmniParser was the #1 trending model on huggingface model hub (starting 10/29/2024).
- [2024/10] Feel free to checkout our demo on [huggingface space](https://huggingface.co/spaces/microsoft/OmniParser)! (stay tuned for OmniParser + Claude Computer Use)

2
docs/Evaluation.md
View File

@@ -1,4 +1,4 @@
# Eval setup for ScreenSpot Pro
We adapt the eval code from ScreenSpot Pro (ss pro) official repo: https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding/tree/main. This folder contains the inference script/results on this benchmark. We are under legal review proces to release omniparser v2. Once it is done, we will update the file so that it can load the v2 model.
We adapt the eval code from ScreenSpot Pro (ss pro) official [repo](https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding/tree/main). This folder contains the inference script/results on this benchmark. We going through legal review proces to release omniparser v2. Once it is done, we will update the file so that it can load the v2 model.
1. eval/ss_pro_gpt4o_omniv2.py: contains the prompt we use, it can be dropped in replacement for this [file](https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding/blob/main/models/gpt4x.py) in the original ss pro repo.
2. eval/logs_sspro_omniv2.json: contains the inferenced results for ss pro using GPT4o+OmniParserv2.

2
eval/ss_pro_gpt4o_omniv2.py
View File

@@ -10,7 +10,7 @@ import openai
from openai import BadRequestError
model_name = "gpt-4o-2024-05-13"
OPENAI_KEY = os.environ.get("OPENAI_API_KEY") # "nEvErGoNnAgIvEyOuUp"
OPENAI_KEY = os.environ.get("OPENAI_API_KEY")
def convert_pil_image_to_base64(image):
buffered = BytesIO()