alihan/OSWorld
1
0
Fork 0
mirror of https://github.com/xlang-ai/OSWorld.git synced 2024-04-29 12:26:03 +03:00
Code Issues Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Tianbao Xie
2024-04-02 20:11:04 +08:00
committed by GitHub
parent ca85a02e4b
commit d7aa3047f2
1 changed files with 14 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
README.md
View File

@@ -5,22 +5,18 @@
<a href="">Paper</a>
</p>
[![OSWorld](https://os-world.github.io/static/videos/main.mp4)](https://os-world.github.io/static/videos/main.mp4)
![Overview]()
## Updates
- 2024-03-28: We released our [paper](), [environment and benchmark](), and [project page](https://os-world.github.io/). Check it out!
- 2024-04-04: We released our [paper](), [environment and benchmark](https://github.com/xlang-ai/OSWorld), and [project page](https://os-world.github.io/). Check it out!
## Install
### Non-virtualized platform
Suppose you are on a system that has not yet been virtualized, meaning you are not on an AWS, Azure, or k8s virtualized environment.
Otherwise, refer to the [virtualized platform](https://github.com/xlang-ai/OSWorld?tab=readme-ov-file##virtualized-platform) part.
Otherwise, refer to the [virtualized platform](https://github.com/xlang-ai/OSWorld?tab=readme-ov-file#virtualized-platform) part.
1. Install [VMware Work Station Pro](https://www.vmware.com/products/workstation-pro/workstation-pro-evaluation.html) (for Apple Chips, it should be [VMware Fusion](https://www.vmware.com/go/getfusion)) and configure `vmrun` command, and verify successful installation by:
```bash
vmrun -T ws list
```
If the installation along with the environment variable set are successful, you will see the message showing the current running virtual machines.
If the installation along with the environment variable set is successful, you will see the message showing the current running virtual machines.
2. Install the environment package, and download the examples and the virtual machine image.
For x86_64 CPU Linux or Windows, you can install the environment package and download the examples and the virtual machine image by running the following commands:
@@ -50,9 +46,12 @@ Run the following minimal example to interact with the environment:
from desktop_env.envs.desktop_env import DesktopEnv
example = {
"id": "94d95f96-9699-4208-98ba-3c3119edf9c2",
"instruction": "I want to install Spotify on my current system. Could you please help me?",
"config": [{"type": "execute", "parameters": {"command": ["python","-c","import pyautogui; import time; pyautogui.click(960, 540); time.sleep(0.5);"]}}], "evaluator": {"func": "check_include_exclude", "result": {"type": "vm_command_line","command": "which spotify"}, "expected": {"type": "rule","rules": {"include": ["spotify"], "exclude": ["not found"]}}}
"id": "94d95f96-9699-4208-98ba-3c3119edf9c2",
"instruction": "I want to install Spotify on my current system. Could you please help me?",
"config": [{"type": "execute", "parameters": {
"command": ["python", "-c", "import pyautogui; import time; pyautogui.click(960, 540); time.sleep(0.5);"]}}],
"evaluator": {"func": "check_include_exclude", "result": {"type": "vm_command_line", "command": "which spotify"},
"expected": {"type": "rule", "rules": {"include": ["spotify"], "exclude": ["not found"]}}}
}
env = DesktopEnv(
path_to_vm="Ubuntu/Ubuntu.vmx",
@@ -65,17 +64,15 @@ obs, reward, done, info = env.step("pyautogui.rightClick()")
## Run Benchmark
### Run the Baseline Agent
If you want to run the baseline agent we use in our paper, you can run the following command as an example:
If you want to run the baseline agent we use in our paper, you can run the following command to run under the GPT-4V pure-screenshot setting as an example:
```bash
python run.py --path_to_vm Ubuntu/Ubuntu.vmx --headless --observation_type screenshot --model gpt-4-vision-preview
```
### Run Evaluation of Your Agent
Please first read through the [agent interface](https://github.com/xlang-ai/OSWorld/mm_agents/README.md) and the [environment interface](https://github.com/xlang-ai/OSWorld/desktop_env/README.md).
And implement the agent interface correctly and import you customized one in the `run.py` file.
Then, you can run the following command to evaluate your agent:
Please first read through the [agent interface](https://github.com/xlang-ai/OSWorld/blob/main/mm_agents/README.md) and the [environment interface](https://github.com/xlang-ai/OSWorld/blob/main/desktop_env/README.md).
Implement the agent interface correctly and import your customized one in the `run.py` file.
Then, you can run a similar command as the previous section to run the benchmark on your agent.
## Citation
If you find this environment useful, please consider citing our work: