alihan/OSWorld
1
0
Fork 0
mirror of https://github.com/xlang-ai/OSWorld.git synced 2024-04-29 12:26:03 +03:00
Code Issues Packages Projects Releases Wiki Activity

Update ROADMAP.md

Browse Source
This commit is contained in:
Tianbao Xie
2024-04-22 12:41:07 +08:00
committed by GitHub
parent 176390cfb9
commit e2d6842535
1 changed files with 9 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
ROADMAP.md
View File

@@ -4,29 +4,30 @@ If you are interested in contributing to the project, please check the [CONTRIBU
## Road Map for Environment Infrastructure
- [x] Explore VMWare, and whether it can be connected and control through mouse package
- [x] Explore VMWare, and whether it can be connected and controlled through the mouse package
- [x] Explore Windows and MacOS, whether it can be installed
- MacOS is closed source and cannot be legally installed
- Windows is available legally and can be installed
- [x] Build gym-like python interface for controlling the VM
- [x] Build a gym-like Python interface for controlling the VM
- [x] Recording of actions (mouse movement, click, keyboard) for humans to annotate, and we can replay it and compress it
- [x] Build a simple task, e.g. open a browser, open a website, click on a button, and close the browser
- [x] Set up a pipeline and build agents implementation (zero-shot) for the task
- [x] Start to design on which tasks inside the DesktopENv to focus on, start to wrap up the environment to be public
- [x] Set up a pipeline and build agent implementation (zero-shot) for the task
- [x] Start to design which tasks inside the DesktopENv to focus on, start to wrap up the environment to be public
- [x] Start to annotate the examples for ~~training~~ and testing
- [x] Error handling during file passing and file opening, etc.
- [x] Add accessibility tree from the OS into the observation space
- [x] Add pre-process and post-process action support for benchmarking setup and evaluation
- [x] Experiment logging and visualization system
- [x] Add more tasks, maybe scale to 300 for v1.0.0, and create a dynamic leaderboard
- [x] Multiprocess support, this can enable the reinforcement learning to be more efficient
- [x] Multiprocess support, can enable reinforcement learning to be more efficient
- [x] Add support for automatic VM download and configuration, enable auto-scaling management
- [ ] Support running on platform that have nested virtualization, e.g. Google Cloud, AWS, etc.
- [ ] VPN setup doc for those who need it
- [ ] Support running on platforms that have nested virtualization, e.g. Google Cloud, AWS, etc.
- [ ] Prepare for the first release of Windows vm image for the environment
## Road Map of Annotation Tool
- [ ] Improve the annotation tool base on DuckTrack, make it more robust which align on accessibility tree
- [ ] Improve the annotation tool base on DuckTrack, and make it more robust which aligns on accessibility tree
- [ ] Annotate the steps of doing the task
- [ ] Crawl all resources we explored from the internet, and make it easy to access
- [ ] Set up ways for community to contribute new examples
- [ ] Set up ways for the crowd-sourcing/community to contribute new examples