alihan/llama-cpp-python
1
0
Fork 0
mirror of https://github.com/abetlen/llama-cpp-python.git synced 2023-09-07 17:34:22 +03:00
Code Issues Packages Projects Releases Wiki Activity

Add Dockerfile + build workflow

Browse Source
This commit is contained in:
Niek van der Maas
2023-04-12 11:53:39 +02:00
parent 9f1e565594
commit c14201dc0f
3 changed files with 48 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@@ -66,6 +66,14 @@ python3 -m llama_cpp.server
Navigate to [http://localhost:8000/docs](http://localhost:8000/docs) to see the OpenAPI documentation.
## Docker image
A Docker image is available on [GHCR](https://ghcr.io/abetlen/llama-cpp-python). To run the server:
```bash
docker run --rm -it -p8000:8000 -v /path/to/models:/models -eMODEL=/models/ggml-vicuna-7b-4bit.bin ghcr.io/abetlen/llama-cpp-python:latest
```
## Low-level API
The low-level API is a direct `ctypes` binding to the C API provided by `llama.cpp`.