deepseekmirror/DeepSeek-V2
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-V2.git synced 2025-04-19 10:09:07 -04:00
Code Issues Actions Packages Projects Releases Wiki Activity

docs: update README for LMDeploy support

Browse Source
This commit is contained in:
zhyncs 2024-06-26 16:22:55 +08:00 committed by GitHub
parent c15bff20d9
commit 0d928f89a4
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 8 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@ -321,6 +321,14 @@ generated_text = [output.outputs[0].text for output in outputs]
print(generated_text)
```
### Inference with LMDeploy (recommended)
To utilize [LMDeploy](https://github.com/InternLM/lmdeploy) for model inference, please install LMDeploy >= v0.5.0. The current LMDeploy PyTorch Engine already supports the [implementation](https://github.com/InternLM/lmdeploy/pull/1621) of the MLA version for DeepSeek V2. If you want to use the pipeline API, please refer to [pipeline](https://lmdeploy.readthedocs.io/en/latest/api/pipeline.html?highlight=pipeline#pipeline).
```bash
# use LMDeploy PyTorch Engine as OpenAI compatible Server
python3 -m lmdeploy serve api_server DeepSeek-V2 --backend pytorch
```
### LangChain Support
Since our API is compatible with OpenAI, you can easily use it in [langchain](https://www.langchain.com/).
Here is an example: