Update vllm info
Browse filesSeems sth wrong with vLLM TP deployment, remove it for now. And add detailed information about vllm docker image.
- deployment_guide.md +1 -19
deployment_guide.md
CHANGED
|
@@ -59,25 +59,7 @@ lmdeploy serve api_server \
|
|
| 59 |
|
| 60 |
## vLLM
|
| 61 |
|
| 62 |
-
-
|
| 63 |
-
|
| 64 |
-
```bash
|
| 65 |
-
# start ray on node 0 and node 1
|
| 66 |
-
|
| 67 |
-
# node 0
|
| 68 |
-
export VLLM_ENGINE_READY_TIMEOUT_S=10000
|
| 69 |
-
vllm serve internlm/Intern-S1-Pro \
|
| 70 |
-
--tensor-parallel-size 16 \
|
| 71 |
-
--enable-expert-parallel \
|
| 72 |
-
--distributed-executor-backend ray \
|
| 73 |
-
--max-model-len 65536 \
|
| 74 |
-
--trust-remote-code \
|
| 75 |
-
--reasoning-parser deepseek_r1 \
|
| 76 |
-
--enable-auto-tool-choice \
|
| 77 |
-
--tool-call-parser hermes
|
| 78 |
-
```
|
| 79 |
-
|
| 80 |
-
- Data Parallelism + Expert Parallelism
|
| 81 |
|
| 82 |
```bash
|
| 83 |
# node 0
|
|
|
|
| 59 |
|
| 60 |
## vLLM
|
| 61 |
|
| 62 |
+
You can use the vLLM nightly-built docker image `vllm/vllm-openai:nightly` to deploy. Refer to [using-docker](https://docs.vllm.ai/en/latest/deployment/docker/?h=docker) for more.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
|
| 64 |
```bash
|
| 65 |
# node 0
|