jack-zxy commited on
Commit
1814c32
·
verified ·
1 Parent(s): 1ba4154

Update vllm info

Browse files

Seems sth wrong with vLLM TP deployment, remove it for now. And add detailed information about vllm docker image.

Files changed (1) hide show
  1. deployment_guide.md +1 -19
deployment_guide.md CHANGED
@@ -59,25 +59,7 @@ lmdeploy serve api_server \
59
 
60
  ## vLLM
61
 
62
- - Tensor Parallelism + Expert Parallelism
63
-
64
- ```bash
65
- # start ray on node 0 and node 1
66
-
67
- # node 0
68
- export VLLM_ENGINE_READY_TIMEOUT_S=10000
69
- vllm serve internlm/Intern-S1-Pro \
70
- --tensor-parallel-size 16 \
71
- --enable-expert-parallel \
72
- --distributed-executor-backend ray \
73
- --max-model-len 65536 \
74
- --trust-remote-code \
75
- --reasoning-parser deepseek_r1 \
76
- --enable-auto-tool-choice \
77
- --tool-call-parser hermes
78
- ```
79
-
80
- - Data Parallelism + Expert Parallelism
81
 
82
  ```bash
83
  # node 0
 
59
 
60
  ## vLLM
61
 
62
+ You can use the vLLM nightly-built docker image `vllm/vllm-openai:nightly` to deploy. Refer to [using-docker](https://docs.vllm.ai/en/latest/deployment/docker/?h=docker) for more.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
 
64
  ```bash
65
  # node 0