vllm 中文文档
OpenAI 兼容服务器部署参数
模型下载
模型下载的渠道很多，这里使用 modelscope 进行下载，InternVL2_5-1B首页，
- 安装 modelscope
pip install modelscope
- 下载模型
from modelscope import snapshot_download
model_dir = snapshot_download('OpenGVLab/InternVL2_5-1B', local_dir="xxx/OpenGVLab/InternVL2_5-1B")
服务部署与请求
在 OpenGVLab/InternVL2_5-1B/config.json 中给定了初始化的超参数，例如temperature、top_p、top_k等，简单部署命令如下，默认使用 8000 端口
vllm serve OpenGVLab/InternVL2_5-1B
or
python -m vllm.entrypoints.openai.api_server --model=OpenGVLab/InternVL2_5-1B
在 http://127.0.0.1:8000/docs 可以看到各种路由信息
请求脚本如下
import base64
import requests
from openai import OpenAI# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"client = OpenAI(# defaults to os.environ.get("OPENAI_API_KEY")api_key=openai_api_key,base_url=openai_api_base,
)models = client.models.list()
model = models.data[0].iddef encode_image_base64_from_url(image_url: str) -> str:"""Encode an image retrieved from a remote url to base64 format."""with requests.get(image_url) as response:response.raise_for_status()result = base64.b64encode(response.content).decode('utf-8')return resultdef image_to_base64(image_path):with open(image_path, "rb") as image_file:image_data = image_file.read()base64_str = base64.b64encode(image_data).decode('utf-8')return base64_str # 添加MIME类型前缀[7](@ref)def single_image_call(image_path):image_base64 = image_to_base64(image_path=image_path)chat_completion_from_base64 = client.chat.completions.create(messages=[{"role":"user","content": [{"type": "text","text": "What’s in this image?"},{"type": "image_url","image_url": {"url": f"data:image/jpeg;base64,{image_base64}"},},],}],model=model,max_tokens=8192,top_p=0.9,temperature=0.0,)return chat_completion_from_base64.choices[0].message.contenttotal_result = []
for i in range(20):result = single_image_call("demo.jpg")total_result.append(result)# 验证多次推理结果是否相同
if len(set(total_result)) == 1:print(True)
else:print(False)
- 请求脚本使用 temperature=0.0 保证每次推理结果相同
- vllm 服务刚启动了，前几个请求始终会出现差异，可能是 bug
- –trust-remote-code 加载用户自己训练的模型，需要该参数
- –port 8765 指定端口号
- –tensor-parallel-size 张量并行数，部署服务需要的显卡数量
- –seed 42 指定随机种子，使用 temperature=0.0，无需该参数也能保证每次推理结果相同
vLLM 示例命令
vllm serve xxx/checkpoint-yyy --port 8567 --trust-remote-code --max-num-batched-tokens 8192 --seed 42 --tensor-parallel-size 8
