前言今天看到vLLM的朋友圈发布了DeepSeek R1的PP支持立刻开始我的捣鼓之旅假如我训练的超大MoE上线了也得做好技术准备工作是不嘛。把踩坑经验给大家分享一下希望能够相比于官方文档更白话一点。Distributed InferenceandServing:https://docs.vllm.ai/en/latest/serving/distributed_serving.html#running-vllm-on-multiple-nodes知乎游凯超 说一定要让整个过程变得丝滑无比我俩配合做了几个验证现在应该只需要 Step0 和 Step3 就可以run起来了如果遇到autoscalar的相关问题可以看Step1可以解决。Step 0 Prepare weights Environment由于权重太大了即使你网速可以也不建议直连下载了。大家可以先从HF及或代理弄一份权重回来直连大概率直接超时或者把公网IP打爆。我们今天展示的多机多卡8xH20 (x2) 部署对应TP size 8PP size 2所以要搞两台这样的机器过来。同时有一个假设两机的网络互通不一定需要IB储存需要共享NAS或OSS均可完成准备工作之后便可以做第一步。Step 1 Setup up Ray Cluster官方文档里面简单带过了这一部分但这个是我被卡时间太久的问题。首先我说一下官方文档的意思就是让你准备好两个节点之间用ray start这个CLI去建立好ray集群。因为后面要用但是比较坑的有两点第一点是启动的命令似乎有点点问题我在前几次尝试的时候都遇到了Ray的autoscaler报错的问题(autoscaler1m19s)Error:No available node types can fulfill resource request{node:33.18.26.153:0.001,GPU:1.0}.Add suitable node types to this cluster to resolve this issue.(autoscaler1m54s)Error:No available node types can fulfill resource request{GPU:1.0,node:33.18.26.153:0.001}.Add suitable node types to this cluster to resolve this issue.(autoscaler2m29s)Error:No available node types can fulfill resource request{GPU:1.0,node:33.18.26.153:0.001}.Add suitable node types to this cluster to resolve this issue.INFO02-0209:39:14ray_utils.py:212]Waitingforcreating a placement group of specsfor150seconds.specs[{node:33.18.26.153:0.001,GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0},{GPU:1.0}].Check ray status to seeifyou have enough resources.这看起来就很奇怪因为vLLM找Ray集群要的Resource是custom resource‘node:33.18.26.153’:0.001这可以理解成vLLM优先要driver节点。但是这个东西我印象中是需要启动ray的时候自己设置的https://docs.ray.io/en/latest/ray-core/scheduling/resources.html#custom-resources像这样才会有这种resource。背后的原因是对于多虚拟网卡的机器会有多个网段vLLM assume使用POD IP来做Ray的master寻址。解法1设置 VLLM_HOST_IP# Get local IP address and set on every node before Ray startVLLM_HOST_IP$(hostname-I|awk{print $1})export VLLM_HOST_IP解法2魔改Ray启动逻辑defget_actual_ip():Get the actual IP address of the current machine.try:# Create a socket to connect to an external server (doesnt actually connect)ssocket.socket(socket.AF_INET,socket.SOCK_DGRAM)s.connect((8.8.8.8,80))ips.getsockname()[0]s.close()returnipexceptException:# Fallback to hostname-based IP resolutionreturnsocket.gethostbyname(socket.gethostname())defstart_ray_cluster():free_portsget_free_ports()portfree_ports[0]node_manager_portfree_ports[1]master_addrget_master_addr()rankget_rank()node_ipget_actual_ip()# Use the new function to get actual IP# Define custom resource based on node IPresource_specf--resources\{{node:{node_ip}: 1}}\ifrank0:cmdfray start --head --port{port}--node-ip-address{master_addr}--node-manager-port{node_manager_port}--node-name{master_addr}{resource_spec}else:cmdfray start --address{master_addr}:{port}--node-manager-port{node_manager_port}--node-name{get_addr()}{resource_spec}ifray.is_initialized():print(Ray is already initialized, skipping node level init.)else:stop_cmdray stopexecute(stop_cmd,checkTrue)print(fExecuting Ray start command:{cmd})execute(cmd,checkTrue)其中execute可以这样写importtimeimportsubprocessdefexecute(cmd,checkFalse,retry1):retsubprocess.run(cmd,shellTrue,capture_outputTrue,textTrue,checkcheck)stateret.returncode0msgret.stdoutifstateelseret.stderrifnotstateandretry1:print(fexecute{cmd}got error{msg}, retry...)time.sleep(1)returnexecute(cmd,check,retry-1)returnstate,msg然后这里我稍微提一下ray的一些基础玩法大家在使用Ray的时候一般都不是在裸机上面的大部分深度学习的资源都是k8s结合kubeflow或者volcano这样的插件分发出来的。环境变量里面会有当前是第几个rank头结点master_addr这样的信息大家可以根据自己的需要把这些函数实现一下。比较坑的 {resource_spec} 这里我已经替大家把坑给填了。Step 2 Other small bugs期间又报了两个错误花了一点时间修复Traceback(most recent call last):File/usr/local/bin/vllm,line5,inmodulefromvllm.scriptsimportmain File/usr/local/lib/python3.10/dist-packages/vllm/__init__.py,line4,inmodulefromvllm.engine.async_llm_engineimportAsyncLLMEngine File/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py,line15,inmodulefromvllm.engine.llm_engineimport(DecoderPromptComponents,LLMEngine,File/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py,line24,inmodulefromvllm.engine.output_processor.interfacesimport(File/usr/local/lib/python3.10/dist-packages/vllm/engine/output_processor/interfaces.py,line6,inmodulefromvllm.engine.output_processor.stop_checkerimportStopChecker File/usr/local/lib/python3.10/dist-packages/vllm/engine/output_processor/stop_checker.py,line6,inmodulefromvllm.transformers_utils.tokenizerimportAnyTokenizer File/usr/local/lib/python3.10/dist-packages/vllm/transformers_utils/tokenizer.py,line13,inmodulefromvllm.transformers_utils.tokenizersimport(BaichuanTokenizer,File/usr/local/lib/python3.10/dist-packages/vllm/transformers_utils/tokenizers/__init__.py,line2,inmodulefromvllm.transformers_utils.tokenizers.mistralimportMistralTokenizer File/usr/local/lib/python3.10/dist-packages/vllm/transformers_utils/tokenizers/mistral.py,line9,inmodulefrommistral_common.tokens.tokenizers.mistralimportChatCompletionRequest File/usr/local/lib/python3.10/dist-packages/mistral_common/tokens/tokenizers/mistral.py,line32,inmodulefrommistral_common.tokens.tokenizers.multimodalimport(File/usr/local/lib/python3.10/dist-packages/mistral_common/tokens/tokenizers/multimodal.py,line6,inmoduleimportcv2 File/usr/local/lib/python3.10/dist-packages/cv2/__init__.py,line181,inmodulebootstrap()File/usr/local/lib/python3.10/dist-packages/cv2/__init__.py,line175,inbootstrapif__load_extra_py_code_for_module(cv2,submodule,DEBUG):File/usr/local/lib/python3.10/dist-packages/cv2/__init__.py,line28,in__load_extra_py_code_for_module py_moduleimportlib.import_module(module_name)File/usr/lib/python3.10/importlib/__init__.py,line126,inimport_modulereturn_bootstrap._gcd_import(name[level:],package,level)File/usr/local/lib/python3.10/dist-packages/cv2/typing/__init__.py,line171,inmoduleLayerIdcv2.dnn.DictValue AttributeError:modulecv2.dnnhas no attributeDictValue一个opencv封建余孽的问题pin住opencv的版本来解决pip install opencv-python-headless4.5.4.58还有一个load之后报TypeError的问题[rank0]:File/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/deepseek_v3.py,line472,inforward[rank0]:kv_c,k_peself.kv_a_proj_with_mqa(hidden_states)[0].split([rank0]:File/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py,line1736,in_wrapped_call_impl[rank0]:returnself._call_impl(*args,**kwargs)[rank0]:File/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py,line1747,in_call_impl[rank0]:returnforward_call(*args,**kwargs)[rank0]:File/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/linear.py,line246,inforward[rank0]:outputself.quant_method.apply(self,x,bias)[rank0]:File/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/fp8.py,line357,inapply[rank0]:returnapply_w8a8_block_fp8_linear([rank0]:File/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py,line61,inapply_w8a8_block_fp8_linear[rank0]:outputw8a8_block_fp8_matmul(q_input,[rank0]:File/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py,line470,inw8a8_block_fp8_matmul[rank0]:configsget_w8a8_block_fp8_configs(N,K,block_size[0],block_size[1])[rank0]:File/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py,line407,inget_w8a8_block_fp8_configs[rank0]:device_namecurrent_platform.get_device_name().replace( ,_)[rank0]:TypeError:abytes-likeobjectisrequired,notstr通过升级 pynvml 解决pip install pynvml-UStep 3 Run the model这一步反而是最简单的vllm serve/your/path/to_checkpoint_deepseek-r1/--tensor-parallel-size8--pipeline-parallel-size2--trust-remote-code--host0.0.0.0由于有了PP加持没有IB的同学也可以尝试把sequence length和bsz给稍微拉大一些拉。用gaoce哥哥贡献的Reasoning Output在同一台机器来试一把或者换一台机器把localhost改了fromopenaiimportOpenAI# Modify OpenAIs API key and API base to use vLLMs API server.openai_api_keyEMPTYopenai_api_basehttp://localhost:8000/v1clientOpenAI(api_keyopenai_api_key,base_urlopenai_api_base,)modelsclient.models.list()modelmodels.data[0].id# Round 1messages[{role:user,content:9.11 and 9.8, which is greater?}]responseclient.chat.completions.create(modelmodel,messagesmessages)reasoning_contentresponse.choices[0].message.reasoning_content contentresponse.choices[0].message.contentprint(reasoning_content:,reasoning_content)print(content:,content)对你不是卡主了是你的钱包不够厚。切到后台可以看到这个prompt里面INFO02-0214:18:52metrics.py:453]Avg prompt throughput:1.7tokens/s,Avg generation throughput:0.1tokens/s,Running:1reqs,Swapped:0reqs,Pending:0reqs,GPU KV cache usage:0.0%,CPU KV cache usage:0.0%.INFO02-0214:18:57metrics.py:453]Avg prompt throughput:0.0tokens/s,Avg generation throughput:20.7tokens/s,Running:1reqs,Swapped:0reqs,Pending:0reqs,GPU KV cacheusage:0.0%,CPU KV cache usage:0.0%.INFO02-0214:19:02metrics.py:453]Avg prompt throughput:0.0tokens/s,Avg generation throughput:20.5tokens/s,Running:1reqs,Swapped:0reqs,Pending:0reqs,GPU KV cacheusage:0.0%,CPU KV cache usage:0.0%.INFO02-0214:19:07metrics.py:453]Avg prompt throughput:0.0tokens/s,Avg generation throughput:20.5tokens/s,Running:1reqs,Swapped:0reqs,Pending:0reqs,GPU KV cacheusage:0.0%,CPU KV cache usage:0.0%.INFO02-0214:19:12metrics.py:453]Avg prompt throughput:0.0tokens/s,Avg generation throughput:20.1tokens/s,Running:1reqs,Swapped:0reqs,Pending:0reqs,GPU KV cacheusage:0.0%,CPU KV cache usage:0.0%.INFO02-0214:19:17metrics.py:453]Avg prompt throughput:0.0tokens/s,Avg generation throughput:19.8tokens/s,Running:1reqs,Swapped:0reqs,Pending:0reqs,GPU KV cacheusage:0.1%,CPU KV cache usage:0.0%.INFO02-0214:19:22metrics.py:453]Avg prompt throughput:0.0tokens/s,Avg generation throughput:19.4tokens/s,Running:1reqs,Swapped:0reqs,Pending:0reqs,GPU KV cacheusage:0.1%,CPU KV cache usage:0.0%.INFO02-0214:19:27metrics.py:453]Avg prompt throughput:0.0tokens/s,Avg generation throughput:19.1tokens/s,Running:1reqs,Swapped:0reqs,Pending:0reqs,GPU KV cacheusage:0.1%,CPU KV cache usage:0.0%.稍等一会他就会告诉你9.8更大了。祝大家捣鼓顺利感谢vLLM社区的工作。https://github.com/vllm-project/vllm/pull/12679凯超真 nb 春节在这做贴身客服哈哈RL仔现在不管原来是主修文还是主修理的都先修infra吧。最后的最后感谢你们的阅读和喜欢作为一位在一线互联网行业奋斗多年的老兵我深知在这个瞬息万变的技术领域中持续学习和进步的重要性。为了帮助更多热爱技术、渴望成长的朋友我特别整理了一份涵盖大模型领域的宝贵资料集。这些资料不仅是我多年积累的心血结晶也是我在行业一线实战经验的总结。这些学习资料不仅深入浅出而且非常实用让大家系统而高效地掌握AI大模型的各个知识点。如果你愿意花时间沉下心来学习相信它们一定能为你提供实质性的帮助。这份完整版的大模型 AI 学习资料已经上传CSDN朋友们如果需要可以微信扫描下方CSDN官方认证二维码免费领取【保证100%免费】DeepSeek全套安装部署资料大模型知识脑图为了成为更好的 AI大模型 开发者这里为大家提供了总的路线图。它的用处就在于你可以按照上面的知识点去找对应的学习资源保证自己学得较为全面。经典书籍阅读阅读AI大模型经典书籍可以帮助读者提高技术水平开拓视野掌握核心技术提高解决问题的能力同时也可以借鉴他人的经验。对于想要深入学习AI大模型开发的读者来说阅读经典书籍是非常有必要的。实战案例光学理论是没用的要学会跟着一起敲要动手实操才能将自己的所学运用到实际当中去这时候可以搞点实战案例来学习。面试资料我们学习AI大模型必然是想找到高薪的工作下面这些面试题都是总结当前最新、最热、最高频的面试题并且每道题都有详细的答案面试前刷完这套面试题资料小小offer不在话下640套AI大模型报告合集这套包含640份报告的合集涵盖了AI大模型的理论研究、技术实现、行业应用等多个方面。无论您是科研人员、工程师还是对AI大模型感兴趣的爱好者这套报告合集都将为您提供宝贵的信息和启示。这份完整版的大模型 AI 学习资料已经上传CSDN朋友们如果需要可以微信扫描下方CSDN官方认证二维码免费领取【保证100%免费】