解决昇腾910B多容器部署Qwen3.6-27b引发的HCCL通信端口冲突问题
环境华为昇腾910B × 8 | vLLM-Ascend v0.20.2rc1 | Docker问题同一台机器启动多个Docker容器运行TP并行推理时HCCL通信端口冲突导致初始化失败一、问题背景在单台8卡昇腾910B服务器上需要同时部署以下模型服务模型用途NPU卡端口TP数BGE-M3向量0号70071Qwen3.6-27B-W8A8视觉/短文本0~3号90094Qwen3.6-27B-W8A8长文本4~7号60064其中视觉模型和长文本模型都使用了 tensor-parallel-size4即4卡张量并行。TP依赖HCCL华为集合通信库对标NCCL在worker进程间建立通信组。报错现象span stylebackground-color:#e1e1e1[ERROR] HCCL: Socket bind failed, address already in use [ERROR] HCCL: Failed to create communicator/span二、问题根因HCCL初始化时会分配Socket端口用于建立卡间通信连接。多个容器/进程同时启动TP推理时如果HCCL分配的端口重叠就会绑定冲突。NVIDIA NCCL在多容器场景通常能自动避让端口冲突但昇腾HCCL不支持自动端口分配必须手动指定互不重叠的端口范围。三、解决方案3.1 核心环境变量HCCL_NPU_SOCKET_PORT_RANGE为每个需要HCCL通信的进程分配独立的、不重叠的端口区间# 容器1 - BGE-M3单卡无TP但仍需预留端口 HCCL_NPU_SOCKET_PORT_RANGE16666-16676 # 容器1 - Qwen视觉模型4卡TP HCCL_NPU_SOCKET_PORT_RANGE16677-16687 # 容器2 - Qwen长文本模型4卡TP HCCL_NPU_SOCKET_PORT_RANGE16688-16698/span端口区间示意16666 ──── 16676 BGE-M3单卡预留 16677 ──── 16687 Qwen视觉模型TP4 16688 ──── 16698 Qwen长文本模型TP4 16699 ──── 16709 预留如需第3个TP实例/span3.2 NPU卡隔离ASCEND_RT_VISIBLE_DEVICES通过运行时环境变量限制每个进程可见的NPU设备等价于NVIDIA的 CUDA_VISIBLE_DEVICES# BGE-M3只用0号卡 ASCEND_RT_VISIBLE_DEVICES0 # 视觉模型用0~3号卡 ASCEND_RT_VISIBLE_DEVICES0,1,2,3 # 长文本模型用4~7号卡 ASCEND_RT_VISIBLE_DEVICES4,5,6,7/span3.3 完整 docker-compose.ymlservices: # 容器1BGE-M3 Qwen视觉模型0~3号卡 model-emb-vl: image: quay.io/ascend/vllm-ascend:v0.20.2rc1 restart: always privileged: true environment: HCCL_BUFFSIZE: 512 OMP_PROC_BIND: false OMP_NUM_THREADS: 1 TASK_QUEUE_ENABLE: 1 PYTORCH_NPU_ALLOC_CONF: expandable_segments:True ports: - 7007:7007 - 9009:9009 devices: - /dev/davinci0 - /dev/davinci1 - /dev/davinci2 - /dev/davinci3 - /dev/davinci4 - /dev/davinci5 - /dev/davinci6 - /dev/davinci7 - /dev/davinci_manager - /dev/devmm_svm - /dev/hisi_hdc volumes: - /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro - /usr/local/dcmi:/usr/local/dcmi:ro - /usr/local/bin/npu-smi:/usr/local/bin/npu-smi:ro - /disk2/bge-m3:/models/bge-m3 - /disk2/Qwen3.6-27B-w8a8:/models/Qwen3.6-27B-w8a8 entrypoint: - /bin/bash - -lc - | # BGE-M3单卡 HCCL_NPU_SOCKET_PORT_RANGE16666-16676 \ ASCEND_RT_VISIBLE_DEVICES0 \ vllm serve /models/bge-m3 \ --served-model-name bge-m3 \ --port 7007 --host 0.0.0.0 \ --gpu-memory-utilization 0.1 \ --max-model-len 8192 # Qwen视觉模型TP4 HCCL_NPU_SOCKET_PORT_RANGE16677-16687 \ ASCEND_RT_VISIBLE_DEVICES0,1,2,3 \ VLLM_WORKER_MULTIPROC_METHODspawn \ vllm serve /models/Qwen3.6-27B-w8a8 \ --served-model-name qwen-vl \ --port 9009 --host 0.0.0.0 \ --tensor-parallel-size 4 \ --quantization ascend \ --max-model-len 25000 \ --max-num-seqs 10 \ --gpu-memory-utilization 0.85 \ --trust-remote-code \ --no-enable-prefix-caching wait shm_size: 10.24gb ipc: host # 容器2Qwen长文本模型4~7号卡 model-text: image: quay.io/ascend/vllm-ascend:v0.20.2rc1 restart: always privileged: true environment: HCCL_BUFFSIZE: 512 OMP_PROC_BIND: false OMP_NUM_THREADS: 1 TASK_QUEUE_ENABLE: 1 PYTORCH_NPU_ALLOC_CONF: expandable_segments:True ports: - 6006:6006 devices: - /dev/davinci0 - /dev/davinci1 - /dev/davinci2 - /dev/davinci3 - /dev/davinci4 - /dev/davinci5 - /dev/davinci6 - /dev/davinci7 - /dev/davinci_manager - /dev/devmm_svm - /dev/hisi_hdc volumes: - /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro - /usr/local/dcmi:/usr/local/dcmi:ro - /usr/local/bin/npu-smi:/usr/local/bin/npu-smi:ro - /disk2/Qwen3.6-27B-w8a8:/models/Qwen3.6-27B-w8a8 entrypoint: - /bin/bash - -lc - | HCCL_NPU_SOCKET_PORT_RANGE16688-16698 \ ASCEND_RT_VISIBLE_DEVICES4,5,6,7 \ vllm serve /models/Qwen3.6-27B-w8a8 \ --served-model-name qwen-long \ --port 6006 --host 0.0.0.0 \ --tensor-parallel-size 4 \ --quantization ascend \ --max-model-len 100000 \ --max-num-seqs 2 \ --gpu-memory-utilization 0.90 \ --trust-remote-code \ --no-enable-prefix-caching shm_size: 10.24gb ipc: host/span四、关键配置说明配置项说明HCCL_NPU_SOCKET_PORT_RANGE最关键每个TP进程组必须分配不重叠端口区间ASCEND_RT_VISIBLE_DEVICES运行时NPU卡隔离等价于 CUDA_VISIBLE_DEVICESipc: host shm_size: 10.24gbTP多进程间共享内存通信必须配置VLLM_WORKER_MULTIPROC_METHODspawnTP1时vLLM多进程启动方式HCCL_BUFFSIZE512HCCL通信缓冲区MB提升TP通信吞吐devices 挂载全部davinci驱动管理设备manager/devmm_svm/hisi_hdc必须全量挂载实际卡隔离靠 ASCEND_RT_VISIBLE_DEVICES五、排查命令# 查看端口占用确认各HCCL端口区间无重叠 ss -tlnp | grep -E 1666[0-9]|1667[0-9]|1668[0-9]|1669[0-9] # 查看NPU状态 npu-smi info # 查看HCCL相关日志 docker logs 容器名 21 | grep -i hccl # 查看NPU拓扑 npu-smi info -t topology/span六、总结昇腾910B多容器并行部署的核心就一句话HCCL_NPU_SOCKET_PORT_RANGE必须为每个TP进程组分配不重叠的端口区间。这是昇腾与NVIDIA部署的最大差异——NCCL能自动避让HCCL不行。配合 ASCEND_RT_VISIBLE_DEVICES 做NPU卡隔离即可稳定运行多容器多模型并行推理。
