你说你模型训出来了权重文件几百个G结果人家问你服务怎么上 你挠挠头“nohup python app.py ” 兄弟那不是生产环境那叫行为艺术。开篇当跑起来不等于上线了上回我们搭完了MLflow模型注册中心和Jenkins训练流水线模型已经在Registry里躺好了。接下来要面对的问题是怎么把这个几百MB甚至几十GB的东西变成一个真正能扛流量的在线服务说到部署模型很多人的第一反应是FastAPI包一层写个app.post(/predict)然后uvicorn main:app --host 0.0.0.0甩上去。这在小规模、单机场景下完全没问题但当你面对这些需求时 需要同时部署10个模型的A/B实验版本 QPS从100飙到10000服务不能挂️ GPU资源有限不能让V100在那空转烧钱 模型更新不停服用户无感知切换 推理延迟要控制在50ms以内这时候你就需要一个正经的模型服务部署方案而Kubernetes就是这件事的最佳载体。今天这篇文章我会从服务化框架选型 → Docker镜像打包 → K8s资源编排 → GPU调度 → 弹性伸缩 → 灰度发布完整走通一条模型上K8s的路径。所有YAML和配置都是可直接复制运行的不整那些此处略去300行的虚的。一、模型服务化FastAPI TorchServe vs Triton Inference Server先把框架选对后面省十倍的心。1.1 三种方案横向对比graph LR subgraph 方案A[方案AFastAPI 手写] A1[app.py] -- A2[加载模型] -- A3[推理函数] -- A4[uvicorn启动] end subgraph 方案B[方案BTorchServe] B1[MAR打包] -- B2[model-store] -- B3[TorchServe Runtime] -- B4[HTTP/gRPC] end subgraph 方案C[方案CTriton Server] C1[model.pt/onnx] -- C2[model-repository] -- C3[Triton Runtime] -- C4[HTTP/gRPC] end style 方案A fill:#FFE4B5,stroke:#FF8C00 style 方案B fill:#87CEEB,stroke:#1E90FF style 方案C fill:#90EE90,stroke:#228B22维度FastAPI手写TorchServeTriton Inference Server上手难度⭐ 低⭐⭐ 中⭐⭐⭐ 较高动态批处理❌ 需手写✅ 内置✅原生支持效果最佳多模型并发❌✅✅✅GPU共享并发执行模型版本管理❌✅✅多框架支持手写适配PyTorch为主PyTorch/TF/ONNX/TRT等GPU利用率30-50%60-70%85-95%生产级监控❌⚠️ 基础✅Prometheus metrics适合场景原型验证PyTorch生态正式生产环境1.2 为什么选Triton简单说Triton在GPU利用率和推理吞吐上碾压其他方案。一个具体的数据同样一块V100跑BERT-base文本分类——FastAPI PyTorch 手写~320 QPSGPU利用率 38%Triton ONNX Dynamic Batching~1200 QPSGPU利用率 89%差了三倍多。这不是玄学是Triton做了三件关键的事Dynamic Batching自动把多个请求攒成batch送进GPU减少kernel launch开销Concurrent Model Execution多个模型实例在GPU上并发跑不互相阻塞Model Analyzer自动帮你找到最优的instance group配置技巧1先用FastAPI快速验证生产切Triton我的建议是POC阶段FastAPI快速验证模型效果没问题后直接上Triton。别在FastAPI上花时间优化批处理——最终还是要迁移的重复造轮子浪费时间。FastAPI快速验证模板拿来即用# app.py - POC验证专用 from fastapi import FastAPI from pydantic import BaseModel import torch app FastAPI() model torch.jit.load(model.pt).cuda().eval() class PredictRequest(BaseModel): text: str app.post(/predict) async def predict(req: PredictRequest): with torch.no_grad(): tokens tokenizer(req.text, return_tensorspt) for k in tokens: tokens[k] tokens[k].cuda() output model(**tokens) return {label: output.argmax().item()}二、Docker镜像构建把模型打成镜像模型部署的第一个工程问题模型文件到底放哪2.1 三种策略策略做法优点缺点打包进镜像Dockerfile COPY模型文件简单一致性高镜像巨大构建慢启动时下载initContainer从S3/OSS拉取镜像小启动慢冷启动噩梦挂载PVCK8s PersistentVolume挂载灵活可共享需要存储基建生产环境推荐「模型放对象存储 initContainer拉取 本地缓存」的混合方案。sequenceDiagram participant K8s as K8s Scheduler participant Init as InitContainer participant OSS as MinIO/OSS participant Main as Triton Container participant GPU as NVIDIA GPU K8s-Init: 1. 调度Pod到GPU节点 Init-OSS: 2. 从对象存储下载模型 OSS--Init: 3. 返回模型文件 Init-Init: 4. 解压到共享EmptyDir Init-K8s: 5. InitContainer完成 K8s-Main: 6. 启动Triton容器 Main-GPU: 7. 加载模型到GPU显存 Main-Main: 8. 开始接收推理请求2.2 完整Dockerfile# Dockerfile - Triton Inference Server with PyTorch backend FROM nvcr.io/nvidia/tritonserver:24.01-py3 # 安装Python依赖如果模型需要自定义Python backend RUN pip install --no-cache-dir \ transformers4.36.0 \ torch2.1.0 \ tokenizers0.15.0 # 创建模型仓库目录 RUN mkdir -p /models # 模型配置文件和预处理脚本放这里 COPY model_config/ /models/ COPY preprocess.py /opt/tritonserver/ # Triton默认端口 EXPOSE 8000 8001 8002 # 启动Triton ENTRYPOINT [tritonserver] CMD [--model-repository/models, \ --log-verbose1, \ --strict-model-configfalse]⚠️避坑1Triton镜像版本别瞎选Triton每个月发一个新版本如24.01, 24.02…但PyTorch backend的CUDA版本是跟镜像绑定的。踩过的坑用了23.12-py3镜像结果它带的PyTorch backend是CUDA 12.1编译的而K8s节点装的驱动只支持到CUDA 11.8——模型加载时报CUDA driver version is insufficient。正确做法先用nvidia-smi确认节点CUDA Driver版本再选匹配的Triton镜像Tag。Triton 24.01起支持CUDA 12.x如果你的环境是11.x用23.09及更早版本。检查命令# 确认节点CUDA版本 kubectl run nvidia-check --rm -it \ --imagenvidia/cuda:11.8-base-ubuntu20.04 \ --restartNever -- nvidia-smi三、K8s完整YAMLDeployment Service Ingress下面是一套可以直接kubectl apply的完整部署配置包含三个资源。3.1 命名空间和ConfigMap# 00-namespace-config.yaml apiVersion: v1 kind: Namespace metadata: name: ai-inference --- apiVersion: v1 kind: ConfigMap metadata: name: triton-config namespace: ai-inference data: model-repo-path: s3://models/text-classifier/v2 s3-endpoint: http://minio.minio:9000 log-level: INFO metrics-interval: 20003.2 Deployment核心# 01-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: text-classifier namespace: ai-inference labels: app: text-classifier version: v2 spec: replicas: 2 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 # 零停机更新 selector: matchLabels: app: text-classifier template: metadata: labels: app: text-classifier version: v2 spec: # --- GPU调度 --- nodeSelector: accelerator: nvidia-tesla-v100 # 指定GPU节点 tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule # --- InitContainer从S3拉取模型 --- initContainers: - name: model-downloader image: amazon/aws-cli:2.15.0 env: - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: minio-creds key: access-key - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: minio-creds key: secret-key command: - sh - -c - | echo Downloading model from MinIO... aws s3 cp s3://models/text-classifier/v2/ /models/ \ --recursive \ --endpoint-url http://minio.minio:9000 \ --no-sign-request echo Model downloaded. Size: $(du -sh /models) volumeMounts: - name: model-storage mountPath: /models # --- 主容器Triton --- containers: - name: triton-server image: registry.example.com/triton-text-classifier:v2 imagePullPolicy: Always ports: - containerPort: 8000 # HTTP name: http protocol: TCP - containerPort: 8001 # gRPC name: grpc protocol: TCP - containerPort: 8002 # Metrics name: metrics protocol: TCP env: - name: CUDA_VISIBLE_DEVICES value: 0 - name: OMP_NUM_THREADS value: 4 resources: requests: memory: 8Gi cpu: 4 nvidia.com/gpu: 1 # 请求1张GPU limits: memory: 16Gi cpu: 8 nvidia.com/gpu: 1 # 限制1张GPU readinessProbe: httpGet: path: /v2/health/ready port: 8000 initialDelaySeconds: 60 periodSeconds: 10 failureThreshold: 5 livenessProbe: httpGet: path: /v2/health/live port: 8000 initialDelaySeconds: 120 periodSeconds: 30 failureThreshold: 3 volumeMounts: - name: model-storage mountPath: /models - name: inference-cache mountPath: /cache - name: shm mountPath: /dev/shm volumes: - name: model-storage emptyDir: sizeLimit: 20Gi # 模型共享存储 medium: Memory # 用内存加速大模型慎用 - name: inference-cache emptyDir: sizeLimit: 2Gi - name: shm emptyDir: medium: Memory sizeLimit: 1Gi # PyTorch多进程需要共享内存3.3 Service# 02-service.yaml apiVersion: v1 kind: Service metadata: name: text-classifier-svc namespace: ai-inference labels: app: text-classifier spec: type: ClusterIP selector: app: text-classifier ports: - name: http port: 8000 targetPort: 8000 protocol: TCP - name: grpc port: 8001 targetPort: 8001 protocol: TCP - name: metrics port: 8002 targetPort: 8002 protocol: TCP sessionAffinity: ClientIP # 尽量让同一客户端打到同一Pod sessionAffinityConfig: clientIP: timeoutSeconds: 300技巧2Service加上sessionAffinity减少模型冷启动Triton第一次收到请求时如果模型还没加载到GPU需要做模型加载冷启动可能耗时几秒到几十秒。 设置sessionAffinity: ClientIP后同一客户端的请求尽量打到同一Pod降低了重复冷启动的几率。 当然这只是一种取巧真正解决冷启动要靠--model-control-modeexplicit做预热。3.4 Ingress# 03-ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: text-classifier-ingress namespace: ai-inference annotations: nginx.ingress.kubernetes.io/proxy-body-size: 16m nginx.ingress.kubernetes.io/proxy-read-timeout: 120 nginx.ingress.kubernetes.io/proxy-send-timeout: 120 nginx.ingress.kubernetes.io/proxy-connect-timeout: 60 # 限流配置 nginx.ingress.kubernetes.io/limit-rps: 200 nginx.ingress.kubernetes.io/limit-burst-multiplier: 2 spec: ingressClassName: nginx tls: - hosts: - api.example.com secretName: api-tls-cert rules: - host: api.example.com http: paths: - path: /v1/classify pathType: Prefix backend: service: name: text-classifier-svc port: number: 8000⚠️避坑2Ingress超时不够长推理丢请求默认Nginx Ingress的proxy-read-timeout是60秒。如果你的模型推理一次要超过60秒比如大模型生成请求会被504。必须根据你的模型最坏情况来调大timeout。上面配置设了120秒如果是LLM流式输出场景建议设到300秒以上并且开启proxy-buffering: off。四、GPU调度与nvidia-device-pluginKubernetes默认不认识GPU。要让Pod里能用nvidia.com/gpu资源需要安装NVIDIA Device Plugin。4.1 前置条件每个GPU节点必须具备 ├── NVIDIA Driver (≥ 450.80.02) ├── nvidia-container-toolkit (原名nvidia-docker2) └── Container Runtime 配置为 nvidia 或 containerd nvidia runtime4.2 安装Device Plugin# 一键安装官方device plugin kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.5/deployments/static/nvidia-device-plugin.yml验证# 看节点是否上报了GPU资源 kubectl describe node gpu-node | grep nvidia.com/gpu # 输出类似 # nvidia.com/gpu: 4 # nvidia.com/gpu: 44.3 GPU调度策略graph TD A[Pod请求GPU] -- B{Node有GPU?} B --|是| C{GPU剩余够?} B --|否| D[Pending等待] C --|够| E[调度到该Node] C --|不够| F[尝试其他Node] F -- G{还有Node?} G --|是| C G --|否| D E -- H{GPU共享模式?} H --|Time-Slicingbr/时间分片| I[多Pod分时共享一张GPU] H --|MIGbr/多实例GPU| J[物理切分为独立GPU实例] H --|MPSbr/多进程服务| K[CUDA MPS共享上下文]三种GPU共享策略对比策略隔离性显存限制适用场景Time-Slicing❌ 弱❌ 不限制开发测试、轻量推理MIG✅ 强✅ 硬隔离A100/A30多租户生产MPS⚠️ 中等❌ 不限制推理密集型单租户⚠️避坑3Time-Slicing模式下显存溢出无人管Time-Slicing只是时间片轮转不做显存隔离。如果你的模型需要6GB显存V100有32GBK8s会让5个Pod都调度上去——结果第4、第5个Pod加载模型时直接OOM崩溃。解决配置device-plugin时设置显存限制或者在Pod里用CUDA_VISIBLE_DEVICES和模型加载时手动限制。A100强烈建议用MIG模式。# device-plugin ConfigMap启用Time-Slicing apiVersion: v1 kind: ConfigMap metadata: name: nvidia-device-plugin-config namespace: kube-system data: config.yaml: | version: v1 sharing: timeSlicing: resources: - name: nvidia.com/gpu replicas: 4 # 一张GPU虚拟化为4份五、HPA自动扩缩让服务自己呼吸固定replicas2在凌晨3点浪费钱在双11秒杀日又扛不住。HPAHorizontal Pod Autoscaler让服务根据负载自动伸缩。5.1 基于CPU/内存的HPA# 04-hpa.yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: text-classifier-hpa namespace: ai-inference spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: text-classifier minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 60 - type: Resource resource: name: memory target: type: Utilization averageUtilization: 75 behavior: scaleUp: stabilizationWindowSeconds: 60 # 扩容快速响应 policies: - type: Pods value: 2 periodSeconds: 60 scaleDown: stabilizationWindowSeconds: 300 # 缩容5分钟冷静期防止抖动 policies: - type: Pods value: 1 periodSeconds: 1205.2 基于自定义指标的HPA推理QPSCPU和内存跟推理服务的实际负载不完全正比。更好的做法是用Prometheus采集Triton的推理QPS指标作为HPA的依据。# 使用Prometheus Adapter的自定义指标HPA apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: text-classifier-hpa-qps namespace: ai-inference spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: text-classifier minReplicas: 2 maxReplicas: 8 metrics: - type: Pods pods: metric: name: nv_inference_request_success_per_second # Triton暴露的请求速率 target: type: AverageValue averageValue: 500 # 每个Pod超过500 QPS就扩容技巧3HPA Cluster Autoscaler联动HPA扩容Pod只是第一步。但如果你只配了HPA没配Cluster AutoscalerCA扩容到一半发现Node不够了——Pod全部Pending白扩。完整链路应该是负载上升 → HPA增加Pod → Pod Pending(资源不足) → CA新增Node → Pod调度成功检查CA是否正常工作kubectl get nodes -l node-typegpu # 确认GPU节点池 kubectl logs -n kube-system deployment/cluster-autoscaler | grep Scale-up六、蓝绿部署模型更新不停服模型迭代频率远高于普通应用——AB实验天天换版本。你需要一个不停服的发布策略。6.1 蓝绿部署架构graph TB subgraph 流量入口 LB[Ingress / API Gatewaybr/api.example.com] end subgraph 蓝环境[ Blue (active)] SVC_B[Service: text-classifier-blue] DEP_B[Deploymentbr/version: v1br/replicas: 2] end subgraph 绿环境[ Green (standby)] SVC_G[Service: text-classifier-green] DEP_G[Deploymentbr/version: v2br/replicas: 2] end LB --|label: colorblue| SVC_B LB -.-|切换label: colorgreen| SVC_G SVC_B -- DEP_B SVC_G -- DEP_G6.2 完整蓝绿部署脚本#!/bin/bash # blue-green-deploy.sh - 模型蓝绿部署 set -e NAMESPACEai-inference APPtext-classifier NEW_VERSION$1 REGISTRYregistry.example.com/triton-${APP} if [ -z $NEW_VERSION ]; then echo Usage: $0 new-version-tag exit 1 fi # 1. 判断当前active环境 CURRENT_COLOR$(kubectl get service ${APP}-svc -n ${NAMESPACE} \ -o jsonpath{.spec.selector.color}) if [ $CURRENT_COLOR blue ]; then TARGET_COLORgreen OLD_COLORblue else TARGET_COLORblue OLD_COLORgreen fi echo echo Current active: ${OLD_COLOR} echo Deploying to: ${TARGET_COLOR} echo New version: ${NEW_VERSION} echo # 2. 部署新版本到备用环境 echo [1/4] Deploying ${TARGET_COLOR} environment with version ${NEW_VERSION}... kubectl apply -f - EOF apiVersion: apps/v1 kind: Deployment metadata: name: ${APP}-${TARGET_COLOR} namespace: ${NAMESPACE} labels: app: ${APP} color: ${TARGET_COLOR} version: ${NEW_VERSION} spec: replicas: 2 selector: matchLabels: app: ${APP} color: ${TARGET_COLOR} template: metadata: labels: app: ${APP} color: ${TARGET_COLOR} version: ${NEW_VERSION} spec: nodeSelector: accelerator: nvidia-tesla-v100 containers: - name: triton-server image: ${REGISTRY}:${NEW_VERSION} ports: - containerPort: 8000 resources: limits: nvidia.com/gpu: 1 readinessProbe: httpGet: path: /v2/health/ready port: 8000 initialDelaySeconds: 90 periodSeconds: 5 EOF # 3. 等待新环境Ready echo [2/4] Waiting for ${TARGET_COLOR} pods to be ready... kubectl wait --forconditionready pod \ -l app${APP},color${TARGET_COLOR} \ -n ${NAMESPACE} \ --timeout300s # 4. 预热模型发送空请求触发模型加载 echo [3/4] Warming up model... POD_NAME$(kubectl get pods -n ${NAMESPACE} \ -l app${APP},color${TARGET_COLOR} \ -o jsonpath{.items[0].metadata.name}) kubectl exec -n ${NAMESPACE} ${POD_NAME} -- \ curl -s -X POST http://localhost:8000/v2/repository/models/text-classifier/load # 5. 切换流量 echo [4/4] Switching traffic from ${OLD_COLOR} to ${TARGET_COLOR}... kubectl patch service ${APP}-svc -n ${NAMESPACE} \ -p {\spec\:{\selector\:{\app\:\${APP}\,\color\:\${TARGET_COLOR}\}}} echo echo ✅ Traffic switched to ${TARGET_COLOR} (version: ${NEW_VERSION}) echo Old ${OLD_COLOR} environment kept running as standby. echo echo Rollback command: echo kubectl patch service ${APP}-svc -n ${NAMESPACE} \\ echo -p {\spec\:{\selector\:{\app\:\${APP}\,\color\:\${OLD_COLOR}\}}}七、Redis推理缓存同样的请求别算两遍很多业务场景下推理请求有高度重复性。比如搜索推荐热门query反复出现内容审核同样的敏感词列表每次都跑一次模型文本分类固定话术反复分类这时候加一层Redis缓存效果立竿见影。sequenceDiagram participant Client as 客户端 participant Ingress as Nginx Ingress participant Cache as Redis缓存 participant Triton as Triton Server Client-Ingress: POST /v1/classify (text你好) Ingress-Cache: GET classify:md5(你好) alt 缓存命中 Cache--Ingress: {label:greeting,score:0.95} Ingress--Client: 直接从缓存返回 (~2ms) else 缓存未命中 Cache--Ingress: nil Ingress-Triton: 转发推理请求 Triton--Ingress: {label:greeting,score:0.95} (~45ms) Ingress-Cache: SETEX classify:md5(你好) 300 {label:greeting} Ingress--Client: 返回推理结果 end7.1 Python实现的推理缓存层# cache_proxy.py - 推理缓存代理 import hashlib import json import redis import httpx from fastapi import FastAPI, Request, HTTPException from fastapi.responses import JSONResponse app FastAPI() r redis.Redis(hostredis.ai-inference, port6379, decode_responsesTrue) TRITON_URL http://text-classifier-svc.ai-inference:8000 CACHE_PREFIX infer:cls: CACHE_TTL 300 # 5分钟过期 app.post(/v1/classify) async def classify_with_cache(request: Request): body await request.json() text body.get(text, ) # 生成缓存Key cache_key f{CACHE_PREFIX}{hashlib.md5(text.encode()).hexdigest()} # 1. 查缓存 cached r.get(cache_key) if cached: result json.loads(cached) result[source] cache return result # 2. 缓存未命中调Triton async with httpx.AsyncClient(timeout30.0) as client: resp await client.post( f{TRITON_URL}/v2/models/text-classifier/infer, jsonbody, ) if resp.status_code ! 200: raise HTTPException(status_coderesp.status_code, detailresp.text) result resp.json() result[source] triton # 3. 写入缓存 r.setex(cache_key, CACHE_TTL, json.dumps(result)) return result # 缓存统计端点 app.get(/cache/stats) async def cache_stats(): info r.info(stats) keys len(r.keys(f{CACHE_PREFIX}*)) return { total_keys: keys, hits: info.get(keyspace_hits, 0), misses: info.get(keyspace_misses, 0), hit_rate: round( info.get(keyspace_hits, 0) / max(info.get(keyspace_hits, 0) info.get(keyspace_misses, 0), 1) * 100, 2 ), }Redis部署一行即可# redis-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: redis-cache namespace: ai-inference spec: replicas: 1 selector: matchLabels: app: redis-cache template: metadata: labels: app: redis-cache spec: containers: - name: redis image: redis:7-alpine ports: - containerPort: 6379 resources: requests: memory: 512Mi cpu: 250m command: - redis-server - --maxmemory 2gb - --maxmemory-policy allkeys-lru # LRU淘汰 - --save # 不做持久化纯缓存 --- apiVersion: v1 kind: Service metadata: name: redis namespace: ai-inference spec: selector: app: redis-cache ports: - port: 6379八、监控与可观测性生产环境没监控等于闭着眼开车。Triton原生暴露Prometheus metrics接入Grafana一条龙。# 05-servicemonitor.yaml - Prometheus Operator apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: triton-monitor namespace: ai-inference labels: release: prometheus spec: selector: matchLabels: app: text-classifier endpoints: - port: metrics interval: 15s path: /metrics关键指标Prometheus Metric含义告警阈值建议nv_inference_request_success推理成功数-nv_inference_request_failure推理失败数 5/minnv_inference_queue_duration_us排队等待时间P99 100msnv_inference_compute_infer_duration_usGPU计算耗时P99 模型基线×2nv_gpu_utilizationGPU使用率 20%资源浪费nv_gpu_memory_used_bytesGPU显存使用 85%九、最终部署一键启动全栈把上面的所有YAML串起来#!/bin/bash # deploy-all.sh - 一键部署整套推理服务 kubectl apply -f 00-namespace-config.yaml kubectl apply -f redis-deployment.yaml kubectl apply -f 01-deployment.yaml kubectl apply -f 02-service.yaml kubectl apply -f 03-ingress.yaml kubectl apply -f 04-hpa.yaml kubectl apply -f 05-servicemonitor.yaml echo ✅ All resources deployed! echo echo Check status: echo kubectl get pods -n ai-inference echo kubectl get hpa -n ai-inference echo echo Test inference: echo curl -X POST https://api.example.com/v1/classify \\ echo -H Content-Type: application/json \\ echo -d {\text\: \今天天气真好\}总结回顾一下我们在这篇文章里从零到一完成了模型推理服务的K8s部署步骤关键技术一句话要点服务化Triton Inference Server比FastAPI手写高3倍吞吐镜像构建Dockerfile InitContainer模型不打包进镜像K8s编排Deployment Service Ingress完整YAML复制即用GPU调度nvidia-device-pluginTime-Slicing/MIG按需选择弹性伸缩HPACPU Prometheus自定义指标双保险蓝绿部署Service Selector切换零停机模型更新推理缓存Redis重复请求热点直接走缓存整套方案已经在生产环境跑了半年多日均推理量2000万次GPU平均利用率从38%提到87%。下一篇预告L5实战四——AI DevOps平台搭建MLOps全链路监控与成本优化我们把Prometheus、Grafana、成本分析和自动降级全部串起来真正实现一个能自己体检和省钱的AI平台。系列文章L5实战一AI DevOps平台搭建——MLflow模型注册中心L5实战二AI DevOps平台搭建——Jenkins训练流水线本文L5实战三Kubernetes模型服务部署← 你在这里L5实战四MLOps全链路监控与成本优化参考资料Triton Inference Server官方文档NVIDIA k8s-device-pluginKubernetes HPA官方文档