模型服务成本先看缓存与伸缩模型服务的成本预算先要拆成可解释的项目请求量、Token、缓存命中、并发和基础资源。本文讨论预算有限时的优先检查顺序不虚构账单或优化收益。若成本上升先核对请求形态、上下文长度、缓存状态和排队时间。相似提问是否适合复用结果也要取决于知识版本、权限和回答风险。语义缓存、限流和弹性伸缩是可选手段不是固定答案。先建立网关侧观测再用代表性请求验证它们是否改善了目标指标。1. 网关层语义缓存与算力防护架构传统的 KV 缓存如 Redis key-value在 LLM 场景下几乎失效因为用户输入的提问文本很难做到逐字匹配。语义缓存的核心在于将文本向量化在向量空间内计算余弦相似度Cosine Similarity。网关可先将请求转成向量并检索候选缓存。命中阈值、知识版本、用户权限和模型版本应一起参与判断未命中后再按队列和上下文长度路由到推理池。2. 线上耗时排查与压测诊断命令在没有任何防护的情况下使用k6进行 50 并发下的 LLM 网关压测观察响应首字延迟TTFT与 Token 吞吐指标。# 1. 使用 k6 模拟高频重复提问与突发流量压测 k6 run --vus 50 --duration 2m - EOF import http from k6/http; import { check, sleep } from k6; export default function () { const payload JSON.stringify({ model: qwen2.5-72b-instruct, messages: [{ role: user, content: 请详细说明 Spring Boot 线程池拒绝策略的配置方法 }], temperature: 0.7 }); const params { headers: { Content-Type: application/json, Authorization: Bearer test-token-0821 }, }; const res http.post(${__ENV.LLM_GATEWAY_URL}/v1/chat/completions, payload, params); check(res, { status is 200: (r) r.status 200, ttft under 800ms: (r) r.timings.waiting 800, }); sleep(0.5); } EOF诊断日志与 GPU 节点监控指标命令# 2. 检查底层 vLLM 容器的 Prompt 缓存命中率与 KV Cache 占用 curl -s ${VLLM_METRICS_URL} | grep -E vllm:num_requests_waiting|vllm:gpu_cache_usage_perc # 3. 抓取网关层 GC 与线程阻断状态 jcmd $(pgrep -f llm-gateway) Thread.print | grep -A 10 BLOCKED通过压测日志发现未加语义缓存前vllm:gpu_cache_usage_perc迅速冲到 98%等待队列vllm:num_requests_waiting堆积超过 120 个请求P99 首字延迟直接恶化到了 4.2 秒。3. 生产级 Gateway 语义缓存 Handler 实现在 Spring Cloud Gateway / Java 网关体系中实现前置语义缓存拦截器。代码中引入向量余弦相似度计算与基于 Redis 的分布式锁防击穿机制。package com.example.gateway.cache; import org.springframework.data.redis.core.StringRedisTemplate; import org.springframework.stereotype.Component; import java.util.List; import java.util.concurrent.CompletableFuture; Component public class SemanticCacheHandler { private final StringRedisTemplate redisTemplate; private final EmbeddingClient embeddingClient; // 相似度判定阈值 private static final double SIMILARITY_THRESHOLD 0.92; private static final String CACHE_PREFIX llm:cache:vector:; public SemanticCacheHandler(StringRedisTemplate redisTemplate, EmbeddingClient embeddingClient) { this.redisTemplate redisTemplate; this.embeddingClient embeddingClient; } public String getCachedResponseIfPresent(String userPrompt) { // 1. 生成当前 Prompt 的 Embedding 向量 float[] queryVector embeddingClient.embed(userPrompt); // 2. 检索临近向量 Key此处示例通过 Redis HGET 批量获取热门 Vector 实体 ListVectorEntity candidates loadHotCacheCandidates(); VectorEntity bestMatch null; double maxSimilarity -1.0; for (VectorEntity candidate : candidates) { double similarity cosineSimilarity(queryVector, candidate.getVector()); if (similarity maxSimilarity) { maxSimilarity similarity; bestMatch candidate; } } // 3. 命中判定 if (maxSimilarity SIMILARITY_THRESHOLD bestMatch ! null) { String cachedCompletion redisTemplate.opsForValue().get(CACHE_PREFIX bestMatch.getId()); if (cachedCompletion ! null) { return cachedCompletion; } } return null; } public void putCacheAsync(String userPrompt, String llmResponse) { // 异步计算 Embedding 并存入 Cache设置 24 小时 TTL CompletableFuture.runAsync(() - { float[] vector embeddingClient.embed(userPrompt); String id java.util.UUID.randomUUID().toString(); saveVectorMetadata(id, vector); redisTemplate.opsForValue().set(CACHE_PREFIX id, llmResponse, java.time.Duration.ofHours(24)); }); } private double cosineSimilarity(float[] vectorA, float[] vectorB) { double dotProduct 0.0; double normA 0.0; double normB 0.0; for (int i 0; i vectorA.length; i) { dotProduct vectorA[i] * vectorB[i]; normA Math.pow(vectorA[i], 2); normB Math.pow(vectorB[i], 2); } if (normA 0 || normB 0) return 0.0; return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB)); } private ListVectorEntity loadHotCacheCandidates() { // 生产环境中可接入 Milvus / Pgvector / RedisSearch 索引 return List.of(); } private void saveVectorMetadata(String id, float[] vector) { // 存储向量索引元数据 } public static class VectorEntity { private String id; private float[] vector; public String getId() { return id; } public float[] getVector() { return vector; } } }4. 基于 Token 指标的 K8s HPA 弹性伸缩配置算力预算优化的另一半在于“用时扩容闲时缩容”。不能仅看 CPU/MEM 利用率必须将 vLLM 的vllm:num_requests_waiting暴露为 Prometheus 自定义指标驱动 K8s HPA。配置PrometheusRule与HorizontalPodAutoscaler清单apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: vllm-metrics-rules namespace: llm-service spec: groups: - name: vllm.rules rules: - record: vllm:requests_waiting_avg expr: avg(vllm:num_requests_waiting{namespacellm-service}) by (deployment) --- apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: vllm-inference-hpa namespace: llm-service spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: vllm-inference-node minReplicas: 2 maxReplicas: 8 metrics: - type: External external: metric: name: vllm_requests_waiting_avg target: type: Value averageValue: 15 behavior: scaleUp: stabilizationWindowSeconds: 30 policies: - type: Percent value: 100 periodSeconds: 15 scaleDown: stabilizationWindowSeconds: 300 policies: - type: Percent value: 25 periodSeconds: 605. 预算治理落地效果与防线配置总结在上线语义缓存与基于 Custom Metrics 的 HPA 之后再次进行 k6 相同规格的压力测试。验证时应在同一批脱敏请求上记录命中率、答案可用性、排队时间和资源占用并保留缓存误命中的回退路径。预算紧时先从观测结果中找无效调用和过长上下文是否启用缓存、裁剪或扩容取决于这轮验证的结果。