AI 算法评测体系如何量化评估大模型的算法能力一、大模型算法能力的评估困境准确率不够用当前评估 LLM 算法能力的常见做法是跑 HumanEval / MBPP报告 pass1。但这个数字掩盖了太多信息一道题第一次通过和第五次通过能力显然不同一道 Easy 通过和一道 Hard 通过权重也不该一样。更关键的是pass1 只衡量能不能写出正确代码不衡量写得有多好——复杂度是否达标、代码是否健壮、边界是否处理。一个完整的算法能力评测体系需要多维度量化正确性、复杂度、鲁棒性、代码质量。只有多维度评估才能发现 LLM 的真实短板指导后续训练和提示优化。二、多维度评测体系的设计2.1 评测维度与指标体系flowchart TD A[LLM算法能力评测] -- B[正确性维度] A -- C[复杂度维度] A -- D[鲁棒性维度] A -- E[代码质量维度] B -- B1[passk: k次采样通过率] B -- B2[首次通过率] B -- B3[用例覆盖率] C -- C1[复杂度达标率] C -- C2[最优解比例] C -- C3[复杂度偏差度] D -- D1[边界用例通过率] D -- D2[异常输入处理率] D -- D3[大规模数据通过率] E -- E1[代码简洁度] E -- E2[可读性评分] E -- E3[风格一致性] B1 B2 B3 C1 C2 C3 D1 D2 D3 E1 E2 E3 -- F[综合能力评分]2.2 passk 的数学定义passk 衡量的是在 k 次采样中至少有一次通过的概率。设 n 次采样中有 c 次通过则passk 1 - C(n-c, k) / C(n, k)当 n10, c5, k1 时pass1 1 - C(5,1)/C(10,1) 1 - 5/10 0.5。这个指标比简单的通过率更稳定因为它考虑了采样方差。2.3 复杂度评估的方法论复杂度评估不能只看代码结构嵌套循环层数必须实际运行测量。方法对同一道题用不同规模n10^3, 10^4, 10^5的输入运行记录耗时用倍率法推断复杂度类别。如果推断结果与题目要求不符即使功能正确也标记为复杂度不达标。三、生产级实现LLM 算法能力评测框架from typing import List, Dict, Tuple, Optional, Callable from dataclasses import dataclass, field from enum import Enum import time import math import ast import textwrap from collections import defaultdict class ComplexityClass(Enum): 复杂度类别 O1 O(1) O_LOG_N O(log n) O_N O(n) O_N_LOG_N O(n log n) O_N2 O(n^2) O_N3 O(n^3) O_2N O(2^n) UNKNOWN unknown dataclass class ProblemSpec: 题目规格 id: str title: str difficulty: str # easy/medium/hard topic: str required_complexity: ComplexityClass test_cases: List[Tuple] # [(input, expected), ...] edge_cases: List[Tuple] # 边界用例 large_input_generator: Optional[Callable] None dataclass class EvaluationMetrics: 评测指标 # 正确性 pass_at_1: float 0.0 first_attempt_rate: float 0.0 test_case_coverage: float 0.0 # 复杂度 complexity_pass_rate: float 0.0 optimal_solution_rate: float 0.0 avg_complexity_deviation: float 0.0 # 鲁棒性 edge_case_pass_rate: float 0.0 large_input_pass_rate: float 0.0 # 代码质量 avg_code_length: float 0.0 readability_score: float 0.0 # 综合 overall_score: float 0.0 dataclass class SingleEvaluation: 单次评测结果 problem_id: str functional_passed: bool complexity_class: ComplexityClass complexity_met: bool edge_cases_passed: int edge_cases_total: int large_input_passed: Optional[bool] code_length: int execution_time_ms: float error_message: Optional[str] None class LLMAlgorithmEvaluator: LLM 算法能力评测框架 多维度评估正确性、复杂度、鲁棒性、代码质量 # 复杂度阶数用于比较大小 COMPLEXITY_ORDER { ComplexityClass.O1: 0, ComplexityClass.O_LOG_N: 1, ComplexityClass.O_N: 2, ComplexityClass.O_N_LOG_N: 3, ComplexityClass.O_N2: 4, ComplexityClass.O_N3: 5, ComplexityClass.O_2N: 10, ComplexityClass.UNKNOWN: 6, } def __init__(self, time_limit_ms: float 3000): self.time_limit_ms time_limit_ms self._results: Dict[str, List[SingleEvaluation]] defaultdict(list) def evaluate_single( self, code: str, spec: ProblemSpec, ) - SingleEvaluation: 评估单次生成的代码 Args: code: LLM 生成的代码 spec: 题目规格 Returns: 评测结果 # 1. 功能测试 functional_passed, error_msg self._run_functional_test( code, spec.test_cases ) # 2. 复杂度评估 complexity self._estimate_complexity(code) complexity_met ( self.COMPLEXITY_ORDER.get(complexity, 6) self.COMPLEXITY_ORDER.get(spec.required_complexity, 6) ) # 3. 边界用例测试 edge_passed, edge_total self._run_edge_tests(code, spec.edge_cases) # 4. 大规模输入测试 large_passed None if spec.large_input_generator and functional_passed: large_passed self._run_large_input_test( code, spec.large_input_generator ) # 5. 代码质量 code_length len(code.strip().split(\n)) return SingleEvaluation( problem_idspec.id, functional_passedfunctional_passed, complexity_classcomplexity, complexity_metcomplexity_met, edge_cases_passededge_passed, edge_cases_totaledge_total, large_input_passedlarge_passed, code_lengthcode_length, execution_time_ms0.0, error_messageerror_msg, ) def evaluate_multi_sample( self, samples: Dict[str, List[str]], specs: Dict[str, ProblemSpec], n_samples: int 10, ) - EvaluationMetrics: 多采样评估计算 passk 等统计指标 Args: samples: {problem_id: [code1, code2, ...]} specs: {problem_id: ProblemSpec} n_samples: 采样次数 Returns: 综合评测指标 all_evals: List[SingleEvaluation] [] for pid, code_list in samples.items(): if pid not in specs: continue spec specs[pid] for code in code_list: result self.evaluate_single(code, spec) self._results[pid].append(result) all_evals.append(result) if not all_evals: return EvaluationMetrics() return self._compute_metrics(all_evals, samples, specs) def _run_functional_test( self, code: str, test_cases: List[Tuple] ) - Tuple[bool, Optional[str]]: 执行功能测试 if not test_cases: return True, None namespace {} try: exec(textwrap.dedent(code), namespace) except Exception as e: return False, f代码执行失败: {e} # 找到函数 func self._find_function(namespace) if not func: return False, 未找到可调用函数 for args, expected in test_cases: try: if isinstance(args, tuple): result func(*args) else: result func(args) if result ! expected: return False, f结果不匹配: 期望{expected}, 实际{result} except Exception as e: return False, f运行异常: {e} return True, None def _run_edge_tests( self, code: str, edge_cases: List[Tuple] ) - Tuple[int, int]: 执行边界用例测试 if not edge_cases: return 0, 0 namespace {} try: exec(textwrap.dedent(code), namespace) except Exception: return 0, len(edge_cases) func self._find_function(namespace) if not func: return 0, len(edge_cases) passed 0 for args, expected in edge_cases: try: if isinstance(args, tuple): result func(*args) else: result func(args) if result expected: passed 1 except Exception: pass return passed, len(edge_cases) def _run_large_input_test( self, code: str, input_generator: Callable ) - bool: 大规模输入测试 namespace {} try: exec(textwrap.dedent(code), namespace) except Exception: return False func self._find_function(namespace) if not func: return False try: large_input input_generator() start time.perf_counter() if isinstance(large_input, tuple): func(*large_input) else: func(large_input) elapsed (time.perf_counter() - start) * 1000 return elapsed self.time_limit_ms except Exception: return False def _estimate_complexity(self, code: str) - ComplexityClass: 基于 AST 预估复杂度 try: tree ast.parse(textwrap.dedent(code)) except SyntaxError: return ComplexityClass.UNKNOWN max_depth self._count_nested_loops(tree) mapping { 0: ComplexityClass.O1, 1: ComplexityClass.O_N, 2: ComplexityClass.O_N2, 3: ComplexityClass.O_N3, } return mapping.get(max_depth, ComplexityClass.O_2N) def _count_nested_loops(self, tree: ast.AST) - int: 统计最大嵌套循环层数 def _walk(node: ast.AST, depth: int) - int: if isinstance(node, (ast.For, ast.While)): depth 1 max_d depth for child in ast.iter_child_nodes(node): max_d max(max_d, _walk(child, depth)) return max_d return _walk(tree, 0) def _find_function(self, namespace: dict) - Optional[Callable]: 从命名空间中找到用户定义的函数 for name, obj in namespace.items(): if callable(obj) and not name.startswith(_): return obj return None def _compute_metrics( self, all_evals: List[SingleEvaluation], samples: Dict[str, List[str]], specs: Dict[str, ProblemSpec], ) - EvaluationMetrics: 计算综合评测指标 metrics EvaluationMetrics() # pass1 计算 problem_results defaultdict(list) for ev in all_evals: problem_results[ev.problem_id].append(ev.functional_passed) pass_at_1_values [] for pid, results in problem_results.items(): n len(results) c sum(results) if n 0: # pass1 1 - C(n-c, 1) / C(n, 1) c / n pass_at_1_values.append(c / n) metrics.pass_at_1 ( sum(pass_at_1_values) / len(pass_at_1_values) if pass_at_1_values else 0.0 ) # 复杂度达标率 complexity_evals [e for e in all_evals if e.functional_passed] if complexity_evals: metrics.complexity_pass_rate ( sum(1 for e in complexity_evals if e.complexity_met) / len(complexity_evals) ) # 边界用例通过率 edge_evals [ e for e in all_evals if e.functional_passed and e.edge_cases_total 0 ] if edge_evals: metrics.edge_case_pass_rate ( sum(e.edge_cases_passed for e in edge_evals) / sum(e.edge_cases_total for e in edge_evals) ) # 大规模输入通过率 large_evals [ e for e in all_evals if e.functional_passed and e.large_input_passed is not None ] if large_evals: metrics.large_input_pass_rate ( sum(1 for e in large_evals if e.large_input_passed) / len(large_evals) ) # 代码长度 metrics.avg_code_length ( sum(e.code_length for e in all_evals) / len(all_evals) ) # 综合评分加权平均 metrics.overall_score ( metrics.pass_at_1 * 40 # 正确性权重 40% metrics.complexity_pass_rate * 25 # 复杂度权重 25% metrics.edge_case_pass_rate * 20 # 鲁棒性权重 20% metrics.large_input_pass_rate * 15 # 性能权重 15% ) return metrics def generate_report(self, metrics: EvaluationMetrics) - str: 生成评测报告 lines [ * 55, LLM 算法能力评测报告, * 55, , 【正确性维度】, f pass1: {metrics.pass_at_1:.1%}, , 【复杂度维度】, f 复杂度达标率: {metrics.complexity_pass_rate:.1%}, , 【鲁棒性维度】, f 边界用例通过率: {metrics.edge_case_pass_rate:.1%}, f 大规模输入通过率: {metrics.large_input_pass_rate:.1%}, , 【代码质量维度】, f 平均代码行数: {metrics.avg_code_length:.0f}, , 【综合评分】, f 总分: {metrics.overall_score:.1f} / 100, , 权重分配: 正确性40% 复杂度25% 鲁棒性20% 性能15%, ] return \n.join(lines) # 使用示例 if __name__ __main__: evaluator LLMAlgorithmEvaluator(time_limit_ms3000) # 定义题目规格 specs { two_sum: ProblemSpec( idtwo_sum, title两数之和, difficultyeasy, topichash_table, required_complexityComplexityClass.O_N, test_cases[ (([2, 7, 11, 15], 9), [0, 1]), (([3, 2, 4], 6), [1, 2]), ], edge_cases[ (([1], 1), []), # 单元素 (([], 0), []), # 空数组 (([3, 3], 6), [0, 1]), # 重复元素 ], large_input_generatorlambda: ( list(range(100000)), 199999 ), ), } # 模拟多次采样 samples { two_sum: [ # 采样1: 正确且复杂度达标 def two_sum(nums, target): seen {} for i, num in enumerate(nums): complement target - num if complement in seen: return [seen[complement], i] seen[num] i return [] , # 采样2: 暴力解法复杂度不达标 def two_sum(nums, target): for i in range(len(nums)): for j in range(i 1, len(nums)): if nums[i] nums[j] target: return [i, j] return [] , # 采样3: 有边界问题 def two_sum(nums, target): seen {} for i, num in enumerate(nums): if target - num in seen: return [seen[target - num], i] seen[num] i return [] , ], } # 执行评测 metrics evaluator.evaluate_multi_sample(samples, specs) print(evaluator.generate_report(metrics))四、评测体系的局限与改进方向4.1 静态复杂度分析的精度问题基于 AST 的嵌套循环分析有两个系统性偏差一是高估——内层循环次数递减时如快排 partition实际复杂度低于静态分析结果二是低估——递归算法的复杂度无法通过 AST 分析得出需要专门的递推式求解。改进方向是引入符号执行或基于运行时的倍率法自动推断。4.2 测试用例的覆盖度自动生成的测试用例只能覆盖已知模式的边界条件。对于语义层面的错误如最大子数组和与最大子序列和的混淆功能测试无法区分——两者在某些用例上结果相同但语义完全不同。形式化验证能解决部分问题但成本过高。4.3 评测偏差与公平性偏差来源影响缓解措施题目选择偏差偏重某类专题均衡覆盖各专题和难度语言偏差Python 比 C 更易通过按语言分组评测采样偏差采样次数少时 passk 不稳定增大采样次数到 50评分偏差权重分配主观多组权重取平均4.4 评测与训练的对抗关系如果 LLM 的训练数据包含了评测题目passk 就失去了意义——模型可能只是记住了答案。需要定期更新评测集或使用动态生成的题目。但动态生成又面临质量控制的挑战自动生成的题目可能存在歧义或无解的情况。五、总结本文构建了 LLM 算法能力的多维度评测框架覆盖正确性passk、复杂度达标率、鲁棒性边界/大规模输入通过率和代码质量四个维度。综合评分采用加权平均权重按工程重要性分配。代码实现包含功能测试、AST 复杂度分析、边界用例检测和大规模输入压力测试。但评测体系存在静态分析精度不足、测试覆盖度有限、评测偏差等局限。评测的最终目标不是排名而是发现短板、指导改进——一个诚实的评测比一个好看的分数更有价值。