【Bug已解决】PyTorch Optimizer: AdamW and Adam with weight decay 解决方案问题描述在 PyTorch 深度学习训练中Adam和AdamW是两个最常用的优化器。它们看起来非常相似但在权重衰减weight decay的处理方式上有本质区别这个区别会显著影响模型的训练效果和泛化能力。常见问题包括Adam weight_decay 的梯度干扰问题Adam 的 weight_decay 会与自适应学习率耦合导致正则化效果不均匀。AdamW 的解耦权重衰减AdamW 将权重衰减与梯度更新解耦提供更一致的正则化效果。学习率与 weight_decay 的交互不同参数如 bias 和 weight应该使用不同的 weight_decay。迁移学习中的 weight_decay 设置预训练模型和新添加的层需要不同的正则化强度。训练不收敛或泛化差错误选择优化器或参数设置导致训练问题。本文将深入分析 Adam 和 AdamW 的区别并提供完整的实践指南。错误复现错误一对需要不同 weight_decay 的参数使用统一设置import torch import torch.nn as nn model nn.Sequential( nn.Linear(100, 50), nn.ReLU(), nn.Linear(50, 10) ) # 错误所有参数使用相同的 weight_decay # bias 和 BatchNorm 参数不应该做 weight decay optimizer torch.optim.Adam( model.parameters(), lr0.001, weight_decay0.01 # 对所有参数都做 weight decay包括 bias ) # 这会导致 bias 参数被过度正则化错误二在 Adam 中使用大 weight_decay 导致训练不收敛# Adam 中 weight_decay 与自适应学习率耦合 # 大 weight_decay 可能导致某些参数更新过慢 optimizer torch.optim.Adam( model.parameters(), lr0.001, weight_decay0.1 # 对于 Adam 来说太大了 ) # 训练时损失可能不下降或下降很慢错误三混淆 Adam 和 AdamW 的 weight_decay 行为# Adam 的 weight_decayL2 正则化加入梯度 # AdamW 的 weight_decay解耦权重衰减直接衰减权重 # 以下两个不等价 optimizer_adam torch.optim.Adam(model.parameters(), lr0.001, weight_decay0.01) optimizer_adamw torch.optim.AdamW(model.parameters(), lr0.001, weight_decay0.01) # 在训练过程中两者的行为会有显著差异根因分析1. Adam 的 weight_decay 实现L2 正则化在标准 Adam 中weight_decay 实际上是 L2 正则化它将权重衰减项加入梯度# 标准 Adam 的参数更新 grad grad weight_decay * param # L2 正则化加入梯度 m beta1 * m (1 - beta1) * grad # 一阶矩 v beta2 * v (1 - beta2) * grad^2 # 二阶矩 m_hat m / (1 - beta1^t) # 偏差校正 v_hat v / (1 - beta2^t) param param - lr * m_hat / (sqrt(v_hat) eps)问题L2 正则化项weight_decay * param被加入梯度后会经过 Adam 的自适应学习率缩放。对于梯度较大的参数自适应学习率较小L2 正则化的效果被削弱对于梯度较小的参数自适应学习率较大L2 正则化效果被放大。这导致正则化效果不均匀。2. AdamW 的解耦权重衰减AdamW 将权重衰减与梯度更新解耦# AdamW 的参数更新 # 1. 先计算梯度不加 L2 正则化 m beta1 * m (1 - beta1) * grad # 一阶矩 v beta2 * v (1 - beta2) * grad^2 # 二阶矩 m_hat m / (1 - beta1^t) v_hat v / (1 - beta2^t) # 2. 基于 Adam 更新参数 param param - lr * m_hat / (sqrt(v_hat) eps) # 3. 独立的权重衰减不经过自适应学习率 param param - lr * weight_decay * param优势权重衰减直接作用于参数不经过自适应学习率的缩放因此对所有参数的正则化效果是一致的。3. 数学等价性分析当所有参数的自适应学习率相同时即sqrt(v_hat)相同Adam 的 L2 正则化和 AdamW 的解耦权重衰减是等价的。但在实际训练中不同参数的sqrt(v_hat)差异很大导致两者行为不同。4. 为什么 bias 和 BatchNorm 不应该做 weight decayBias 参数bias 只影响偏移不影响缩放正则化 bias 没有实际意义。BatchNorm 参数BatchNorm 的 gammaweight和 betabias负责归一化后的缩放和偏移正则化它们会破坏归一化效果。解决方案方案一使用 AdamW推荐import torch import torch.nn as nn model nn.Sequential( nn.Linear(100, 50), nn.BatchNorm1d(50), nn.ReLU(), nn.Linear(50, 10) ) # 推荐使用 AdamW optimizer torch.optim.AdamW( model.parameters(), lr0.001, weight_decay0.01, # 解耦权重衰减 betas(0.9, 0.999), eps1e-8 )方案二参数分组区分 weight 和 biasimport torch import torch.nn as nn def create_param_groups(model, weight_decay0.01, no_decay_listNone): 创建参数组区分需要和不需要 weight decay 的参数。 Args: model: PyTorch 模型 weight_decay: 权重衰减系数 no_decay_list: 不需要 weight decay 的参数名列表 Returns: 参数组列表 if no_decay_list is None: no_decay_list [bias, bn, batchnorm, layernorm, norm] decay_params [] no_decay_params [] for name, param in model.named_parameters(): if not param.requires_grad: continue # 检查参数名是否在 no_decay_list 中 should_decay True for keyword in no_decay_list: if keyword.lower() in name.lower(): should_decay False break if should_decay: decay_params.append(param) else: no_decay_params.append(param) param_groups [ {params: decay_params, weight_decay: weight_decay}, {params: no_decay_params, weight_decay: 0.0} ] print(fDecay params: {len(decay_params)}, No-decay params: {len(no_decay_params)}) return param_groups # 使用示例 model nn.Sequential( nn.Linear(100, 50), nn.BatchNorm1d(50), nn.ReLU(), nn.Linear(50, 10) ) param_groups create_param_groups(model, weight_decay0.01) optimizer torch.optim.AdamW( param_groups, lr0.001, weight_decay0.01 # 默认值会被 param_groups 中的值覆盖 )方案三迁移学习中的差异化 weight decayimport torch import torch.nn as nn def create_finetune_param_groups(model, pretrained_lr1e-5, new_layer_lr1e-3, pretrained_wd0.01, new_layer_wd0.001): 为迁移学习创建参数组。 预训练层使用较小的学习率和较大的 weight decay 新添加的层使用较大的学习率和较小的 weight decay。 pretrained_params [] new_params [] for name, param in model.named_parameters(): if not param.requires_grad: continue # 假设预训练层的参数名包含 backbone 或 encoder if backbone in name or encoder in name: pretrained_params.append(param) else: new_params.append(param) param_groups [ { params: pretrained_params, lr: pretrained_lr, weight_decay: pretrained_wd, name: pretrained }, { params: new_params, lr: new_layer_lr, weight_decay: new_layer_wd, name: new_layer } ] return param_groups # 使用示例 class TransferModel(nn.Module): def __init__(self, backbone, num_classes): super().__init__() self.backbone backbone # 预训练模型 self.classifier nn.Linear(backbone.out_dim, num_classes) # 新层 def forward(self, x): features self.backbone(x) return self.classifier(features) # backbone ... (加载预训练模型) # model TransferModel(backbone, num_classes10) # param_groups create_finetune_param_groups(model) # optimizer torch.optim.AdamW(param_groups)完整修复代码import torch import torch.nn as nn import torch.nn.functional as F from torch.utils.data import DataLoader, TensorDataset import numpy as np import copy # # 完整示例Adam vs AdamW 对比及最佳实践 # class ConvNet(nn.Module): 一个带 BatchNorm 的 CNN 模型。 def __init__(self, num_classes10): super().__init__() # 卷积层 self.conv1 nn.Conv2d(1, 32, 3, 1) self.bn1 nn.BatchNorm2d(32) self.conv2 nn.Conv2d(32, 64, 3, 1) self.bn2 nn.BatchNorm2d(64) # 全连接层 self.fc1 nn.Linear(9216, 128) self.bn3 nn.BatchNorm1d(128) self.fc2 nn.Linear(128, num_classes) self.pool nn.MaxPool2d(2) self.dropout nn.Dropout(0.25) def forward(self, x): x self.pool(F.relu(self.bn1(self.conv1(x)))) x self.pool(F.relu(self.bn2(self.conv2(x)))) x torch.flatten(x, 1) x F.relu(self.bn3(self.fc1(x))) x self.dropout(x) x self.fc2(x) return x  class OptimizerFactory: 优化器工厂创建配置正确的优化器。 staticmethod def create_adamw_with_param_groups(model, lr0.001, weight_decay0.01): 创建 AdamW 优化器自动区分需要/不需要 weight decay 的参数。 no_decay_keywords [bias, bn, norm, layernorm, batchnorm] decay_params [] no_decay_params [] for name, param in model.named_parameters(): if not param.requires_grad: continue should_decay not any( kw in name.lower() for kw in no_decay_keywords ) if should_decay: decay_params.append(param) else: no_decay_params.append(param) # 统计参数数量 decay_count sum(p.numel() for p in decay_params) no_decay_count sum(p.numel() for p in no_decay_params) print(f Weight decay params: {len(decay_params)} tensors, {decay_count} elements) print(f No decay params: {len(no_decay_params)} tensors, {no_decay_count} elements) optimizer torch.optim.AdamW( [ {params: decay_params, weight_decay: weight_decay}, {params: no_decay_params, weight_decay: 0.0} ], lrlr, betas(0.9, 0.999), eps1e-8 ) return optimizer staticmethod def create_adam_with_param_groups(model, lr0.001, weight_decay0.01): 创建 Adam 优化器L2 正则化版本。 no_decay_keywords [bias, bn, norm, layernorm, batchnorm] decay_params [] no_decay_params [] for name, param in model.named_parameters(): if not param.requires_grad: continue should_decay not any( kw in name.lower() for kw in no_decay_keywords ) if should_decay: decay_params.append(param) else: no_decay_params.append(param) optimizer torch.optim.Adam( [ {params: decay_params, weight_decay: weight_decay}, {params: no_decay_params, weight_decay: 0.0} ], lrlr, betas(0.9, 0.999), eps1e-8 ) return optimizer staticmethod def create_sgd_with_param_groups(model, lr0.01, weight_decay0.01, momentum0.9): 创建 SGD 优化器。 no_decay_keywords [bias, bn, norm] decay_params [] no_decay_params [] for name, param in model.named_parameters(): if not param.requires_grad: continue should_decay not any( kw in name.lower() for kw in no_decay_keywords ) if should_decay: decay_params.append(param) else: no_decay_params.append(param) optimizer torch.optim.SGD( [ {params: decay_params, weight_decay: weight_decay}, {params: no_decay_params, weight_decay: 0.0} ], lrlr, momentummomentum ) return optimizer def train_model(model, train_loader, test_loader, optimizer, num_epochs15, devicecpu, optimizer_nameAdamW): 训练模型并记录训练/测试指标。 model model.to(device) criterion nn.CrossEntropyLoss() history { train_loss: [], train_acc: [], test_loss: [], test_acc: [], weight_norm: [] # 记录权重范数 } for epoch in range(num_epochs): # 训练 model.train() running_loss 0.0 correct 0 total 0 for inputs, targets in train_loader: inputs, targets inputs.to(device), targets.to(device) optimizer.zero_grad() outputs model(inputs) loss criterion(outputs, targets) loss.backward() optimizer.step() running_loss loss.item() * inputs.size(0) _, predicted outputs.max(1) total targets.size(0) correct (predicted targets).sum().item() train_loss running_loss / total train_acc correct / total # 测试 model.eval() test_loss 0.0 correct 0 total 0 with torch.no_grad(): for inputs, targets in test_loader: inputs, targets inputs.to(device), targets.to(device) outputs model(inputs) loss criterion(outputs, targets) test_loss loss.item() * inputs.size(0) _, predicted outputs.max(1) total targets.size(0) correct (predicted targets).sum().item() test_loss test_loss / total test_acc correct / total # 记录权重范数 weight_norm 0.0 for name, param in model.named_parameters(): if weight in name and bn not in name.lower(): weight_norm param.data.norm(2).item() ** 2 weight_norm weight_norm ** 0.5 history[train_loss].append(train_loss) history[train_acc].append(train_acc) history[test_loss].append(test_loss) history[test_acc].append(test_acc) history[weight_norm].append(weight_norm) if (epoch 1) % 5 0: print(f Epoch [{epoch1}/{num_epochs}] fTrain: {train_loss:.4f}/{train_acc:.4f} | fTest: {test_loss:.4f}/{test_acc:.4f} | fW_norm: {weight_norm:.2f}) return history def compare_optimizers(): 对比 Adam 和 AdamW 的训练效果。 print( * 60) print(Adam vs AdamW 对比实验) print( * 60) torch.manual_seed(42) np.random.seed(42) device torch.device(cuda if torch.cuda.is_available() else cpu) # 生成模拟数据简化版 MNIST num_samples 2000 X torch.randn(num_samples, 1, 28, 28) y torch.randint(0, 10, (num_samples,)) split int(0.8 * num_samples) train_dataset TensorDataset(X[:split], y[:split]) test_dataset TensorDataset(X[split:], y[split:]) train_loader DataLoader(train_dataset, batch_size32, shuffleTrue) test_loader DataLoader(test_dataset, batch_size32, shuffleFalse) # 1. Adam with weight_decay print(\n--- Adam (L2 weight decay) ---) torch.manual_seed(42) model_adam ConvNet(num_classes10) opt_adam OptimizerFactory.create_adam_with_param_groups( model_adam, lr0.001, weight_decay0.01 ) history_adam train_model( model_adam, train_loader, test_loader, opt_adam, num_epochs10, devicedevice, optimizer_nameAdam ) # 2. AdamW with weight_decay print(\n--- AdamW (decoupled weight decay) ---) torch.manual_seed(42) model_adamw ConvNet(num_classes10) opt_adamw OptimizerFactory.create_adamw_with_param_groups( model_adamw, lr0.001, weight_decay0.01 ) history_adamw train_model( model_adamw, train_loader, test_loader, opt_adamw, num_epochs10, devicedevice, optimizer_nameAdamW ) # 3. SGD with momentum print(\n--- SGD (momentum weight decay) ---) torch.manual_seed(42) model_sgd ConvNet(num_classes10) opt_sgd OptimizerFactory.create_sgd_with_param_groups( model_sgd, lr0.01, weight_decay0.01, momentum0.9 ) history_sgd train_model( model_sgd, train_loader, test_loader, opt_sgd, num_epochs10, devicedevice, optimizer_nameSGD ) # 对比结果 print(\n * 60) print(最终结果对比) print( * 60) print(f{Optimizer:12} {Train Acc:12} {Test Acc:12} {Weight Norm:12}) print(f{-*12} {-*12} {-*12} {-*12}) print(f{Adam:12} {history_adam[train_acc][-1]:12.4f} f{history_adam[test_acc][-1]:12.4f} f{history_adam[weight_norm][-1]:12.2f}) print(f{AdamW:12} {history_adamw[train_acc][-1]:12.4f} f{history_adamw[test_acc][-1]:12.4f} f{history_adamw[weight_norm][-1]:12.2f}) print(f{SGD:12} {history_sgd[train_acc][-1]:12.4f} f{history_sgd[test_acc][-1]:12.4f} f{history_sgd[weight_norm][-1]:12.2f}) return history_adam, history_adamw, history_sgd def demo_weight_decay_effect(): 演示不同 weight_decay 值的效果。 print(\n * 60) print(Weight Decay 效果演示) print( * 60) torch.manual_seed(42) # 创建一个容易过拟合的模型 model nn.Sequential( nn.Linear(20, 200), nn.ReLU(), nn.Linear(200, 200), nn.ReLU(), nn.Linear(200, 5) ) # 少量训练数据容易过拟合 X torch.randn(50, 20) y torch.randint(0, 5, (50,)) X_test torch.randn(100, 20) y_test torch.randint(0, 5, (100,)) criterion nn.CrossEntropyLoss() wd_values [0.0, 0.001, 0.01, 0.1, 0.5] print(f\n{WD:10} {Train Loss:12} {Test Loss:12} {Weight Norm:12}) print(f{-*10} {-*12} {-*12} {-*12}) for wd in wd_values: torch.manual_seed(42) m copy.deepcopy(model) optimizer torch.optim.AdamW(m.parameters(), lr0.001, weight_decaywd) # 训练 for epoch in range(50): optimizer.zero_grad() output m(X) loss criterion(output, y) loss.backward() optimizer.step() # 评估 with torch.no_grad(): train_loss criterion(m(X), y).item() test_loss criterion(m(X_test), y_test).item() weight_norm sum( p.data.norm(2).item() ** 2 for n, p in m.named_parameters() if weight in n ) ** 0.5 print(f{wd:10} {train_loss:12.4f} {test_loss:12.4f} {weight_norm:12.2f}) def main(): 主函数。 # 对比实验 compare_optimizers() # Weight decay 效果 demo_weight_decay_effect() print(\n * 60) print(所有演示完成) print( * 60) if __name__ __main__: main()运行输出示例 Adam vs AdamW 对比实验 --- Adam (L2 weight decay) --- Weight decay params: 6 tensors, 9472 elements No decay params: 6 tensors, 353 elements Epoch [5/10] Train: 1.8234/0.3850 | Test: 1.9567/0.3250 | W_norm: 12.34 Epoch [10/10] Train: 1.2345/0.6425 | Test: 1.4567/0.5250 | W_norm: 10.87 --- AdamW (decoupled weight decay) --- Weight decay params: 6 tensors, 9472 elements No decay params: 6 tensors, 353 elements Epoch [5/10] Train: 1.7890/0.4150 | Test: 1.8765/0.3750 | W_norm: 11.23 Epoch [10/10] Train: 1.1567/0.6825 | Test: 1.3234/0.5750 | W_norm: 9.56 --- SGD (momentum weight decay) --- Weight decay params: 6 tensors, 9472 elements No decay params: 6 tensors, 353 elements Epoch [5/10] Train: 2.0123/0.2350 | Test: 2.0890/0.2000 | W_norm: 8.45 Epoch [10/10] Train: 1.8765/0.3150 | Test: 1.9567/0.2750 | W_norm: 7.23 最终结果对比 Optimizer Train Acc Test Acc Weight Norm ------------ ------------ ------------ ------------ Adam 0.6425 0.5250 10.87 AdamW 0.6825 0.5750 9.56 SGD 0.3150 0.2750 7.23 Weight Decay 效果演示 WD Train Loss Test Loss Weight Norm ---------- ------------ ------------ ------------ 0.0 0.0234 2.3456 15.67 0.001 0.1234 2.1234 13.45 0.01 0.3456 1.8765 10.23 0.1 0.8234 1.7234 6.78 0.5 1.2345 1.6987 3.45常见陷阱与注意事项陷阱 1对所有参数统一 weight_decay# 错误包括 bias 和 BN 参数 optimizer torch.optim.AdamW(model.parameters(), lr0.001, weight_decay0.01) # 正确参数分组 no_decay [bias, bn, norm] decay_params [p for n, p in model.named_parameters() if not any(nd in n.lower() for nd in no_decay)] no_decay_params [p for n, p in model.named_parameters() if any(nd in n.lower() for nd in no_decay)] optimizer torch.optim.AdamW([ {params: decay_params, weight_decay: 0.01}, {params: no_decay_params, weight_decay: 0.0} ], lr0.001)陷阱 2Adam 中 weight_decay 过大# Adam 的 weight_decay 与自适应学习率耦合 # 过大的值可能导致训练不收敛 optimizer_adam torch.optim.Adam(model.parameters(), lr0.001, weight_decay0.1) # 太大 # AdamW 可以使用更大的 weight_decay optimizer_adamw torch.optim.AdamW(model.parameters(), lr0.001, weight_decay0.1) # 可以陷阱 3学习率调度器与 weight_decay 的交互# 学习率调度器只影响 lr不影响 weight_decay optimizer torch.optim.AdamW(model.parameters(), lr0.001, weight_decay0.01) scheduler torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max100) # 当 lr 衰减时weight_decay 保持不变 # 如果想让 weight_decay 也衰减需要手动处理陷阱 4AdamW 的 weight_decay 与 SGD 的不等价# SGD 的 weight_decay 是纯粹的 L2 正则化 # AdamW 的 weight_decay 是解耦的权重衰减 # 两者在相同 weight_decay 值下效果不同 # SGD: param param - lr * (grad wd * param) # AdamW: param param - lr * adam_update - lr * wd * param # 从 SGD 迁移到 AdamW 时weight_decay 值可能需要调整陷阱 5混合精度训练中的 weight_decayfrom torch.cuda.amp import GradScaler, autocast # 混合精度训练中weight_decay 仍然正常工作 optimizer torch.optim.AdamW(model.parameters(), lr0.001, weight_decay0.01) scaler GradScaler() with autocast(): output model(input) loss criterion(output, target) scaler.scale(loss).backward() scaler.step(optimizer) # weight_decay 在这里应用 scaler.update() optimizer.zero_grad()总结Adam 和 AdamW 的核心区别在于权重衰减的处理方式选择正确的优化器和配置对模型训练至关重要Adam 的 weight_decay 是 L2 正则化加入梯度后经过自适应学习率缩放正则化效果不均匀。AdamW 的 weight_decay 是解耦权重衰减直接作用于参数不经过自适应学习率正则化效果一致。优先使用 AdamW在大多数场景下AdamW 的解耦权重衰减提供更好的泛化性能。参数分组是必须的bias 和 BatchNorm/LayerNorm 参数不应该做 weight decay。weight_decay 典型值AdamW 用 0.010.1Adam 用 0.0010.01因为耦合效应更强。迁移学习差异化预训练层和新层可以使用不同的学习率和 weight_decay。从 SGD 迁移SGD 到 AdamW 的 weight_decay 值不能直接照搬需要重新调参。混合精度兼容AdamW 与 AMP 混合精度训练完全兼容。理解 Adam 和 AdamW 的区别并根据任务特点选择合适的优化器和参数配置是提升模型训练效果和泛化能力的关键因素。