PyTorch nn.CrossEntropyLoss 实战3种权重设置与标签平滑对比附代码交叉熵损失函数是深度学习分类任务中最核心的组件之一。PyTorch框架中的nn.CrossEntropyLoss不仅实现了标准的交叉熵计算还提供了weight和label_smoothing两个关键参数来解决实际训练中的类别不平衡和过拟合问题。本文将深入解析这些高级参数的实现原理并通过完整的代码对比展示不同配置下的训练效果差异。1. 交叉熵损失函数核心原理回顾在进入高级参数实战之前我们需要明确几个关键概念Logits模型最后一层线性层的原始输出值未经过softmaxSoftmax将logits转换为概率分布的函数$p_i \frac{e^{z_i}}{\sum_j e^{z_j}}$交叉熵衡量预测概率分布与真实分布的差异$H(p,q) -\sum p_i \log q_i$PyTorch的nn.CrossEntropyLoss实际上同时完成了softmax和交叉熵计算其数学表达式为loss(x, class) -log(exp(x[class]) / sum(exp(x[i]))) -x[class] log(sum(exp(x[i])))这种实现方式在数值稳定性上优于分开计算softmax和交叉熵。2. 权重参数的三种设置方式类别不平衡是现实数据中的常见问题。weight参数允许我们为不同类别分配不同的损失权重其核心公式变为$$ loss(x, class) weight[class] \times (-x[class] \log(\sum_j e^{x[j]})) $$2.1 手动指定权重最直接的方式是根据领域知识手动设置各类别权重。例如在医疗诊断场景中罕见病类别的权重通常更高import torch import torch.nn as nn # 假设3分类问题类别0、1、2的权重分别为1.0, 2.5, 1.8 weights torch.tensor([1.0, 2.5, 1.8]) criterion nn.CrossEntropyLoss(weightweights) # 示例计算 logits torch.tensor([[2.0, 1.0, 0.5], [0.5, 2.0, 1.5]]) # 2个样本的logits targets torch.tensor([0, 1]) # 真实类别 loss criterion(logits, targets) print(f加权损失值: {loss.item():.4f})2.2 自动计算类别权重更科学的方式是根据训练集统计自动计算权重常用方法是反比于类别频率def calculate_weights(train_labels): class_counts torch.bincount(train_labels) num_classes len(class_counts) weights 1. / (class_counts.float() / class_counts.float().sum()) return weights / weights.sum() * num_classes # 归一化 # 假设训练集中3个类别的样本数分别为100, 30, 20 train_labels torch.cat([ torch.zeros(100), torch.ones(30), torch.full((20,), 2) ]).long() weights calculate_weights(train_labels) print(f自动计算权重: {weights})2.3 类别平衡权重在极端不平衡场景下可以使用更激进的平衡策略def balanced_weights(train_labels, beta0.9): class_counts torch.bincount(train_labels) effective_num 1.0 - torch.pow(beta, class_counts) weights (1.0 - beta) / effective_num return weights / weights.sum() * len(weights) weights balanced_weights(train_labels) print(f平衡权重: {weights})三种权重计算方式对比表方法类型优点缺点适用场景手动指定灵活可控依赖领域知识类别重要性明确自动计算数据驱动对极端少数类敏感一般不平衡数据平衡权重强调少数类可能过度补偿极端不平衡数据3. 标签平滑技术解析与实现标签平滑(Label Smoothing)是一种正则化技术通过软化one-hot编码来防止模型对标签的过度自信。其数学表达为$$ y (1 - \alpha) \times y \alpha / K $$其中$K$是类别数$\alpha$是平滑系数。3.1 基础实现PyTorch 2.0直接支持label_smoothing参数criterion nn.CrossEntropyLoss(label_smoothing0.1) # 等效手动实现 def smooth_one_hot(labels, alpha, num_classes): smoothed torch.full((labels.size(0), num_classes), alpha / (num_classes-1)) smoothed.scatter_(1, labels.unsqueeze(1), 1.0 - alpha) return smoothed smoothed_targets smooth_one_hot(targets, 0.1, 3)3.2 与权重的组合使用标签平滑可以与权重参数协同工作criterion nn.CrossEntropyLoss( weighttorch.tensor([1.0, 2.0, 1.5]), label_smoothing0.1 )3.3 不同平滑系数的效果对比我们通过一个简单的实验观察不同$\alpha$值的影响import matplotlib.pyplot as plt def test_smoothing_effects(logits, targets): alphas [0, 0.05, 0.1, 0.2, 0.3] losses [] for alpha in alphas: criterion nn.CrossEntropyLoss(label_smoothingalpha) loss criterion(logits, targets) losses.append(loss.item()) plt.plot(alphas, losses, markero) plt.xlabel(Smoothing Alpha) plt.ylabel(Loss Value) plt.title(Label Smoothing Effect on Loss) plt.grid(True) plt.show() test_smoothing_effects(logits, targets)典型情况下损失值会随$\alpha$增大而先降后升最优值通常在0.05-0.2之间。4. 完整训练案例对比现在我们实现一个完整的图像分类训练流程对比不同参数配置的效果。使用CIFAR-10数据集人为制造类别不平衡import torchvision from torch.utils.data import WeightedRandomSampler # 创建不平衡的CIFAR-10数据集 transform torchvision.transforms.Compose([ torchvision.transforms.ToTensor(), torchvision.transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5)) ]) dataset torchvision.datasets.CIFAR10(root./data, trainTrue, downloadTrue, transformtransform) # 人为制造不平衡各类别样本数指数递减 class_counts [5000] [5000 // (2**i) for i in range(1,10)] print(各类别样本数:, class_counts) # 创建带权重的DataLoader weights 1. / torch.tensor(class_counts, dtypetorch.float) samples_weights weights[torch.tensor(dataset.targets)] sampler WeightedRandomSampler(samples_weights, len(samples_weights)) train_loader torch.utils.data.DataLoader(dataset, batch_size64, samplersampler)定义测试函数def train_and_evaluate(criterion, epochs10): model torchvision.models.resnet18(num_classes10) optimizer torch.optim.Adam(model.parameters(), lr1e-3) train_losses [] accuracies [] for epoch in range(epochs): model.train() total_loss 0 for inputs, labels in train_loader: optimizer.zero_grad() outputs model(inputs) loss criterion(outputs, labels) loss.backward() optimizer.step() total_loss loss.item() # 评估 model.eval() correct 0 total 0 with torch.no_grad(): for inputs, labels in test_loader: outputs model(inputs) _, predicted torch.max(outputs.data, 1) total labels.size(0) correct (predicted labels).sum().item() acc correct / total train_losses.append(total_loss/len(train_loader)) accuracies.append(acc) print(fEpoch {epoch1}: Loss{train_losses[-1]:.4f}, Acc{acc:.4f}) return train_losses, accuracies配置对比实验# 实验配置 configs [ {name: Baseline, criterion: nn.CrossEntropyLoss()}, {name: Weighted, criterion: nn.CrossEntropyLoss(weightweights)}, {name: Smoothing(0.1), criterion: nn.CrossEntropyLoss(label_smoothing0.1)}, {name: WeightedSmoothing, criterion: nn.CrossEntropyLoss(weightweights, label_smoothing0.1)} ] # 运行实验 results {} for config in configs: print(f\nRunning {config[name]}...) losses, accs train_and_evaluate(config[criterion]) results[config[name]] {loss: losses, acc: accs}可视化结果plt.figure(figsize(12,5)) plt.subplot(1,2,1) for name, res in results.items(): plt.plot(res[loss], labelname) plt.xlabel(Epoch) plt.ylabel(Training Loss) plt.legend() plt.subplot(1,2,2) for name, res in results.items(): plt.plot(res[acc], labelname) plt.xlabel(Epoch) plt.ylabel(Test Accuracy) plt.legend() plt.tight_layout() plt.show()典型实验结果会显示加权方法能显著提升少数类的识别率标签平滑有助于提高最终准确率组合使用可能获得最佳平衡5. 高级技巧与注意事项5.1 权重归一化当使用较大权重值时建议进行归一化以避免梯度爆炸weights weights / weights.sum() * len(weights)5.2 动态权重调整在训练过程中可以动态调整权重def dynamic_weight_adjustment(epoch, max_epoch): base_weights torch.tensor([...]) # 基础权重 # 随训练进程逐渐降低权重影响 factor 1.0 - min(epoch / max_epoch, 0.8) return base_weights * factor5.3 标签平滑与MixUp的结合MixUp数据增强与标签平滑有协同效应def mixup_data(x, y, alpha0.2): lam np.random.beta(alpha, alpha) batch_size x.size(0) index torch.randperm(batch_size) mixed_x lam * x (1 - lam) * x[index] y_a, y_b y, y[index] return mixed_x, y_a, y_b, lam # 在训练循环中 inputs, targets_a, targets_b, lam mixup_data(inputs, targets) outputs model(inputs) loss lam * criterion(outputs, targets_a) (1 - lam) * criterion(outputs, targets_b)5.4 类别权重与学习率对于高权重类别可以相应降低学习率optimizer torch.optim.SGD([ {params: model.base.parameters()}, {params: model.classifier.parameters(), lr: 1e-3 * weights.mean()} ], lr1e-3)
