3D ResNet-101 PyTorch 复现指南从 2D 到 3D 卷积的 5 个关键改动点1. 理解 3D 卷积的核心差异在视频分析和医学影像处理中传统 2D 卷积神经网络无法捕捉时序信息。3D 卷积通过在空间维度H×W基础上增加时间维度 T形成真正的三维卷积核K×K×K。这种结构能同时提取空间和时间特征但对计算资源的需求呈指数级增长。关键参数对比表参数类型2D 卷积3D 卷积输入维度(C, H, W)(C, D, H, W)卷积核形状(K, K)(K, K, K)输出特征图(C, H, W)(C, D, H, W)计算复杂度O(K²·C·C·H·W)O(K³·C·C·D·H·W)提示在 PyTorch 中3D 卷积使用nn.Conv3d其参数顺序为 (in_channels, out_channels, kernel_size, stride, padding)2. 基础结构改造卷积与池化层2.1 卷积层升级将原始 ResNet-101 的所有nn.Conv2d替换为nn.Conv3d特别注意第一个卷积层的改动# 原始 2D 版本 self.conv1 nn.Conv2d(3, 64, kernel_size7, stride2, padding3) # 3D 改造版本 self.conv1 nn.Conv3d(3, 64, kernel_size(3,7,7), stride(1,2,2), padding(1,3,3))这里的时间维度 stride 设为 1避免过早压缩时序信息。2.2 池化层适配最大池化层需要同步升级# 2D 池化 nn.MaxPool2d(kernel_size3, stride2, padding1) # 3D 池化 nn.MaxPool3d(kernel_size(1,3,3), stride(1,2,2), padding(0,1,1))3. 残差块的时空改造3.1 Bottleneck 结构调整ResNet-101 的核心是 Bottleneck 块其 3D 改造需要特别注意维度匹配class Bottleneck3D(nn.Module): expansion 4 def __init__(self, in_planes, planes, stride1, downsampleNone): super().__init__() self.conv1 nn.Conv3d(in_planes, planes, kernel_size1, biasFalse) self.bn1 nn.BatchNorm3d(planes) self.conv2 nn.Conv3d(planes, planes, kernel_size3, stridestride, padding1, biasFalse) self.bn2 nn.BatchNorm3d(planes) self.conv3 nn.Conv3d(planes, planes*self.expansion, kernel_size1, biasFalse) self.bn3 nn.BatchNorm3d(planes*self.expansion) self.relu nn.ReLU(inplaceTrue) self.downsample downsample3.2 下采样处理当需要进行维度匹配时下采样模块也需要 3D 化downsample nn.Sequential( nn.Conv3d(self.in_planes, planes * block.expansion, kernel_size1, stridestride, biasFalse), nn.BatchNorm3d(planes * block.expansion) )4. 批归一化与初始化调整4.1 3D 批归一化所有nn.BatchNorm2d需替换为nn.BatchNorm3dself.bn1 nn.BatchNorm3d(64) # 替换原 bn14.2 参数初始化3D 卷积建议使用 Kaiming 初始化for m in self.modules(): if isinstance(m, nn.Conv3d): nn.init.kaiming_normal_(m.weight, modefan_out) elif isinstance(m, nn.BatchNorm3d): nn.init.constant_(m.weight, 1) nn.init.constant_(m.bias, 0)5. 输出层与全局池化5.1 时空池化策略3D 特征图需要特殊处理全局池化last_duration int(math.ceil(sample_duration / 16)) # 时间维度下采样 last_size int(math.ceil(sample_size / 32)) # 空间维度下采样 self.avgpool nn.AvgPool3d((last_duration, last_size, last_size))5.2 全连接层调整输出特征需要展平为 (batch_size, features)x x.view(x.size(0), -1) # 保持 batch 维度 x self.fc(x)完整模型实现以下是整合后的 3D ResNet-101 核心代码class ResNet3D(nn.Module): def __init__(self, block, layers, sample_size112, sample_duration16, num_classes400): super().__init__() self.in_planes 64 self.conv1 nn.Conv3d(3, 64, kernel_size(3,7,7), stride(1,2,2), padding(1,3,3), biasFalse) self.bn1 nn.BatchNorm3d(64) self.relu nn.ReLU(inplaceTrue) self.maxpool nn.MaxPool3d(kernel_size(1,3,3), stride(1,2,2), padding(0,1,1)) self.layer1 self._make_layer(block, 64, layers[0]) self.layer2 self._make_layer(block, 128, layers[1], stride2) self.layer3 self._make_layer(block, 256, layers[2], stride2) self.layer4 self._make_layer(block, 512, layers[3], stride2) last_duration math.ceil(sample_duration / 16) last_size math.ceil(sample_size / 32) self.avgpool nn.AvgPool3d((last_duration, last_size, last_size)) self.fc nn.Linear(512 * block.expansion, num_classes) for m in self.modules(): if isinstance(m, nn.Conv3d): nn.init.kaiming_normal_(m.weight, modefan_out) elif isinstance(m, nn.BatchNorm3d): nn.init.constant_(m.weight, 1) nn.init.constant_(m.bias, 0) def _make_layer(self, block, planes, blocks, stride1): downsample None if stride ! 1 or self.in_planes ! planes * block.expansion: downsample nn.Sequential( nn.Conv3d(self.in_planes, planes * block.expansion, kernel_size1, stridestride, biasFalse), nn.BatchNorm3d(planes * block.expansion) ) layers [] layers.append(block(self.in_planes, planes, stride, downsample)) self.in_planes planes * block.expansion for _ in range(1, blocks): layers.append(block(self.in_planes, planes)) return nn.Sequential(*layers)实际应用建议输入预处理视频数据建议采样固定帧数如16/32帧医学影像需保持切片间距一致显存优化使用梯度检查点gradient checkpointing尝试混合精度训练减小初始输入分辨率迁移学习2D 预训练权重可通过重复扩展至3D时间维度权重初始化为均值或中心高斯分布def inflate_2d_to_3d(conv2d, time_dim3, centerTrue): weight_2d conv2d.weight.data if center: weight_3d torch.zeros(*((time_dim,) weight_2d.shape)) middle time_dim // 2 weight_3d[middle] weight_2d.clone() else: weight_3d weight_2d.unsqueeze(2).repeat(1,1,time_dim,1,1) / time_dim return weight_3d
)
