SE-ResNet vs ResNet: PyTorch Code Implementation Differences

SE-ResNet and ResNet are both widely used neural network architectures in deep learning. SE-ResNet enhances ResNet by incorporating the SE (Squeeze-and-Excitation) module, resulting in improved model performance.

In PyTorch code, the primary distinction between SE-ResNet and ResNet lies in the network structure definition. SE-ResNet introduces an SEBlock class, encapsulating the Squeeze and Excitation operations. This class can be integrated into the ResNet architecture to implement SE-ResNet.

Specifically, SE-ResNet incorporates the SEBlock after each basic block (BasicBlock) and bottleneck block (Bottleneck). As an illustration, consider the implementation of BasicBlock for SE-ResNet:

class BasicBlock(nn.Module):
    expansion = 1

    def __init__(self, in_planes, planes, stride=1):
        super(BasicBlock, self).__init__()
        self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(planes)
        self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(planes)
        self.se = SEBlock(planes)  # Add SEBlock

        self.shortcut = nn.Sequential()
        if stride != 1 or in_planes != self.expansion*planes:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_planes, self.expansion*planes, kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(self.expansion*planes)
            )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = self.se(out)  # Apply SEBlock
        out += self.shortcut(x)
        out = F.relu(out)
        return out

As shown in the code, an SEBlock is added within the class initialization and applied in the forward function.

The implementation of SE-ResNet for the Bottleneck block follows a similar approach.

In essence, the difference between SE-ResNet and ResNet lies solely in their network structure definitions. Consequently, PyTorch implementation simply involves adding the SEBlock to the existing ResNet architecture.