add_param_group

add_param_group() is a method in PyTorch that allows users to add a new parameter group to an optimizer. A parameter group is a dictionary that describes the parameters, hyperparameters and optimization options for a specific set of model parameters. It is useful when we want to use different learning rates, weight decay or momentum for different layers or sets of parameters in a neural network.

The add_param_group() method takes in a dictionary as input which should contain the following keys:

• params: a list of parameter tensors to be optimized.
• lr: the learning rate for the parameter group.
• weight_decay: the weight decay for the parameter group.
• momentum: the momentum factor for the parameter group.
• dampening: the dampening for momentum correction for the parameter group.
• nesterov: whether to use Nesterov momentum for the parameter group.

This method can be called multiple times to add different parameter groups to the optimizer.

Example:

import torch.optim as optim

optimizer = optim.SGD(model.parameters(), lr=0.1)

# add a new parameter group with a different learning rate and weight decay
optimizer.add_param_group({'params': model.fc.parameters(), 'lr': 0.01, 'weight_decay': 0.001})

# add another parameter group with a different momentum
optimizer.add_param_group({'params': model.conv.parameters(), 'lr': 0.1, 'momentum': 0.9})
``