Understanding Process Rank in Transformer Models: Values & Usage

The process rank in a transformer model refers to the number of parallel processes used during training or inference. Typical values for process rank in transformer models can range from 1 to several hundred, depending on the size of the model and the available computing resources. For example, a small transformer model may use a process rank of 1 or 2, while a large transformer model may use a process rank of 128 or more. The process rank is often set based on the available memory and number of GPUs or CPUs that can be used for parallel processing.