Process rank in transformer model what if this value set to -1

In a transformer model, the process rank is typically used to assign a unique identifier to each process in a distributed computing environment. This identifier is used to determine the order in which processes should execute and communicate with each other during training or inference.

If the process rank is set to -1, it usually means that the process is not part of a distributed computing environment and is running on a single machine. In this case, the process rank is not relevant and can be ignored.

However, if the process rank is set to -1 in a distributed computing environment, it may indicate an error or misconfiguration. Each process in the environment should have a unique process rank, typically starting from 0 and incrementing by 1 for each subsequent process. If the process rank is set to -1, it may cause communication errors or other issues during training or inference