Understanding Process Rank in Transformer Models: What Does -1 Mean?

In a transformer model, 'process rank' is typically used to assign a unique identifier to each process within a distributed computing environment. This identifier dictates the order in which processes execute and communicate during training or inference.

Setting the process rank to -1 usually signifies that the process is not part of a distributed setup and operates on a single machine. In such cases, process rank is inconsequential and can be disregarded.

However, encountering a process rank of -1 within a distributed computing environment may signal an error or misconfiguration. Each process in the distributed environment should possess a distinct process rank, generally starting from 0 and incrementing by 1 for each subsequent process. A process rank of -1 can lead to communication errors or other problems during training or inference.