UniCL: A Highly Efficient Baseline for Cross-Supervised Learning with Noisy Labels

Numerous approaches have been proposed to address the 'learning-with-noisy-labels' (LNL) problem using supervised learning. These approaches include the development of robust loss functions, estimation of transition matrices, and utilization of deep neural networks (DNNs) to enhance classification robustness. However, some existing methods require a small set of clean samples for training, which limits their applicability to practical problems. Additionally, it has been observed that DNNs tend to learn simple patterns before fitting noisy labels, leading to the notion that samples with lower loss are more likely to be clean. Building upon this observation, a two-component Beta Mixture Model (BMM) was fitted in a previous study to distinguish between clean and noisy samples. However, this approach resulted in undesirable flat distributions under asymmetric noise. MentorNet introduced curriculum learning to identify correct labels for student network training, while Co-teaching and Co-teaching+ trained two neural networks simultaneously and shared each other's samples with small loss for joint updates. In comparison, DivideMix combined two networks using implicit and explicit methods and employed a Gaussian Mixture Model (GMM) to model the loss distribution of samples, dividing the dataset into a labeled set (mostly clean) and an unlabeled set (mostly noisy). An improved MixMatch method was then used for semi-supervised training. Although these supervised learning-based LNL methods have made significant progress in distinguishing clean and noisy samples, they do not leverage unlabeled samples and are not directly applicable to semi-supervised learning tasks. In contrast, UniCL proposes a highly efficient baseline for cross-supervised learning scenarios that utilizes both labeled samples (correct and incorrect) and unlabeled samples.