Dealing with Class Imbalance: Resampling, Weighting, and Ensembles

Class imbalance is a very common problem in practical applications—real-world data is never as high-quality as public datasets, and the number of positive and negative samples is often wildly lopsided. There are three main categories of methods for handling class imbalance: Resampling, Weight balancing, and Ensembles.

Resampling

Resampling goes in two directions: undersampling the class with many samples, or oversampling the class with few samples.

Resampling the data
Undersampling (left) removes majority-class samples; oversampling (right) adds minority-class samples.

A classic undersampling algorithm is Tomek Links: find the majority-class samples that are closest to minority-class samples and remove them, making the decision boundary clearer. But this trick doesn’t always work—it can also erase subtle decision boundaries and backfire.

Undersampling with Tomek Links
Tomek Links: remove majority-class samples hugging the minority class to clean up the boundary.

A classic oversampling algorithm is SMOTE: first compute the K nearest neighbors of some minority-class sample xix_i, randomly pick one neighbor x^i\hat{x}_i from those K, then randomly generate a new sample on the line segment between xix_i and x^i\hat{x}_i; repeat until enough new samples have been generated.

Oversampling with SMOTE
SMOTE: synthesize new samples on the line segment between a minority-class sample and its neighbors.

Weight balancing

Another category of methods assigns different weights to samples of different classes; the representative example is Focal Loss:

FL(pt)=−αt (1−pt)γlog⁡(pt)\text{FL}(p_t) = -\alpha_t\,(1-p_t)^{\gamma}\log(p_t)

Here α\alpha handles the positive/negative imbalance—assigning different weights to the losses of positive and negative samples; γ\gamma handles the easy/hard imbalance—the larger γ\gamma is, the smaller the loss for high-confidence “easy samples,” so the loss function focuses more on the hard-to-classify “hard samples.”

Focal Loss makes the loss focus more on hard samples
The larger γ is, the more the loss for easy samples (high confidence) is suppressed, focusing the loss on hard samples.

Ensembles

There is also ensemble learning. The common Bagging (bootstrap aggregating) samples different subsets of the data to train multiple classifiers, then votes on the results and takes the one with the most votes as the final result—a combination of multiple weak classifiers is often more robust than a single one.

The Bagging algorithm
Bagging: train multiple classifiers on different subsets of the data, then aggregate by voting.

References

  • He, Haibo, Ma, Yunqian. Imbalanced Learning: Foundations, Algorithms, and Applications. Wiley, 2013.
  • Chawla, Nitesh V., et al. SMOTE: Synthetic Minority Over-sampling Technique. JAIR, 2002.
  • Lin, Tsung-Yi, et al. Focal Loss for Dense Object Detection. ICCV, 2017.
  • Technology