presentsUNAIVERSE
The UN AI timeline

    Learn AI /Model Quality and Evaluation /Class imbalance

    Class imbalance

    Also called: imbalanced data, rare class.

    Detail

    In five seconds

    What it is

    A dataset has this problem when one class appears far more often than another, which can make overall accuracy look strong while a model misses rare cases.

    In plain words

    When the thing you want to detect is rare, a system can seem good by mostly predicting the common case. So you should not rely on one overall score.

    An everyday example

    A drawer contains 100 batteries: 98 are charged and two are flat. A checker that declares every battery charged is right 98% of the time. That sounds excellent until someone needs it to find the flat batteries. Its overall accuracy is 98%, but its recall for flat batteries is zero because it misses both of them.

    Where the story stops working

    A guard can change behaviour immediately after instructions, while a trained model follows patterns learned from data and does not adapt on the spot unless it is retrained or redesigned. Also, fake tickets are intentionally deceptive, but rare classes in data are not necessarily deceptive; they are simply less frequent.

    Remember thisWhen important cases are rare, compare the model with the majority-class baseline and examine recall, precision and the confusion matrix. Overall accuracy may mostly reflect routine cases.

    Understand it

    Understand it

    Why it matters

    Non-specialists often see one number, such as accuracy, and assume the system works. With imbalanced data, that can lead to poor purchasing, unsafe deployment, or false confidence in a model that fails on the cases that matter most. In practice, many important tasks are imbalanced: fraud is rarer than legitimate transactions, serious disease is rarer than no disease, and damaged roads are rarer than intact roads. If you ask the right questions early, you can avoid approving a system that performs well only on the easy majority cases. This helps with quality control, risk management, and transparent reporting.

    Why accuracy can mislead under class imbalance

    Show why a high overall score can hide failure on a rare but important class, and what extra checks are needed.

    1. 01

      Imbalanced data

      Show 1,000 cases: 950 common-class cases and 50 rare-class cases. The rare class is the smaller group people may care most about finding.

      Draw two stacked bars or icon grids labelled Common 950 and Rare 50, with strong colour contrast and text labels on each section.

    2. 02

      Naive prediction

      A model predicts every case as the common class. Because most cases really are common, many predictions look correct at first glance.

      In the left panel, show one output bar labelled Predicted common: 1,000 and Predicted rare: 0, with arrows from both input groups feeding into Predicted common.

    3. 03

      Misleading accuracy

      This model gets 950 out of 1,000 correct, so accuracy is 95%. That sounds good, but it found none of the 50 rare cases.

      Place a prominent badge reading Accuracy 95% in the left panel, and beneath it a warning note: Rare-class recall 0/50.

    4. 04

      Look inside the errors

      Break results into a confusion matrix view: 950 true common, 50 missed rare, 0 found rare. This reveals what the headline score hid.

      In the right panel, draw a simple 2x2 matrix with counts and highlight the cell for missed rare cases in a warning colour.

    5. 05

      Use better checks

      For imbalanced tasks, inspect recall, precision, F1 score, and the confusion matrix, and consider whether the training data or sampling strategy should change.

      Add a checklist under the matrix: Recall, Precision, F1 score, Confusion matrix, Class distribution, each with small icons and short labels.

    Two side-by-side panels compare the same 1,000-case dataset. The left panel shows a naive model predicting only the common class and getting high accuracy. The right panel shows the class counts broken into correct and incorrect predictions for the rare class. Read left to right: first the class distribution, then the tempting score, then the missing rare cases and the better metrics to inspect.

    Hover or tap each panel to reveal the same data expressed as accuracy, recall, precision, and counts; a toggle switches between percentages and raw numbers.

    Describe this without the diagram

    The diagram compares how the same classifier can look good or bad depending on what is measured. Step 1 shows a dataset of 1,000 cases, with 950 common cases and 50 rare cases. Step 2 shows a model that predicts every case as the common class. Step 3 states that this gives 95 percent accuracy because 950 of 1,000 predictions match the common class. Step 4 reveals the hidden problem in a confusion matrix: all 50 rare cases are missed, so rare-class recall is zero. Step 5 lists better checks for this situation: recall, precision, F1 score, confusion matrix, and class distribution.

    With reduced motion: A static two-panel comparison remains on screen at once: the left panel shows the imbalanced dataset and the 95 percent accuracy badge for predicting only the common class; the right panel shows the confusion matrix and checklist of better evaluation measures.

    In classification, a system sorts items into categories such as "damaged road" or "not damaged road". Class imbalance means one category is much more common than the other in the data used to train or test the system. The rarer category is often the one people most need to detect.

    What goes in is a set of labelled examples. If 9,900 satellite images show no road damage and 100 show damage, the system sees many more examples of "no damage" than "damage". During training, many models can reduce their overall error simply by favouring the common class.

    What happens next is that a headline score, especially accuracy, can look strong even if the model performs badly on the rare class. A model that predicts "no damage" for every image would be 99% accurate in that example, but it would find none of the damaged roads. This is why people also check measures such as precision, recall, F1 score, and the confusion matrix.

    What comes out is not just a model but a risk of misleading evaluation. The problem is about the data distribution and the metric choice, not automatically about unfair treatment between social groups. It can contribute to harmful outcomes, but it is different from algorithmic bias, which is about systematic differences in behaviour across people or groups.

    How “The two flat batteries” maps onto the real thing

    In the storyIn the AI system
    Most visitors have valid ticketsThe majority class dominates the dataset
    A few visitors have fake ticketsThe minority or rare class
    The guard waves everyone throughA model predicts only the common class
    The organisers count caught fake tickets separatelyEvaluation uses recall, precision, or a confusion matrix instead of accuracy alone

    A worked example

    An information management officer in a field presence is testing a model that prioritises satellite images for an access map due with the 5pm situation report.

    Starting from: Test set: 10,000 images, including 9,700 with no visible road damage and 300 with damage.

    1. A classifier that labels every image “no damage” would achieve 97% accuracy while finding none of the damaged-road images.
    2. The tested model labels 150 images “damage” and 9,850 “no damage”.
    3. Label checks find 120 true positives, 30 false positives, 9,670 true negatives and 180 false negatives.
    4. Its overall accuracy is 97.9%. For the damage class, precision is 80% and recall is 40%.

    Result: The model improves accuracy by only 0.9 percentage points over the trivial majority-class classifier and still misses 60% of the damaged-road images. The headline figure is therefore unsuitable for deciding which images can be left unreviewed.

    In a United Nations workplace

    A monitoring and evaluation officer in a regional bureau is reviewing an implementing partner’s progress report before the consolidated update goes to the front office at close of business. The report cites strong overall accuracy for a model that flags workplan activities likely to miss their milestones.

    Because most activities are on schedule, predicting “on schedule” every time may already produce impressive accuracy while identifying none of the delayed activities. Before clearance, the officer requests the label distribution, confusion matrix, recall and precision for delayed activities, and comparison with that majority-class baseline.

    Before you try thisDo not upload underlying monitoring records, including personal data or confidential information about partner performance, to a tool that has not been approved for that material. Unauthorised disclosure could harm individuals, prejudice programme decisions or breach confidentiality. Model flags should support review, not determine escalation without human assessment.

    Where you may hear it

    • In a regional bureau results meeting reviewing an implementing partner’s progress report
    • During front office clearance of an update that quotes model accuracy without class-level results
    • At a vendor demonstration to an ICT governance board, when the performance slide omits minority-class recall
    • During an evaluation office’s first reading of a model validation report

    Avoid confusion

    Avoid confusion

    Often confused with

    Common misconceptions

    • If accuracy is high, imbalance cannot be a serious problem.

      High accuracy can come almost entirely from the majority class. When rare cases are important, a model may score well overall while missing most of them. You need to inspect the confusion matrix and class-specific measures such as recall and precision.

    • Class imbalance means the model is automatically biased against protected groups.

      These are different ideas. Class imbalance describes uneven category counts in data, such as many more negatives than positives. Algorithmic bias concerns systematic differences in outcomes across people or groups. The two can interact, but one does not automatically prove the other.

    • The solution is always to collect exactly equal numbers for every class.

      Balanced data can help, but it is not always the only or best fix. Real-world frequencies may genuinely be uneven, and forcing perfect balance can distort evaluation. Other responses include choosing suitable metrics, reweighting classes, resampling carefully, adjusting thresholds, and checking performance on realistic test sets.

    Quick check

    A dataset has 9,900 normal cases and 100 rare cases. A model predicts every case as normal and gets 99% accuracy. What is the best interpretation?

    Go deeper

    Go deeper

    The simple version

    When the thing you want to detect is rare, a system can seem good by mostly predicting the common case. So you should not rely on one overall score.

    The technical version

    In a classification task, class imbalance means the label distribution is highly skewed, such as many more negative than positive examples. Under this condition, optimising or reporting aggregate accuracy can be misleading because the loss and the metric may be dominated by majority-class performance. A trivial classifier that always predicts the majority class can achieve high accuracy while having zero recall for the minority class. Proper evaluation usually includes the confusion matrix and class-sensitive metrics such as precision, recall, F1 score, balanced accuracy, or area-under-precision-recall measures, depending on the task. Mitigation may involve class weighting, over-sampling or under-sampling, threshold adjustment, stratified sampling, and collecting more representative minority-class data. These methods help, but they also have trade-offs, such as overfitting duplicated rare examples or creating unrealistic class proportions in testing, so the test set should reflect the deployment context.

    Reviewed external resources

    Where this came from

    Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

    The one thing to remember

    When important cases are rare, compare the model with the majority-class baseline and examine recall, precision and the confusion matrix. Overall accuracy may mostly reflect routine cases.