Understand it in plain language

Overfitting happens when a model adapts too closely to training-specific patterns and performs less well on relevant new examples. Underfitting means the model fails to capture useful structure even in training. A train–validation gap is a clue, not a complete diagnosis: leakage, distribution shift, and inconsistent preprocessing also need checking. Regularization constrains fitting; its strength must be chosen using suitable validation.

How the AI approach works

A model can learn the signal we need and also accidental details of the training sample. Overfitting emphasizes those sample-specific details at the expense of relevant new cases. Underfitting occurs when the model or optimization fails to capture useful structure even in training. Diagnose both with appropriate losses, baselines, learning curves, and error inspection.

Regularization changes the fitting tradeoff. L2 penalizes large weights; L1 can encourage sparse coefficients. Early stopping selects a checkpoint before validation quality deteriorates. Dropout randomly removes some activations during training in suitable networks. Data augmentation creates task-preserving variations, such as small image transformations when the label should remain unchanged. These methods are not interchangeable, and invalid augmentation can teach the wrong task.

Choose regularization strength and model capacity on validation data or suitable cross-validation. All learned preprocessing belongs inside each training fold. Split by person, object, or time when those dependencies matter. A random split of duplicate records may look excellent while measuring memorization rather than useful generalization.

A performance drop can also come from distribution shift: the deployment population, camera, vocabulary, or policy has changed. Stronger regularization does not automatically fix that. Compare the data-generating conditions, check leakage, and inspect important slices before deciding whether to simplify, collect data, adapt, or retrain.

Break the idea into steps

1. Compare training and validation behaviour across model complexity.
2. Check splitting and preprocessing before blaming only the model.
3. Try simpler models, more representative data, or regularization, then evaluate again.

Worked example: connect the method to the task

A leaf classifier scores 99% on training and 70% on validation. First verify that labels and preprocessing match and the split is realistic. If the training–validation gap persists, test reduced capacity, augmentation, weight decay, or more representative data. Compare changes on the same validation protocol.

Now suppose both scores are good, but a new phone camera performs poorly. That is evidence to investigate a domain shift, not simply proof of overfitting. Collect representative phone images and evaluate them separately before choosing a remedy.

Apply the idea: check your reasoning

Before you move on

• Explain the main idea without repeating the technical label.
• Identify the approach, its assumptions, and the evidence needed to trust the result.
• Complete the challenge, then compare your reasoning with the answer.

Sources & a deeper next step

The refresh uses primary papers and official documentation. These optional links are in English; you can understand this lesson without reading them first. Product versions and availability can change.

• scikit-learn: common pitfalls and leakage ↗
• scikit-learn: model evaluation ↗