Leakage: the bug that makes every metric lie

In short: leakage is when information from the test set, or from the answer itself, sneaks into training. The three kinds I check for every time are the same person on both sides of the split, preprocessing fit on all the data, and a feature that already knows the label. Each one has a cheap test that catches it.

Leakage is the worst bug in machine learning because it doesn't crash. It makes everything look better. Accuracy goes up, the curve looks beautiful, the slide deck writes itself. Then the model meets new data and falls apart, usually in front of the people you most wanted to impress.

In health and voice data it's especially common, because the data has structure that a random split ignores: patients, visits, sites, devices. These are the three patterns I check for every time, and the usual way each one gets found.

Leak one: the same person on both sides

You have several recordings per patient. You shuffle all recordings and split 80/20. Now most patients have some recordings in train and some in test. The model doesn't need to learn anything about the disease. It learns to recognise voices, and it remembers which voices belonged to which label.

The same thing happens with hospital data split by visit instead of by patient, with sensor data split by window instead of by wearer, and with images split by slice instead of by scan. Anything where one source produces many rows.

How it usually gets found: the model is tested on people from a new site, or a new batch, and the score drops hard. Or someone re-runs the evaluation with a grouped split and the number falls well below what was reported. The fix is to split by the thing that would be new in real life. In medicine that's almost always the patient.

Leak two: preprocessing that saw the test set

This one is quieter. You scale features, impute missing values, select the top features, or oversample the minority class, and you do it on the whole dataset before splitting. Each step uses statistics from rows that end up in the test set. Feature selection is the worst offender: picking the features that correlate best with the label, using all labels, and then testing on some of those same labels is grading your own homework.

How it usually gets found: someone moves the preprocessing inside the cross-validation loop, usually during a refactor, and the score drops. The fix is structural. Put every step that learns anything into one pipeline, and give that pipeline only the training fold.

from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold, cross_val_score

# Every step that learns from data lives inside the pipeline,
# so it only ever sees the training fold.
model = make_pipeline(
    SimpleImputer(strategy="median"),
    StandardScaler(),
    SelectKBest(k=20),
    LogisticRegression(max_iter=1000),
)

# Split by patient, not by row.
cv = GroupKFold(n_splits=5)
scores = cross_val_score(model, X, y, groups=patient_id, cv=cv, scoring="average_precision")

# A cheap guard you can keep in your test suite.
for train_idx, test_idx in cv.split(X, y, groups=patient_id):
    assert not set(patient_id[train_idx]) & set(patient_id[test_idx])

Leak three: a feature that knows the answer

The last kind is the hardest to see, because nothing is wrong with the split. One of the inputs simply carries the label. A medication field that only exists after diagnosis. A free-text note written by a clinician who already knew the outcome. In voice data, the recording setup itself: if patients were recorded in a clinic and healthy controls at home, the model can learn the room, the microphone or the background hum, and never listen to the voice at all.

How it usually gets found: someone looks at what the model relies on. A SHAP or permutation importance plot shows a top feature that makes no clinical sense, like a timestamp, a device ID or the length of silence before speech. Another good test is to train a model on metadata alone: site, device, date, file length. If that model does well, your real model might be doing the same thing.

A ten-minute leak check

Before trusting any number, I run through this list. None of it takes long.

1. Name the unit that would be new in production. Patient, speaker, site, device, week. Split by that, and assert that no group appears on both sides.
2. Find every step that calls fit. Each one belongs inside the pipeline, after the split. That includes resampling.
3. Sort features by importance and read the top ten out loud. If you can't explain one to a clinician, investigate it.
4. Train on metadata only. If it beats chance by much, the recording conditions differ between classes.
5. Hold out one whole site or one time period if you have more than one. It's the closest thing to the real world you'll get before deployment.

Pick the model you're proudest of right now and do step one today. If the number survives a grouped split, you'll trust it more. If it doesn't, you found out before your users did.