A model that performs suspiciously well

A logistic regression predicting loan default, trained on four features: income, credit score, loan amount, and the number of collections calls made on the account.

model.fit(X_train, y_train)
accuracy_score(y_test, model.predict(X_test))

1.000

Perfect accuracy on held-out test data. Not 99%, not 95%, every single prediction correct. In real machine learning work, this result should read as an alarm, not a win. A genuinely hard prediction problem, forecasting who will default before it happens, does not resolve to perfect accuracy from four ordinary features. Something else is going on.

The feature doing the actual work

collections_calls, the number of collections calls made on an account, is a real, ordinary-sounding column. It's also a feature that only takes a non-zero value after an account has already started defaulting. Collections calls don't predict a default. They're a direct, downstream consequence of one that's already underway, sometimes already logged in the same data pipeline before the "did this loan default" label even gets finalized.

Included as a predictor, the model isn't learning to forecast default risk from income and credit history. It's learning "if collections_calls is greater than zero, predict default," which is trivially, perfectly true by construction, because collections calls only ever happen once a default has already occurred. The model looks excellent because it's been handed the answer disguised as an input.

Removing it and re-measuring

evaluate(["income", "credit_score", "loan_amount"], "WITHOUT the leaky feature")

accuracy = 0.854

85.4%, a real, plausible number for this kind of problem, using only information that would genuinely be available before a loan is issued. The eleven-and-a-half point drop from the leaky version isn't the model getting worse. It's the model finally being evaluated on the actual task, predicting default from information available in advance, instead of a much easier task in disguise, recognizing default from information that already describes it.

Why this is easy to miss

Nothing about collections_calls looks suspicious sitting in a feature list next to income and credit_score. It's a real, legitimate column that genuinely exists in the data. The leakage isn't a data quality problem, the values aren't wrong, and it isn't a coding bug, the training code ran correctly. It's a timing problem: the feature's value depends on information that, in a real production setting, wouldn't exist yet at the moment a prediction actually needs to be made. That distinction, "does this feature reflect the state of the world before or after the outcome," doesn't show up by inspecting a dataframe. It shows up by asking, for every single feature, exactly when its value becomes known relative to the event being predicted.

The takeaway

Suspiciously good validation performance is a real, checkable warning sign, not a result to celebrate without first asking why. Any feature worth including in a model deserves one specific question: would this value actually be available at the moment a real prediction needs to be made, before the outcome, or does it only exist because the outcome already happened. A perfect accuracy score is far more often evidence of the second case than proof of a genuinely excellent model.