The split everyone learns first

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=1
)

The standard first move in almost every machine learning tutorial: shuffle the data, hold out 20% at random, train on the rest. For data where each row is genuinely independent of the others, this is exactly right. For a time series, a daily revenue number, a sensor reading, anything where "day 500" and "day 501" are related, it silently measures a different, easier problem than the one that actually matters.

A real, growing series

Daily revenue for a business with a genuine growth trend, some weekly seasonality, and real noise. Forecast it with two features, the day index and day of week, using the same model, evaluated two different ways.

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=1, shuffle=True
)
model.fit(X_train, y_train)
mean_absolute_error(y_test, model.predict(X_test))

9.47

An average error of under 10, on revenue numbers running from roughly 80 to 900. That looks like a genuinely strong forecasting model.

The same model, evaluated the way it will actually be used

split_point = int(len(df) * 0.8)
X_train, X_test = X.iloc[:split_point], X.iloc[split_point:]
y_train, y_test = y.iloc[:split_point], y.iloc[split_point:]

model.fit(X_train, y_train)
mean_absolute_error(y_test, model.predict(X_test))

85.11

Same model, same data, same 80/20 proportion. The only change is that training now uses the first 80% of days in order, and testing uses the final 20%, the actual shape of a real forecasting task: train on the past, predict the future. The error is nine times larger.

Why the random split hid this

print("training revenue range:", y_train.min(), "-", y_train.max())
print("test revenue range:", y_test.min(), "-", y_test.max())

training revenue range: 80 - 752
test revenue range (never seen at training time): 718 - 919

With the chronological split, the test period's real revenue is genuinely higher than anything the model saw during training, because the business kept growing. A tree-based model like this one can't extrapolate past the range of values it was trained on; it predicts by averaging similar training examples, and it has no similar examples for "revenue this high," because chronologically, that level of revenue hadn't happened yet at training time.

The random split hid this completely. Shuffling scatters both early, low-revenue days and late, high-revenue days into both the training set and the test set, so every test point ends up with genuinely similar training neighbors on both sides of it. The model isn't forecasting in that setup, it's interpolating between values it already has direct examples of, which is a real, much easier task that a random split makes indistinguishable from genuine forecasting.

The takeaway

train_test_split's default shuffle is correct for data where row order carries no information. For a time series, it's a real, specific way to overestimate performance, and the gap isn't small: nine times worse here, and the direction of the error is entirely predictable, validation looking better than production ever will, once real time has moved the data somewhere the model was never trained on. A chronological split, train strictly on the past, test strictly on the future, is the one that actually resembles how the model will be used once it's deployed, and it's worth comparing both directly, the way it's compared here, rather than trusting whichever one a library's default happens to produce.