An ordinary aggregation
df.groupby("region")["revenue"].sum()
About as standard a pandas operation as exists: group rows by a category, sum a numeric column within each group. Nothing about this line looks like it could silently discard real data.
Real source data, with a few real gaps
df = pd.DataFrame({
"region": ["West", "East", np.nan, "West", "East", np.nan],
"revenue": [100, 200, 300, 150, 250, 400],
})
Two of the six rows have a missing region, a completely realistic situation: a record synced from a system that didn't always capture that field, a manual entry someone left blank, a join that didn't find a match for every row. The revenue values are all real and all present.
What the groupby actually returns
grouped = df.groupby("region")["revenue"].sum()
print(grouped)
print("total from groupby result:", grouped.sum())
print("total from raw column:", df["revenue"].sum())
region
East 450
West 250
Name: revenue, dtype: int64
total from groupby result: 700
total from raw column: 1400
Half. The grouped total is exactly half the real total, because the two rows with a missing region, worth 300 and 400, a real 700 in revenue, simply don't appear anywhere in the output. Not as an "Unknown" group, not as a warning, not as a NaN row that at least signals something was excluded. They're just gone, and the only way to notice is to independently check the grouped sum against the raw column's sum, the way it's checked above.
Why
groupby() excludes rows with a missing value in the grouping column by default. This is real, documented, and deliberate behavior on pandas' part, not a bug, but it's also exactly the kind of default that's easy to never read about until it costs something. A NaN in a grouping column doesn't get its own group unless explicitly asked for; it gets left out of the result entirely.
The fix
grouped = df.groupby("region", dropna=False)["revenue"].sum()
region
East 450
West 250
NaN 700
Name: revenue, dtype: int64
total: 1400 (matches the raw column exactly)
One keyword argument, and the missing-region rows show up as their own real, visible group instead of vanishing. Whether the right long-term fix is dropna=False, or going back and finding out why region is missing for those rows in the first place, depends on the data. Either way, the number is now visible enough to make that decision instead of silently absent from the total.
The takeaway
Any groupby() on a column that can realistically contain a missing value is worth a direct check: does the grouped total match the raw column's total? If it doesn't, the gap is dropna's default doing exactly what it's documented to do, quietly, on rows that had nothing wrong with their actual data, only with the category they were supposed to be grouped into.
Comments
Loading comments...