A completely reasonable thing to want

An inventory table, stored as Parquet, one row updated:

df.write.mode("overwrite").parquet(path)              # initial write

updated_row = spark.createDataFrame(
    [(1, "warehouse-a", 95)], ["item_id", "location", "stock"]
)
updated_row.write.mode("append").parquet(path)          # "update" item 1

Reasonable intent: item 1's stock count changed from 120 to 95, write the new value. append is the mode that exists for adding new data without destroying what's already there.

What's actually in the table afterward

spark.read.parquet(path).orderBy("item_id").show()

item_id | location    | stock
--------+-------------+------
      1 | warehouse-a | 120
      1 | warehouse-a |  95
      2 | warehouse-b |  75

real row count: 3

Item 1 now appears twice, once with the old stock count, once with the new one. Plain Parquet has no concept of "update this specific row." It's a real, plain file format: a write either creates new files (overwrite, which replaces everything) or adds more files (append, which adds rows, never modifies or removes existing ones). There is no operation in between, no real way to say "this one row changed, leave the rest alone."

The same scenario, on Delta Lake

Delta Lake is a real, open-source table format that adds a transaction log on top of the exact same Parquet files, and with it, real operations Parquet alone can't express:

from delta.tables import DeltaTable

dt = DeltaTable.forPath(spark, path)
updates = spark.createDataFrame(
    [(1, "warehouse-a", 95), (3, "warehouse-c", 40)],
    ["item_id", "location", "stock"]
)
dt.alias("t").merge(
    updates.alias("s"), "t.item_id = s.item_id"
).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute()

item_id | location    | stock
--------+-------------+------
      1 | warehouse-a |   95
      2 | warehouse-b |   75
      3 | warehouse-c |   40

Item 1 genuinely updated in place, item 3 genuinely inserted as new, in one real, atomic operation. No duplicate row, because MERGE is a real operation Delta Lake actually implements, matching source rows to target rows and deciding update-versus-insert per row, exactly the operation plain Parquet has no way to express at all.

The part that's easy to miss: nothing was thrown away

spark.read.format("delta").option("versionAsOf", 0).load(path).orderBy("item_id").show()

item_id | location    | stock
--------+-------------+------
      1 | warehouse-a |  120
      2 | warehouse-b |   75

The table exactly as it existed before the MERGE, still queryable, on demand. Delta Lake doesn't overwrite old data files when a row changes, it writes new files and records, in a real transaction log, which files make up which version of the table. MERGE looks like an in-place update from the outside, and the old data is still genuinely there underneath, recoverable, the same mechanism that also makes the whole operation atomic, a MERGE that fails partway through never leaves the table in a half-updated state.

dt.history().select("version", "operation").show()

version | operation
--------+----------
      1 | MERGE
      0 | WRITE

A real, complete audit trail of every change, for free, as a direct consequence of how the format is built.

The takeaway

Plain Parquet is a real, efficient way to store data, but it's a file format, not a database, no update, no delete, no transaction guarantee, only "write more files" and "replace everything." Delta Lake sits on top of the same physical Parquet files and adds exactly the operations a real, changing dataset needs: atomic updates, a genuine audit trail, and the ability to query an earlier version directly. The duplicate row above is what happens when a dataset that changes over time is stored in a format that was never designed to represent change at all.