“Thus, five-year-old data will still be there in the original encoding, unless you have explicitly rewritten it since then. This observation is sometimes summed up as data outlives code.”Martin Kleppmann · Designing Data-Intensive Applications · 2017 · Designing Data-Intensive Applications (2017), cap. 4 Encoding and Evolution — Dataflow Through Databases
Add, backfill, switch, only then drop — never change in place.
An application is replaced in five minutes; its data sits for five years. Hence two compatibilities: backward (new code reads old data) and forward (old code reads new data — because nothing is ever updated all at once). The rules that keep both are few and strict: add optional columns, never reuse a name with a different meaning, never change a type in place. A type change takes four steps: add the new column, fill it (backfill), move the readers, only then drop the old one. Formats with a schema (Avro, Protobuf, Parquet) make the rules checkable; free-form JSON leaves them to luck. The data contract is the same idea, raised to the level of teams: the producer and the warehouse agree on schema, semantics, SLA, owner and change policy, and the agreement is checked automatically on every publish. A change that breaks the contract is published as a new version, alongside the old. The reason is that the worst defects are silent: a column renamed at the source arrives as NULL, the report drops by 30% and nobody gets an error. The contract catches it at the boundary, where it is cheap.
Why it matters An unannounced schema change does not raise an error; it produces a wrong report, discovered a month later. The contract moves the discovery to the boundary and to the minute.