Book
Designing Data-Intensive Applications
by Martin Kleppmann · 2017 · 1 reading card
1 card
- 01
Lesson 12 · Data outlives code: every schema change has to be readable by the old code and the new.
Designing Data-Intensive Applications · 2017
Add, backfill, switch, only then drop — never change in place.
An application is replaced in five minutes; its data sits for five years. Hence two compatibilities: backward (new code reads old data) and forward (old code reads new data — because nothing is ever updated all at once). The rules that keep both are few and strict: add optional columns, never reuse a name with a different meaning, never change a type in place. A type change takes four steps: add the new column, fill it (backfill), move the readers, only then drop the old one. Formats with a schema (Avro, Protobuf, Parquet) make the rules checkable; free-form JSON leaves them to luck. The data contract is the same idea, raised to the level of teams: the producer and the warehouse agree on schema, semantics, SLA, owner and change policy, and the agreement is checked automatically on every publish. A change that breaks the contract is published as a new version, alongside the old. The reason is that the worst defects are silent: a column renamed at the source arrives as NULL, the report drops by 30% and nobody gets an error. The contract catches it at the boundary, where it is cheap.
“Thus, five-year-old data will still be there in the original encoding, unless you have explicitly rewritten it since then. This observation is sometimes summed up as data outlives code.”