Jay Kreps

1 card

1. 01

   Lesson 9 · The log is the source of truth; the database is a cache of the latest value in the log.

   The Log: What every software engineer should know about real-time data's unifying abstraction · 2013

   A periodic SELECT does not see deletions; the transaction log sees every one.

   A log is a list you only append to, in order. It sounds trivial, but it is the abstraction that ties together replication, streaming and the data warehouse. Every relational database has one inside — the transaction log — and every insert, update or delete passes through it before it reaches the tables. Change data capture (CDC) reads exactly that log and emits every change as an event: no periodic queries loading the source, no window lost between two runs and, above all, deletions included — a nightly SELECT never sees the row that vanished at noon. Events land in a distributed log; consumers read with an offset, so they can resume where they stopped or, if needed, replay from the beginning. In the warehouse, bronze receives the raw events and silver applies "latest value per key" (merge) or keeps the full history. The realistic semantics is at-least-once plus idempotency: every event carries a key and a version, and applying it twice yields the same result. Order matters per key, not across keys.

   “A log is perhaps the simplest possible storage abstraction. It is an append-only, totally-ordered sequence of records ordered by time.”

   Open the card →