“A log is perhaps the simplest possible storage abstraction. It is an append-only, totally-ordered sequence of records ordered by time.”Jay Kreps · The Log: What every software engineer should know about real-time data's unifying abstraction · 2013 · The Log: What every software engineer should know about real-time data's unifying abstraction, LinkedIn Engineering (2013) — Part One: What Is a Log?
A periodic SELECT does not see deletions; the transaction log sees every one.
A log is a list you only append to, in order. It sounds trivial, but it is the abstraction that ties together replication, streaming and the data warehouse. Every relational database has one inside — the transaction log — and every insert, update or delete passes through it before it reaches the tables. Change data capture (CDC) reads exactly that log and emits every change as an event: no periodic queries loading the source, no window lost between two runs and, above all, deletions included — a nightly SELECT never sees the row that vanished at noon. Events land in a distributed log; consumers read with an offset, so they can resume where they stopped or, if needed, replay from the beginning. In the warehouse, bronze receives the raw events and silver applies "latest value per key" (merge) or keeps the full history. The realistic semantics is at-least-once plus idempotency: every event carries a key and a version, and applying it twice yields the same result. Order matters per key, not across keys.
If the log is the source of truth, an unusual consequence follows: the tables are no longer sacred. They become a materialized view — a result computed from events, rebuildable from scratch at any time. This changes how you think about mistakes. A botched migration or faulty logic is no longer a catastrophe, but a restart: drop the table, fix the transformation, replay the log. Just as you can rewrite a recipe and cook again from the same ingredients, having lost nothing. The database stops being the home of truth and becomes a cache of the latest value, reconstructible from the journal. But if the log can rebuild anything, an obvious question arises: how long must that log be kept?
The previous passage ended with a question: how long must the log be kept? The answer is surprising: not forever. Since any table can be rebuilt from any point, a snapshot plus the events after it is enough. Many systems go further and keep only the latest value per key in the log, through a process called compaction. Think of your phone's contact list: you don't keep every old version of every number, just the current state, yet you can still call anyone. Full history becomes a choice, not an obligation — you keep it only if you want to answer questions about the past. But if the log is this central, a practical problem appears: what happens when several services want to write to it at the same time?
Why it matters Periodic extraction with SELECT misses deletions and loads the source; a warehouse fed that way slowly loses accuracy without raising a single error.