Book
Fundamentals of Data Engineering
by Joe Reis, Matt Housley · 2022 · 1 reading card
1 card
Fundamentals of Data Engineering · 2022
A rerun for the same window must give the same result — that is idempotency.
ETL transforms data in a separate tool before loading it; ELT loads it raw and transforms it with SQL inside the warehouse, where compute is cheap, the SQL lives in git and lineage can be read from the queries. The new order makes extraction thin (copy faithfully) and transformation rich (model, test, document). Orchestration is the graph of tasks: dependencies, schedule, retries, alerts, a view of what ran and what did not. The parameter of a run is the time window, never "now" — otherwise yesterday's run can no longer be repeated. Idempotency is the property that makes everything repairable: rerunning for the same window produces exactly the same rows. You get it by deleting and reloading the partition, or by merging on a key — never by blind appends, which double the data on the second run. With it, a backfill is just a loop over old windows, and a 3 a.m. failure is fixed with a rerun, not an investigation. Incremental loads carry a watermark and a window for late-arriving data; small files get compacted; everything is tested in CI, like any code.
“Data engineering is the development, implementation, and maintenance of systems and processes that take in raw data and produce high-quality, consistent information that supports downstream use cases, such as analysis and machine learning.”