“Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.”W. Edwards Deming · Out of the Crisis · 1986 · Out of the Crisis (1986), cap. 2 — cele 14 puncte pentru management, punctul 3
A test at intake costs one line; the same error found in the report costs a week.
Data quality has dimensions that can be measured: completeness (are rows or values missing?), uniqueness (duplicate keys?), validity (values from the allowed set, correct types), consistency (do the lines sum to the order total?), timeliness (did it arrive on time?) and accuracy (does it match reality?). Deming's point applied here: tests are code and sit next to the transformations, not in a quarterly audit report. At intake, contracts (lesson 12). On every model: not null, unique, accepted values, relationships between tables, freshness, expected volume. At the output, anomaly monitors on the figures that matter — rows per day, distributions, sums. What you do when a test fails depends on what is at stake: for critical figures, the load stops (fail-closed) and someone looks; for the rest, suspect rows go to quarantine and the remainder passes. Every dataset has an owner and an SLA, and the cause is fixed upstream, not in the report. The report is where you find out; it is not where you fix.
Quality is not only built into the pipeline; the pipeline can learn. Every quarantined row is a signal, not just trash to discard. If the same check keeps failing, the upstream contract is too loose and deserves tightening. Yesterday's test becomes today's rule, and inspection keeps moving earlier until it nearly disappears. Think of a store receiving electronic invoices. The first time, a wrong total is caught by a test and the row goes to quarantine. The third time, you add a contract rule that the total must match the line items, and the supplier gets notified. Next month, the error never reaches the system at all. This changes the data team's role: less detective, more rule architect. The report becomes history, not an alarm. The next step is asking who has the authority to tighten the contract — and what happens when the supplier refuses.
Why it matters The same error costs one line of test at intake and a week of reconciliation when found in the report. The difference is the place, not the effort.