Author
Michael Armbrust, Ali Ghodsi, Reynold Xin, Matei Zaharia
1 reading card from 1 book · 2021.
1 card
Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics · 2021
Bronze is the evidence, silver is the cleaned truth, gold is the answer.
The data lake promised cheap storage for anything: Parquet files on object storage, no enforced schema, no transactions. It often delivered a swamp: nobody knows what is clean, two jobs write over the same directory, a read catches half of a write. The classic warehouse has transactions, schema and speed, but in a closed format, as a copy of the data. The lakehouse puts an open table format between them (Delta, Iceberg, Hudi): a metadata log over the Parquet files that brings ACID transactions, "time travel" to an earlier version, schema evolution and, above all, several engines — SQL, Spark, an ML framework — reading the same files without copies. The medallion architecture organises the layers: bronze (data exactly as it arrived, never modified), silver (cleaned, deduplicated, correctly typed), gold (aggregated, modelled for consumption — usually as a star). The golden rule is that bronze is the evidence: anything in silver and gold must be rebuildable from it.
“This paper argues that the data warehouse architecture as we know it today will wither in the coming years and be replaced by a new architectural pattern, the Lakehouse, which will (i) be based on open direct-access data formats, such as Apache Parquet, (ii) have first-class support for machine learning and data science, and (iii) offer state-of-the-art performance.”