Open Data Lakehouse With Trino and Apache Iceberg: 2023 Workshop
In a hands-on workshop at the Data Science Innovators Conference 2023, Starburst's Matt Fuller and Tom Nats explained how Trino and Apache Iceberg combine into an open lakehouse. A live demo showed merges, updates, deletes and snapshot time travel.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- Trino brings the compute. A coordinator plans queries and workers process data in parallel across lake and non-lake sources.
- Iceberg is a metadata layer. Snapshots enable time travel, rollback, concurrent inserts and schema evolution on open file formats.
- Open stack avoids lock-in. Object storage, open file formats, an open table format and an open engine make up the stack.
- Lake data can be changed. The demo ran merge, update and delete statements, relevant to deletion requests such as GDPR.
- Time travel is built in. Querying an earlier snapshot showed the data before an update.
In a hands-on workshop at the Data Science Innovators Conference 2023, "Building an Open Data Lake House Using Trino and Apache Iceberg," two Starburst executives explained how the two open source technologies fit together and demonstrated the approach live. Matt Fuller, co-founder of Starburst, covered Trino. Tom Nats, VP of Product and Director of Customer Solutions at Starburst, covered Apache Iceberg. The session combined a short overview with a SQL walkthrough using Starburst Galaxy and Tabular.
What is Trino and what is it used for?
Fuller described Trino, formerly known as Presto, as a fast distributed SQL query engine for large datasets spread across heterogeneous sources. He listed four main use cases: interactive analytics from SQL or BI tools, high-performance analytics on object storage, federated access to many data sources, and batch ETL processing.
A cluster has a coordinator and any number of workers. "The coordinator is the brain, the workers are really the muscle," Fuller said. The coordinator parses, plans and schedules queries, and the workers process pieces of the data in parallel. A connector model lets Trino query sources outside the lake, such as MySQL, Postgres, Elasticsearch and MongoDB, so teams do not always have to replicate data into the lake first. Fuller summarized Trino as "bring your own data" and Iceberg as "bring your own compute."
Why did Apache Iceberg replace the Hive approach?
Nats traced the history. Early Hadoop data sat in raw files, then columnar formats such as Parquet and ORC sped up analytics, but updates, inserts and managing huge file counts remained hard. The Hive metastore became a bottleneck, he said, and Hive partitioning depended on column order, forcing users to write filters on partition columns or trigger massive scans. Directory-style partitioning and file listing were also a poor and costly fit for cloud object storage.
Iceberg, created at Netflix by Ryan Blue and Daniel Weeks, according to Nats, is a metadata layer over existing file formats. Each change creates a new snapshot, which enables time travel and rollback. Optimistic concurrency lets inserts proceed without blocking, and columns are tracked by ID so schema changes are simpler. File-level statistics let engines skip files, and partitioning can be changed later. Optimize and vacuum commands compact small files and clean up old snapshots.
What does an open lakehouse stack look like?
Nats said he came from 20-plus years of data warehousing and wanted ACID transactions on the lake. Warehouses offered fine-grained access control, data quality and concurrency, while lakes brought separated storage and compute, scale on commodity hardware, open formats and flexibility with semi-structured data. The lakehouse combines both.
The stack he described has four layers: object storage, open file formats, a table format like Iceberg and a compute engine like Trino. Because the middle layers are open source and not owned by a company, he said, teams avoid lock-in, such as data stored in a vendor-controlled account that charges every time it is touched.
How do you build a lakehouse with Trino and Iceberg?
Fuller worked in a browser with Starburst Galaxy, Starburst's managed Trino service, and Tabular, a managed Iceberg service on S3. He attached a Tabular catalog to a Trino cluster, created a partitioned table and loaded New York taxi trip data with an insert-select statement. Iceberg partitioned the data by month automatically, even though some records had impossible dates.
Nats then explained a merge statement that deduplicates source rows with a row-number function and inserts only rows that are not already present in the target. Merges are idempotent, he said, and adding date predicates cut the rows read in the demo. Fuller loaded several months, about 7.8 million rows for one of them.
What can you do with updates, time travel and deletes?
The demo showed that data can be changed directly in the lake. An update rounded trip distances to the nearest mile, and a history table let Fuller query the earlier snapshot to see the original values. "There's nothing magic we're doing here," he said, noting Spark could read the same Iceberg snapshots.
Fuller also deleted rows for one pickup location, a use case he tied to GDPR requests, which he described as previously very difficult or impossible on traditional data lakes. Then he queried an older snapshot and found about 13,520 rows that had matched. The result, he said, is "the best of both worlds."
In Q&A, Fuller said Trino applies across industries and that Starburst has many health services customers. At the time, he said, Starburst offered a free Galaxy cluster plus $500 in credits for paid tiers.
Where can you learn more about enterprise data infrastructure?
Teams building open data platforms for AI can compare notes at COLLIDE 2027 on Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. Organizers call it the largest AI conference in the Southern U.S. Details are at /events/collide/; registration is coming soon.
Frequently asked questions
What is a data lakehouse?
Nats described it as combining the benefits of a data warehouse, such as ACID transactions, with those of a data lake, such as separated storage and compute and open formats.
What does Apache Iceberg do?
It is a metadata layer over Parquet, ORC and Avro files that adds snapshots, time travel, schema evolution, concurrency control and file-level statistics.
Can you update or delete data in a data lake?
With Iceberg, yes. The demo ran update and delete statements and then queried an earlier snapshot to see the original data.
What is Trino?
A fast distributed SQL query engine, formerly Presto, that can query lakes and other data sources through connectors.
This article is based on remarks by Matt Fuller, Co-founder, Starburst; Tom Nats, VP of Product & Director of Customer Solutions, Starburst at Data Science Innovators Conference 2023 on April 12, 2023. More coverage: More AI Connect Events.
Filed under: Data & AI Infrastructure
