Git for Data: Zero-Copy Clones of Production Data with Dremio Arctic
At COLLIDE 2023 in Atlanta, Dremio's Jeremiah Morrow demonstrated how branching in the Dremio Arctic Iceberg catalog gives data scientists isolated, zero-copy clones of production data. The demo showed a branch, a tagged change, a merge to main and a one-command rollback.
By The AI Enterprise Magazine Team4 min read
Key takeaways
- Clones without copies. Branches use metadata pointers, so data scientists get production-quality data without duplicating it.
- Isolation protects dashboards. Changes on a branch are not visible to production users until a merge to main.
- Governance is built in. Morrow said every change is tracked in a commit audit, with access controls for sensitive data.
- Merges are atomic. Changes land as all commits or none, which suits ETL and data quality checks.
- Rollback is a single command. Moving the branch header to an earlier commit reverts a bad change.
At COLLIDE Data Conference 2023 in Atlanta, Dremio Product Marketing Director Jeremiah Morrow showed how "Git for data" can give data scientists a zero-copy clone of production data in seconds. In a pre-recorded demo of Dremio Arctic, an Apache Iceberg catalog, he created a branch, changed a table, merged the change and rolled it back.
Why is it hard to give data scientists production data?
Data scientists need production-quality data in an environment where they will not break dashboards for other data consumers, Morrow said. In many companies that means standing up a separate environment with a copy of the production data.
That copy can take hours, days or weeks depending on the data team's backlog, he said. It also becomes a new asset to manage, and the complexity grows across production, development and test environments.
What is Git for data?
Morrow described Git for data as applying software development practices to data. It uses branching, a set of metadata pointers, to create a "no copy clone" of production data. Asked whether it duplicates data, he said it is a snapshot of metadata pointers and the data stays where it lived.
In his example, the main branch serves data consumers. One branch is for data science, where scientists can break things without anyone seeing it. Another is an ETL branch, where new data is ingested and quality-checked before being exposed to users through a merge.
Morrow listed three advantages. Isolation means changes are not exposed until a merge to main. Version control gives the organization one consistent view of production data. For governance, every change is tracked in a commit audit, and access controls limit who sees sensitive data.
How did the demo work?
Morrow set the demo in a Ford dealership in Annapolis, with an Iceberg table of July truck and SUV sales. He wanted to model what July would look like if the Ford Excursion returned. The old way, he said, would be to ask the data team for a copy and a separate environment, since no one knows who uses the sales data in their dashboards.
Instead he ran a create-branch command to make a "Ford test" branch and switched his session to it. He added a row for the Excursion, changed the number of Expeditions sold, and ran a select to show the changes. A query on main showed the data had not changed.
What can teams do with branches, tags and merges?
Morrow tagged the view "July sales with Excursion" so others could find it, rather than relying on a 20-character commit ID. Tags persist after a branch is deleted, he said. Branches are not meant to last forever, other than main, because production data changes and teams will want a fresh branch.
He also toured the Arctic catalog, which shows data, commit history, tags and branches, along with privilege controls and table optimization. He described the optimization as Iceberg compaction, which rewrites small files into larger ones to fix the small-file problem in micro-batching or streaming workloads.
A merge to main exposed the change to production users. Morrow said all changes are made atomically, "all of the commits or none of the commits."
How do you undo a bad change?
Morrow staged a scenario in which a colleague complains that the dashboard broke. Rolling back, he said, takes a single command that moves the branch header to the commit before the change. After the rollback, a select on main showed no changes. "It is like nothing ever happened," he said.
Who is Dremio and where does Arctic fit?
Morrow described Dremio as a SQL query engine and unified access layer that lets teams join data from separate sources and query it as if it were in one location. He said it has two services, Dremio Sonar, the query engine and semantic layer, and Dremio Arctic, an Iceberg catalog that uses Iceberg's snapshot features to deliver Git for data. He framed the pitch against the "cooperative architecture" of multiple data warehouses and data lakes with no easy way to join the data for BI and reporting.
These were Dremio's claims at the time of the 2023 talk, and the session was a vendor demonstration of Dremio's own platform.
Where to learn about data infrastructure next
COLLIDE 2027, the largest AI conference in the Southern U.S. with more than 1,000 enterprise data and AI leaders, takes place Wednesday, October 13, 2027, at the Sandy Springs Performing Arts Center in Atlanta. See COLLIDE. AI Connect also runs 24 free live webinars on its 2027 schedule; see webinars.
Frequently asked questions
What is Git for data?
Morrow described it as applying software development practices such as branching and merging to data, using Iceberg metadata pointers.
Does a zero-copy clone duplicate data?
No. Morrow said it is a snapshot of metadata pointers, and the data stays where it lived.
How do you roll back a bad data change?
Morrow said you move the branch header to the commit before the change with a single command.
Do tags last after a branch is deleted?
Yes, according to Morrow, a tag on a view persists even after the branch is thrown away.
This article is based on remarks by Jeremiah Morrow, Product Marketing Director, Dremio Arctic and Iceberg, Dremio at COLLIDE Data + AI Conference 2023 on October 3, 2023. More coverage: COLLIDE AI Conference.
Filed under: Data & AI Infrastructure, Observability & Data Quality
