Delta Lake, Apache Iceberg, or Apache Hudi: Choosing an Open Table Format

Delta Lake, Apache Iceberg, or Apache Hudi: Choosing an Open Table Format

By Rplus Analytics · Data Engineering · Open Source

The single most consequential decision in a modern data lakehouse is which open table format you write into. It determines what engines can read your data, how schema changes propagate, what your time-travel and rollback story looks like, and — most importantly — how easily you can change your mind later. (PRJ8500)

Three formats now dominate: Delta Lake, Apache Iceberg, and Apache Hudi. All three solve roughly the same problem — bringing ACID transactions, schema evolution, and consistent reads to Parquet files in object storage — but they make different trade-offs along the way. This post is a practical comparison aimed at architects making the choice for the first time, or revisiting it because the original choice no longer fits.

What all three give you

Before the differences, the common ground. All three formats provide:

• ACID transactions on top of object storage, with snapshot isolation
• Schema evolution — adding, renaming, and (with caveats) dropping columns without rewriting data
• Time travel — querying historical versions of a table as of a timestamp or version
• Hidden partitioning logic that doesn't require consumers to know the partition columns
• Compaction and optimisation mechanisms to keep small-file problems in check
• Open file format underneath — Parquet, readable by any tool that speaks Parquet

If your only question is "do I need an open table format?", the answer is yes, and any of the three is dramatically better than writing raw Parquet directories.

Delta Lake

Origin and stewardship. Originated at Databricks, donated to the Linux Foundation in 2022. Still has the strongest gravitational pull toward Databricks, but the open-source format itself is genuinely open and increasingly well-supported by other engines.

Strengths.

• Tightest integration with Spark, including Photon and Delta Live Tables
• Mature handling of streaming writes via structured streaming
• Excellent support for change data feeds (CDC out of the table itself)
• Best-in-class performance characteristics when paired with Databricks runtime
• Liquid Clustering offers a more flexible alternative to traditional partitioning

Where it fits well. Platforms standardising on Spark as the primary processing engine, especially those that already use or plan to use Databricks. Streaming-heavy workloads benefit from the structured streaming integration. (DSA)

Where to watch out. Read support outside Spark is improving but historically lagged Iceberg. Some advanced features are easier on Databricks than on open-source Spark.

Apache Iceberg

Origin and stewardship. Originated at Netflix, now an Apache project with a genuinely diverse contributor base — Netflix, Apple, Snowflake, AWS, Google, and others. The most "neutral" of the three in terms of vendor backing.

Strengths.

• Broadest engine support — Spark, Trino, Flink, Snowflake, BigQuery, Dremio, Athena, Starburst all read it natively
• Hidden partitioning and partition evolution are first-class concepts
• Snapshot semantics are very clean, which makes branching and tagging workflows straightforward
• Excellent for organisations expecting to read the same tables from multiple engines

Where it fits well. Platforms with heterogeneous engine usage — analysts on Trino, ML on Spark, ad-hoc queries from Snowflake or BigQuery — all reading the same tables. Anywhere "engine-agnosticism" is a higher priority than maximum performance on any single engine.

Where to watch out. Streaming write story is less mature than Delta's. Compaction and small-file management require more deliberate operational attention.

Apache Hudi

Origin and stewardship. Originated at Uber, where it was built specifically to solve incremental data processing problems at very large scale. Apache project.

Strengths.

• Best-in-class for incremental upserts and merge-on-read patterns
• Designed from day one for near-real-time ingestion of changing data
• Strong record-level indexing makes point updates and lookups efficient
• Battle-tested at extreme scale (Uber-scale and beyond)

Where it fits well. Workloads dominated by frequent upserts of small changes against large tables — CDC from operational databases, near-real-time event aggregation, slowly-changing-dimension patterns at scale. The original design point shows.

Where to watch out. Smaller community and engine ecosystem than Delta or Iceberg. Steeper learning curve for teams without specific incremental-processing requirements.

A practical decision framework

Three questions usually settle it.

1. What's your primary processing engine?

• Spark-first, especially with Databricks → Delta Lake is the path of least resistance
• Multiple engines reading the same tables → Iceberg
• Streaming ingestion with heavy upserts → Hudi (or Delta with care)

2. What's your existing skill base?

The open-source ecosystem around Delta and Iceberg is now large enough that hiring isn't a serious differentiator. Hudi expertise is rarer. Don't pick a format your team can't operate.

3. How locked-in does the format leave you?

All three are open. But the practical portability differs. Iceberg has the broadest cross-engine read support today, which makes it the safest choice if engine flexibility is your top concern. Delta is catching up quickly. Hudi is the most engine-specific in practice.

What we recommend, in plain terms

For most UK public sector lakehouse builds, we lean toward Delta Lake when the platform is Databricks-centric, and Iceberg when it isn't — particularly when Snowflake, Trino, or multiple BI engines are in the picture.

For workloads dominated by high-frequency upserts against very large tables — typically operational reporting on top of CDC streams — Hudi remains worth a serious look despite the smaller ecosystem.

The wrong answer is to delay the decision and write raw Parquet "for now." That always becomes a migration project later, and migrations of this layer are painful regardless of which format you eventually pick.

---

Rplus Analytics helps UK public sector teams design and operate open-format data lakehouses. If you'd like a sounding board on the format choice for your platform, get in touch.