The expansion of Delta Lake 4.x in a multi-engine environment

The rivalry between Databricks and Snowflake persists as Apache Polaris provides a unified catalog for both Iceberg and Delta Lake tables. Polaris graduated to a top-level Apache project in February 2026, which allows it to manage access control and read/write interoperability across different formats. I find that this reduces the complexity of managing multiple catalogs for separate table protocols. Databricks addressed similar concerns with catalog-managed table enhancements in Delta Lake 4.1.0, which established the catalog as the source of truth for table state. The introduction of catalog-managed tables allows Databricks to shift commit coordination from the filesystem to the catalog, which aligns Delta Lake with the catalog-oriented approach Iceberg has used since its inception. To handle non-Iceberg formats, Polaris introduced generic table entities that manage Delta Lake and Apache Hudi through new REST APIs. A new Spark plugin enables Spark to communicate with these Polaris APIs to manage both Iceberg and Delta Lake tables in one place.

Delta Lake 4.0 introduced the Variant data type to store semi-structured payloads in typed binary form. This eliminates the need for expensive JSON parsing. This release also brought Type Widening as a table feature, which allows users to change certain column types without rewriting all existing files. Delta Lake 4.1.0 added support for Apache Spark 4.1.0 and introduced server-side planning in preview mode. This specific feature delegates scan planning, such as file discovery and predicate filtering, to an external catalog like Unity Catalog via the Iceberg REST Catalog protocol. You should consider these upgrades if your current workflows rely on outdated Spark runtimes. The new Spark V2 connector in Delta Lake 4.2.0-RC0 supports batch writes and vectorized Deletion Vector reads. I notice that Delta Lake 4.2.0-RC0 also includes UniForm, which publishes Iceberg metadata through Unity Catalog to provide Iceberg readers a consistent view of Delta data. This interoperability feature lets Iceberg-compatible engines read Delta tables without data conversion, though it only works for read-only access. Delta 4.0 brought Delta Connect, which allows DeltaTable APIs to communicate with a remote Spark server via Spark Connect. 4.2.0-RC0 includes atomic REPLACE TABLE operations. Delta Kernel improvements support geometry and geography types and make schema parsing lazy for faster reads.

Iceberg and Delta Lake both use metadata to manage Parquet files, but they handle updates differently. Iceberg uses a multi-layered metadata tree. This tree includes a metadata file that points to a manifest list, which points to manifest files that list the actual data files. Query engines use this to prune irrelevant manifests and data files using stored column statistics without opening a single file. Delta Lake relies on the _delta_log directory. Readers replay the transaction log to find the current table state. Iceberg provides native partition evolution and lets teams change how they partition data without rewriting existing files. Delta Lake uses liquid clustering to organize data based on clustering keys, which replaces traditional Hive-style partitioning.

If you rely on community-driven libraries like delta-rs, you may encounter significant problems because the latest Databricks Runtime protocol versions do not match the protocol support level offered by delta-rs. This mismatch makes many Delta features incompatible with the delta-rs library. Will the community ever bridge this protocol gap for non-JVM users? Delta Lake 4.2.0-RC0 adds streaming reads on Unity Catalog managed tables as an experimental feature. This addition helps support real-time pipelines on catalog-owned data.