Databricks Unity Catalog governance and Iceberg REST Catalog limits

Unity Catalog reached a critical inflection point in May 2026 when managed Apache Iceberg tables and the Iceberg REST Catalog hit general availability. This move closes the format gap that previously locked users into Delta Lake, allowing external engines like Spark and Flink to read and write directly to Databricks storage. I view this as the single most important shift for governance because it decouples data portability from vendor lock-in. However, the governance layer remains tightly coupled to the Databricks runtime, creating a persistent blind spot for hybrid stacks. The catalog now acts as a neutral hub for interoperability, yet its enforcement mechanisms do not travel with the data bytes.

Where Governance Stops at the Boundary

The core limitation is that fine-grained policies do not cross the platform boundary. If you apply row filters or column masks to a table, external clients cannot access those protected assets via the Iceberg REST API. Databricks introduced OpenSharing in June 2026 to handle reads outside the account, but providers still cannot share tables carrying table-level row filters or column masks. This means the rule protecting your data stays inside Databricks while the data itself moves out. You can use the cross-engine attribute-based access control feature, currently in Beta, to let an external Iceberg client read a filtered table. In that scenario, Unity Catalog evaluates the policy during scan planning and returns a narrowed scan plan. It is a clever workaround, but it only supports reads and requires specific client versions and the EXTERNAL USE SCHEMA privilege. For organizations running multi-region estates, the situation gets worse because each region requires its own metastore. Access across regions relies on sharing rather than a unified catalog graph, which fragments lineage and complicates compliance audits.

AI Integration and Operational Costs

Generative AI models are now treated as governed data assets within the catalog. When you build a Retrieval-Augmented Generation chatbot, it connects to a Vector Index built from governed "Gold" tables. Because security is embedded in the table, the AI inherits the governance rules. If a user lacks permission to see supplier contracts, the RAG model will not retrieve chunks from that document. This prevents data leaks by design. The Model Registry ensures that production models are frozen and versioned, so a junior data scientist cannot overwrite a live asset. On the cost side, Governance Hub entered Beta in July 2026 and went public in August. It provides a spend explorer that tracks costs by product, workspace, or tag. While it offers read-only recommendations, it does not apply changes automatically. Teams needing deeper attribution must look at third-party tools like LakeSentry, which charges $499 per month for up to five users. The native tooling gives you visibility, but you still need manual intervention to optimize compute usage.

Does the current enforcement model scale to fully decentralized data mesh architectures?