Unity Catalog was designed for the open lakehouse. Beforehand, knowledge groups have been caught in silos, usually pressured to duplicate knowledge throughout platforms simply to make use of the instruments they wished. Each new platform or instrument meant copying datasets, rebuilding entry insurance policies from scratch, and maintaining the whole lot in sync. The outcome was elevated prices from redundant storage, insurance policies that drifted out of sync, and fragmented knowledge entry and discovery.
Once we open sourced Unity Catalog and launched Open APIs, we broke down the silos that beforehand stored prospects locked-in. Enterprises may lastly preserve one copy of knowledge, use any compute engine, and govern the whole lot from one place. The UC ecosystem has thrived since. Right now, hundreds of consumers use Unity Catalog to control and entry Delta Lake and Apache Iceberg tables, with dozens of integrations within the rising Unity Catalog ecosystem — from Apache Spark and Trino to DuckDB and Confluent Tableflow.
Exterior Entry to Managed Tables, Now in Beta
UC managed tables are the place openness meets efficiency. These superior tables use Predictive Optimization and Liquid Clustering to mechanically tune knowledge layouts, run compaction and vacuuming, and preserve statistics contemporary — delivering as much as 20× quicker queries and 50% decrease storage prices, whereas staying totally accessible by way of open APIs.
Now in Beta, exterior engines, comparable to Apache Spark, Flink, and DuckDB, can create and write to UC managed Delta tables with centralized governance and computerized optimizations.
With the Beta, exterior engines can:
- Create managed tables — Get up new UC managed tables straight from an exterior engine.
- Batch learn and write — Learn and write to managed tables with full transactional security.
- Stream to and from managed tables — Use managed tables as each a streaming supply and sink, enabling end-to-end real-time pipelines on Apache Spark.
As a result of each operation flows by way of UC managed tables constructed on catalog commits, you get serialized commits that stop log corruption and full auditability of each learn and write. Predictive Optimization continues to run seamlessly, even on tables accessed by exterior engines. Catalog commits additionally lay the groundwork for options like multi-statement, multi-table transactions that require a centralized commit coordinator.
The thriving UC ecosystem is continuous to develop as engines broaden help for exterior entry to managed tables. Delta Kernel — the open supply Java and Rust library for studying, writing, and committing to Delta tables — abstracts the low-level protocol particulars so connector builders can give attention to UC integration, not Delta implementation. Apache Spark, Delta Flink, and DuckDB have all leveraged Delta Kernel to help exterior writes to UC managed tables and combine with catalog-managed commits, and the ecosystem continues to develop. By dealing with the low-level protocol complexity, Delta Kernel makes it easy for any engine to combine with Unity Catalog which contributes to a rising ecosystem of connectors.
Safe Exterior Entry Made Doable By Credential Merchandising
For an exterior engine to entry knowledge in UC, it wants a safe method to authenticate and get scoped entry to cloud storage with out requiring broad, static permissions or credentials tied to a selected account. Unity Catalog handles this by way of credential merchandising, which is now usually obtainable (GA): UC points short-lived, scoped credentials to exterior engines on demand, with entry insurance policies enforced centrally.
Hundreds of consumers have used UC Open APIs and two additions make it production-ready at enterprise scale. Exterior engines can now authenticate to UC utilizing machine-to-machine (M2M) OAuth, assembly enterprise safety necessities with out counting on personalised entry tokens (PATs), that are per-user, long-lived, and arduous to rotate. And credentials are refreshed mechanically by engines by way of the UC credential merchandising APIs, so pipelines that run for hours full reliably with out tokens expiring mid-job.
With credential merchandising, enterprises can learn, write, and create managed and exterior tables in Unity Catalog from any suitable engine or instrument. These credentials are short-lived, scoped to the requested useful resource, and ruled by UC privileges. This implies your platform group retains full management over which principals can entry knowledge externally and what they’ll do with it.
With Unity Catalog’s Open APIs, we have empowered our groups to make use of their most popular instruments whereas sustaining governance and knowledge consistency. We will leverage the advantages of managed tables inside a really interoperable knowledge and AI platform that works throughout a number of compute engines.— Sudipta Das, Director of Enterprise Information Operations at PepsiCo
Credential Merchandising for Volumes
Credential merchandising extends not solely to tables but in addition unstructured knowledge. Quantity credential merchandising is now in Public Preview, so exterior shoppers can request momentary, scoped credentials to entry pictures, PDFs, and movies saved in volumes with Unity Catalog governance. The identical entry management mannequin, audit path, and scoped credentials apply whether or not you are querying a desk or processing a uncooked video file externally.
What’s Subsequent?
We’re persevering with to put money into making exterior entry extra succesful. Credential merchandising right this moment governs coarse-grained entry controls for exterior engines. We have additionally developed performance to implement attribute-based entry controls (ABAC) for exterior reads, which makes governance fine-grained. This makes it attainable to implement row and column degree ABAC insurance policies when UC managed tables are learn rom exterior engines.
Get Began Right now
To get began with credential merchandising, see our documentation. To make use of the Beta of exterior entry to managed Delta tables:
- Enroll in “Exterior Entry to Unity Catalog Managed Delta Desk” within the Databricks preview portal (see Handle Databricks previews)
- Allow exterior knowledge entry in your metastore and grant EXTERNAL_USE_SCHEMA on the schema containing the tables you wish to entry.
- Create a brand new UC managed desk. To maneuver present knowledge, see the migration information for changing exterior tables to managed.
- Use Delta-Spark 4.2 with Unity Catalog 0.4.1 to create, learn, and write to managed tables from exterior compute. See the exterior entry documentation.
Be part of us at Information and AI Summit 2026
Information and AI Summit 2026 is nearly right here! Be part of us June 15-18, 2026 on the Moscone Heart in San Francisco, California to find out how main organizations are utilizing Unity Catalog to control knowledge and AI throughout engines. Register right this moment to get a primary take a look at what’s coming subsequent for open, unified governance.
