Guide, read 10 Nov 2026
Table catalogs for lakehouse engines: Glue, Unity Catalog, Polaris and Nessie
All guidesOpen the Compute Bill
What an engine needs from a table catalog
In Snowflake's description of the Iceberg design, a catalog forms the first layer of the table specification and must store the current metadata pointer for each table, mapping a table name to the location of its current metadata file. Apache Iceberg publishes a REST catalog specification so that engines can talk to different catalogs through one interface.
For a Spark batch team the practical questions are which catalogs the engine can write through, who is allowed to commit, and what other engines can read the same tables. Data catalogs in the governance sense, with search and lineage, are a separate product category that this site does not cover.
AWS Glue Data Catalog
The Glue Data Catalog is a central metadata repository that integrates with Athena, Lake Formation, EMR and SageMaker. Its Iceberg REST endpoint supports API operations from the Iceberg REST specification, supports table specifications v1 and v2 (v2 by default), and for us-east-1 is the endpoint glue.us-east-1.amazonaws.com/iceberg (an API address, not a web page).
Athena documents that its Iceberg support works only with tables in the Glue Data Catalog and uses Glue optimistic locking. Glue also hosts table optimizers for compaction, snapshot retention and orphan file deletion. Compaction is off by default for new tables and is enabled per table with an IAM role, at $0.44 per DPU-hour in us-east-1.
Amazon S3 Tables
S3 Tables stores tables in the Iceberg format in table buckets, and AWS exposes an Iceberg REST endpoint for engines outside the AWS analytics integration. Maintenance is built in and on by default for every table: compaction with a 512 MB default target and snapshot management with a 120-hour default maximum snapshot age. The service fails snapshot management for a table that has user-defined tags or branches, a restriction to check if you use Iceberg branches.
Unity Catalog open source
The Unity Catalog project describes itself as open governance for tables, multimodal data, functions and models, licensed Apache 2.0 and an LF AI & Data project, with Delta Lake tables and an Iceberg REST Catalog API. Its 0.7 announcement is titled as adding native Apache Iceberg tables. The GitHub repository was created June 13, 2024 and last pushed November 10, 2026.
Databricks documents the commercial Unity Catalog separately, and several Databricks features, such as predictive optimization and serverless access to external data, depend on it. The open-source project and the Databricks product are different deployments.
Apache Polaris
Apache Polaris is described as an open-source, fully featured catalog for Apache Iceberg that implements Iceberg's REST API. Its site names Apache Doris, Apache Flink, Apache Spark, Dremio OSS, StarRocks and Trino as engines that work with it. The repository is Apache-2.0 licensed, was created May 29, 2024 and was last pushed November 10, 2026.
Two vendors build on it. Dremio's Open Catalog is built on Apache Polaris. Snowflake Open Catalog is a managed service for Apache Polaris, but new customers who had not already created an account are directed to Snowflake Horizon Catalog.
Project Nessie
Project Nessie describes itself as a transactional catalog for data lakes with Git-like semantics, offering cross-table transactions and working with Iceberg tables. The main Nessie server serves the repository through the Iceberg REST API and its own native REST API. The documentation read on 2026-11-10 is for version 0.110.0, and the repository, Apache-2.0 licensed, was created April 9, 2020.
Catalogs that vendors host
Vendors in alphabetical order. Dremio's Open Catalog is built on Apache Polaris, supports the Iceberg REST Catalog API, and lists the Glue Data Catalog as a connectable catalog; for Iceberg tables in it, Dremio runs automatic optimization on a dedicated engine it configures. Onehouse documents Glue catalog sync as part of migrating from EMR. Snowflake supports an external catalog integration such as AWS Glue for Iceberg tables, and its own Open Catalog is the Polaris service described above.
Hosting a catalog adds a dependency that table files alone never carry. If the catalog is vendor-hosted, ask what export or replication the vendor documents before relying on it as the only copy of table pointers.
What to check
- Which engines on your runtime can commit through the catalog, not only read.
- Whether the catalog runs maintenance for you (Glue and S3 Tables do for Iceberg) or leaves it to your jobs.
- Whether tags and branches are used, given the S3 Tables restriction above.
- Who hosts it: AWS-managed, vendor-managed or a service you run.
- How a second engine authenticates against the REST endpoint.
Sources
- Apache Iceberg REST catalog specification, read 10 Nov 2026
- Snowflake Open Catalog overview, read 10 Nov 2026
- AWS Glue Data Catalog, read 10 Nov 2026
- AWS Glue Iceberg REST endpoint, read 10 Nov 2026
- AWS Glue compaction optimization, read 10 Nov 2026
- Amazon S3 Tables overview, read 10 Nov 2026
- Amazon S3 Tables maintenance, read 10 Nov 2026
- Unity Catalog open source, read 10 Nov 2026
- Unity Catalog repository (GitHub API), read 10 Nov 2026
- Databricks Unity Catalog, read 10 Nov 2026
- Apache Polaris, read 10 Nov 2026
- Apache Polaris repository (GitHub API), read 10 Nov 2026
- Project Nessie, read 10 Nov 2026
- Project Nessie 0.110.0 documentation, read 10 Nov 2026
- Project Nessie repository (GitHub API), read 10 Nov 2026
- Dremio Cloud data formats, read 10 Nov 2026
- Dremio Cloud table optimization, read 10 Nov 2026
- S3 Tables with AWS Glue, read 10 Nov 2026
- Athena Iceberg optimization, read 10 Nov 2026
- Onehouse EMR migration, read 10 Nov 2026
- Snowflake Iceberg tables, read 10 Nov 2026
Last reviewed 10 Nov 2026.