Situation, read 10 Nov 2026
Leaving a managed platform for an open stack: what to check on a lakehouse platform
All situationsOpen the Compute Bill
5 things to check, with the published facts that answer each
What table maintenance would the team own after the move?
A managed platform runs compaction and cleanup for you. On an open stack, someone schedules it and pays for it.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR plus Athena on Iceberg | Athena compaction is manual: run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM yourself on Iceberg tables; S3 Tables compaction is on by default. | AWS documentation, 10 Nov 2026 |
| Dremio | Dremio runs automatic optimization on Iceberg tables in its Open Catalog: compaction, row-level delete handling, clustering, partition evolution and manifest rewriting. | Dremio documentation, 10 Nov 2026 |
| Google BigQuery and BigLake | BigQuery Iceberg managed tables run automatic storage optimization (adaptive file sizing, clustering, garbage collection); it is a separate billed item. | Google Cloud documentation, 10 Nov 2026 |
| Onehouse | Onehouse claims automatic scaling and automated async table services; its docs pages give no per-job Spark tuning list. | Onehouse documentation, 10 Nov 2026 |
| Open stack on EMR and EKS | Iceberg documentation says expiring snapshots, removing old metadata files and deleting orphan files are recommended, and compacting with rewriteDataFiles is optional, all as jobs you schedule. | Apache Iceberg documentation, 10 Nov 2026 |
| Starburst | Starburst Galaxy runs Iceberg maintenance with ALTER TABLE ... EXECUTE commands: optimize, drop_extended_stats, expire_snapshots and remove_orphan_files. | Starburst documentation, 10 Nov 2026 |
Can the same tables stay readable from both platforms during the move?
A staged move needs one copy of the data readable by both engines. Check which format bridges the platforms document.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR plus Athena on Iceberg | Athena supports Iceberg 1.4.2 only for tables in the AWS Glue Data Catalog, with Glue optimistic locking. | AWS documentation, 10 Nov 2026 |
| Databricks | Databricks UniForm generates Iceberg metadata for Delta tables from Runtime 14.3 LTS, so Iceberg clients read Delta data without rewriting files. | Databricks documentation, 10 Nov 2026 |
| Dremio | Dremio Cloud documentation lists Iceberg and Delta Lake and a Polaris-based Open Catalog with the Iceberg REST Catalog API, and lists AWS Glue Data Catalog as connectable. | Dremio documentation, 10 Nov 2026 |
| Google BigQuery and BigLake | BigQuery Iceberg managed tables export Iceberg V2 snapshots for open engines. | Google Cloud documentation, 10 Nov 2026 |
| Microsoft Fabric | Fabric OneLake handles Delta and Iceberg through metadata virtualization. | Microsoft Learn documentation, 10 Nov 2026 |
| Onehouse | Apache XTable (formerly OneTable) translates table metadata between Hudi, Delta Lake and Iceberg; it is incubating at the Apache Software Foundation, and Onehouse says it created Hudi and XTable. | Apache XTable site, 10 Nov 2026 |
| Snowflake | Snowflake supports Iceberg tables in customer-managed storage with an external catalog integration such as AWS Glue, and reads Delta files through Delta Direct. | Snowflake documentation, 10 Nov 2026 |
| Starburst | Starburst Galaxy creates and writes Iceberg tables and reads Delta Lake; Hudi tables are read-only. | Starburst documentation, 10 Nov 2026 |
Which Spark behavior changes when the code leaves the managed runtime?
Each platform documents gaps against open-source Spark. The list of gaps is the migration work.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR plus Athena on Iceberg | AWS's documentation states 100% API compatibility for the EMR runtime for Spark; that is AWS's claim about its own runtime. | AWS page, 10 Nov 2026 |
| Databricks | On Databricks serverless compute, Spark RDD APIs, R, Maven coordinates, global temp views and Databricks Container Services are unsupported. | Databricks documentation, 10 Nov 2026 |
| Onehouse | Onehouse documentation says EMR code can be reused unless it uses AWS-specific features, and lists the EMR step and catalog work. | Onehouse documentation, 10 Nov 2026 |
| Open stack on EMR and EKS | EMR on EKS supports the Kubernetes Operator for Apache Spark on your own EKS clusters; the Kubeflow Spark Operator is Apache-2.0 licensed with Helm installation. | AWS documentation, 10 Nov 2026 |
| Snowflake | Snowflake documentation says Spark DataFrame and Spark SQL code can be reused with minimal migration, and that Spark Connect defers plan resolution, which changes behavior versus Spark Classic. | Snowflake documentation, 10 Nov 2026 |
Which rates are published, and which are unreadable?
An exit is justified by a bill, so the rates must be readable. Some managed platforms render their rates by script, so the line says which were read.
| Option | Published fact | Source, read |
|---|---|---|
| Databricks | Databricks bills DBUs per second with pay-as-you-go or committed-use discounts; the data file behind the pricing calculator lists Jobs Compute on AWS at $0.15 per DBU on the Premium plan, with 4.08 DBU per node-hour for r6i.4xlarge (read 2026-11-10). | Databricks pricing data file (AWS), loaded by its pricing calculator, 10 Nov 2026 |
| Google BigQuery and BigLake | BigQuery capacity pricing is per slot-hour; the pricing page text read on 2026-11-10 lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate. | Google Cloud documentation, 10 Nov 2026 |
| Microsoft Fabric | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus. | Azure retail price API, 10 Nov 2026 |
| Open stack on EMR and EKS | AWS's EMR pricing example in us-east-1 shows an EMR charge of $0.105 per hour next to EC2 at $0.398 per hour for a c4.2xlarge, billed per second with a one-minute minimum. | AWS page, 10 Nov 2026 |
| Snowflake | Snowflake's Service Consumption Table, effective 2026-11-09, lists $3.00 per credit for the Enterprise edition on AWS US East (Northern Virginia); warehouses bill credits per second with a 60-second minimum. | Snowflake Service Consumption Table, 10 Nov 2026 |
| Starburst | Starburst Galaxy credit prices start at $0.50 (Pro) and $0.75 (Enterprise) per credit, with a Free tier of up to 3 clusters. | Starburst page, 10 Nov 2026 |
Which clouds and regions does each option document?
A move off one platform can tie the team to a single cloud when the target runs in only one.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR plus Athena on Iceberg | The Amazon EMR, Athena, Glue and S3 services run only on AWS. | AWS page, 10 Nov 2026 |
| Databricks | Databricks lists many AWS regions and is also hosted on Azure and Google Cloud. | Databricks documentation, 10 Nov 2026 |
| Dremio | Dremio Cloud runs on AWS; Dremio Enterprise can be deployed on Kubernetes, on premises, or in AWS, Azure or GCP. | Dremio documentation, 10 Nov 2026 |
| Google BigQuery and BigLake | BigQuery Omni reads S3 data in six AWS regions, and runs on Google Cloud. | Google Cloud documentation, 10 Nov 2026 |
| Onehouse | Onehouse runs in the customer's AWS or GCP account; Azure is not listed in its billing documentation. | Onehouse documentation, 10 Nov 2026 |
| Snowflake | Snowflake runs on AWS, Azure and Google Cloud, with each account in a single region. | Snowflake documentation, 10 Nov 2026 |
| Starburst | Starburst Galaxy prices are listed for AWS, Azure and GCP regions. | Starburst page, 10 Nov 2026 |
Sources
- AWS documentation, read 10 Nov 2026
- Dremio documentation, read 10 Nov 2026
- Google Cloud documentation, read 10 Nov 2026
- Onehouse documentation, read 10 Nov 2026
- Apache Iceberg documentation, read 10 Nov 2026
- Starburst documentation, read 10 Nov 2026
- AWS documentation, read 10 Nov 2026
- Databricks documentation, read 10 Nov 2026
- Dremio documentation, read 10 Nov 2026
- Microsoft Learn documentation, read 10 Nov 2026
- Apache XTable site, read 10 Nov 2026
- Snowflake documentation, read 10 Nov 2026
- Starburst documentation, read 10 Nov 2026
- AWS page, read 10 Nov 2026
- Databricks documentation, read 10 Nov 2026
- Onehouse documentation, read 10 Nov 2026
- AWS documentation, read 10 Nov 2026
- Snowflake documentation, read 10 Nov 2026
- Databricks pricing data file (AWS), loaded by its pricing calculator, read 10 Nov 2026
- Google Cloud documentation, read 10 Nov 2026
- Azure retail price API, read 10 Nov 2026
- AWS page, read 10 Nov 2026
- Snowflake Service Consumption Table, read 10 Nov 2026
- Starburst page, read 10 Nov 2026
- Databricks documentation, read 10 Nov 2026
- Dremio documentation, read 10 Nov 2026
- Google Cloud documentation, read 10 Nov 2026
- Onehouse documentation, read 10 Nov 2026
- Snowflake documentation, read 10 Nov 2026
- Starburst page, read 10 Nov 2026
Last reviewed 10 Nov 2026.