# Lakehouse Compare: every page as markdown Ten ways to run Spark batch work on a lakehouse, Iceberg, Delta Lake and Hudi side by side, and a Compute Bill priced from published AWS prices. --- # Lakehouse platforms compared, with the bill for the same work on each Compared, read 10 Nov 2026 **The short answer.** Ten ways to run Spark batch work on a lakehouse, from Amazon EMR and Databricks to Onehouse and Snowflake, sit side by side with 3 table formats: Iceberg, Delta Lake and Hudi. The Compute Bill has 12 lines for one reference job, 10 r6i.4xlarge nodes for 2 hours in us-east-1, each priced from published rates. - Amazon EMR on EC2: $25.20 - Amazon EMR on EKS (EC2 nodes): $24.42 - Amazon EMR Serverless: $31.63 - Databricks on AWS (Jobs Compute): $32.40 - Open stack: Spark on plain EC2: $20.16 - BigQuery (Enterprise edition slots): $19.20 - Dremio Cloud (Large engine): $102.40 - DualBird on Amazon EMR (F2 instances): $49.50 - Microsoft Fabric (F128 capacity): $46.08 - Onehouse (Quanton, bring-your-own-cloud): No total - Snowflake (XL standard warehouse): $96.00 - Starburst Galaxy (Pro tier): $60.00 ## The job on every line: Nightly Spark batch job, 10 nodes for 2 hours (us-east-1) The bill beside the headline prices the same ten EC2 nodes first, then the platforms whose own billing unit is mapped to the job by a stated assumption. Both groups are alphabetical. Lines in the second group bill different units and engines, so each total is read against its own assumption, not against the other lines. 1 of 12 lines has no total (Onehouse); the Compute Bill page says why. - **Assumption 1:** Region us-east-1; all AWS prices are on-demand Linux prices from the AWS Price List API read on 2026-11-10. - **Assumption 2:** Worker node: r6i.4xlarge, 16 vCPU and 128 GiB. Ten workers for two hours is 20 node-hours, 160 vCPU and 1,280 GiB. - **Assumption 3:** The driver or primary node is not counted. - **Assumption 4:** Storage, S3 requests, EBS, data transfer and public IPv4 charges are excluded. - **Assumption 5:** No discounts, commitments, Savings Plans or Spot. ## Four ways into the facts ### Asset: Lakehouse Compute Bill (12 lines, 11 with a total) Published unit prices times one stated workload, with the formula and what each line leaves out. [Open the Compute Bill](/compute-bill/) ### Formats: Iceberg, Delta Lake, Hudi (3 formats, 8 rows each) The three open table formats in the same rows: releases, engines, compaction, catalogs. [Read the formats](/formats/) ### Upkeep: Compaction and table maintenance (5 pages) What compaction, snapshot expiry and table services do, and which platforms run them for you. [Read the maintenance pages](/maintenance/iceberg-compaction/) ### Profiles: The platforms, A to Z (10 entries) Same fields in the same order for every entry: what it does well, where it is limited, the published facts. [Read the profiles](/platforms/) ## What each platform does well, and where it is limited | Platform | Kind | Does well | Limited | | --- | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | Spark and SQL engines on AWS | Per AWS documentation, the EMR runtime for Spark is 100% API-compatible with open-source Spark, so existing jobs need no rewrite (read 2026-11-10). | Per AWS documentation, Athena compaction is manual (you run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM), and Glue Iceberg compaction is off by default for new tables. | | [Databricks](/platforms/databricks/) | Managed lakehouse built around Delta Lake | Per Databricks documentation, Photon keeps the Spark Catalyst planner, works with existing Spark API code and falls back to the Spark runtime for unsupported operations (read 2026-11-10). | Per Databricks documentation, serverless compute supports only Spark Connect APIs, so RDD APIs, R and Maven coordinates are unsupported, and ANSI mode on by default can change results of existing jobs. | | [Dremio](/platforms/dremio/) | Lakehouse with its own SQL engine | Per Dremio documentation, Iceberg tables in the Open Catalog get automatic optimization: data file compaction, row-level delete handling, clustering, partition evolution and manifest rewriting (read 2026-11-10). | Per Dremio documentation, queries run on its own columnar engine, not on Spark, and the pages read describe no mode that runs existing Spark jobs. | | [DualBird](/platforms/dualbird/) | FPGA Spark plugin for AWS F2 | DualBird says onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin, with Spark still handling planning and scheduling (read 2026-11-10). | DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | Serverless Google Cloud data warehouse | Per Google documentation, BigQuery Iceberg managed tables store data in buckets you own, support DML and streaming, and run automatic compaction, clustering and garbage collection (read 2026-11-10). | Per Google documentation, BigQuery Omni on S3 does not support DML or CREATE TABLE, so it does not run write-heavy Spark batch ETL on S3. | | [Microsoft Fabric](/platforms/microsoft-fabric/) | SaaS analytics on Azure with Spark | Per Microsoft Learn, the native execution engine (Apache Gluten and Velox) needs no code changes, falls back to JVM Spark for unsupported operators and supports Parquet and Delta (read 2026-11-10). | Per Microsoft Learn, Fabric runs over OneLake on Azure and the region page lists Azure regions only, so Spark compute does not run inside AWS. | | [Onehouse](/platforms/onehouse/) | Managed lakehouse in your cloud account | Per Onehouse documentation, its jobs are fully compatible with Apache Spark and EMR code can be reused unless it uses AWS-specific features (read 2026-11-10). | Per Onehouse documentation, billing is in OCUs on a bring-your-own-cloud model, and no dollar rate per OCU was found on 2026-11-10. | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | Self-run Spark with open table formats | Per AWS documentation, EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three formats are available (read 2026-11-10). | Per the Apache Iceberg documentation, users run table maintenance themselves: expire snapshots, remove orphan files and compact with rewriteDataFiles as jobs they schedule (read 2026-11-10). | | [Snowflake](/platforms/snowflake/) | Cloud data platform with PySpark support | Per Snowflake documentation, Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect, and batch jobs are submitted with the Snowpark Submit CLI (read 2026-11-10). | Per Snowflake documentation, Snowpark Connect supports Apache Spark 3.5 only, and DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported (read 2026-11-10). | | [Starburst](/platforms/starburst/) | Managed Trino service for Iceberg | Per Starburst documentation, Galaxy creates and writes Iceberg tables (Parquet, ORC or Avro) and runs maintenance with ALTER TABLE ... EXECUTE optimize, expire_snapshots and remove_orphan_files (read 2026-11-10). | Per Starburst documentation, Galaxy runs SQL on Trino, not Spark, so DataFrame code, UDFs and PySpark would be rewritten as Trino SQL or kept on Spark (read 2026-11-10). | Alphabetical. First sentence of each; the full text and sources are on the platform's profile. ## Five situations - [Spark bill doubled on EMR or EKS](/situations/bill-doubled/): What to check when an EMR or EKS Spark bill doubled: billing units, cost claims, code changes, settings that move spend, and region, with published facts. - [Nightly pipelines finish after the morning deadline](/situations/late-pipelines/): For nightly pipelines that miss the morning: published runtimes, who ran them, whether a reader can repeat them, failure behavior on large jobs, and Spark versions. - [AI and feature pipelines wait on data preparation](/situations/ai-data-prep/): For AI training and feature pipelines that wait on data prep: table formats, unchanged Spark code, UDF and fallback behavior, and where data has to live. - [Leaving Databricks or Snowflake for an open stack](/situations/leaving-managed/): For a team weighing a move from Databricks or Snowflake to an open stack: what you own afterward, shared table formats, Spark changes and cloud coverage. - [Large Iceberg tables with daily compaction cost](/situations/iceberg-compaction/): For teams paying for daily Iceberg compaction: who runs it, the published unit prices, strategies, delete handling and the benchmarks that exist. ## What a lakehouse is A lakehouse keeps data as open files, usually Parquet, in object storage such as Amazon S3, and adds a table format over the files. The table format records which files make up a table, so engines can read and write it with transactions. Apache Iceberg, Delta Lake and Apache Hudi are the three open table formats covered here. Any engine that understands the format can use the same table. The Apache Iceberg home page names Spark, Trino, Flink, Presto, Hive and Impala as engines that work with the same tables at the same time. The Delta Lake home page lists a similar set, and the Hudi home page lists Spark, Flink and others under data processing. The platforms on this site differ in who runs the engine, who maintains the tables (compaction, snapshot cleanup, file layout) and what they bill for. A team that owns the Spark cluster bill needs those three facts side by side, each with its source and read date. Data catalogs in the governance sense, lineage, semantic layers and MCP servers for data are outside the subject of this site. ## Guides and glossary - [What is a data lakehouse: definition, origins, 2021 paper](/guides/what-is-a-data-lakehouse/) - [Databricks vs Snowflake: published facts side by side](/guides/databricks-vs-snowflake-facts/) - [Iceberg vs Delta Lake: documented facts, no pick](/guides/iceberg-vs-delta-lake/) - [EMR cost optimization for an Iceberg lakehouse](/guides/emr-cost-optimization-lakehouse/) - [Choosing a table format: what to check for Iceberg, Delta, Hudi](/guides/choosing-a-table-format/) - [The open stack on AWS: Spark, table formats and catalogs](/guides/open-stack-on-aws/) - [All guides](/guides/) - [Glossary: 35 terms](/glossary/) ## Sources - Apache Iceberg home page: https://iceberg.apache.org/ (read 10 Nov 2026) - Delta Lake home page: https://delta.io/ (read 10 Nov 2026) - Apache Hudi home page: https://hudi.apache.org/ (read 10 Nov 2026) - AWS Price List API, Amazon EC2, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json (read 10 Nov 2026) - AWS Price List API, Amazon EMR, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/us-east-1/index.json (read 10 Nov 2026) - Amazon EKS pricing page: https://aws.amazon.com/eks/pricing/ (read 10 Nov 2026) - Databricks pricing data file loaded by its pricing calculator, AWS: https://www.databricks.com/en-pricing-assets/data/pricing/AWS.json (read 10 Nov 2026) - Google BigQuery pricing page: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Azure Retail Prices API, Microsoft Fabric, eastus: https://prices.azure.com/api/retail/prices?$filter=serviceName eq 'Microsoft Fabric' and armRegionName eq 'eastus' (read 10 Nov 2026) - Starburst Galaxy pricing details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Snowflake Service Consumption Table, effective 2026-11-09: https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf (read 10 Nov 2026) - Dremio pricing page: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio Cloud engines documentation: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/ --- # The Lakehouse Compute Bill: the same Spark batch job, priced on each platform Compute Bill, read 10 Nov 2026 **The short answer.** The Lakehouse Compute Bill prices one job, Nightly Spark batch job, 10 nodes for 2 hours in us-east-1, on 12 lines from published on-demand prices read 10 Nov 2026. 11 lines have a total; 1 does not, because a needed price is not published. Every formula and assumption is on this page. ## The stated cost on each line - Amazon EMR on EC2: $25.20 - Amazon EMR on EKS (EC2 nodes): $24.42 - Amazon EMR Serverless: $31.63 - Databricks on AWS (Jobs Compute): $32.40 - Open stack: Spark on plain EC2: $20.16 - BigQuery (Enterprise edition slots): $19.20 - Dremio Cloud (Large engine): $102.40 - DualBird on Amazon EMR (F2 instances): $49.50 - Microsoft Fabric (F128 capacity): $46.08 - Onehouse (Quanton, bring-your-own-cloud): No total - Snowflake (XL standard warehouse): $96.00 - Starburst Galaxy (Pro tier): $60.00 ## The reference workload One nightly Spark batch job on AWS us-east-1 that needs ten worker nodes of 16 vCPU and 128 GiB memory (r6i.4xlarge as the EC2 baseline) for two hours, on-demand, reading and writing Apache Iceberg tables on S3. The same job is priced on every line. - Region us-east-1; all AWS prices are on-demand Linux prices from the AWS Price List API read on 2026-11-10. - Worker node: r6i.4xlarge, 16 vCPU and 128 GiB. Ten workers for two hours is 20 node-hours, 160 vCPU and 1,280 GiB. - The driver or primary node is not counted. - Storage, S3 requests, EBS, data transfer and public IPv4 charges are excluded. - No discounts, commitments, Savings Plans or Spot. - Where a platform bills in its own unit (credits, slots, capacity units, DCUs), the mapping from the reference job to that unit is an assumption stated on that line. - No performance number is derived: every line assumes the job takes two hours. ## Lines that price the same EC2 nodes | Line | How it is billed | Formula | Stated cost | What is in and out | | --- | --- | --- | --- | --- | | [Amazon EMR on EC2](/platforms/aws-emr-athena-iceberg/) | EC2 instance-hours plus an EMR price per instance-hour, each billed per second with a one-minute minimum. | 20 node-hours x ($1.008 EC2 + $0.252 EMR) = 20 x $1.26 = $25.20. | $25.20 | Includes EC2 and the EMR price for ten r6i.4xlarge workers. Excludes the primary (driver) node, EBS volumes, S3 requests, data transfer and public IPv4 charges. | | [Amazon EMR on EKS (EC2 nodes)](/platforms/aws-emr-athena-iceberg/) | EC2 instance-hours for the nodes, plus an EMR charge per vCPU-hour and per GB-hour of the pods. AWS also charges $0.10 per hour for each EKS cluster; that fee is excluded from this line. | EC2: 20 node-hours x $1.008 = $20.16. EMR vCPU: 160 vCPU x 2 hours = 320 vCPU-hours x $0.007084 = $2.26688. EMR memory: 1,280 GB x 2 hours = 2,560 GB-hours x $0.000777875 = $1.99136. Total $20.16 + $2.26688 + $1.99136 = $24.41824, which rounds to $24.42. | $24.42 | Includes EC2 and the EMR uplift for pods that request the full vCPU and memory of all ten nodes. Excludes the EKS cluster fee ($0.10 per hour per cluster, as on the open stack line), because it depends on how long the cluster stays up, not on the two-hour job; and it excludes the driver pod, EBS, S3 requests, data transfer and public IPv4 charges. | | [Amazon EMR Serverless](/platforms/aws-emr-athena-iceberg/) | Per vCPU-hour and per GB-hour of the workers while they run, plus storage beyond the free ephemeral amount. No instances to manage. | vCPU: 160 x 2 = 320 vCPU-hours x $0.052624 = $16.83968. Memory: 1,280 GB x 2 = 2,560 GB-hours x $0.0057785 = $14.79296. Total $16.83968 + $14.79296 = $31.63264, which rounds to $31.63. | $31.63 | Includes vCPU and memory charges for workers that total 160 vCPU and 1,280 GB for two hours. Excludes extra ephemeral storage (the $0.000111 per GB-hour rate applies only beyond the free amount), the driver, S3 requests and data transfer. Workers are billed only while they run, so a job that scales in and out would be billed for less than the full two hours; that is not modeled. | | [Databricks on AWS (Jobs Compute)](/platforms/databricks/) | DBUs consumed per node-hour (rate by instance type) times a price per DBU for the compute type and plan, plus the EC2 bill for the same nodes in the customer's account. | DBUs: 20 node-hours x 4.08 DBU per node-hour = 81.6 DBU. 81.6 x $0.15 = $12.24. EC2: 20 x $1.008 = $20.16. Total $12.24 + $20.16 = $32.40. | $32.40 | Includes the Databricks charge at the Premium plan Jobs Compute rate and the EC2 bill for the same ten r6i.4xlarge nodes, which is paid to AWS separately. Both Databricks figures come from the JSON data file that the Databricks pricing calculator page loads, because the pricing pages themselves are drawn by script; that file does not label a region, and the line assumes it applies to us-east-1. The Enterprise plan rate in the same file is $0.20 per DBU. Excludes the driver node, storage and S3 requests, and the Photon, serverless and committed-use options. | | [Open stack: Spark on plain EC2](/platforms/open-stack-emr-eks/) | EC2 instance-hours only. There is no managed-service charge; the team runs open-source Spark itself. | 10 nodes x 2 hours = 20 node-hours. 20 x $1.008 = $20.16. | $20.16 | Includes EC2 on-demand compute for the ten workers. Excludes the driver node, EBS volumes, S3 requests, data transfer, and the engineering time to operate Spark, Iceberg and the cluster manager. If the same open stack ran on a self-managed EKS cluster, AWS would add $0.10 per hour for each EKS cluster (Amazon EKS pricing page, read 2026-11-10). That fee is excluded here, as on the EMR on EKS line, because it depends on how long the cluster stays up, not on the two-hour job. | Alphabetical. ## Lines whose capacity is mapped by a stated assumption These platforms bill in their own units (slots, capacity units, credits, DCUs, OCUs) or on different hardware. The mapping from the reference job to that unit is the assumption written in each formula, not a published equivalence, and no line here claims the job runs in the same two hours. | Line | How it is billed | Formula | Stated cost | What is in and out | | --- | --- | --- | --- | --- | | [BigQuery (Enterprise edition slots)](/platforms/bigquery-biglake/) | Per slot-hour for the edition used, billed per second with a one-minute minimum. A slot is described by Google as a virtual CPU. | 160 slots (one per vCPU of the reference job) x 2 hours = 320 slot-hours. 320 x $0.06 = $19.20. | $19.20 | Includes slot compute only. Excludes storage, and excludes any data movement into or out of AWS. BigQuery runs SQL on Google's slots, not Spark on EC2, so this line prices the same amount of vCPU time, not the same Spark job. The text of the pricing page lists one rate for the Enterprise edition without showing which region the rate belongs to, so the region is not confirmed here. | | [Dremio Cloud (Large engine)](/platforms/dremio/) | DCUs consumed per hour by engine size while the engine runs, times a price per DCU. | Large engine = 256 DCUs per hour. 256 x 2 hours = 512 DCUs x $0.20 = $102.40. | $102.40 | Includes engine DCUs for two hours of one Large replica. Excludes storage, LLM token charges and data transfer. The engines page lists executors per size but not the vCPU or memory of an executor, so which size equals the reference workload is an assumption. An XLarge engine (16 executors, 512 DCUs per hour) would give $204.80. Dremio Cloud runs SQL in its own engine, not Spark on EC2. The Dremio pricing page returned an access error to the fetch tool on 2026-11-10 and the $0.20 rate was read through a second fetch method. | | [DualBird on Amazon EMR (F2 instances)](/platforms/dualbird/) | EC2 F2 instance-hours plus the EMR price for F2 instance-hours. DualBird's plugin price: none found on 2026-11-10. | Infrastructure only, at the same node count and hours as the other lines: 20 node-hours x ($1.98 EC2 + $0.495 EMR) = 20 x $2.475 = $49.50. | $49.50 | Includes only EC2 and EMR charges for ten f2.6xlarge nodes for two hours. The DualBird plugin price: none found on 2026-11-10, so it is not included. Excludes the primary node, EBS, S3 requests and data transfer. This line prices instances at the node count and hours of the reference workload. It does not say that ten F2 nodes are needed, or that fewer would do; DualBird sizing guidance: none found on 2026-11-10, and its benchmarks are its own (see the benchmarks list). DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. | | [Microsoft Fabric (F128 capacity)](/platforms/microsoft-fabric/) | Pay-as-you-go capacity units (CU) per hour for the whole capacity SKU while it is running, not per job. One CU maps to two Spark VCores, per Microsoft. | 160 vCPU treated as 160 Spark VCores = 80 CU needed. The next SKU down, F64, has 64 CU, which is fewer than the 80 CU needed, so F64 is too small and the smallest F SKU with at least 80 CU is F128 (128 CU). 128 CU x 2 hours = 256 CU-hours x $0.18 = $46.08. | $46.08 | Includes the F128 capacity for two hours. F128 is billed in full even though 80 CU are used, because F64 (64 CU) is too small for the stated mapping. Excludes OneLake storage, the Power BI license items Microsoft lists separately, and any pause and resume overhead. Fabric runs on Azure, not AWS, so this line prices the Azure US East capacity for the same vCPU count. The Fabric pricing page showed no rate in its text on 2026-11-10; the rate comes from the Azure Retail Prices API. | | [Onehouse (Quanton, bring-your-own-cloud)](/platforms/onehouse/) | Onehouse Compute Units (OCU), where 1 OCU is a 4-core instance for 1 hour in the customer's account, plus support costs. The customer also pays AWS for the instances. | OCU quantity: 160 vCPU / 4 cores = 40 OCU per hour x 2 hours = 80 OCU. Price per OCU: None published on 2026-11-10, so the Onehouse charge is not computed. AWS instance part, if the same r6i.4xlarge nodes were used: 20 x $1.008 = $20.16. | No total | None published on 2026-11-10: a dollar price per OCU. The Onehouse pricing URL returned HTTP 404 and the billing documentation states the OCU definition but no dollar rate. The $1.008 EC2 price is shown only to size the AWS part; Onehouse's documentation lists its own standard instance types (Graviton m8g sizes), so the instances actually used would differ. Support costs are also unpublished. | | [Snowflake (XL standard warehouse)](/platforms/snowflake/) | Credits per warehouse-hour by size, billed per second with a 60-second minimum, times a price per credit by edition, cloud and region. | XL warehouse = 16 credits per hour. 16 x 2 hours = 32 credits x $3.00 = $96.00. | $96.00 | Includes warehouse credits for two hours of an XL standard warehouse at the Enterprise on-demand rate. Excludes storage, cloud services credits beyond the daily allowance, and data transfer. The table lists credits per hour for each size but not the vCPU or memory behind each size, so which size equals the reference workload is an assumption. For the same hours, the Standard edition ($2.00 per credit) gives $64.00 and Business Critical ($4.00) gives $128.00; a 2XL warehouse (32 credits per hour) doubles each figure. Snowflake runs SQL and Snowpark in its own engine, not Spark on EC2. | | [Starburst Galaxy (Pro tier)](/platforms/starburst/) | Credits per cluster-hour, by number of workers, times a price per credit that depends on tier, cloud and region. Galaxy runs Trino SQL, not Spark. | The published table is 1 worker = 6 credits per hour, 8 workers = 48, 16 workers = 96 (linear). Ten workers is not listed; at the same 6 per worker it is 60 credits per hour. 60 x 2 hours = 120 credits x $0.50 = $60.00. | $60.00 | Includes Galaxy credits for a ten-worker cluster for two hours at the Pro tier. Excludes storage, S3 requests, data transfer and the tier features. Enterprise tier is $0.75 per credit and Mission Critical $1.00 on the same page, which would give $90.00 and $120.00 for the same credits. The worker size in a Galaxy cluster is not stated on the page, so ten Galaxy workers are not shown to equal ten r6i.4xlarge nodes. | Alphabetical. ## Change the workload Calculator: set worker nodes and hours; each total rescales the published arithmetic. Fixed-size capacity lines scale by hours only. | Line | Reference total | Your total | Scales with | | --- | --- | --- | --- | | [Amazon EMR on EC2](/platforms/aws-emr-athena-iceberg/) | $25.20 | $25.20 | Nodes and hours | | [Amazon EMR on EKS (EC2 nodes)](/platforms/aws-emr-athena-iceberg/) | $24.42 | $24.42 | Nodes and hours | | [Amazon EMR Serverless](/platforms/aws-emr-athena-iceberg/) | $31.63 | $31.63 | Nodes and hours | | [Databricks on AWS (Jobs Compute)](/platforms/databricks/) | $32.40 | $32.40 | Nodes and hours | | [Open stack: Spark on plain EC2](/platforms/open-stack-emr-eks/) | $20.16 | $20.16 | Nodes and hours | | [BigQuery (Enterprise edition slots)](/platforms/bigquery-biglake/) | $19.20 | $19.20 | Nodes and hours | | [Dremio Cloud (Large engine)](/platforms/dremio/) | $102.40 | $102.40 | Hours only (fixed capacity size) | | [DualBird on Amazon EMR (F2 instances)](/platforms/dualbird/) | $49.50 | $49.50 | Nodes and hours | | [Microsoft Fabric (F128 capacity)](/platforms/microsoft-fabric/) | $46.08 | $46.08 | Hours only (fixed capacity size) | | [Onehouse (Quanton, bring-your-own-cloud)](/platforms/onehouse/) | No total | No total | Nodes and hours | | [Snowflake (XL standard warehouse)](/platforms/snowflake/) | $96.00 | $96.00 | Hours only (fixed capacity size) | | [Starburst Galaxy (Pro tier)](/platforms/starburst/) | $60.00 | $60.00 | Nodes and hours | Your total updates as you type. Without JavaScript the reference totals stand. ## Published benchmarks for Spark batch work on these platforms | Published | Test | Ran by | Who ran it | Result as published | | --- | --- | --- | --- | --- | | Not stated | [Amazon EMR runtime for Spark, vendor claim](https://aws.amazon.com/emr/features/spark/) | AWS | Vendor-run | AWS says up to 4.5x faster than open-source equivalents and 2.7x faster Iceberg write performance. No workload or method is stated on the page. | | Not stated | [Single task performance, same cluster size](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=asset) | DualBird | Vendor-run | DualBird's home page says X15-X20 speedup for the same cluster size and cost reduced by over 50%. No hardware, baseline, date or method is stated on the page. | | Not stated | [Fabric native execution engine vs open-source Spark](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime) | Microsoft | Vendor-run | Microsoft says up to six times faster than open-source Spark and approximately 83% compute cost savings on a fixed-size Fabric cluster. No workload is stated in the runtime documentation. | | 2026-11-10 | [Spark UDF pipeline, EMR vs Quanton at a B2B SaaS customer](https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs) | Unnamed B2B SaaS customer, published by Onehouse | Another party | p50 (9.5M rows): EMR 110 s, Quanton 93 s on 24 nodes each. p97 (1.1B rows): 4,850 s vs 3,490 s. p99 (2.1B rows): 10,400 s vs 6,650 s on 22 nodes. p100 (21B rows): EMR did not finish on 180 nodes (out of local disk, aborted after 3.7 hours); Quanton took 6,950 s on 180 nodes; EMR took 6,760 s on 300 nodes. Nodes were AWS m8gd in the customer's account. | | 2026-06-11 | [Sort-based Iceberg compaction, 100 GB, on AWS F2 instances](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=asset) | DualBird | Vendor-run | DualBird says about 12x to 20x faster Spark task execution and 55% to 85% lower EC2 cost, with about $2 per TB for DualBird against about $6 per TB (rising past $13 per TB) for CPU-based clusters. Baselines were vanilla Spark and an unnamed C++ accelerated Spark; instance types and counts are not stated, and the method is behind a sign-up form. | | 2025-12-02 | [Onehouse Quanton vs Amazon EMR 7.12, 10 TB TPC-DS and 1 TB LakeLoader merges](https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads) | Onehouse | Vendor-run | Onehouse says it still comes out with a 2.5x better price/performance advantage against EMR 7.12, and reports EMR 7.12 as 32% better than its previous versions. Hardware: latest general-purpose Graviton instances. Results are shown as charts in the post. | | 2025-11-26 | [Amazon EMR 7.12 vs open-source Spark 3.5.6 on Iceberg, TPC-DS 3 TB](https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/) | AWS | Vendor-run | Average runtime: EMR 7.12 1349.62 s; EMR 7.5 1535.62 s; open-source Spark 3.5.6 with Iceberg 1.10.0 6113.92 s. Cost as published: EMR 7.12 $4.81; EMR 7.5 $5.47; open-source Spark $17.65. Setup: 9 x r5d.4xlarge per cluster (1 primary, 8 workers), 104 SparkSQL queries, 3 rounds, Iceberg Hadoop catalog. | | 2025-11-12 | [Apache Iceberg on Quanton vs open-source Spark](https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads) | Onehouse | Vendor-run | TPC-DS up to about 3.1x better performance than open-source Spark; TPCx-BB 2x; TPC-DI 2x (3x to 4x on MERGE INTO only); LakeLoader 4.1x better write performance, as published. TPC-DI and LakeLoader at 1 TB. | Newest first. No number here is derived; runtimes come from different hardware and data. ## Download and cite CSV: [/datasets/lakehouse-compute-bill.csv](/datasets/lakehouse-compute-bill.csv). JSON: [/datasets/lakehouse-compute-bill.json](/datasets/lakehouse-compute-bill.json). Cite this: Lakehouse Compare, "Lakehouse Compute Bill", edition 10 Nov 2026, https://datalakehousecompare.com/datasets/ (CC BY 4.0). ## Sources - AWS Price List API, Amazon EC2, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json (read 10 Nov 2026) - AWS Price List API, Amazon EMR, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/us-east-1/index.json (read 10 Nov 2026) - Amazon EKS pricing page: https://aws.amazon.com/eks/pricing/ (read 10 Nov 2026) - Databricks pricing data file loaded by its pricing calculator, AWS: https://www.databricks.com/en-pricing-assets/data/pricing/AWS.json (read 10 Nov 2026) - Google BigQuery pricing page: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Azure Retail Prices API, Microsoft Fabric, eastus: https://prices.azure.com/api/retail/prices?$filter=serviceName eq 'Microsoft Fabric' and armRegionName eq 'eastus' (read 10 Nov 2026) - Starburst Galaxy pricing details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Snowflake Service Consumption Table, effective 2026-11-09: https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf (read 10 Nov 2026) - Dremio pricing page: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio Cloud engines documentation: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - AWS Big Data Blog, Run Apache Spark and Iceberg 4.5x faster: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Amazon EMR features, Spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Onehouse blog, Quanton vs the latest AWS EMR: https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads (read 10 Nov 2026) - Onehouse blog, Apache Iceberg on Quanton: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026) - Onehouse blog, EMR migration to Quanton for Spark UDF jobs: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - DualBird blog, Iceberg compaction benchmark: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) - Microsoft Learn, Fabric runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compute-bill/ --- # How the Lakehouse Compute Bill is calculated Compute Bill method, read 10 Nov 2026 **The short answer.** The Lakehouse Compute Bill prices one reference job, ten r6i.4xlarge nodes for two hours in us-east-1, from prices read on 2026-11-10. Both groups are alphabetical, and mapped-group totals are read against their own assumptions, not against each other. A line with no published price shows No total. ## The reference workload One nightly Spark batch job in AWS us-east-1: ten worker nodes of the r6i.4xlarge class (16 vCPU, 128 GiB), running for two hours, on-demand, reading and writing Apache Iceberg tables on S3. That is 20 node-hours, 160 vCPU and 1,280 GiB. The same job is used on every line. ## The price sources Amazon prices come from the AWS Price List API for EC2 and EMR, published 2026-11-10: r6i.4xlarge is $1.008 per hour on EC2 and $0.252 per hour for the EMR price. Other prices come from each provider's own page or price API: the Snowflake Service Consumption Table dated 2026-11-09, Google's BigQuery page, the Azure Retail Prices API for Fabric, and the Starburst and Dremio pricing pages. Every unit price on the page names its source and read date. ## The arithmetic Each line shows its formula in plain numbers: published unit price times the stated quantity, rounded to cents. Nothing is estimated and no performance number is derived. Platforms that do not bill by the instance (Snowflake, BigQuery, Fabric, Starburst, Dremio) need a mapping from the reference job to their unit, such as a warehouse size or a slot count. Those mappings are assumptions, and each line says so. ## What is excluded - The driver or primary node, EBS volumes, S3 storage and requests, data transfer and public IPv4 charges. - Support plans, enterprise discounts, Savings Plans, Reserved Instances and Spot. - Engineering time to run the cluster, tune it and maintain Iceberg tables. - Any price that is not published: those lines carry no total and say "None published on 2026-11-10". ## Why both groups are alphabetical The bill has two groups. The first holds lines that price the same ten EC2 nodes: the open stack on EC2, EMR on EKS, EMR on EC2, EMR Serverless (which bills the same 160 vCPU and 1,280 GB as vCPU-hours and GB-hours, not as instances) and Databricks on AWS. The second holds lines whose own billing unit is mapped to the job by an assumption stated on the line: BigQuery, Microsoft Fabric, DualBird on EMR (ten f2.6xlarge nodes, a different instance type), Starburst, Snowflake, Dremio and Onehouse. Inside each group, lines are alphabetical by name, and a line with no total is shown in place. Totals in the first group share one set of nodes. Totals in the second group bill different units and engines, so each is read against its own stated assumption, not against the other lines. A different warehouse size, slot mapping or node count would move lines, and the platforms differ in what the price includes: some include a managed runtime, some are SQL engines rather than Spark, and Fabric runs on Azure. Read the cost note on each line before comparing two totals. ## How the calculator works The reader changes the node count and the hours. The calculator multiplies those two numbers into the same formulas shown on each line. Prices stay as published; the calculator has no field for a price, a discount or a performance factor, and it does not change the mappings used for Snowflake, BigQuery, Fabric, Starburst or Dremio. ## How often prices are re-read Prices on this page were read on 2026-11-10. The date beside each price is the date it was read. Cloud prices change without notice, so check the linked source before using a figure in a budget. ## Sources - AWS Price List API, Amazon EMR, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/us-east-1/index.json (read 10 Nov 2026) - AWS Price List API, Amazon EC2, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json (read 10 Nov 2026) - Amazon EMR pricing page: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EKS pricing page: https://aws.amazon.com/eks/pricing/ (read 10 Nov 2026) - Snowflake Service Consumption Table, effective 2026-11-09: https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf (read 10 Nov 2026) - Google BigQuery pricing page: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Azure Retail Prices API, Microsoft Fabric, eastus: https://prices.azure.com/api/retail/prices?$filter=serviceName eq 'Microsoft Fabric' and armRegionName eq 'eastus' (read 10 Nov 2026) - Microsoft Learn, Spark concurrency in Fabric: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Starburst Galaxy pricing details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Dremio pricing page: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio Cloud engines documentation: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - Onehouse usage and billing documentation: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Databricks AWS pricing page: https://www.databricks.com/product/aws-pricing (read 10 Nov 2026) - Databricks pricing data file (AWS), loaded by its pricing calculator: https://www.databricks.com/en-pricing-assets/data/pricing/AWS.json (read 10 Nov 2026) - Databricks Lakeflow Jobs pricing page: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) - Amazon EC2 F2 instances: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compute-bill/method/ --- # 10 ways to run Spark batch work on a lakehouse, listed A to Z Platforms, read 10 Nov 2026 **The short answer.** Ten ways to run Spark batch work on a lakehouse, listed alphabetically: Amazon EMR plus Athena on Iceberg, Databricks, Dremio, DualBird, Google BigQuery and BigLake, Microsoft Fabric, Onehouse, the open stack on EMR and EKS, Snowflake and Starburst. Each entry gives what it does, its published limits and its sources. ## Every entry on this site | Platform | Kind | Compute Bill lines | Summary | | --- | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | Spark and SQL engines on AWS | 3 | Spark on EMR, SQL and Spark on Athena, and Iceberg tables in Glue or S3 Tables, priced per use on AWS and priced here in us-east-1. | | [Databricks](/platforms/databricks/) | Managed lakehouse built around Delta Lake | 1 | Managed Spark with the Photon engine and serverless jobs, Delta Lake tables, and Iceberg reads of Delta data through UniForm, on AWS, Azure and Google Cloud. | | [Dremio](/platforms/dremio/) | Lakehouse with its own SQL engine | 1 | Arrow-based SQL engine with automatic Iceberg table optimization and a Polaris-based Open Catalog, billed in DCUs on AWS, owned by SAP since July 2026. | | [DualBird](/platforms/dualbird/) | FPGA Spark plugin for AWS F2 | 1 | Spark plugin for AWS EC2 F2 FPGA instances, with a DualBird-run Iceberg compaction benchmark; it is an accelerator for EMR and EKS, not a lakehouse. | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | Serverless Google Cloud data warehouse | 1 | Serverless warehouse with Iceberg managed tables, BigLake external tables, Spark stored procedures and BigQuery Omni for S3 data. | | [Microsoft Fabric](/platforms/microsoft-fabric/) | SaaS analytics on Azure with Spark | 1 | Azure SaaS analytics with Spark, OneLake storage and capacity pricing; S3 shortcuts read AWS data in place, read-only. | | [Onehouse](/platforms/onehouse/) | Managed lakehouse in your cloud account | 1 | Quanton engine for Spark and SQL jobs in your own AWS or GCP account, with managed table services for Hudi, Iceberg and Delta Lake. | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | Self-run Spark with open table formats | 1 | Spark on EMR or EKS with open table formats: unchanged Spark code, with table maintenance, tuning and operations owned by the team. | | [Snowflake](/platforms/snowflake/) | Cloud data platform with PySpark support | 1 | PySpark DataFrame code on Snowflake warehouses via Snowpark Connect, with Iceberg tables in customer-managed storage, on AWS, Azure and Google Cloud. | | [Starburst](/platforms/starburst/) | Managed Trino service for Iceberg | 1 | Managed Trino lakehouse service that writes Iceberg tables, runs Iceberg maintenance commands and bills in credits, with prices by AWS region. | Alphabetical by name. Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/platforms/ --- # Amazon EMR plus Athena on Iceberg, Spark and SQL engines on AWS: the published facts Platform profile, read 10 Nov 2026 **The short answer.** Amazon EMR runs open-source Spark on EC2, EKS or Serverless, and Athena queries Iceberg tables in the AWS Glue Data Catalog. AWS describes the EMR runtime as 100% API-compatible with open-source Spark. EMR 7.x lists Spark 3.5.x builds, and S3 Tables compaction is on by default with a 512 MB target file size. - **Kind:** Spark and SQL engines on AWS - **Website:** [aws.amazon.com/emr](https://aws.amazon.com/emr/) - **Name checked:** Amazon EMR, Amazon Athena, AWS Glue and Amazon S3 Tables all still exist under these names; EMR 7.x lists Spark 3.5.x and the EMR Spark page lists Spark 4.0 and Iceberg v3 support. - **Read:** 10 Nov 2026 ## What it does well - Per AWS documentation, the EMR runtime for Spark is 100% API-compatible with open-source Spark, so existing jobs need no rewrite (read 2026-11-10). - AWS publishes the setup for its TPC-DS 3 TB Iceberg benchmark in GitHub repositories, so a reader can repeat the run (read 2026-11-10). ## Where it is limited - Per AWS documentation, Athena compaction is manual (you run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM), and Glue Iceberg compaction is off by default for new tables. - Every service in this entry runs only on AWS, and Athena supports Iceberg 1.4.2 only for tables in the Glue Data Catalog (read 2026-11-10). ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | AWS states its runtime executes up to 4.5x faster than open-source Spark and gives 2.7x faster Iceberg write performance; the page gives no workload, scale or configuration. | vendor | [aws.amazon.com/emr/features/spark](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | ## Published prices and cost results | Fact | Source type | Source, read | | --- | --- | --- | | EMR on EC2 adds a per-instance uplift in addition to EC2: the AWS price list shows $0.048 per hour for EMR m5.xlarge in us-east-1 (publication 2026-11-10). | cloud provider docs | [pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json), 10 Nov 2026 | | EMR on EKS uplift in us-east-1 is $0.007084 per vCPU-hour and $0.000777875 per GB-hour, in addition to the EC2 or Fargate cost. | cloud provider docs | [pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json), 10 Nov 2026 | | The EMR pricing page worked example for EMR Serverless uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour; a job with 100 vCPU for 30 minutes plus 200 more vCPU for 15 minutes costs $9.596275 in that example. | cloud provider docs | [aws.amazon.com/emr/pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | Athena SQL costs $5.00 per TB scanned in us-east-1; Athena for Apache Spark costs $0.35 per DPU-hour; provisioned capacity costs $0.30 per DPU-hour. | cloud provider docs | [pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/ind](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json), 10 Nov 2026 | | Glue Iceberg compaction costs $0.44 per DPU-hour in us-east-1; Glue 6.0+ ETL jobs cost $0.308 per DPU-hour and older Glue ETL $0.44. | cloud provider docs | [pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.js](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json), 10 Nov 2026 | | S3 Tables in us-east-1: compaction is $0.005 per GB processed plus $0.002 per 1,000 objects compacted; sort or Z-order compaction is $0.01 per GB plus $0.002 per 1,000 objects; storage is $0.0265 per GB-month for the first 50 TB. | cloud provider docs | [pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.j](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json), 10 Nov 2026 | ## Code changes and migration | Fact | Source type | Source, read | | --- | --- | --- | | The EMR runtime for Spark is described as 100% API-compatible with open-source Spark, so existing Spark jobs need no rewrite; EMR is native Spark on EC2, EKS or Serverless. | cloud provider docs | [aws.amazon.com/emr/features/spark](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | Athena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client; PySpark engine version 3 offers the console notebook experience. | cloud provider docs | [docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html](https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html), 10 Nov 2026 | ## Tuning and table upkeep | Fact | Source type | Source, read | | --- | --- | --- | | EMR enables adaptive query execution, dynamic partition pruning, bloom filter join and optimized join reorder; docs say AQE adaptive join conversion is on by default and can be switched off with a Spark setting. | cloud provider docs | [docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html), 10 Nov 2026 | | Athena compaction is manual: run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM yourself on Iceberg tables. | cloud provider docs | [docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html), 10 Nov 2026 | | Glue Data Catalog Iceberg compaction is disabled by default for new tables and must be enabled per table with an IAM role. | cloud provider docs | [docs.aws.amazon.com/glue/latest/dg/enable-compaction.html](https://docs.aws.amazon.com/glue/latest/dg/enable-compaction.html), 10 Nov 2026 | | S3 Tables compaction is on by default for all tables with a 512 MB default target file size (configurable 64 to 512 MB) and also applies row-level deletes. | cloud provider docs | [docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html), 10 Nov 2026 | ## Spark versions and table formats | Fact | Source type | Source, read | | --- | --- | --- | | EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0); the EMR Spark feature page says EMR supports Apache Iceberg v3 and Spark 4.0. | cloud provider docs | [docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html), 10 Nov 2026 | | Athena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only with tables in the AWS Glue Data Catalog and Glue optimistic locking; DDL is not supported on tables registered with Lake Formation. | cloud provider docs | [docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | All of these are AWS services, so they run only on AWS; EMR, Athena, Glue and S3 are priced in us-east-1 on the AWS price list. | cloud provider docs | [aws.amazon.com/emr/features/spark](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | ## On the Compute Bill | Line | How it is billed | Formula | Stated cost | What is in and out | | --- | --- | --- | --- | --- | | [Amazon EMR on EKS (EC2 nodes)](/platforms/aws-emr-athena-iceberg/) | EC2 instance-hours for the nodes, plus an EMR charge per vCPU-hour and per GB-hour of the pods. AWS also charges $0.10 per hour for each EKS cluster; that fee is excluded from this line. | EC2: 20 node-hours x $1.008 = $20.16. EMR vCPU: 160 vCPU x 2 hours = 320 vCPU-hours x $0.007084 = $2.26688. EMR memory: 1,280 GB x 2 hours = 2,560 GB-hours x $0.000777875 = $1.99136. Total $20.16 + $2.26688 + $1.99136 = $24.41824, which rounds to $24.42. | $24.42 | Includes EC2 and the EMR uplift for pods that request the full vCPU and memory of all ten nodes. Excludes the EKS cluster fee ($0.10 per hour per cluster, as on the open stack line), because it depends on how long the cluster stays up, not on the two-hour job; and it excludes the driver pod, EBS, S3 requests, data transfer and public IPv4 charges. | | [Amazon EMR on EC2](/platforms/aws-emr-athena-iceberg/) | EC2 instance-hours plus an EMR price per instance-hour, each billed per second with a one-minute minimum. | 20 node-hours x ($1.008 EC2 + $0.252 EMR) = 20 x $1.26 = $25.20. | $25.20 | Includes EC2 and the EMR price for ten r6i.4xlarge workers. Excludes the primary (driver) node, EBS volumes, S3 requests, data transfer and public IPv4 charges. | | [Amazon EMR Serverless](/platforms/aws-emr-athena-iceberg/) | Per vCPU-hour and per GB-hour of the workers while they run, plus storage beyond the free ephemeral amount. No instances to manage. | vCPU: 160 x 2 = 320 vCPU-hours x $0.052624 = $16.83968. Memory: 1,280 GB x 2 = 2,560 GB-hours x $0.0057785 = $14.79296. Total $16.83968 + $14.79296 = $31.63264, which rounds to $31.63. | $31.63 | Includes vCPU and memory charges for workers that total 160 vCPU and 1,280 GB for two hours. Excludes extra ephemeral storage (the $0.000111 per GB-hour rate applies only beyond the free amount), the driver, S3 requests and data transfer. Workers are billed only while they run, so a job that scales in and out would be billed for less than the full two hours; that is not modeled. | ## Searched for and none found on 10 Nov 2026 - AWS blog post with TPC-DS method, scale, instance types and total runtime for EMR vs OSS Spark (the guessed blog URL returned 404) (searched: aws.amazon.com/emr/features/spark, docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html, docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance-benchm) - Published scripts for AWS benchmarks (searched: aws.amazon.com/emr/features/spark) ## Amazon EMR plus Athena on Iceberg side by side with each other entry - [Amazon EMR plus Athena on Iceberg and Databricks](/compare/aws-emr-athena-iceberg-vs-databricks/) - [Amazon EMR plus Athena on Iceberg and Dremio](/compare/aws-emr-athena-iceberg-vs-dremio/) - [Amazon EMR plus Athena on Iceberg and DualBird](/compare/aws-emr-athena-iceberg-vs-dualbird/) - [Amazon EMR plus Athena on Iceberg and Google BigQuery and BigLake](/compare/aws-emr-athena-iceberg-vs-bigquery-biglake/) - [Amazon EMR plus Athena on Iceberg and Microsoft Fabric](/compare/aws-emr-athena-iceberg-vs-microsoft-fabric/) - [Amazon EMR plus Athena on Iceberg and Onehouse](/compare/aws-emr-athena-iceberg-vs-onehouse/) - [Amazon EMR plus Athena on Iceberg and Open stack on EMR and EKS](/compare/aws-emr-athena-iceberg-vs-open-stack-emr-eks/) - [Amazon EMR plus Athena on Iceberg and Snowflake](/compare/aws-emr-athena-iceberg-vs-snowflake/) - [Amazon EMR plus Athena on Iceberg and Starburst](/compare/aws-emr-athena-iceberg-vs-starburst/) ## Sources - Amazon EMR plus Athena on Iceberg: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR plus Athena on Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Amazon EMR plus Athena on Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Amazon EMR plus Athena on Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - Amazon EMR plus Athena on Iceberg: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR plus Athena on Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/ind: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026) - Amazon EMR plus Athena on Iceberg: docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026) - Amazon EMR plus Athena on Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html (read 10 Nov 2026) - Amazon EMR plus Athena on Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Amazon EMR plus Athena on Iceberg: docs.aws.amazon.com/glue/latest/dg/enable-compaction.html: https://docs.aws.amazon.com/glue/latest/dg/enable-compaction.html (read 10 Nov 2026) - Amazon EMR plus Athena on Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.js: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - Amazon EMR plus Athena on Iceberg: docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Amazon EMR plus Athena on Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.j: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - AWS Price List API, Amazon EC2, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json (read 10 Nov 2026) - AWS Price List API, Amazon EMR, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/us-east-1/index.json (read 10 Nov 2026) - Amazon EKS pricing page: https://aws.amazon.com/eks/pricing/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/platforms/aws-emr-athena-iceberg/ --- # Databricks, Managed lakehouse built around Delta Lake: the published facts Platform profile, read 10 Nov 2026 **The short answer.** Per Databricks documentation, Databricks runs existing Spark code on its managed runtime, with Photon on by default for classic jobs compute and serverless jobs as an option. Runtime 18 LTS ships Apache Spark 4.1.0 and Runtime 19 ships Spark 4.2.0. Delta UniForm lets Iceberg clients read Delta tables from Runtime 14.3 LTS. - **Kind:** Managed lakehouse built around Delta Lake - **Website:** [www.databricks.com](https://www.databricks.com/) - **Name checked:** Databricks docs and pricing pages still use the name Databricks on AWS. Jobs product is now called Lakeflow Jobs and pipelines are called Lakeflow Pipelines on the vendor pricing and docs pages. - **Read:** 10 Nov 2026 ## What it does well - Per Databricks documentation, Photon keeps the Spark Catalyst planner, works with existing Spark API code and falls back to the Spark runtime for unsupported operations (read 2026-11-10). - Per Databricks documentation, Delta tables can generate Iceberg metadata from Runtime 14.3 LTS, so Iceberg clients read Delta data without rewriting files (read 2026-11-10). ## Where it is limited - Per Databricks documentation, serverless compute supports only Spark Connect APIs, so RDD APIs, R and Maven coordinates are unsupported, and ANSI mode on by default can change results of existing jobs. - Per Databricks, compute is billed in DBUs in addition to the cloud bill for the same instances, so a Jobs Compute run pays Databricks and AWS separately; the Compute Bill line adds $12.24 of DBUs to $20.16 of EC2 (read 2026-11-10). ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | Docs claim Photon gives up to 5x better price/performance for data and analytics workloads versus other cloud data warehouses; no method is given on this page. | vendor docs | [docs.databricks.com/aws/en/compute/photon](https://docs.databricks.com/aws/en/compute/photon), 10 Nov 2026 | | Databricks announced an audited 100TB TPC-DS result of 32,941,245 QphDS for Databricks SQL, 2.2x the previous record held by Alibaba, with total system cost 10% lower on list pricing. This is a SQL warehouse result, not a Spark batch ETL result. | vendor | [databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing](https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html), 10 Nov 2026 | | The same Databricks blog says research from Barcelona Supercomputing Center found Databricks 2.7x faster and 12x better in price-performance than Snowflake on a TPC-DS derivative. This is a vendor-reported claim of a third-party study; the study itself was not fetched. | vendor | [databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing](https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html), 10 Nov 2026 | ## Published prices and cost results | Fact | Source type | Source, read | | --- | --- | --- | | Databricks bills compute in DBUs at per-second granularity with pay-as-you-go or committed-use discounts, and no up-front costs. The pricing pages render their rates by script; the AWS pricing data file those pages load lists Jobs Compute at $0.15 per DBU on the plan it names Premium (r6i.4xlarge: 4.08 DBU per hour). The file names no region. | vendor | [databricks.com/product/aws-pricing](https://www.databricks.com/product/aws-pricing), 10 Nov 2026 | | Jobs pricing offers serverless (Performance Optimized and Standard modes) and classic clusters; the jobs pricing page maps plan names across clouds (its Premium tier on Azure corresponds to Enterprise on AWS and GCP), while the AWS pricing data file lists the $0.15 Jobs Compute rate under a plan it names Premium. Photon instances consume DBUs at a different rate. | vendor | [databricks.com/product/pricing/jobs](https://www.databricks.com/product/pricing/jobs), 10 Nov 2026 | ## Code changes and migration | Fact | Source type | Source, read | | --- | --- | --- | | Photon is a native C++ vectorized engine that keeps Spark's Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime. | vendor docs | [docs.databricks.com/aws/en/compute/photon](https://docs.databricks.com/aws/en/compute/photon), 10 Nov 2026 | | Serverless compute for notebooks and jobs supports only Spark Connect APIs; Spark RDD APIs, R, Maven coordinates, global temp views, DBFS mounts with instance profiles and Databricks Container Services are not supported. ANSI mode is on by default and the default time zone is UTC, which can change results of existing jobs. | vendor docs | [docs.databricks.com/aws/en/compute/serverless/limitations](https://docs.databricks.com/aws/en/compute/serverless/limitations), 10 Nov 2026 | ## Tuning and table upkeep | Fact | Source type | Source, read | | --- | --- | --- | | Photon is on by default for classic all-purpose compute, jobs compute and classic pipelines; API users must set runtime_engine to PHOTON. Photon instance types use DBUs at a different rate than the same instance without Photon. | vendor docs | [docs.databricks.com/aws/en/compute/photon](https://docs.databricks.com/aws/en/compute/photon), 10 Nov 2026 | | On serverless compute Databricks allocates and manages compute so users do not provision clusters; the Spark UI and Spark logs are not available, query profile and client-side logs are used instead. | vendor docs | [docs.databricks.com/aws/en/compute/serverless/limitations](https://docs.databricks.com/aws/en/compute/serverless/limitations), 10 Nov 2026 | | Serverless compute runs workloads without provisioning compute in your cloud account, and Databricks says this reduces the need to manage compute resources. It requires Unity Catalog to reach external data sources. | vendor docs | [docs.databricks.com/aws/en/compute/serverless](https://docs.databricks.com/aws/en/compute/serverless/), 10 Nov 2026 | ## Spark versions and table formats | Fact | Source type | Source, read | | --- | --- | --- | | Databricks Runtime 18 LTS ships Apache Spark 4.1.0 (released June 10, 2026); Runtime 17.3 LTS ships Spark 4.0.0; Runtime 16.4 LTS and 15.4 LTS ship Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). | vendor docs | [docs.databricks.com/aws/en/release-notes/runtime](https://docs.databricks.com/aws/en/release-notes/runtime/), 10 Nov 2026 | | Delta Lake tables can generate Iceberg metadata (UniForm, Iceberg reads) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files. | vendor docs | [docs.databricks.com/aws/en/delta/uniform](https://docs.databricks.com/aws/en/delta/uniform), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Databricks on AWS lists many AWS regions including us-east-1 (US East, Northern Virginia), us-east-2, us-west-1, eu-west-1, eu-central-1, ap-southeast-2 and us-gov-west-1; the platform is also hosted on Azure and Google Cloud. Some features are only in a subset of regions. | vendor docs | [docs.databricks.com/aws/en/resources/supported-regions](https://docs.databricks.com/aws/en/resources/supported-regions), 10 Nov 2026 | ## On the Compute Bill | Line | How it is billed | Formula | Stated cost | What is in and out | | --- | --- | --- | --- | --- | | [Databricks on AWS (Jobs Compute)](/platforms/databricks/) | DBUs consumed per node-hour (rate by instance type) times a price per DBU for the compute type and plan, plus the EC2 bill for the same nodes in the customer's account. | DBUs: 20 node-hours x 4.08 DBU per node-hour = 81.6 DBU. 81.6 x $0.15 = $12.24. EC2: 20 x $1.008 = $20.16. Total $12.24 + $20.16 = $32.40. | $32.40 | Includes the Databricks charge at the Premium plan Jobs Compute rate and the EC2 bill for the same ten r6i.4xlarge nodes, which is paid to AWS separately. Both Databricks figures come from the JSON data file that the Databricks pricing calculator page loads, because the pricing pages themselves are drawn by script; that file does not label a region, and the line assumes it applies to us-east-1. The Enterprise plan rate in the same file is $0.20 per DBU. Excludes the driver node, storage and S3 requests, and the Photon, serverless and committed-use options. | ## Searched for and none found on 10 Nov 2026 - Repeatable scripts for the TPC-DS record (searched: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing) - Spark batch ETL benchmark (not SQL warehouse) with published method (searched: docs.databricks.com/aws/en/compute/photon) - Per-DBU dollar rate for Photon and serverless jobs on AWS us-east-1 in a readable page (the Jobs Compute classic rate was read from the pricing data file) (searched: databricks.com/product/pricing/jobs, databricks.com/product/aws-pricing) ## Databricks side by side with each other entry - [Databricks and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-databricks/) - [Databricks and Dremio](/compare/databricks-vs-dremio/) - [Databricks and DualBird](/compare/databricks-vs-dualbird/) - [Databricks and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-databricks/) - [Databricks and Microsoft Fabric](/compare/databricks-vs-microsoft-fabric/) - [Databricks and Onehouse](/compare/databricks-vs-onehouse/) - [Databricks and Open stack on EMR and EKS](/compare/databricks-vs-open-stack-emr-eks/) - [Databricks and Snowflake](/compare/databricks-vs-snowflake/) - [Databricks and Starburst](/compare/databricks-vs-starburst/) ## Sources - Databricks: docs.databricks.com/aws/en/compute/photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks: docs.databricks.com/aws/en/compute/serverless/limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks: docs.databricks.com/aws/en/compute/serverless: https://docs.databricks.com/aws/en/compute/serverless/ (read 10 Nov 2026) - Databricks: docs.databricks.com/aws/en/release-notes/runtime: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Databricks: docs.databricks.com/aws/en/delta/uniform: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Databricks: docs.databricks.com/aws/en/resources/supported-regions: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Databricks: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Databricks: databricks.com/product/aws-pricing: https://www.databricks.com/product/aws-pricing (read 10 Nov 2026) - Databricks: databricks.com/product/pricing/jobs: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) - AWS Price List API, Amazon EC2, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json (read 10 Nov 2026) - Databricks pricing data file loaded by its pricing calculator, AWS: https://www.databricks.com/en-pricing-assets/data/pricing/AWS.json (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/platforms/databricks/ --- # Dremio, Lakehouse with its own SQL engine: the published facts Platform profile, read 10 Nov 2026 **The short answer.** Dremio Cloud runs SQL on its own Arrow-based engine and manages Iceberg tables in a built-in Open Catalog based on Apache Polaris. It bills $0.20 per Dremio Compute Unit, only while engines run. SAP announced on July 6, 2026 that it completed its acquisition of Dremio. - **Kind:** Lakehouse with its own SQL engine - **Website:** [www.dremio.com](https://www.dremio.com/) - **Name checked:** Still sold as Dremio. SAP agreed to acquire Dremio on May 4, 2026 and completed the acquisition on July 6, 2026. The Dremio website now carries the banner 'Dremio is now part of SAP'. - **Read:** 10 Nov 2026 ## What it does well - Per Dremio documentation, Iceberg tables in the Open Catalog get automatic optimization: data file compaction, row-level delete handling, clustering, partition evolution and manifest rewriting (read 2026-11-10). - Dremio publishes DCU rates by engine size, from 16 DCUs per hour for 2XSmall to 2048 for 3XLarge, with a 60-second minimum (read 2026-11-10). ## Where it is limited - Per Dremio documentation, queries run on its own columnar engine, not on Spark, and the pages read describe no mode that runs existing Spark jobs. - Dremio Cloud documentation lists Iceberg and Delta Lake as table formats; Hudi support: none found on 2026-11-10. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | Dremio's founder reported a full 1 TB TPC-DS run (all 99 queries) in 22 seconds on an 8-node m7gd.4xlarge cluster, with no caching, and attributes it to Autonomous Reflections and automatic Iceberg clustering. This is a BI query benchmark run by the vendor, not a Spark batch ETL or compaction test. | vendor | [dremio.com/blog/breakthrough-announcement-dremio-is-the](https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/), 10 Nov 2026 | ## Published prices and cost results | Fact | Source type | Source, read | | --- | --- | --- | | Dremio Cloud is priced at $0.20 per DCU (Dremio Compute Unit). Dremio Enterprise (self-managed) pricing is 'contact sales'. The free trial for Dremio Cloud is a $400 credit for 30 days. | vendor | [dremio.com/pricing](https://www.dremio.com/pricing), 10 Nov 2026 | | DCUs are consumed only while engines run. Usage equals engine and replica uptime in seconds (60-second minimum) times the DCU rate for the engine size. Published rates per hour: 2XSmall 16, XSmall 32, Small 64, Medium 128, Large 256, XLarge 512, 2XLarge 1024, 3XLarge 2048. Storage billed in TB-months only for Dremio-hosted storage; LLM tokens billed separately. | vendor docs | [docs.dremio.com/dremio-cloud/admin/engines](https://docs.dremio.com/dremio-cloud/admin/engines/), 10 Nov 2026 | | Billable usage is DCUs (compute), LLM tokens (AI features) and storage. Compute is billed in DCUs only while engines are running. | vendor docs | [docs.dremio.com/dremio-cloud/admin/subscription/usage](https://docs.dremio.com/dremio-cloud/admin/subscription/usage), 10 Nov 2026 | ## Code changes and migration | Fact | Source type | Source, read | | --- | --- | --- | | Dremio runs queries on its own columnar execution engine built on Apache Arrow, not on Spark. Docs describe Spark and Flink only as external engines that can use the Open Catalog (Iceberg REST) alongside Dremio. The pages read describe no mode that runs existing Spark jobs, so Spark ETL would stay on Spark or be rewritten as Dremio SQL. | vendor docs | [docs.dremio.com/current/what-is-dremio/architecture](https://docs.dremio.com/current/what-is-dremio/architecture), 10 Nov 2026 | ## Tuning and table upkeep | Fact | Source type | Source, read | | --- | --- | --- | | For Iceberg tables in the Open Catalog, Dremio runs automatic optimization on a dedicated engine it configures: data file compaction, row-level delete handling (position deletes on v2, deletion vectors on v3), clustering, partition evolution and manifest rewriting. The same operations can be run by hand with OPTIMIZE TABLE. Optimization works in batches sized to engine resources. | vendor docs | [docs.dremio.com/dremio-cloud/manage-govern/optimization](https://docs.dremio.com/dremio-cloud/manage-govern/optimization), 10 Nov 2026 | | Engines have settings the admin chooses: size (executors per replica), min and max replicas, last-replica auto-stop (default 2 hours), enqueued time limit (default 5 minutes) and query runtime limit. Default concurrency per replica ranges from 2 (2XSmall) to 20 (3XLarge). | vendor docs | [docs.dremio.com/dremio-cloud/admin/engines](https://docs.dremio.com/dremio-cloud/admin/engines/), 10 Nov 2026 | ## Spark versions and table formats | Fact | Source type | Source, read | | --- | --- | --- | | The Dremio Cloud data formats page lists Apache Iceberg and Delta Lake as table formats (plus Parquet and text formats). Hudi is not listed on that page. The Open Catalog is built on Apache Polaris and supports the Iceberg REST Catalog API; the docs list AWS Glue Data Catalog as a connectable catalog. | vendor docs | [docs.dremio.com/dremio-cloud/developer/data-formats](https://docs.dremio.com/dremio-cloud/developer/data-formats/), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Dremio Cloud runs on AWS (Azure listed as coming soon). Supported regions: us-east-1, us-east-2, us-west-2, eu-west-1 and eu-central-1. Dremio Enterprise can be deployed on Kubernetes, on premises, or in AWS, Azure or GCP. | vendor docs | [docs.dremio.com/dremio-cloud/about/regions](https://docs.dremio.com/dremio-cloud/about/regions), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | Dremio published a 2020 TPC-DS methodology post (SF1000 and SF10000, Parquet on S3, Apache JMeter test plans, 58 of 99 queries that ran unmodified on Dremio and Presto distributions). The 2025 22-second TPC-DS post pages read give cluster type but no scripts, scale-factor files or configuration repository. | vendor | [dremio.com/blog/dremio-benchmarking-methodology](https://www.dremio.com/blog/dremio-benchmarking-methodology/), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | SAP and Dremio announced on May 4, 2026 that SAP agreed to acquire Dremio. Terms were not disclosed and the deal was pending regulatory approval at announcement. SAP said Business Data Cloud will become an Apache Iceberg-native lakehouse using Dremio. | vendor | [news.sap.com/2026/05/sap-to-acquire-dremio](https://news.sap.com/2026/05/sap-to-acquire-dremio/), 10 Nov 2026 | | SAP announced on July 6, 2026 that it completed the acquisition of Dremio. | vendor | [news.sap.com/2026/07/sap-completes-dremio-acquisition](https://news.sap.com/2026/07/sap-completes-dremio-acquisition/), 10 Nov 2026 | | The dremio-oss GitHub repository is licensed Apache-2.0 and its last push was 2025-09-26, so the open-source repository has been quiet while the commercial products (Dremio Cloud, Dremio Enterprise, Dremio Community Edition) continue. | open source project | [api.github.com/repos/dremio/dremio-oss](https://api.github.com/repos/dremio/dremio-oss), 10 Nov 2026 | ## On the Compute Bill | Line | How it is billed | Formula | Stated cost | What is in and out | | --- | --- | --- | --- | --- | | [Dremio Cloud (Large engine)](/platforms/dremio/) | DCUs consumed per hour by engine size while the engine runs, times a price per DCU. | Large engine = 256 DCUs per hour. 256 x 2 hours = 512 DCUs x $0.20 = $102.40. | $102.40 | Includes engine DCUs for two hours of one Large replica. Excludes storage, LLM token charges and data transfer. The engines page lists executors per size but not the vCPU or memory of an executor, so which size equals the reference workload is an assumption. An XLarge engine (16 executors, 512 DCUs per hour) would give $204.80. Dremio Cloud runs SQL in its own engine, not Spark on EC2. The Dremio pricing page returned an access error to the fetch tool on 2026-11-10 and the $0.20 rate was read through a second fetch method. | ## Searched for and none found on 10 Nov 2026 - Published cost for the same Spark batch work (ETL, joins, compaction) on Dremio versus Spark (searched: dremio.com/pricing, docs.dremio.com/dremio-cloud/admin/subscription/usage, dremio.com/post-sitemap.xml) - Apache Hudi table support in Dremio Cloud docs (searched: docs.dremio.com/dremio-cloud/developer/data-formats, docs.dremio.com/llms.txt) - Independent third-party Dremio benchmark (searched: dremio.com/post-sitemap.xml, news.google.com/rss/search?q=Dremio+acquires+OR+acquired+OR+funding) ## Dremio side by side with each other entry - [Dremio and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-dremio/) - [Dremio and Databricks](/compare/databricks-vs-dremio/) - [Dremio and DualBird](/compare/dremio-vs-dualbird/) - [Dremio and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-dremio/) - [Dremio and Microsoft Fabric](/compare/dremio-vs-microsoft-fabric/) - [Dremio and Onehouse](/compare/dremio-vs-onehouse/) - [Dremio and Open stack on EMR and EKS](/compare/dremio-vs-open-stack-emr-eks/) - [Dremio and Snowflake](/compare/dremio-vs-snowflake/) - [Dremio and Starburst](/compare/dremio-vs-starburst/) ## Sources - Dremio: news.sap.com/2026/05/sap-to-acquire-dremio: https://news.sap.com/2026/05/sap-to-acquire-dremio/ (read 10 Nov 2026) - Dremio: news.sap.com/2026/07/sap-completes-dremio-acquisition: https://news.sap.com/2026/07/sap-completes-dremio-acquisition/ (read 10 Nov 2026) - Dremio: api.github.com/repos/dremio/dremio-oss: https://api.github.com/repos/dremio/dremio-oss (read 10 Nov 2026) - Dremio: dremio.com/pricing: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/engines: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/subscription/usage: https://docs.dremio.com/dremio-cloud/admin/subscription/usage (read 10 Nov 2026) - Dremio: docs.dremio.com/current/what-is-dremio/architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/developer/data-formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/manage-govern/optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/about/regions: https://docs.dremio.com/dremio-cloud/about/regions (read 10 Nov 2026) - Dremio: dremio.com/blog/breakthrough-announcement-dremio-is-the: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026) - Dremio: dremio.com/blog/dremio-benchmarking-methodology: https://www.dremio.com/blog/dremio-benchmarking-methodology/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/platforms/dremio/ --- # DualBird, FPGA Spark plugin for AWS F2: the published facts Platform profile, read 10 Nov 2026 **The short answer.** DualBird says its Spark plugin runs sort, join and shuffle work on FPGAs in AWS EC2 F2 instances, and that onboarding is two changes: switch the instance type to F2 and add the plugin. Its one worked example with a blog write-up is a 100 GB sort-based Iceberg compaction run, dated June 11, 2026, with the method behind a form. - **Kind:** FPGA Spark plugin for AWS F2 - **Website:** [dualbird.io](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile) - **Name checked:** Site dualbird.io is live under the name DualBird (written 'Dualbird' in the footer). dualbird.net serves the same site. No rename or acquisition found. - **Read:** 10 Nov 2026 ## What it does well - DualBird says onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin, with Spark still handling planning and scheduling (read 2026-11-10). - DualBird published a 100 GB sort-based Iceberg compaction run on June 11, 2026, run by DualBird, and states about $2 per TB for itself against about $6 per TB for CPU clusters. ## Where it is limited - DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. - A price for the plugin and a benchmark run by a party other than DualBird: none found on 2026-11-10. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. | vendor | [dualbird.io](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. | vendor | [dualbird.io](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' | vendor | [dualbird.io/product](https://dualbird.io/product?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. | vendor | [dualbird.io](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. | vendor | [dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | vendor | [dualbird.io](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Published prices and cost results | Fact | Source type | Source, read | | --- | --- | --- | | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | vendor | [dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Code changes and migration | Fact | Source type | Source, read | | --- | --- | --- | | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. | vendor | [dualbird.io](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | vendor | [dualbird.io/resources/blog/a-simpler-spark](https://dualbird.io/resources/blog/a-simpler-spark?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Tuning and table upkeep | Fact | Source type | Source, read | | --- | --- | --- | | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | vendor | [dualbird.io/product](https://dualbird.io/product?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Spark versions and table formats | Fact | Source type | Source, read | | --- | --- | --- | | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | vendor | [dualbird.io](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). | cloud provider docs | [aws.amazon.com/ec2/instance-types/f2](https://aws.amazon.com/ec2/instance-types/f2/), 10 Nov 2026 | | The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. | cloud provider docs | [docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html), 10 Nov 2026 | | The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | vendor | [dualbird.io/product](https://dualbird.io/product?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | vendor | [dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper](https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Company and breadth | Fact | Source type | Source, read | | --- | --- | --- | | DualBird's own blog (Feb 5, 2026) announces a $25M raise led by Lightspeed Venture Partners with Angular Ventures, Uncork Capital and Bessemer Venture Partners. It says general availability was planned for early 2026 and that early deployments with design partners showed hours-to-minutes gains. The funding is confirmed by the company itself only. Lightspeed's page returned HTTP 403 and no press article was reachable. | vendor | [dualbird.io/resources/blog/dualbird-raises-25m](https://dualbird.io/resources/blog/dualbird-raises-25m?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | Use cases on the home page: AI data infrastructure (feature engineering and AI data prep), big data pipelines, and accelerating data analytics (fresher data for dashboards). The Iceberg compaction benchmark is the one worked example with published method. | vendor | [dualbird.io](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | | About page: founded and led by semiconductor and distributed data systems veterans; positions FPGA-based acceleration as a drop-in Spark plugin on standard AWS instances. Blog posts date from Dec 1, 2025 to Sep 1, 2026. Footer copyright reads 2025. Contact: contact@dualbird.io. | vendor | [dualbird.io/about](https://dualbird.io/about?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=profile), 10 Nov 2026 | ## On the Compute Bill | Line | How it is billed | Formula | Stated cost | What is in and out | | --- | --- | --- | --- | --- | | [DualBird on Amazon EMR (F2 instances)](/platforms/dualbird/) | EC2 F2 instance-hours plus the EMR price for F2 instance-hours. DualBird's plugin price: none found on 2026-11-10. | Infrastructure only, at the same node count and hours as the other lines: 20 node-hours x ($1.98 EC2 + $0.495 EMR) = 20 x $2.475 = $49.50. | $49.50 | Includes only EC2 and EMR charges for ten f2.6xlarge nodes for two hours. The DualBird plugin price: none found on 2026-11-10, so it is not included. Excludes the primary node, EBS, S3 requests and data transfer. This line prices instances at the node count and hours of the reference workload. It does not say that ten F2 nodes are needed, or that fewer would do; DualBird sizing guidance: none found on 2026-11-10, and its benchmarks are its own (see the benchmarks list). DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. | ## Searched for and none found on 10 Nov 2026 - Third-party (non-DualBird) benchmark or review of DualBird by another party (searched: dualbird.io/resources, github.com/dualbird, lsvp.com/dualbird) - Supported Spark versions, Databricks support, table format support list (searched: dualbird.io, dualbird.io/product, dualbird.io/about) - Published price or pricing model for the DualBird plugin (searched: dualbird.io, dualbird.io/product, aws.amazon.com/marketplace/search/results?searchTerms=dualbird) - Named customers (only unnamed design partners) and press coverage of the funding round (searched: dualbird.io, dualbird.io/resources/blog/dualbird-raises-25m, dualbird.io/team) ## DualBird side by side with each other entry - [DualBird and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-dualbird/) - [DualBird and Databricks](/compare/databricks-vs-dualbird/) - [DualBird and Dremio](/compare/dremio-vs-dualbird/) - [DualBird and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-dualbird/) - [DualBird and Microsoft Fabric](/compare/dualbird-vs-microsoft-fabric/) - [DualBird and Onehouse](/compare/dualbird-vs-onehouse/) - [DualBird and Open stack on EMR and EKS](/compare/dualbird-vs-open-stack-emr-eks/) - [DualBird and Snowflake](/compare/dualbird-vs-snowflake/) - [DualBird and Starburst](/compare/dualbird-vs-starburst/) ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/dualbird-raises-25m: https://dualbird.io/resources/blog/dualbird-raises-25m (read 10 Nov 2026) - DualBird: dualbird.io/about: https://dualbird.io/about (read 10 Nov 2026) - AWS Price List API, Amazon EC2, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json (read 10 Nov 2026) - AWS Price List API, Amazon EMR, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/us-east-1/index.json (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/platforms/dualbird/ --- # Google BigQuery and BigLake, Serverless Google Cloud data warehouse: the published facts Platform profile, read 10 Nov 2026 **The short answer.** Google BigQuery is a serverless warehouse on Google Cloud with Iceberg managed tables, BigLake tables over Cloud Storage and S3, and Spark stored procedures that run PySpark or Scala code from SQL. BigQuery Omni reads S3 data in six AWS regions, but it does not support DML or CREATE TABLE. - **Kind:** Serverless Google Cloud data warehouse - **Website:** [cloud.google.com/bigquery](https://cloud.google.com/bigquery) - **Name checked:** BigQuery and BigLake still exist. Dataproc, which runs Spark, is now titled Managed Service for Apache Spark (formerly Dataproc) on its pricing page; Serverless Spark inside BigQuery is documented as stored procedures for Apache Spark. - **Read:** 10 Nov 2026 ## What it does well - Per Google documentation, BigQuery Iceberg managed tables store data in buckets you own, support DML and streaming, and run automatic compaction, clustering and garbage collection (read 2026-11-10). - Per Google documentation, stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud (read 2026-11-10). ## Where it is limited - Per Google documentation, BigQuery Omni on S3 does not support DML or CREATE TABLE, so it does not run write-heavy Spark batch ETL on S3. - Google's pricing page text lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate (read 2026-11-10). Per-TiB on-demand and Spark DCU rates are not used on this site. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | Google states Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name; no method is given on the pricing page. | vendor | [cloud.google.com/dataproc-serverless/pricing](https://cloud.google.com/dataproc-serverless/pricing), 10 Nov 2026 | ## Published prices and cost results | Fact | Source type | Source, read | | --- | --- | --- | | BigQuery offers on-demand pricing per TiB scanned (first 1 TiB per month free) or capacity pricing per slot-hour. BigQuery Omni has its own on-demand per TiB pricing for AWS regions. The page text shows $0.06 per slot-hour for Enterprise edition pay-as-you-go compute, with no region shown beside the rate (read 2026-11-10). | vendor docs | [cloud.google.com/bigquery/pricing](https://cloud.google.com/bigquery/pricing), 10 Nov 2026 | | Google's Spark service (Managed Service for Apache Spark, formerly Dataproc) bills serverless Spark per second in Data Compute Units with a 1-minute minimum, plus shuffle storage; Standard tier is for batch ETL, Premium tier uses Lightning Engine. | vendor docs | [cloud.google.com/dataproc-serverless/pricing](https://cloud.google.com/dataproc-serverless/pricing), 10 Nov 2026 | ## Code changes and migration | Fact | Source type | Source, read | | --- | --- | --- | | On BigQuery Omni, DML statements and CREATE TABLE are not supported, BigQuery ML statements are not supported, and only EXPORT DATA queries are allowed for destination tables, so it does not run write-heavy Spark batch ETL on S3. Standard and Enterprise Plus editions do not support Omni regions. | cloud provider docs | [cloud.google.com/bigquery/docs/omni-introduction](https://cloud.google.com/bigquery/docs/omni-introduction), 10 Nov 2026 | | BigQuery stored procedures for Apache Spark let you run PySpark or Scala/Java JAR code from SQL via CREATE PROCEDURE ... OPTIONS(engine = "SPARK", runtime_version = "2.2"), using a Cloud Storage file or inline code and a connection; this runs on Google Cloud, so code must be wrapped as procedures. | cloud provider docs | [cloud.google.com/bigquery/docs/spark-procedures](https://cloud.google.com/bigquery/docs/spark-procedures), 10 Nov 2026 | ## Tuning and table upkeep | Fact | Source type | Source, read | | --- | --- | --- | | Iceberg managed tables run automatic storage optimization (adaptive file sizing, clustering, garbage collection), so compaction needs no job configuration on Google Cloud; storage optimization is a separate billed item. | cloud provider docs | [cloud.google.com/bigquery/docs/iceberg-tables](https://cloud.google.com/bigquery/docs/iceberg-tables), 10 Nov 2026 | ## Spark versions and table formats | Fact | Source type | Source, read | | --- | --- | --- | | BigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, export Iceberg V2 snapshots for open engines, and do automatic compaction, clustering and garbage collection. | cloud provider docs | [cloud.google.com/bigquery/docs/iceberg-tables](https://cloud.google.com/bigquery/docs/iceberg-tables), 10 Nov 2026 | | BigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security. | cloud provider docs | [cloud.google.com/bigquery/docs/biglake-intro](https://cloud.google.com/bigquery/docs/biglake-intro), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | BigQuery Omni lets BigQuery analytics run on data in Amazon S3 through BigLake tables, in AWS regions us-east-1, us-west-2, ap-northeast-2, ap-southeast-2, eu-west-1 and eu-central-1. | cloud provider docs | [cloud.google.com/bigquery/docs/omni-introduction](https://cloud.google.com/bigquery/docs/omni-introduction), 10 Nov 2026 | ## On the Compute Bill | Line | How it is billed | Formula | Stated cost | What is in and out | | --- | --- | --- | --- | --- | | [BigQuery (Enterprise edition slots)](/platforms/bigquery-biglake/) | Per slot-hour for the edition used, billed per second with a one-minute minimum. A slot is described by Google as a virtual CPU. | 160 slots (one per vCPU of the reference job) x 2 hours = 320 slot-hours. 320 x $0.06 = $19.20. | $19.20 | Includes slot compute only. Excludes storage, and excludes any data movement into or out of AWS. BigQuery runs SQL on Google's slots, not Spark on EC2, so this line prices the same amount of vCPU time, not the same Spark job. The text of the pricing page lists one rate for the Enterprise edition without showing which region the rate belongs to, so the region is not confirmed here. | ## Searched for and none found on 10 Nov 2026 - Dollar rates for BigQuery per TiB, Omni and DCU in readable page text (the Enterprise slot-hour rate, $0.06, was read) (searched: cloud.google.com/bigquery/pricing, cloud.google.com/dataproc-serverless/pricing) - Published scripts for Lightning Engine benchmark (searched: cloud.google.com/dataproc-serverless/pricing) - Spark version supported by BigQuery stored procedures beyond runtime_version 2.2 example (searched: cloud.google.com/bigquery/docs/spark-procedures) ## Google BigQuery and BigLake side by side with each other entry - [Google BigQuery and BigLake and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-bigquery-biglake/) - [Google BigQuery and BigLake and Databricks](/compare/bigquery-biglake-vs-databricks/) - [Google BigQuery and BigLake and Dremio](/compare/bigquery-biglake-vs-dremio/) - [Google BigQuery and BigLake and DualBird](/compare/bigquery-biglake-vs-dualbird/) - [Google BigQuery and BigLake and Microsoft Fabric](/compare/bigquery-biglake-vs-microsoft-fabric/) - [Google BigQuery and BigLake and Onehouse](/compare/bigquery-biglake-vs-onehouse/) - [Google BigQuery and BigLake and Open stack on EMR and EKS](/compare/bigquery-biglake-vs-open-stack-emr-eks/) - [Google BigQuery and BigLake and Snowflake](/compare/bigquery-biglake-vs-snowflake/) - [Google BigQuery and BigLake and Starburst](/compare/bigquery-biglake-vs-starburst/) ## Sources - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/omni-introduction: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/spark-procedures: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/iceberg-tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/biglake-intro: https://cloud.google.com/bigquery/docs/biglake-intro (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/dataproc-serverless/pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/platforms/bigquery-biglake/ --- # Microsoft Fabric, SaaS analytics on Azure with Spark: the published facts Platform profile, read 10 Nov 2026 **The short answer.** Microsoft Fabric is a SaaS analytics platform on Azure with a Spark lakehouse over OneLake and capacity pricing. The Azure retail price API lists Spark usage at $0.18 per capacity-unit hour in eastus. OneLake S3 shortcuts give read-only access to data in S3 buckets without copying it. - **Kind:** SaaS analytics on Azure with Spark - **Website:** [www.microsoft.com/en-us/microsoft-fabric](https://www.microsoft.com/en-us/microsoft-fabric) - **Name checked:** Microsoft Fabric still exists under this name. It runs on Azure (OneLake is built on Azure Data Lake Storage), not on AWS; AWS S3 data is reached only through OneLake shortcuts. - **Read:** 10 Nov 2026 ## What it does well - Per Microsoft Learn, the native execution engine (Apache Gluten and Velox) needs no code changes, falls back to JVM Spark for unsupported operators and supports Parquet and Delta (read 2026-11-10). - Per Microsoft Learn, OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying it (read 2026-11-10). ## Where it is limited - Per Microsoft Learn, Fabric runs over OneLake on Azure and the region page lists Azure regions only, so Spark compute does not run inside AWS. - Writes to S3 through a OneLake shortcut are not possible, and the method for Microsoft's native engine benchmark: none found on 2026-11-10. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | Microsoft reports that on TPC-DS scale factor 1000 with Delta the native engine was up to six times faster than open-source Spark, with about 83% compute cost savings on a fixed-size cluster. | vendor | [learn.microsoft.com/en-us/fabric/data-engineering/runtime](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime), 10 Nov 2026 | ## Published prices and cost results | Fact | Source type | Source, read | | --- | --- | --- | | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus (Spark Memory Optimized and autoscale for Spark entries, pay-as-you-go). | vendor docs | [prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft](https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'), 10 Nov 2026 | | In Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling. | vendor docs | [learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-](https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing), 10 Nov 2026 | | Autoscale Billing for Spark is an opt-in pay-as-you-go model that runs Spark on dedicated serverless resources at standard PAYG CU-hour rates; reserved capacity discounts do not apply and a base capacity is still required. | vendor docs | [azure.microsoft.com/en-us/pricing/details/microsoft-fabric](https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/), 10 Nov 2026 | ## Code changes and migration | Fact | Source type | Source, read | | --- | --- | --- | | The Fabric native execution engine, based on Apache Gluten and Velox, needs no code changes, falls back to JVM Spark for unsupported operators, and supports Parquet and Delta. | vendor docs | [learn.microsoft.com/en-us/fabric/data-engineering/runtime](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime), 10 Nov 2026 | ## Tuning and table upkeep | Fact | Source type | Source, read | | --- | --- | --- | | Starter pools start in 5 to 10 seconds with no setup; custom Spark pools let you tune node size and scaling, and native execution can be toggled per application with a Spark setting. | vendor docs | [learn.microsoft.com/en-us/fabric/data-engineering/spark-compute](https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute), 10 Nov 2026 | ## Spark versions and table formats | Fact | Source type | Source, read | | --- | --- | --- | | Fabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1; OneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and vice versa. | vendor docs | [learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables](https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Fabric workloads run over OneLake, described as built on Azure Data Lake Storage; the region availability page lists Azure regions and no AWS regions, so Spark compute does not run inside AWS. | vendor docs | [learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview](https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview), 10 Nov 2026 | | OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; file caching can reduce cross-cloud egress costs. Writes to S3 are not possible through the shortcut. | vendor docs | [learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut](https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut), 10 Nov 2026 | ## On the Compute Bill | Line | How it is billed | Formula | Stated cost | What is in and out | | --- | --- | --- | --- | --- | | [Microsoft Fabric (F128 capacity)](/platforms/microsoft-fabric/) | Pay-as-you-go capacity units (CU) per hour for the whole capacity SKU while it is running, not per job. One CU maps to two Spark VCores, per Microsoft. | 160 vCPU treated as 160 Spark VCores = 80 CU needed. The next SKU down, F64, has 64 CU, which is fewer than the 80 CU needed, so F64 is too small and the smallest F SKU with at least 80 CU is F128 (128 CU). 128 CU x 2 hours = 256 CU-hours x $0.18 = $46.08. | $46.08 | Includes the F128 capacity for two hours. F128 is billed in full even though 80 CU are used, because F64 (64 CU) is too small for the stated mapping. Excludes OneLake storage, the Power BI license items Microsoft lists separately, and any pause and resume overhead. Fabric runs on Azure, not AWS, so this line prices the Azure US East capacity for the same vCPU count. The Fabric pricing page showed no rate in its text on 2026-11-10; the rate comes from the Azure Retail Prices API. | ## Searched for and none found on 10 Nov 2026 - Pay-as-you-go F SKU dollar price from the Azure pricing page (table renders as $-) (searched: azure.microsoft.com/en-us/pricing/details/microsoft-fabric) - Method and scripts for the native engine benchmark (searched: learn.microsoft.com/en-us/fabric/data-engineering/runtime, learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engin) ## Microsoft Fabric side by side with each other entry - [Microsoft Fabric and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-microsoft-fabric/) - [Microsoft Fabric and Databricks](/compare/databricks-vs-microsoft-fabric/) - [Microsoft Fabric and Dremio](/compare/dremio-vs-microsoft-fabric/) - [Microsoft Fabric and DualBird](/compare/dualbird-vs-microsoft-fabric/) - [Microsoft Fabric and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-microsoft-fabric/) - [Microsoft Fabric and Onehouse](/compare/microsoft-fabric-vs-onehouse/) - [Microsoft Fabric and Open stack on EMR and EKS](/compare/microsoft-fabric-vs-open-stack-emr-eks/) - [Microsoft Fabric and Snowflake](/compare/microsoft-fabric-vs-snowflake/) - [Microsoft Fabric and Starburst](/compare/microsoft-fabric-vs-starburst/) ## Sources - Microsoft Fabric: learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview: https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - Microsoft Fabric: prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption' (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Microsoft Fabric: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-compute: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute (read 10 Nov 2026) - Azure Retail Prices API, Microsoft Fabric, eastus: https://prices.azure.com/api/retail/prices?$filter=serviceName eq 'Microsoft Fabric' and armRegionName eq 'eastus' (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/platforms/microsoft-fabric/ --- # Onehouse, Managed lakehouse in your cloud account: the published facts Platform profile, read 10 Nov 2026 **The short answer.** Onehouse runs Spark and SQL jobs on its Quanton engine in your own AWS or GCP account, and the jobs read and write Hudi, Iceberg and Delta Lake. It bills in Onehouse Compute Units, where 1 OCU is a 4-core instance for 1 hour. Onehouse says its founder created Apache Hudi. - **Kind:** Managed lakehouse in your cloud account - **Website:** [www.onehouse.ai](https://www.onehouse.ai/) - **Name checked:** Onehouse still trades under this name. Its Spark and SQL engine is called Quanton and runs on the Onehouse Compute Runtime (announced May 20, 2025). The table format translation project OneTable was renamed Apache XTable and is incubating at the Apache Software Foundation. - **Read:** 10 Nov 2026 ## What it does well - Per Onehouse documentation, its jobs are fully compatible with Apache Spark and EMR code can be reused unless it uses AWS-specific features (read 2026-11-10). - Onehouse published the setup for its 10 TB TPC-DS and 1 TB LakeLoader comparison against EMR 7.12, and open-sourced LakeLoader under Apache-2.0 (read 2026-11-10). ## Where it is limited - Per Onehouse documentation, billing is in OCUs on a bring-your-own-cloud model, and no dollar rate per OCU was found on 2026-11-10. - Onehouse's results against EMR 7.12 are its own and shown as charts; a confirmation by another party: none found on 2026-11-10. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | Onehouse benchmarked Quanton against open-source Spark on Iceberg: TPC-DS up to about 3.1x, TPCx-BB 2x, TPC-DI (1 TB) 2x with 3-4x on MERGE INTO, and LakeLoader 4.1x for writes. The baseline is open-source Spark, not EMR. | vendor | [onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-worklo](https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads), 10 Nov 2026 | | Onehouse reran its benchmarks against EMR 7.12 after AWS announced it: it reports EMR improved 32% over earlier versions and that Onehouse 'still comes out with a 2.5x better price/performance advantage'. Setup: 10 TB TPC-DS for read/transform, 1 TB LakeLoader merges for load, latest general-purpose Graviton instances. Results are shown as charts; Onehouse also reports Hudi and Delta query performance on EMR 7.12 was very similar to Iceberg. | vendor | [onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark](https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads), 10 Nov 2026 | | A B2B SaaS customer (unnamed) ran its UDF-heavy pipeline on EMR and on Quanton in its own AWS account on m8gd nodes. Quanton was faster at p50, p97 and p99 tenant sizes; on the largest tenant (21 billion rows) EMR filled local disks on 180 nodes and was aborted after 3.7 hours, and finished on 300 nodes. The write-up is published by Onehouse. | vendor | [onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs](https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs), 10 Nov 2026 | ## Published prices and cost results | Fact | Source type | Source, read | | --- | --- | --- | | Onehouse bills in Onehouse Compute Units (OCU) on a bring-your-own-cloud model: 1 OCU is a 4-core instance for 1 hour in your account, plus support costs; no separate volume or storage fee, and the customer pays its own AWS or GCP infrastructure bill. The docs page does not state a dollar rate per OCU. | vendor docs | [docs.onehouse.ai/product/project-settings/usage-and-billing](https://docs.onehouse.ai/product/project-settings/usage-and-billing), 10 Nov 2026 | | Vendor claims at the Quanton launch: 2-3x better price-performance for ETL, and a guaranteed greater-than-50% cost reduction for existing Hudi users moving Spark jobs from AWS EMR or open-source Spark. These are guarantees stated by the vendor, not results from an independent party. | vendor | [onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtim](https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton), 10 Nov 2026 | ## Code changes and migration | Fact | Source type | Source, read | | --- | --- | --- | | Onehouse docs say its Jobs are fully compatible with Apache Spark and that EMR code can be reused unless it uses AWS-specific features. Migration work listed: map EMR steps to Onehouse Jobs created through its API or Airflow operators (CREATE CLUSTER, CREATE JOB, RUN JOB, DELETE CLUSTER), and configure Glue catalog sync or contact support to make Glue the primary catalog. | vendor docs | [docs.onehouse.ai/product/jobs/emr-migration](https://docs.onehouse.ai/product/jobs/emr-migration), 10 Nov 2026 | | At the May 20, 2025 launch Onehouse said users can run existing Spark and SQL jobs on new cluster types and that existing Hudi/Spark jobs can be pointed at Onehouse 'with no code changes'. The Quanton Kubernetes Operator is described as a 'drop-in replacement' for an existing Spark Kubernetes operator. | vendor | [onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtim](https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton), 10 Nov 2026 | ## Tuning and table upkeep | Fact | Source type | Source, read | | --- | --- | --- | | Each cluster needs min and max OCU limits. Docs warn that if usage reaches Max OCU the cluster stops scaling up, which can delay processing; Min OCU keeps a cluster warm. The Onehouse docs pages read give no per-job Spark tuning list; Onehouse claims automatic scaling and automated async table services. | vendor docs | [docs.onehouse.ai/product/compute/create-cluster](https://docs.onehouse.ai/product/compute/create-cluster), 10 Nov 2026 | ## Spark versions and table formats | Fact | Source type | Source, read | | --- | --- | --- | | Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake. Docs: Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog (AWS Glue and Snowflake Open Catalog are documented); for Delta Lake the user installs Delta Spark as a dependency in the JAR or Python job. In the Nov 12, 2025 Iceberg post Onehouse said Delta acceleration is 'up next'. | vendor docs | [docs.onehouse.ai/product/jobs](https://docs.onehouse.ai/product/jobs/), 10 Nov 2026 | | Apache XTable (formerly OneTable) translates table metadata between Hudi, Delta Lake and Iceberg without being a new format. It is incubating at the Apache Software Foundation; Onehouse says it created Hudi and XTable. | open source project | [xtable.apache.org](https://xtable.apache.org/), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Onehouse runs EC2 or Google Compute Engine instances inside the customer's AWS or GCP account (Azure not listed in the billing docs). Standard instance types map to Graviton m8g sizes on AWS (oh-general-4 = m8g.xlarge, oh-general-8 = m8g.2xlarge, oh-general-16 = m8g.4xlarge). A project is scoped to one region and account; adding a region needs a support ticket. | vendor docs | [docs.onehouse.ai/product/project-settings/usage-and-billing](https://docs.onehouse.ai/product/project-settings/usage-and-billing), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | Onehouse open-sourced LakeLoader, a tool that generates incremental update/insert loads for benchmarking write performance. The repository is Apache-2.0 licensed, last pushed 2026-10-30. The Quanton engine itself and the full Iceberg benchmark configurations were not found as a repository in the pages read. | open source project | [api.github.com/repos/onehouseinc/lake-loader](https://api.github.com/repos/onehouseinc/lake-loader), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | Onehouse was founded by Vinoth Chandar, creator and PMC chair of Apache Hudi, who built the first data lakehouse at Uber. Its product claims also include a Table Optimizer, OneSync catalog sync, and a Quanton Kubernetes Operator to run Quanton Spark on the customer's own Kubernetes. | vendor | [onehouse.ai/about-us](https://www.onehouse.ai/about-us), 10 Nov 2026 | ## On the Compute Bill | Line | How it is billed | Formula | Stated cost | What is in and out | | --- | --- | --- | --- | --- | | [Onehouse (Quanton, bring-your-own-cloud)](/platforms/onehouse/) | Onehouse Compute Units (OCU), where 1 OCU is a 4-core instance for 1 hour in the customer's account, plus support costs. The customer also pays AWS for the instances. | OCU quantity: 160 vCPU / 4 cores = 40 OCU per hour x 2 hours = 80 OCU. Price per OCU: None published on 2026-11-10, so the Onehouse charge is not computed. AWS instance part, if the same r6i.4xlarge nodes were used: 20 x $1.008 = $20.16. | No total | None published on 2026-11-10: a dollar price per OCU. The Onehouse pricing URL returned HTTP 404 and the billing documentation states the OCU definition but no dollar rate. The $1.008 EC2 price is shown only to size the AWS part; Onehouse's documentation lists its own standard instance types (Graviton m8g sizes), so the instances actually used would differ. Support costs are also unpublished. | ## Searched for and none found on 10 Nov 2026 - Dollar price per OCU or a published list price for Onehouse (searched: onehouse.ai/pricing, (404), docs.onehouse.ai/product/project-settings/usage-and-billing) - Independent third-party confirmation of Quanton benchmarks (searched: news.google.com/rss/search?q=Onehouse+Quanton, news.google.com/rss/search?q=Onehouse+funding+Hudi) - Spark versions supported by Quanton clusters (searched: docs.onehouse.ai/product/jobs, docs.onehouse.ai/product/compute/create-cluster, docs.onehouse.ai/product/dataplane-setup/prerequisites/aws) ## Onehouse side by side with each other entry - [Onehouse and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-onehouse/) - [Onehouse and Databricks](/compare/databricks-vs-onehouse/) - [Onehouse and Dremio](/compare/dremio-vs-onehouse/) - [Onehouse and DualBird](/compare/dualbird-vs-onehouse/) - [Onehouse and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-onehouse/) - [Onehouse and Microsoft Fabric](/compare/microsoft-fabric-vs-onehouse/) - [Onehouse and Open stack on EMR and EKS](/compare/onehouse-vs-open-stack-emr-eks/) - [Onehouse and Snowflake](/compare/onehouse-vs-snowflake/) - [Onehouse and Starburst](/compare/onehouse-vs-starburst/) ## Sources - Onehouse: docs.onehouse.ai/product/jobs/emr-migration: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Onehouse: onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtim: https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton (read 10 Nov 2026) - Onehouse: docs.onehouse.ai/product/project-settings/usage-and-billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Onehouse: docs.onehouse.ai/product/jobs: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - Onehouse: xtable.apache.org: https://xtable.apache.org/ (read 10 Nov 2026) - Onehouse: docs.onehouse.ai/product/compute/create-cluster: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Onehouse: onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-worklo: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026) - Onehouse: onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark: https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads (read 10 Nov 2026) - Onehouse: onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - Onehouse: api.github.com/repos/onehouseinc/lake-loader: https://api.github.com/repos/onehouseinc/lake-loader (read 10 Nov 2026) - Onehouse: onehouse.ai/about-us: https://www.onehouse.ai/about-us (read 10 Nov 2026) - AWS Price List API, Amazon EC2, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/platforms/onehouse/ --- # Open stack on EMR and EKS, Self-run Spark with open table formats: the published facts Platform profile, read 10 Nov 2026 **The short answer.** The open stack is Apache Spark on Amazon EMR (EC2, EKS or Serverless) or self-run on EKS, with Iceberg, Delta Lake or Hudi tables you maintain yourself. EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2, Iceberg 1.10.1 and Hudi 1.0.2. AWS prices the EMR uplift in us-east-1. - **Kind:** Self-run Spark with open table formats - **Website:** [spark.apache.org](https://spark.apache.org/) - **Name checked:** Not a vendor. Components are current: Amazon EMR release 7.14.0 ships Spark 3.5.8-amzn-0, Iceberg 1.10.1-amzn-1, Delta 3.3.2-amzn-3 and Hudi 1.0.2-amzn-3; Iceberg docs show latest 1.12.0; the Kubeflow Spark Operator (Apache-2.0) was pushed on 2026-11-10. - **Read:** 10 Nov 2026 ## What it does well - Per AWS documentation, EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three formats are available (read 2026-11-10). - AWS ran TPC-DS 3 TB on 9 x r5d.4xlarge and reports $4.81 for EMR 7.12 against $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0, with repositories to repeat it. ## Where it is limited - Per the Apache Iceberg documentation, users run table maintenance themselves: expire snapshots, remove orphan files and compact with rewriteDataFiles as jobs they schedule (read 2026-11-10). - A cost comparison of open-stack Spark against managed lakehouses by a party other than AWS or a vendor: none found on 2026-11-10. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | AWS ran TPC-DS 3 TB (104 SparkSQL queries, 3 rounds, 9 x r5d.4xlarge, Hadoop catalog) and reports EMR 7.12 averaged 1,349.62 s versus 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0 and 1,535.62 s for EMR 7.5. AWS notes results are not comparable to official TPC-DS. The run is by the vendor of EMR. | vendor | [aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t](https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/), 10 Nov 2026 | ## Published prices and cost results | Fact | Source type | Source, read | | --- | --- | --- | | EMR on EC2 charges an EMR price in addition to the EC2 price (and EBS), billed per second with a one-minute minimum. AWS's example in us-east-1: c4.2xlarge EMR charge $0.105/hour next to EC2 $0.398/hour; one master and two core nodes for a month total $1,101.57. Spot instances are up to a 90% discount versus On-Demand. | cloud provider docs | [aws.amazon.com/emr/pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | EMR on EKS adds an uplift per vCPU-hour and per GB-hour to EKS/EC2 or Fargate costs, plus $0.10 per hour for each EKS cluster. AWS's worked example uses $0.007084 per vCPU-hour and $0.000777875 per GB-hour (100 vCPU and 300 GB for 30 minutes = $0.4709 EMR uplift). The uplift applies to any pod using an EMR runtime image, including with the Spark operator. | cloud provider docs | [aws.amazon.com/emr/pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | EMR Serverless is billed per vCPU-hour and GB-hour. AWS's worked example uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour for a job with 100 vCPU and 750 GB for 30 minutes, then 200 more vCPU for 15 minutes, totaling $9.596275. | cloud provider docs | [aws.amazon.com/emr/pricing](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | In the same AWS benchmark the cost was $4.81 for EMR 7.12, $5.47 for EMR 7.5 and $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0. Cost formula uses r5d.4xlarge at $1.152/hour plus $0.27/hour EMR plus EBS, On-Demand. | vendor | [aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t](https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/), 10 Nov 2026 | ## Code changes and migration | Fact | Source type | Source, read | | --- | --- | --- | | The EMR runtime for Spark provides 'full API compatibility with open source Apache Spark and Apache Iceberg', so existing Spark jobs run as is. Optimized runtimes are used by EMR on EC2, EMR Serverless, EMR on EKS, EMR on Outposts and AWS Glue. Self-run Spark on EKS uses the open-source Spark operator and needs no job rewrite. | vendor | [aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t](https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/), 10 Nov 2026 | ## Tuning and table upkeep | Fact | Source type | Source, read | | --- | --- | --- | | Iceberg docs say users must run maintenance: expire snapshots, remove old metadata files and delete orphan files are 'recommended'; compacting data files with the Spark rewriteDataFiles action (for example target-file-size-bytes of 500 MB) and rewriting manifests are 'optional' maintenance, run as jobs you schedule. | open source project | [iceberg.apache.org/docs/latest/maintenance](https://iceberg.apache.org/docs/latest/maintenance/), 10 Nov 2026 | | Hudi merge-on-read tables need compaction to merge log files into base files and bound read latency. Compaction has two steps, scheduling (plan written to the timeline) and execution; the docs describe strategies for choosing which file slices to compact. | open source project | [hudi.apache.org/docs/compaction](https://hudi.apache.org/docs/compaction), 10 Nov 2026 | | Delta Lake OSS docs list file management features the user applies: compaction (bin-packing) with OPTIMIZE, auto compaction, data skipping, Z-ordering, multi-part checkpointing, log compactions and optimized write. | open source project | [docs.delta.io/latest/optimizations-oss.html](https://docs.delta.io/latest/optimizations-oss.html), 10 Nov 2026 | ## Spark versions and table formats | Fact | Source type | Source, read | | --- | --- | --- | | Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. All three table formats are available on EMR. | cloud provider docs | [docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html), 10 Nov 2026 | | EMR on EKS supports the Kubernetes Operator for Apache Spark (Spark operator) as a job submission model with the EMR runtime on your own EKS clusters. The Kubeflow Spark Operator is Apache-2.0 licensed with Helm installation. | cloud provider docs | [docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operato](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Amazon EMR has regional endpoints in the US (us-east-1, us-east-2, us-west-1, us-west-2) and many other AWS regions per the AWS general reference endpoints table. Spark runs on EC2 instances the user chooses (the AWS benchmark used r5d.4xlarge). | cloud provider docs | [docs.aws.amazon.com/general/latest/gr/emr.html](https://docs.aws.amazon.com/general/latest/gr/emr.html), 10 Nov 2026 | ## How open its benchmarks are | Fact | Source type | Source, read | | --- | --- | --- | | AWS publishes the setup in GitHub repositories (aws-samples/emr-on-eks-benchmark branch tpcds-v2.13_iceberg, and aws-samples/emr-spark-benchmark) with build instructions, so the TPC-DS Iceberg benchmark can be repeated. | vendor | [github.com/aws-samples/emr-on-eks-benchmark](https://github.com/aws-samples/emr-on-eks-benchmark), 10 Nov 2026 | ## On the Compute Bill | Line | How it is billed | Formula | Stated cost | What is in and out | | --- | --- | --- | --- | --- | | [Open stack: Spark on plain EC2](/platforms/open-stack-emr-eks/) | EC2 instance-hours only. There is no managed-service charge; the team runs open-source Spark itself. | 10 nodes x 2 hours = 20 node-hours. 20 x $1.008 = $20.16. | $20.16 | Includes EC2 on-demand compute for the ten workers. Excludes the driver node, EBS volumes, S3 requests, data transfer, and the engineering time to operate Spark, Iceberg and the cluster manager. If the same open stack ran on a self-managed EKS cluster, AWS would add $0.10 per hour for each EKS cluster (Amazon EKS pricing page, read 2026-11-10). That fee is excluded here, as on the EMR on EKS line, because it depends on how long the cluster stays up, not on the two-hour job. | ## Searched for and none found on 10 Nov 2026 - Independent third-party cost comparison (by a named party that is not AWS, Onehouse or another vendor) of open-stack Spark on EMR or EKS against managed lakehouses (searched: news.google.com/rss/search?q=EMR+7.12+Spark+performance+Iceberg+AWS+big+, awslabs.github.io/data-on-eks/docs/blueprints/data-analytics/spark-opera, (404)) - Published benchmark of Iceberg, Delta or Hudi compaction time or cost on EMR/EKS (searched: iceberg.apache.org/docs/latest/maintenance, hudi.apache.org/docs/compaction, docs.delta.io/latest/optimizations-oss.html) ## Open stack on EMR and EKS side by side with each other entry - [Open stack on EMR and EKS and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and Databricks](/compare/databricks-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and Dremio](/compare/dremio-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and DualBird](/compare/dualbird-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and Microsoft Fabric](/compare/microsoft-fabric-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and Onehouse](/compare/onehouse-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and Snowflake](/compare/open-stack-emr-eks-vs-snowflake/) - [Open stack on EMR and EKS and Starburst](/compare/open-stack-emr-eks-vs-starburst/) ## Sources - Open stack on EMR and EKS: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Open stack on EMR and EKS: aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Open stack on EMR and EKS: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Open stack on EMR and EKS: docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operato: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) - Open stack on EMR and EKS: iceberg.apache.org/docs/latest/maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Open stack on EMR and EKS: hudi.apache.org/docs/compaction: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Open stack on EMR and EKS: docs.delta.io/latest/optimizations-oss.html: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Open stack on EMR and EKS: docs.aws.amazon.com/general/latest/gr/emr.html: https://docs.aws.amazon.com/general/latest/gr/emr.html (read 10 Nov 2026) - Open stack on EMR and EKS: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - AWS Price List API, Amazon EC2, us-east-1 (published 2026-11-10): https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/platforms/open-stack-emr-eks/ --- # Snowflake, Cloud data platform with PySpark support: the published facts Platform profile, read 10 Nov 2026 **The short answer.** Per Snowflake documentation, Snowflake runs existing PySpark DataFrame code on its own warehouses through Snowpark Connect for Apache Spark, with no Spark cluster to manage. Snowpark Connect supports Spark 3.5 only. Warehouses bill credits per second with a 60-second minimum, and Iceberg tables can sit in customer-managed S3 storage. - **Kind:** Cloud data platform with PySpark support - **Website:** [www.snowflake.com](https://www.snowflake.com/) - **Name checked:** Snowflake docs still use the name Snowflake. Snowpark Connect for Apache Spark exists and is documented. - **Read:** 10 Nov 2026 ## What it does well - Per Snowflake documentation, Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect, and batch jobs are submitted with the Snowpark Submit CLI (read 2026-11-10). - Per Snowflake documentation, Iceberg tables can keep data in customer-managed S3, GCS or Azure storage through external volumes and a catalog integration such as AWS Glue. ## Where it is limited - Per Snowflake documentation, Snowpark Connect supports Apache Spark 3.5 only, and DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported (read 2026-11-10). - On-demand credit prices on AWS US East (Northern Virginia) are $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical) per the Snowflake Service Consumption Table effective 2026-11-09 (read 2026-11-10); the Snowflake pricing page itself renders by script. ## Published prices and cost results | Fact | Source type | Source, read | | --- | --- | --- | | Warehouses bill credits per second with a 60-second minimum on each start; each size step roughly doubles credits per hour (Small is 2 credits per hour, Medium is 4). Snowflake's Service Consumption Table (effective 2026-11-09) lists on-demand credits on AWS US East at $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical). | vendor docs | [docs.snowflake.com/en/user-guide/cost-understanding-compute](https://docs.snowflake.com/en/user-guide/cost-understanding-compute), 10 Nov 2026 | ## Code changes and migration | Fact | Source type | Source, read | | --- | --- | --- | | Snowpark Connect for Apache Spark runs existing PySpark DataFrame code against the Snowflake engine over the Spark Connect protocol; there is no Spark cluster, all workloads run on a Snowflake warehouse. Batch jobs are submitted with the Snowpark Submit CLI. | vendor docs | [docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark), 10 Nov 2026 | | Because Snowflake rather than Spark is the engine, there are semantic differences: DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported, and ByteType, ShortType and IntegerType are returned as LongType. The docs list unsupported APIs for DataFrame and Dataset. | vendor docs | [docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md), 10 Nov 2026 | | Docs say the Spark DataFrame and Spark SQL code can be reused with minimal migration or changes; the compatibility notes say Spark Connect defers plan resolution, which changes behavior versus Spark Classic. | vendor docs | [docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark), 10 Nov 2026 | ## Tuning and table upkeep | Fact | Source type | Source, read | | --- | --- | --- | | Docs provide a separate Optimizing Snowpark Connect for Spark workloads section and a Troubleshooting section covering known limitations, so some job-level adjustment is expected after migration. | vendor docs | [docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md), 10 Nov 2026 | ## Spark versions and table formats | Fact | Source type | Source, read | | --- | --- | --- | | Snowpark Connect supports Apache Spark 3.5 only; workloads written for 3.4 and below or 4.0 and above are not supported and may produce protocol errors or missing API failures. | vendor docs | [docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md), 10 Nov 2026 | | Snowflake supports Iceberg tables (Parquet) with data in customer-managed S3, GCS or Azure storage via external volumes, an external catalog integration such as AWS Glue, and Delta files read through Delta Direct; Hudi was not listed on the page. | vendor docs | [docs.snowflake.com/en/user-guide/tables-iceberg](https://docs.snowflake.com/en/user-guide/tables-iceberg), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Snowflake runs on AWS, Azure and Google Cloud; AWS regions include us-east-1, us-west-2, eu-west-1 and ca-central-1. Each account is hosted in a single region. | vendor docs | [docs.snowflake.com/en/user-guide/intro-regions.md](https://docs.snowflake.com/en/user-guide/intro-regions.md), 10 Nov 2026 | ## On the Compute Bill | Line | How it is billed | Formula | Stated cost | What is in and out | | --- | --- | --- | --- | --- | | [Snowflake (XL standard warehouse)](/platforms/snowflake/) | Credits per warehouse-hour by size, billed per second with a 60-second minimum, times a price per credit by edition, cloud and region. | XL warehouse = 16 credits per hour. 16 x 2 hours = 32 credits x $3.00 = $96.00. | $96.00 | Includes warehouse credits for two hours of an XL standard warehouse at the Enterprise on-demand rate. Excludes storage, cloud services credits beyond the daily allowance, and data transfer. The table lists credits per hour for each size but not the vCPU or memory behind each size, so which size equals the reference workload is an assumption. For the same hours, the Standard edition ($2.00 per credit) gives $64.00 and Business Critical ($4.00) gives $128.00; a 2XL warehouse (32 credits per hour) doubles each figure. Snowflake runs SQL and Snowpark in its own engine, not Spark on EC2. | ## Searched for and none found on 10 Nov 2026 - Published benchmark for Spark batch jobs on Snowpark Connect with method (searched: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-, docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-) - Repeatable benchmark scripts (searched: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-) ## Snowflake side by side with each other entry - [Snowflake and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-snowflake/) - [Snowflake and Databricks](/compare/databricks-vs-snowflake/) - [Snowflake and Dremio](/compare/dremio-vs-snowflake/) - [Snowflake and DualBird](/compare/dualbird-vs-snowflake/) - [Snowflake and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-snowflake/) - [Snowflake and Microsoft Fabric](/compare/microsoft-fabric-vs-snowflake/) - [Snowflake and Onehouse](/compare/onehouse-vs-snowflake/) - [Snowflake and Open stack on EMR and EKS](/compare/open-stack-emr-eks-vs-snowflake/) - [Snowflake and Starburst](/compare/snowflake-vs-starburst/) ## Sources - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/cost-understanding-compute: https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/tables-iceberg: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/intro-regions.md: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md (read 10 Nov 2026) - Snowflake Service Consumption Table, effective 2026-11-09: https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/platforms/snowflake/ --- # Starburst, Managed Trino service for Iceberg: the published facts Platform profile, read 10 Nov 2026 **The short answer.** Starburst describes Galaxy as a managed data lake analytics platform built on Trino. It creates and writes Iceberg tables and reads Delta Lake and Hudi tables. Credit prices on AWS in US East (N. Virginia) start at $0.50 for the Pro tier. Work runs as Trino SQL, not as Spark jobs. - **Kind:** Managed Trino service for Iceberg - **Website:** [www.starburst.io](https://www.starburst.io/) - **Name checked:** Still sold as Starburst; Galaxy is the managed product and Starburst Enterprise Platform the self-managed one. No acquisition of Starburst found in the news feed checked on 2026-11-10. Starburst's August 2026 blog comments on SAP's acquisition of Dremio. - **Read:** 10 Nov 2026 ## What it does well - Per Starburst documentation, Galaxy creates and writes Iceberg tables (Parquet, ORC or Avro) and runs maintenance with ALTER TABLE ... EXECUTE optimize, expire_snapshots and remove_orphan_files (read 2026-11-10). - Starburst publishes credit prices by AWS region, for example $0.50 (Pro) and $0.75 (Enterprise) in US East (N. Virginia) and $0.60 and $0.90 in Frankfurt (read 2026-11-10). ## Where it is limited - Per Starburst documentation, Galaxy runs SQL on Trino, not Spark, so DataFrame code, UDFs and PySpark would be rewritten as Trino SQL or kept on Spark (read 2026-11-10). - Per Starburst documentation, Hudi tables are read-only in Galaxy: it cannot create new Hudi tables or write to them. ## Published speed results | Fact | Source type | Source, read | | --- | --- | --- | | Starburst reported GPU-accelerated Trino results (its proprietary distribution, not open-source Trino): TPC-H up to 6x on accelerated queries and ClickBench 3.4x overall, single-node development environment. This is a SQL query test, not Spark batch work. | vendor | [starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia](https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/), 10 Nov 2026 | ## Published prices and cost results | Fact | Source type | Source, read | | --- | --- | --- | | Galaxy credit prices start at $0.50 (Pro), $0.75 (Enterprise) and $1.00 (Mission Critical) per credit. A Free tier allows up to 3 clusters. Pricing depends on compute usage, tier and the cloud provider and region. Annual commitments may get discounts. | vendor | [starburst.io/pricing](https://www.starburst.io/pricing/), 10 Nov 2026 | | On-demand list prices per credit (Pro / Enterprise / Mission Critical) on AWS: US East N. Virginia, US East Ohio and US West Oregon $0.50 / $0.75 / $1.00; Europe Ireland $0.56 / $0.84 / $1.12; Europe Frankfurt $0.60 / $0.90 / $1.20; Asia Pacific Tokyo $0.64 / $0.97 / $1.29; Brazil (Sao Paulo) $0.80 / $1.19 / $1.59. | vendor | [starburst.io/pricing/pricing-details](https://www.starburst.io/pricing/pricing-details/), 10 Nov 2026 | | Cluster compute credit consumption: 1 worker 6 credits per hour, 2 workers 12, 4 workers 24, 8 workers 48, 16 workers 96. Managed ingest: file ingest 0.60 credits per hour; table maintenance and data transforms 0.30 credits per hour. Billing tracks cluster compute hours, data ingestion compute and private link data transfer. | vendor | [starburst.io/pricing/pricing-details](https://www.starburst.io/pricing/pricing-details/), 10 Nov 2026 | ## Code changes and migration | Fact | Source type | Source, read | | --- | --- | --- | | Galaxy is 'a fully-managed data lake analytics platform built in addition to Trino'. Work runs as SQL on Trino, not Spark, so Spark jobs (DataFrame code, UDFs, PySpark) would have to be rewritten as Trino SQL, or kept on Spark against the same Iceberg tables. The Galaxy docs read contain no Spark execution mode. | vendor docs | [docs.starburst.io/starburst-galaxy/index.html](https://docs.starburst.io/starburst-galaxy/index.html), 10 Nov 2026 | ## Tuning and table upkeep | Fact | Source type | Source, read | | --- | --- | --- | | Iceberg table maintenance in Galaxy is run with ALTER TABLE ... EXECUTE commands: optimize (compaction; files under file_size_threshold, default 100MB, are merged), drop_extended_stats, expire_snapshots, remove_orphan_files. Live table maintenance for ingest tables is a separate Galaxy page. | vendor docs | [docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la](https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html), 10 Nov 2026 | | For batch-style work Galaxy offers a fault tolerant execution mode that spools intermediate data and retries failed parts of a query. Docs say 'No other configuration is required' to enable it, but not to use it if most queries run under about a minute, and that it can be slightly slower than normal execution. FTE and Accelerated modes are not available on the Free cluster size and not all catalogs support FTE. | vendor docs | [docs.starburst.io/starburst-galaxy/data-engineering/optimization-perform](https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html), 10 Nov 2026 | ## Spark versions and table formats | Fact | Source type | Source, read | | --- | --- | --- | | Galaxy's Great Lakes connectivity lists Iceberg, Delta Lake, Hive and Hudi table format pages. Hudi tables are read-only: 'Galaxy cannot create new Hudi tables or write to them.' Iceberg tables can be created and written; the Iceberg data file format can be Parquet, ORC or Avro. | vendor docs | [docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la](https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html), 10 Nov 2026 | ## Where it runs | Fact | Source type | Source, read | | --- | --- | --- | | Galaxy prices are listed for AWS, Azure and GCP regions. AWS regions priced include us-east-1, us-east-2, us-west-1, us-west-2, ca-central, Ireland, London, Frankfurt, Zurich, Singapore, Sydney, Tokyo, Cape Town, UAE, Tel Aviv and Sao Paulo. A cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature. | vendor | [starburst.io/pricing/pricing-details](https://www.starburst.io/pricing/pricing-details/), 10 Nov 2026 | ## Profile | Fact | Source type | Source, read | | --- | --- | --- | | Starburst's August 7, 2026 blog on SAP's acquisition of Dremio is competitor commentary about SAP's acquisition of Dremio; it does not announce any change to Starburst's own ownership. | vendor | [starburst.io/blog/5-questions-every-dremio-customer-should-ask-about-the](https://www.starburst.io/blog/5-questions-every-dremio-customer-should-ask-about-the-sap-deal/), 10 Nov 2026 | ## On the Compute Bill | Line | How it is billed | Formula | Stated cost | What is in and out | | --- | --- | --- | --- | --- | | [Starburst Galaxy (Pro tier)](/platforms/starburst/) | Credits per cluster-hour, by number of workers, times a price per credit that depends on tier, cloud and region. Galaxy runs Trino SQL, not Spark. | The published table is 1 worker = 6 credits per hour, 8 workers = 48, 16 workers = 96 (linear). Ten workers is not listed; at the same 6 per worker it is 60 credits per hour. 60 x 2 hours = 120 credits x $0.50 = $60.00. | $60.00 | Includes Galaxy credits for a ten-worker cluster for two hours at the Pro tier. Excludes storage, S3 requests, data transfer and the tier features. Enterprise tier is $0.75 per credit and Mission Critical $1.00 on the same page, which would give $90.00 and $120.00 for the same credits. The worker size in a Galaxy cluster is not stated on the page, so ten Galaxy workers are not shown to equal ten r6i.4xlarge nodes. | ## Searched for and none found on 10 Nov 2026 - Spark-related features in Starburst Galaxy (running or accelerating Spark jobs) (searched: docs.starburst.io/starburst-galaxy/index.html, docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la, docs.starburst.io/starburst-galaxy/data-engineering/optimization-perform) - Published cost for the same Spark batch work on Starburst (searched: starburst.io/pricing, starburst.io/pricing/pricing-details, starburst.io/sitemap.xml) - Starburst benchmark of ETL, joins or compaction on Galaxy CPU clusters with data size and scripts (searched: starburst.io/sitemap.xml, starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia) - Funding total and license of the Starburst distribution from a fetched primary source (searched: starburst.io/about, news.google.com/rss/search?q=Starburst+Data+Trino+2026) ## Starburst side by side with each other entry - [Starburst and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-starburst/) - [Starburst and Databricks](/compare/databricks-vs-starburst/) - [Starburst and Dremio](/compare/dremio-vs-starburst/) - [Starburst and DualBird](/compare/dualbird-vs-starburst/) - [Starburst and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-starburst/) - [Starburst and Microsoft Fabric](/compare/microsoft-fabric-vs-starburst/) - [Starburst and Onehouse](/compare/onehouse-vs-starburst/) - [Starburst and Open stack on EMR and EKS](/compare/open-stack-emr-eks-vs-starburst/) - [Starburst and Snowflake](/compare/snowflake-vs-starburst/) ## Sources - Starburst: starburst.io/pricing: https://www.starburst.io/pricing/ (read 10 Nov 2026) - Starburst: starburst.io/pricing/pricing-details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Starburst: docs.starburst.io/starburst-galaxy/index.html: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - Starburst: docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - Starburst: docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - Starburst: docs.starburst.io/starburst-galaxy/data-engineering/optimization-perform: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026) - Starburst: starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026) - Starburst: starburst.io/blog/5-questions-every-dremio-customer-should-ask-about-the: https://www.starburst.io/blog/5-questions-every-dremio-customer-should-ask-about-the-sap-deal/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/platforms/starburst/ --- # 45 pairs of platforms, side by side in the same rows Side by side, read 10 Nov 2026 **The short answer.** Forty-five side-by-side pages, one for each pair of the ten platforms, put two platforms next to each other, field by field: how each runs Spark code, what it bills, which table formats it supports, where it runs and what it publishes. Three more pages pair the table formats. Each page ends with facts, not a pick. ## Amazon EMR plus Athena on Iceberg - [Amazon EMR plus Athena on Iceberg and Databricks](/compare/aws-emr-athena-iceberg-vs-databricks/) - [Amazon EMR plus Athena on Iceberg and Dremio](/compare/aws-emr-athena-iceberg-vs-dremio/) - [Amazon EMR plus Athena on Iceberg and DualBird](/compare/aws-emr-athena-iceberg-vs-dualbird/) - [Amazon EMR plus Athena on Iceberg and Google BigQuery and BigLake](/compare/aws-emr-athena-iceberg-vs-bigquery-biglake/) - [Amazon EMR plus Athena on Iceberg and Microsoft Fabric](/compare/aws-emr-athena-iceberg-vs-microsoft-fabric/) - [Amazon EMR plus Athena on Iceberg and Onehouse](/compare/aws-emr-athena-iceberg-vs-onehouse/) - [Amazon EMR plus Athena on Iceberg and Open stack on EMR and EKS](/compare/aws-emr-athena-iceberg-vs-open-stack-emr-eks/) - [Amazon EMR plus Athena on Iceberg and Snowflake](/compare/aws-emr-athena-iceberg-vs-snowflake/) - [Amazon EMR plus Athena on Iceberg and Starburst](/compare/aws-emr-athena-iceberg-vs-starburst/) ## Databricks - [Databricks and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-databricks/) - [Databricks and Dremio](/compare/databricks-vs-dremio/) - [Databricks and DualBird](/compare/databricks-vs-dualbird/) - [Databricks and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-databricks/) - [Databricks and Microsoft Fabric](/compare/databricks-vs-microsoft-fabric/) - [Databricks and Onehouse](/compare/databricks-vs-onehouse/) - [Databricks and Open stack on EMR and EKS](/compare/databricks-vs-open-stack-emr-eks/) - [Databricks and Snowflake](/compare/databricks-vs-snowflake/) - [Databricks and Starburst](/compare/databricks-vs-starburst/) ## Dremio - [Dremio and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-dremio/) - [Dremio and Databricks](/compare/databricks-vs-dremio/) - [Dremio and DualBird](/compare/dremio-vs-dualbird/) - [Dremio and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-dremio/) - [Dremio and Microsoft Fabric](/compare/dremio-vs-microsoft-fabric/) - [Dremio and Onehouse](/compare/dremio-vs-onehouse/) - [Dremio and Open stack on EMR and EKS](/compare/dremio-vs-open-stack-emr-eks/) - [Dremio and Snowflake](/compare/dremio-vs-snowflake/) - [Dremio and Starburst](/compare/dremio-vs-starburst/) ## DualBird - [DualBird and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-dualbird/) - [DualBird and Databricks](/compare/databricks-vs-dualbird/) - [DualBird and Dremio](/compare/dremio-vs-dualbird/) - [DualBird and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-dualbird/) - [DualBird and Microsoft Fabric](/compare/dualbird-vs-microsoft-fabric/) - [DualBird and Onehouse](/compare/dualbird-vs-onehouse/) - [DualBird and Open stack on EMR and EKS](/compare/dualbird-vs-open-stack-emr-eks/) - [DualBird and Snowflake](/compare/dualbird-vs-snowflake/) - [DualBird and Starburst](/compare/dualbird-vs-starburst/) ## Google BigQuery and BigLake - [Google BigQuery and BigLake and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-bigquery-biglake/) - [Google BigQuery and BigLake and Databricks](/compare/bigquery-biglake-vs-databricks/) - [Google BigQuery and BigLake and Dremio](/compare/bigquery-biglake-vs-dremio/) - [Google BigQuery and BigLake and DualBird](/compare/bigquery-biglake-vs-dualbird/) - [Google BigQuery and BigLake and Microsoft Fabric](/compare/bigquery-biglake-vs-microsoft-fabric/) - [Google BigQuery and BigLake and Onehouse](/compare/bigquery-biglake-vs-onehouse/) - [Google BigQuery and BigLake and Open stack on EMR and EKS](/compare/bigquery-biglake-vs-open-stack-emr-eks/) - [Google BigQuery and BigLake and Snowflake](/compare/bigquery-biglake-vs-snowflake/) - [Google BigQuery and BigLake and Starburst](/compare/bigquery-biglake-vs-starburst/) ## Microsoft Fabric - [Microsoft Fabric and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-microsoft-fabric/) - [Microsoft Fabric and Databricks](/compare/databricks-vs-microsoft-fabric/) - [Microsoft Fabric and Dremio](/compare/dremio-vs-microsoft-fabric/) - [Microsoft Fabric and DualBird](/compare/dualbird-vs-microsoft-fabric/) - [Microsoft Fabric and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-microsoft-fabric/) - [Microsoft Fabric and Onehouse](/compare/microsoft-fabric-vs-onehouse/) - [Microsoft Fabric and Open stack on EMR and EKS](/compare/microsoft-fabric-vs-open-stack-emr-eks/) - [Microsoft Fabric and Snowflake](/compare/microsoft-fabric-vs-snowflake/) - [Microsoft Fabric and Starburst](/compare/microsoft-fabric-vs-starburst/) ## Onehouse - [Onehouse and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-onehouse/) - [Onehouse and Databricks](/compare/databricks-vs-onehouse/) - [Onehouse and Dremio](/compare/dremio-vs-onehouse/) - [Onehouse and DualBird](/compare/dualbird-vs-onehouse/) - [Onehouse and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-onehouse/) - [Onehouse and Microsoft Fabric](/compare/microsoft-fabric-vs-onehouse/) - [Onehouse and Open stack on EMR and EKS](/compare/onehouse-vs-open-stack-emr-eks/) - [Onehouse and Snowflake](/compare/onehouse-vs-snowflake/) - [Onehouse and Starburst](/compare/onehouse-vs-starburst/) ## Open stack on EMR and EKS - [Open stack on EMR and EKS and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and Databricks](/compare/databricks-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and Dremio](/compare/dremio-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and DualBird](/compare/dualbird-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and Microsoft Fabric](/compare/microsoft-fabric-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and Onehouse](/compare/onehouse-vs-open-stack-emr-eks/) - [Open stack on EMR and EKS and Snowflake](/compare/open-stack-emr-eks-vs-snowflake/) - [Open stack on EMR and EKS and Starburst](/compare/open-stack-emr-eks-vs-starburst/) ## Snowflake - [Snowflake and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-snowflake/) - [Snowflake and Databricks](/compare/databricks-vs-snowflake/) - [Snowflake and Dremio](/compare/dremio-vs-snowflake/) - [Snowflake and DualBird](/compare/dualbird-vs-snowflake/) - [Snowflake and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-snowflake/) - [Snowflake and Microsoft Fabric](/compare/microsoft-fabric-vs-snowflake/) - [Snowflake and Onehouse](/compare/onehouse-vs-snowflake/) - [Snowflake and Open stack on EMR and EKS](/compare/open-stack-emr-eks-vs-snowflake/) - [Snowflake and Starburst](/compare/snowflake-vs-starburst/) ## Starburst - [Starburst and Amazon EMR plus Athena on Iceberg](/compare/aws-emr-athena-iceberg-vs-starburst/) - [Starburst and Databricks](/compare/databricks-vs-starburst/) - [Starburst and Dremio](/compare/dremio-vs-starburst/) - [Starburst and DualBird](/compare/dualbird-vs-starburst/) - [Starburst and Google BigQuery and BigLake](/compare/bigquery-biglake-vs-starburst/) - [Starburst and Microsoft Fabric](/compare/microsoft-fabric-vs-starburst/) - [Starburst and Onehouse](/compare/onehouse-vs-starburst/) - [Starburst and Open stack on EMR and EKS](/compare/open-stack-emr-eks-vs-starburst/) - [Starburst and Snowflake](/compare/snowflake-vs-starburst/) ## Table formats side by side - [Delta Lake and Apache Hudi](/formats/delta-lake-vs-hudi/) - [Delta Lake and Apache Iceberg](/formats/delta-lake-vs-iceberg/) - [Apache Hudi and Apache Iceberg](/formats/hudi-vs-iceberg/) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/ --- # Amazon EMR plus Athena on Iceberg and Databricks: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR plus Athena on Iceberg and Databricks are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Amazon EMR plus Athena on Iceberg | Databricks | | --- | --- | --- | | Kind | Spark and SQL engines on AWS | Managed lakehouse built around Delta Lake | | Does well | Per AWS documentation, the EMR runtime for Spark is 100% API-compatible with open-source Spark, so existing jobs need no rewrite (read 2026-11-10). AWS publishes the setup for its TPC-DS 3 TB Iceberg benchmark in GitHub repositories, so a reader can repeat the run (read 2026-11-10). | Per Databricks documentation, Photon keeps the Spark Catalyst planner, works with existing Spark API code and falls back to the Spark runtime for unsupported operations (read 2026-11-10). Per Databricks documentation, Delta tables can generate Iceberg metadata from Runtime 14.3 LTS, so Iceberg clients read Delta data without rewriting files (read 2026-11-10). | | Limited | Per AWS documentation, Athena compaction is manual (you run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM), and Glue Iceberg compaction is off by default for new tables. Every service in this entry runs only on AWS, and Athena supports Iceberg 1.4.2 only for tables in the Glue Data Catalog (read 2026-11-10). | Per Databricks documentation, serverless compute supports only Spark Connect APIs, so RDD APIs, R and Maven coordinates are unsupported, and ANSI mode on by default can change results of existing jobs. Per Databricks, compute is billed in DBUs in addition to the cloud bill for the same instances, so a Jobs Compute run pays Databricks and AWS separately; the Compute Bill line adds $12.24 of DBUs to $20.16 of EC2 (read 2026-11-10). | | Published speed results | AWS states its runtime executes up to 4.5x faster than open-source Spark and gives 2.7x faster Iceberg write performance; the page gives no workload, scale or configuration. | Docs claim Photon gives up to 5x better price/performance for data and analytics workloads versus other cloud data warehouses; no method is given on this page. Databricks announced an audited 100TB TPC-DS result of 32,941,245 QphDS for Databricks SQL, 2.2x the previous record held by Alibaba, with total system cost 10% lower on list pricing. This is a SQL warehouse result, not a Spark batch ETL result. The same Databricks blog says research from Barcelona Supercomputing Center found Databricks 2.7x faster and 12x better in price-performance than Snowflake on a TPC-DS derivative. This is a vendor-reported claim of a third-party study; the study itself was not fetched. | | Published prices and cost results | EMR on EC2 adds a per-instance uplift in addition to EC2: the AWS price list shows $0.048 per hour for EMR m5.xlarge in us-east-1 (publication 2026-11-10). EMR on EKS uplift in us-east-1 is $0.007084 per vCPU-hour and $0.000777875 per GB-hour, in addition to the EC2 or Fargate cost. The EMR pricing page worked example for EMR Serverless uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour; a job with 100 vCPU for 30 minutes plus 200 more vCPU for 15 minutes costs $9.596275 in that example. Athena SQL costs $5.00 per TB scanned in us-east-1; Athena for Apache Spark costs $0.35 per DPU-hour; provisioned capacity costs $0.30 per DPU-hour. Glue Iceberg compaction costs $0.44 per DPU-hour in us-east-1; Glue 6.0+ ETL jobs cost $0.308 per DPU-hour and older Glue ETL $0.44. S3 Tables in us-east-1: compaction is $0.005 per GB processed plus $0.002 per 1,000 objects compacted; sort or Z-order compaction is $0.01 per GB plus $0.002 per 1,000 objects; storage is $0.0265 per GB-month for the first 50 TB. | Databricks bills compute in DBUs at per-second granularity with pay-as-you-go or committed-use discounts, and no up-front costs. The pricing pages render their rates by script; the AWS pricing data file those pages load lists Jobs Compute at $0.15 per DBU on the plan it names Premium (r6i.4xlarge: 4.08 DBU per hour). The file names no region. Jobs pricing offers serverless (Performance Optimized and Standard modes) and classic clusters; the jobs pricing page maps plan names across clouds (its Premium tier on Azure corresponds to Enterprise on AWS and GCP), while the AWS pricing data file lists the $0.15 Jobs Compute rate under a plan it names Premium. Photon instances consume DBUs at a different rate. | | Code changes and migration | The EMR runtime for Spark is described as 100% API-compatible with open-source Spark, so existing Spark jobs need no rewrite; EMR is native Spark on EC2, EKS or Serverless. Athena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client; PySpark engine version 3 offers the console notebook experience. | Photon is a native C++ vectorized engine that keeps Spark's Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime. Serverless compute for notebooks and jobs supports only Spark Connect APIs; Spark RDD APIs, R, Maven coordinates, global temp views, DBFS mounts with instance profiles and Databricks Container Services are not supported. ANSI mode is on by default and the default time zone is UTC, which can change results of existing jobs. | | Tuning and table upkeep | EMR enables adaptive query execution, dynamic partition pruning, bloom filter join and optimized join reorder; docs say AQE adaptive join conversion is on by default and can be switched off with a Spark setting. Athena compaction is manual: run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM yourself on Iceberg tables. Glue Data Catalog Iceberg compaction is disabled by default for new tables and must be enabled per table with an IAM role. S3 Tables compaction is on by default for all tables with a 512 MB default target file size (configurable 64 to 512 MB) and also applies row-level deletes. | Photon is on by default for classic all-purpose compute, jobs compute and classic pipelines; API users must set runtime_engine to PHOTON. Photon instance types use DBUs at a different rate than the same instance without Photon. On serverless compute Databricks allocates and manages compute so users do not provision clusters; the Spark UI and Spark logs are not available, query profile and client-side logs are used instead. Serverless compute runs workloads without provisioning compute in your cloud account, and Databricks says this reduces the need to manage compute resources. It requires Unity Catalog to reach external data sources. | | Spark versions and table formats | EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0); the EMR Spark feature page says EMR supports Apache Iceberg v3 and Spark 4.0. Athena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only with tables in the AWS Glue Data Catalog and Glue optimistic locking; DDL is not supported on tables registered with Lake Formation. | Databricks Runtime 18 LTS ships Apache Spark 4.1.0 (released June 10, 2026); Runtime 17.3 LTS ships Spark 4.0.0; Runtime 16.4 LTS and 15.4 LTS ship Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). Delta Lake tables can generate Iceberg metadata (UniForm, Iceberg reads) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files. | | Where it runs | All of these are AWS services, so they run only on AWS; EMR, Athena, Glue and S3 are priced in us-east-1 on the AWS price list. | Databricks on AWS lists many AWS regions including us-east-1 (US East, Northern Virginia), us-east-2, us-west-1, eu-west-1, eu-central-1, ap-southeast-2 and us-gov-west-1; the platform is also hosted on Azure and Google Cloud. Some features are only in a subset of regions. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | Amazon EMR on EKS (EC2 nodes): $24.42; Amazon EMR on EC2: $25.20; Amazon EMR Serverless: $31.63 | Databricks on AWS (Jobs Compute): $32.40 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/ind: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/glue/latest/dg/enable-compaction.html: https://docs.aws.amazon.com/glue/latest/dg/enable-compaction.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.js: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.j: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless/limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless: https://docs.databricks.com/aws/en/compute/serverless/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/release-notes/runtime: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/delta/uniform: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/resources/supported-regions: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/aws-pricing: https://www.databricks.com/product/aws-pricing (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/pricing/jobs: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/aws-emr-athena-iceberg-vs-databricks/ --- # Amazon EMR plus Athena on Iceberg and Dremio: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR plus Athena on Iceberg and Dremio are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Amazon EMR plus Athena on Iceberg | Dremio | | --- | --- | --- | | Kind | Spark and SQL engines on AWS | Lakehouse with its own SQL engine | | Does well | Per AWS documentation, the EMR runtime for Spark is 100% API-compatible with open-source Spark, so existing jobs need no rewrite (read 2026-11-10). AWS publishes the setup for its TPC-DS 3 TB Iceberg benchmark in GitHub repositories, so a reader can repeat the run (read 2026-11-10). | Per Dremio documentation, Iceberg tables in the Open Catalog get automatic optimization: data file compaction, row-level delete handling, clustering, partition evolution and manifest rewriting (read 2026-11-10). Dremio publishes DCU rates by engine size, from 16 DCUs per hour for 2XSmall to 2048 for 3XLarge, with a 60-second minimum (read 2026-11-10). | | Limited | Per AWS documentation, Athena compaction is manual (you run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM), and Glue Iceberg compaction is off by default for new tables. Every service in this entry runs only on AWS, and Athena supports Iceberg 1.4.2 only for tables in the Glue Data Catalog (read 2026-11-10). | Per Dremio documentation, queries run on its own columnar engine, not on Spark, and the pages read describe no mode that runs existing Spark jobs. Dremio Cloud documentation lists Iceberg and Delta Lake as table formats; Hudi support: none found on 2026-11-10. | | Published speed results | AWS states its runtime executes up to 4.5x faster than open-source Spark and gives 2.7x faster Iceberg write performance; the page gives no workload, scale or configuration. | Dremio's founder reported a full 1 TB TPC-DS run (all 99 queries) in 22 seconds on an 8-node m7gd.4xlarge cluster, with no caching, and attributes it to Autonomous Reflections and automatic Iceberg clustering. This is a BI query benchmark run by the vendor, not a Spark batch ETL or compaction test. | | Published prices and cost results | EMR on EC2 adds a per-instance uplift in addition to EC2: the AWS price list shows $0.048 per hour for EMR m5.xlarge in us-east-1 (publication 2026-11-10). EMR on EKS uplift in us-east-1 is $0.007084 per vCPU-hour and $0.000777875 per GB-hour, in addition to the EC2 or Fargate cost. The EMR pricing page worked example for EMR Serverless uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour; a job with 100 vCPU for 30 minutes plus 200 more vCPU for 15 minutes costs $9.596275 in that example. Athena SQL costs $5.00 per TB scanned in us-east-1; Athena for Apache Spark costs $0.35 per DPU-hour; provisioned capacity costs $0.30 per DPU-hour. Glue Iceberg compaction costs $0.44 per DPU-hour in us-east-1; Glue 6.0+ ETL jobs cost $0.308 per DPU-hour and older Glue ETL $0.44. S3 Tables in us-east-1: compaction is $0.005 per GB processed plus $0.002 per 1,000 objects compacted; sort or Z-order compaction is $0.01 per GB plus $0.002 per 1,000 objects; storage is $0.0265 per GB-month for the first 50 TB. | Dremio Cloud is priced at $0.20 per DCU (Dremio Compute Unit). Dremio Enterprise (self-managed) pricing is 'contact sales'. The free trial for Dremio Cloud is a $400 credit for 30 days. DCUs are consumed only while engines run. Usage equals engine and replica uptime in seconds (60-second minimum) times the DCU rate for the engine size. Published rates per hour: 2XSmall 16, XSmall 32, Small 64, Medium 128, Large 256, XLarge 512, 2XLarge 1024, 3XLarge 2048. Storage billed in TB-months only for Dremio-hosted storage; LLM tokens billed separately. Billable usage is DCUs (compute), LLM tokens (AI features) and storage. Compute is billed in DCUs only while engines are running. | | Code changes and migration | The EMR runtime for Spark is described as 100% API-compatible with open-source Spark, so existing Spark jobs need no rewrite; EMR is native Spark on EC2, EKS or Serverless. Athena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client; PySpark engine version 3 offers the console notebook experience. | Dremio runs queries on its own columnar execution engine built on Apache Arrow, not on Spark. Docs describe Spark and Flink only as external engines that can use the Open Catalog (Iceberg REST) alongside Dremio. The pages read describe no mode that runs existing Spark jobs, so Spark ETL would stay on Spark or be rewritten as Dremio SQL. | | Tuning and table upkeep | EMR enables adaptive query execution, dynamic partition pruning, bloom filter join and optimized join reorder; docs say AQE adaptive join conversion is on by default and can be switched off with a Spark setting. Athena compaction is manual: run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM yourself on Iceberg tables. Glue Data Catalog Iceberg compaction is disabled by default for new tables and must be enabled per table with an IAM role. S3 Tables compaction is on by default for all tables with a 512 MB default target file size (configurable 64 to 512 MB) and also applies row-level deletes. | For Iceberg tables in the Open Catalog, Dremio runs automatic optimization on a dedicated engine it configures: data file compaction, row-level delete handling (position deletes on v2, deletion vectors on v3), clustering, partition evolution and manifest rewriting. The same operations can be run by hand with OPTIMIZE TABLE. Optimization works in batches sized to engine resources. Engines have settings the admin chooses: size (executors per replica), min and max replicas, last-replica auto-stop (default 2 hours), enqueued time limit (default 5 minutes) and query runtime limit. Default concurrency per replica ranges from 2 (2XSmall) to 20 (3XLarge). | | Spark versions and table formats | EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0); the EMR Spark feature page says EMR supports Apache Iceberg v3 and Spark 4.0. Athena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only with tables in the AWS Glue Data Catalog and Glue optimistic locking; DDL is not supported on tables registered with Lake Formation. | The Dremio Cloud data formats page lists Apache Iceberg and Delta Lake as table formats (plus Parquet and text formats). Hudi is not listed on that page. The Open Catalog is built on Apache Polaris and supports the Iceberg REST Catalog API; the docs list AWS Glue Data Catalog as a connectable catalog. | | Where it runs | All of these are AWS services, so they run only on AWS; EMR, Athena, Glue and S3 are priced in us-east-1 on the AWS price list. | Dremio Cloud runs on AWS (Azure listed as coming soon). Supported regions: us-east-1, us-east-2, us-west-2, eu-west-1 and eu-central-1. Dremio Enterprise can be deployed on Kubernetes, on premises, or in AWS, Azure or GCP. | | How open its benchmarks are | None found on 10 Nov 2026. | Dremio published a 2020 TPC-DS methodology post (SF1000 and SF10000, Parquet on S3, Apache JMeter test plans, 58 of 99 queries that ran unmodified on Dremio and Presto distributions). The 2025 22-second TPC-DS post pages read give cluster type but no scripts, scale-factor files or configuration repository. | | Compute Bill | Amazon EMR on EKS (EC2 nodes): $24.42; Amazon EMR on EC2: $25.20; Amazon EMR Serverless: $31.63 | Dremio Cloud (Large engine): $102.40 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/ind: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/glue/latest/dg/enable-compaction.html: https://docs.aws.amazon.com/glue/latest/dg/enable-compaction.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.js: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.j: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - Dremio: dremio.com/pricing: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/engines: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/subscription/usage: https://docs.dremio.com/dremio-cloud/admin/subscription/usage (read 10 Nov 2026) - Dremio: docs.dremio.com/current/what-is-dremio/architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/developer/data-formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/manage-govern/optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/about/regions: https://docs.dremio.com/dremio-cloud/about/regions (read 10 Nov 2026) - Dremio: dremio.com/blog/breakthrough-announcement-dremio-is-the: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026) - Dremio: dremio.com/blog/dremio-benchmarking-methodology: https://www.dremio.com/blog/dremio-benchmarking-methodology/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/aws-emr-athena-iceberg-vs-dremio/ --- # Amazon EMR plus Athena on Iceberg and DualBird: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR plus Athena on Iceberg and DualBird are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Amazon EMR plus Athena on Iceberg | DualBird | | --- | --- | --- | | Kind | Spark and SQL engines on AWS | FPGA Spark plugin for AWS F2 | | Does well | Per AWS documentation, the EMR runtime for Spark is 100% API-compatible with open-source Spark, so existing jobs need no rewrite (read 2026-11-10). AWS publishes the setup for its TPC-DS 3 TB Iceberg benchmark in GitHub repositories, so a reader can repeat the run (read 2026-11-10). | DualBird says onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin, with Spark still handling planning and scheduling (read 2026-11-10). DualBird published a 100 GB sort-based Iceberg compaction run on June 11, 2026, run by DualBird, and states about $2 per TB for itself against about $6 per TB for CPU clusters. | | Limited | Per AWS documentation, Athena compaction is manual (you run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM), and Glue Iceberg compaction is off by default for new tables. Every service in this entry runs only on AWS, and Athena supports Iceberg 1.4.2 only for tables in the Glue Data Catalog (read 2026-11-10). | DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. A price for the plugin and a benchmark run by a party other than DualBird: none found on 2026-11-10. | | Published speed results | AWS states its runtime executes up to 4.5x faster than open-source Spark and gives 2.7x faster Iceberg write performance; the page gives no workload, scale or configuration. | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | | Published prices and cost results | EMR on EC2 adds a per-instance uplift in addition to EC2: the AWS price list shows $0.048 per hour for EMR m5.xlarge in us-east-1 (publication 2026-11-10). EMR on EKS uplift in us-east-1 is $0.007084 per vCPU-hour and $0.000777875 per GB-hour, in addition to the EC2 or Fargate cost. The EMR pricing page worked example for EMR Serverless uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour; a job with 100 vCPU for 30 minutes plus 200 more vCPU for 15 minutes costs $9.596275 in that example. Athena SQL costs $5.00 per TB scanned in us-east-1; Athena for Apache Spark costs $0.35 per DPU-hour; provisioned capacity costs $0.30 per DPU-hour. Glue Iceberg compaction costs $0.44 per DPU-hour in us-east-1; Glue 6.0+ ETL jobs cost $0.308 per DPU-hour and older Glue ETL $0.44. S3 Tables in us-east-1: compaction is $0.005 per GB processed plus $0.002 per 1,000 objects compacted; sort or Z-order compaction is $0.01 per GB plus $0.002 per 1,000 objects; storage is $0.0265 per GB-month for the first 50 TB. | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | | Code changes and migration | The EMR runtime for Spark is described as 100% API-compatible with open-source Spark, so existing Spark jobs need no rewrite; EMR is native Spark on EC2, EKS or Serverless. Athena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client; PySpark engine version 3 offers the console notebook experience. | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | | Tuning and table upkeep | EMR enables adaptive query execution, dynamic partition pruning, bloom filter join and optimized join reorder; docs say AQE adaptive join conversion is on by default and can be switched off with a Spark setting. Athena compaction is manual: run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM yourself on Iceberg tables. Glue Data Catalog Iceberg compaction is disabled by default for new tables and must be enabled per table with an IAM role. S3 Tables compaction is on by default for all tables with a 512 MB default target file size (configurable 64 to 512 MB) and also applies row-level deletes. | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | | Spark versions and table formats | EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0); the EMR Spark feature page says EMR supports Apache Iceberg v3 and Spark 4.0. Athena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only with tables in the AWS Glue Data Catalog and Glue optimistic locking; DDL is not supported on tables registered with Lake Formation. | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | | Where it runs | All of these are AWS services, so they run only on AWS; EMR, Athena, Glue and S3 are priced in us-east-1 on the AWS price list. | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | | How open its benchmarks are | None found on 10 Nov 2026. | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | | Compute Bill | Amazon EMR on EKS (EC2 nodes): $24.42; Amazon EMR on EC2: $25.20; Amazon EMR Serverless: $31.63 | DualBird on Amazon EMR (F2 instances): $49.50 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/ind: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/glue/latest/dg/enable-compaction.html: https://docs.aws.amazon.com/glue/latest/dg/enable-compaction.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.js: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.j: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/aws-emr-athena-iceberg-vs-dualbird/ --- # Amazon EMR plus Athena on Iceberg and Google BigQuery and BigLake: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR plus Athena on Iceberg and Google BigQuery and BigLake are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Amazon EMR plus Athena on Iceberg | Google BigQuery and BigLake | | --- | --- | --- | | Kind | Spark and SQL engines on AWS | Serverless Google Cloud data warehouse | | Does well | Per AWS documentation, the EMR runtime for Spark is 100% API-compatible with open-source Spark, so existing jobs need no rewrite (read 2026-11-10). AWS publishes the setup for its TPC-DS 3 TB Iceberg benchmark in GitHub repositories, so a reader can repeat the run (read 2026-11-10). | Per Google documentation, BigQuery Iceberg managed tables store data in buckets you own, support DML and streaming, and run automatic compaction, clustering and garbage collection (read 2026-11-10). Per Google documentation, stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud (read 2026-11-10). | | Limited | Per AWS documentation, Athena compaction is manual (you run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM), and Glue Iceberg compaction is off by default for new tables. Every service in this entry runs only on AWS, and Athena supports Iceberg 1.4.2 only for tables in the Glue Data Catalog (read 2026-11-10). | Per Google documentation, BigQuery Omni on S3 does not support DML or CREATE TABLE, so it does not run write-heavy Spark batch ETL on S3. Google's pricing page text lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate (read 2026-11-10). Per-TiB on-demand and Spark DCU rates are not used on this site. | | Published speed results | AWS states its runtime executes up to 4.5x faster than open-source Spark and gives 2.7x faster Iceberg write performance; the page gives no workload, scale or configuration. | Google states Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name; no method is given on the pricing page. | | Published prices and cost results | EMR on EC2 adds a per-instance uplift in addition to EC2: the AWS price list shows $0.048 per hour for EMR m5.xlarge in us-east-1 (publication 2026-11-10). EMR on EKS uplift in us-east-1 is $0.007084 per vCPU-hour and $0.000777875 per GB-hour, in addition to the EC2 or Fargate cost. The EMR pricing page worked example for EMR Serverless uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour; a job with 100 vCPU for 30 minutes plus 200 more vCPU for 15 minutes costs $9.596275 in that example. Athena SQL costs $5.00 per TB scanned in us-east-1; Athena for Apache Spark costs $0.35 per DPU-hour; provisioned capacity costs $0.30 per DPU-hour. Glue Iceberg compaction costs $0.44 per DPU-hour in us-east-1; Glue 6.0+ ETL jobs cost $0.308 per DPU-hour and older Glue ETL $0.44. S3 Tables in us-east-1: compaction is $0.005 per GB processed plus $0.002 per 1,000 objects compacted; sort or Z-order compaction is $0.01 per GB plus $0.002 per 1,000 objects; storage is $0.0265 per GB-month for the first 50 TB. | BigQuery offers on-demand pricing per TiB scanned (first 1 TiB per month free) or capacity pricing per slot-hour. BigQuery Omni has its own on-demand per TiB pricing for AWS regions. The page text shows $0.06 per slot-hour for Enterprise edition pay-as-you-go compute, with no region shown beside the rate (read 2026-11-10). Google's Spark service (Managed Service for Apache Spark, formerly Dataproc) bills serverless Spark per second in Data Compute Units with a 1-minute minimum, plus shuffle storage; Standard tier is for batch ETL, Premium tier uses Lightning Engine. | | Code changes and migration | The EMR runtime for Spark is described as 100% API-compatible with open-source Spark, so existing Spark jobs need no rewrite; EMR is native Spark on EC2, EKS or Serverless. Athena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client; PySpark engine version 3 offers the console notebook experience. | On BigQuery Omni, DML statements and CREATE TABLE are not supported, BigQuery ML statements are not supported, and only EXPORT DATA queries are allowed for destination tables, so it does not run write-heavy Spark batch ETL on S3. Standard and Enterprise Plus editions do not support Omni regions. BigQuery stored procedures for Apache Spark let you run PySpark or Scala/Java JAR code from SQL via CREATE PROCEDURE ... OPTIONS(engine = "SPARK", runtime_version = "2.2"), using a Cloud Storage file or inline code and a connection; this runs on Google Cloud, so code must be wrapped as procedures. | | Tuning and table upkeep | EMR enables adaptive query execution, dynamic partition pruning, bloom filter join and optimized join reorder; docs say AQE adaptive join conversion is on by default and can be switched off with a Spark setting. Athena compaction is manual: run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM yourself on Iceberg tables. Glue Data Catalog Iceberg compaction is disabled by default for new tables and must be enabled per table with an IAM role. S3 Tables compaction is on by default for all tables with a 512 MB default target file size (configurable 64 to 512 MB) and also applies row-level deletes. | Iceberg managed tables run automatic storage optimization (adaptive file sizing, clustering, garbage collection), so compaction needs no job configuration on Google Cloud; storage optimization is a separate billed item. | | Spark versions and table formats | EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0); the EMR Spark feature page says EMR supports Apache Iceberg v3 and Spark 4.0. Athena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only with tables in the AWS Glue Data Catalog and Glue optimistic locking; DDL is not supported on tables registered with Lake Formation. | BigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, export Iceberg V2 snapshots for open engines, and do automatic compaction, clustering and garbage collection. BigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security. | | Where it runs | All of these are AWS services, so they run only on AWS; EMR, Athena, Glue and S3 are priced in us-east-1 on the AWS price list. | BigQuery Omni lets BigQuery analytics run on data in Amazon S3 through BigLake tables, in AWS regions us-east-1, us-west-2, ap-northeast-2, ap-southeast-2, eu-west-1 and eu-central-1. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | Amazon EMR on EKS (EC2 nodes): $24.42; Amazon EMR on EC2: $25.20; Amazon EMR Serverless: $31.63 | BigQuery (Enterprise edition slots): $19.20 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/ind: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/glue/latest/dg/enable-compaction.html: https://docs.aws.amazon.com/glue/latest/dg/enable-compaction.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.js: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.j: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/omni-introduction: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/spark-procedures: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/iceberg-tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/biglake-intro: https://cloud.google.com/bigquery/docs/biglake-intro (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/dataproc-serverless/pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/aws-emr-athena-iceberg-vs-bigquery-biglake/ --- # Amazon EMR plus Athena on Iceberg and Microsoft Fabric: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR plus Athena on Iceberg and Microsoft Fabric are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Amazon EMR plus Athena on Iceberg | Microsoft Fabric | | --- | --- | --- | | Kind | Spark and SQL engines on AWS | SaaS analytics on Azure with Spark | | Does well | Per AWS documentation, the EMR runtime for Spark is 100% API-compatible with open-source Spark, so existing jobs need no rewrite (read 2026-11-10). AWS publishes the setup for its TPC-DS 3 TB Iceberg benchmark in GitHub repositories, so a reader can repeat the run (read 2026-11-10). | Per Microsoft Learn, the native execution engine (Apache Gluten and Velox) needs no code changes, falls back to JVM Spark for unsupported operators and supports Parquet and Delta (read 2026-11-10). Per Microsoft Learn, OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying it (read 2026-11-10). | | Limited | Per AWS documentation, Athena compaction is manual (you run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM), and Glue Iceberg compaction is off by default for new tables. Every service in this entry runs only on AWS, and Athena supports Iceberg 1.4.2 only for tables in the Glue Data Catalog (read 2026-11-10). | Per Microsoft Learn, Fabric runs over OneLake on Azure and the region page lists Azure regions only, so Spark compute does not run inside AWS. Writes to S3 through a OneLake shortcut are not possible, and the method for Microsoft's native engine benchmark: none found on 2026-11-10. | | Published speed results | AWS states its runtime executes up to 4.5x faster than open-source Spark and gives 2.7x faster Iceberg write performance; the page gives no workload, scale or configuration. | Microsoft reports that on TPC-DS scale factor 1000 with Delta the native engine was up to six times faster than open-source Spark, with about 83% compute cost savings on a fixed-size cluster. | | Published prices and cost results | EMR on EC2 adds a per-instance uplift in addition to EC2: the AWS price list shows $0.048 per hour for EMR m5.xlarge in us-east-1 (publication 2026-11-10). EMR on EKS uplift in us-east-1 is $0.007084 per vCPU-hour and $0.000777875 per GB-hour, in addition to the EC2 or Fargate cost. The EMR pricing page worked example for EMR Serverless uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour; a job with 100 vCPU for 30 minutes plus 200 more vCPU for 15 minutes costs $9.596275 in that example. Athena SQL costs $5.00 per TB scanned in us-east-1; Athena for Apache Spark costs $0.35 per DPU-hour; provisioned capacity costs $0.30 per DPU-hour. Glue Iceberg compaction costs $0.44 per DPU-hour in us-east-1; Glue 6.0+ ETL jobs cost $0.308 per DPU-hour and older Glue ETL $0.44. S3 Tables in us-east-1: compaction is $0.005 per GB processed plus $0.002 per 1,000 objects compacted; sort or Z-order compaction is $0.01 per GB plus $0.002 per 1,000 objects; storage is $0.0265 per GB-month for the first 50 TB. | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus (Spark Memory Optimized and autoscale for Spark entries, pay-as-you-go). In Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling. Autoscale Billing for Spark is an opt-in pay-as-you-go model that runs Spark on dedicated serverless resources at standard PAYG CU-hour rates; reserved capacity discounts do not apply and a base capacity is still required. | | Code changes and migration | The EMR runtime for Spark is described as 100% API-compatible with open-source Spark, so existing Spark jobs need no rewrite; EMR is native Spark on EC2, EKS or Serverless. Athena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client; PySpark engine version 3 offers the console notebook experience. | The Fabric native execution engine, based on Apache Gluten and Velox, needs no code changes, falls back to JVM Spark for unsupported operators, and supports Parquet and Delta. | | Tuning and table upkeep | EMR enables adaptive query execution, dynamic partition pruning, bloom filter join and optimized join reorder; docs say AQE adaptive join conversion is on by default and can be switched off with a Spark setting. Athena compaction is manual: run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM yourself on Iceberg tables. Glue Data Catalog Iceberg compaction is disabled by default for new tables and must be enabled per table with an IAM role. S3 Tables compaction is on by default for all tables with a 512 MB default target file size (configurable 64 to 512 MB) and also applies row-level deletes. | Starter pools start in 5 to 10 seconds with no setup; custom Spark pools let you tune node size and scaling, and native execution can be toggled per application with a Spark setting. | | Spark versions and table formats | EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0); the EMR Spark feature page says EMR supports Apache Iceberg v3 and Spark 4.0. Athena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only with tables in the AWS Glue Data Catalog and Glue optimistic locking; DDL is not supported on tables registered with Lake Formation. | Fabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1; OneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and vice versa. | | Where it runs | All of these are AWS services, so they run only on AWS; EMR, Athena, Glue and S3 are priced in us-east-1 on the AWS price list. | Fabric workloads run over OneLake, described as built on Azure Data Lake Storage; the region availability page lists Azure regions and no AWS regions, so Spark compute does not run inside AWS. OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; file caching can reduce cross-cloud egress costs. Writes to S3 are not possible through the shortcut. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | Amazon EMR on EKS (EC2 nodes): $24.42; Amazon EMR on EC2: $25.20; Amazon EMR Serverless: $31.63 | Microsoft Fabric (F128 capacity): $46.08 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/ind: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/glue/latest/dg/enable-compaction.html: https://docs.aws.amazon.com/glue/latest/dg/enable-compaction.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.js: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.j: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview: https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - Microsoft Fabric: prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption' (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Microsoft Fabric: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-compute: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/aws-emr-athena-iceberg-vs-microsoft-fabric/ --- # Amazon EMR plus Athena on Iceberg and Onehouse: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR plus Athena on Iceberg and Onehouse are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Amazon EMR plus Athena on Iceberg | Onehouse | | --- | --- | --- | | Kind | Spark and SQL engines on AWS | Managed lakehouse in your cloud account | | Does well | Per AWS documentation, the EMR runtime for Spark is 100% API-compatible with open-source Spark, so existing jobs need no rewrite (read 2026-11-10). AWS publishes the setup for its TPC-DS 3 TB Iceberg benchmark in GitHub repositories, so a reader can repeat the run (read 2026-11-10). | Per Onehouse documentation, its jobs are fully compatible with Apache Spark and EMR code can be reused unless it uses AWS-specific features (read 2026-11-10). Onehouse published the setup for its 10 TB TPC-DS and 1 TB LakeLoader comparison against EMR 7.12, and open-sourced LakeLoader under Apache-2.0 (read 2026-11-10). | | Limited | Per AWS documentation, Athena compaction is manual (you run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM), and Glue Iceberg compaction is off by default for new tables. Every service in this entry runs only on AWS, and Athena supports Iceberg 1.4.2 only for tables in the Glue Data Catalog (read 2026-11-10). | Per Onehouse documentation, billing is in OCUs on a bring-your-own-cloud model, and no dollar rate per OCU was found on 2026-11-10. Onehouse's results against EMR 7.12 are its own and shown as charts; a confirmation by another party: none found on 2026-11-10. | | Published speed results | AWS states its runtime executes up to 4.5x faster than open-source Spark and gives 2.7x faster Iceberg write performance; the page gives no workload, scale or configuration. | Onehouse benchmarked Quanton against open-source Spark on Iceberg: TPC-DS up to about 3.1x, TPCx-BB 2x, TPC-DI (1 TB) 2x with 3-4x on MERGE INTO, and LakeLoader 4.1x for writes. The baseline is open-source Spark, not EMR. Onehouse reran its benchmarks against EMR 7.12 after AWS announced it: it reports EMR improved 32% over earlier versions and that Onehouse 'still comes out with a 2.5x better price/performance advantage'. Setup: 10 TB TPC-DS for read/transform, 1 TB LakeLoader merges for load, latest general-purpose Graviton instances. Results are shown as charts; Onehouse also reports Hudi and Delta query performance on EMR 7.12 was very similar to Iceberg. A B2B SaaS customer (unnamed) ran its UDF-heavy pipeline on EMR and on Quanton in its own AWS account on m8gd nodes. Quanton was faster at p50, p97 and p99 tenant sizes; on the largest tenant (21 billion rows) EMR filled local disks on 180 nodes and was aborted after 3.7 hours, and finished on 300 nodes. The write-up is published by Onehouse. | | Published prices and cost results | EMR on EC2 adds a per-instance uplift in addition to EC2: the AWS price list shows $0.048 per hour for EMR m5.xlarge in us-east-1 (publication 2026-11-10). EMR on EKS uplift in us-east-1 is $0.007084 per vCPU-hour and $0.000777875 per GB-hour, in addition to the EC2 or Fargate cost. The EMR pricing page worked example for EMR Serverless uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour; a job with 100 vCPU for 30 minutes plus 200 more vCPU for 15 minutes costs $9.596275 in that example. Athena SQL costs $5.00 per TB scanned in us-east-1; Athena for Apache Spark costs $0.35 per DPU-hour; provisioned capacity costs $0.30 per DPU-hour. Glue Iceberg compaction costs $0.44 per DPU-hour in us-east-1; Glue 6.0+ ETL jobs cost $0.308 per DPU-hour and older Glue ETL $0.44. S3 Tables in us-east-1: compaction is $0.005 per GB processed plus $0.002 per 1,000 objects compacted; sort or Z-order compaction is $0.01 per GB plus $0.002 per 1,000 objects; storage is $0.0265 per GB-month for the first 50 TB. | Onehouse bills in Onehouse Compute Units (OCU) on a bring-your-own-cloud model: 1 OCU is a 4-core instance for 1 hour in your account, plus support costs; no separate volume or storage fee, and the customer pays its own AWS or GCP infrastructure bill. The docs page does not state a dollar rate per OCU. Vendor claims at the Quanton launch: 2-3x better price-performance for ETL, and a guaranteed greater-than-50% cost reduction for existing Hudi users moving Spark jobs from AWS EMR or open-source Spark. These are guarantees stated by the vendor, not results from an independent party. | | Code changes and migration | The EMR runtime for Spark is described as 100% API-compatible with open-source Spark, so existing Spark jobs need no rewrite; EMR is native Spark on EC2, EKS or Serverless. Athena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client; PySpark engine version 3 offers the console notebook experience. | Onehouse docs say its Jobs are fully compatible with Apache Spark and that EMR code can be reused unless it uses AWS-specific features. Migration work listed: map EMR steps to Onehouse Jobs created through its API or Airflow operators (CREATE CLUSTER, CREATE JOB, RUN JOB, DELETE CLUSTER), and configure Glue catalog sync or contact support to make Glue the primary catalog. At the May 20, 2025 launch Onehouse said users can run existing Spark and SQL jobs on new cluster types and that existing Hudi/Spark jobs can be pointed at Onehouse 'with no code changes'. The Quanton Kubernetes Operator is described as a 'drop-in replacement' for an existing Spark Kubernetes operator. | | Tuning and table upkeep | EMR enables adaptive query execution, dynamic partition pruning, bloom filter join and optimized join reorder; docs say AQE adaptive join conversion is on by default and can be switched off with a Spark setting. Athena compaction is manual: run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM yourself on Iceberg tables. Glue Data Catalog Iceberg compaction is disabled by default for new tables and must be enabled per table with an IAM role. S3 Tables compaction is on by default for all tables with a 512 MB default target file size (configurable 64 to 512 MB) and also applies row-level deletes. | Each cluster needs min and max OCU limits. Docs warn that if usage reaches Max OCU the cluster stops scaling up, which can delay processing; Min OCU keeps a cluster warm. The Onehouse docs pages read give no per-job Spark tuning list; Onehouse claims automatic scaling and automated async table services. | | Spark versions and table formats | EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0); the EMR Spark feature page says EMR supports Apache Iceberg v3 and Spark 4.0. Athena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only with tables in the AWS Glue Data Catalog and Glue optimistic locking; DDL is not supported on tables registered with Lake Formation. | Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake. Docs: Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog (AWS Glue and Snowflake Open Catalog are documented); for Delta Lake the user installs Delta Spark as a dependency in the JAR or Python job. In the Nov 12, 2025 Iceberg post Onehouse said Delta acceleration is 'up next'. Apache XTable (formerly OneTable) translates table metadata between Hudi, Delta Lake and Iceberg without being a new format. It is incubating at the Apache Software Foundation; Onehouse says it created Hudi and XTable. | | Where it runs | All of these are AWS services, so they run only on AWS; EMR, Athena, Glue and S3 are priced in us-east-1 on the AWS price list. | Onehouse runs EC2 or Google Compute Engine instances inside the customer's AWS or GCP account (Azure not listed in the billing docs). Standard instance types map to Graviton m8g sizes on AWS (oh-general-4 = m8g.xlarge, oh-general-8 = m8g.2xlarge, oh-general-16 = m8g.4xlarge). A project is scoped to one region and account; adding a region needs a support ticket. | | How open its benchmarks are | None found on 10 Nov 2026. | Onehouse open-sourced LakeLoader, a tool that generates incremental update/insert loads for benchmarking write performance. The repository is Apache-2.0 licensed, last pushed 2026-10-30. The Quanton engine itself and the full Iceberg benchmark configurations were not found as a repository in the pages read. | | Compute Bill | Amazon EMR on EKS (EC2 nodes): $24.42; Amazon EMR on EC2: $25.20; Amazon EMR Serverless: $31.63 | Onehouse (Quanton, bring-your-own-cloud): no total (see the Compute Bill for why) | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/ind: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/glue/latest/dg/enable-compaction.html: https://docs.aws.amazon.com/glue/latest/dg/enable-compaction.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.js: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.j: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs/emr-migration: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtim: https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/project-settings/usage-and-billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): xtable.apache.org: https://xtable.apache.org/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/compute/create-cluster: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-worklo: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark: https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): api.github.com/repos/onehouseinc/lake-loader: https://api.github.com/repos/onehouseinc/lake-loader (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/aws-emr-athena-iceberg-vs-onehouse/ --- # Amazon EMR plus Athena on Iceberg and Open stack on EMR and EKS: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR plus Athena on Iceberg and Open stack on EMR and EKS are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Amazon EMR plus Athena on Iceberg | Open stack on EMR and EKS | | --- | --- | --- | | Kind | Spark and SQL engines on AWS | Self-run Spark with open table formats | | Does well | Per AWS documentation, the EMR runtime for Spark is 100% API-compatible with open-source Spark, so existing jobs need no rewrite (read 2026-11-10). AWS publishes the setup for its TPC-DS 3 TB Iceberg benchmark in GitHub repositories, so a reader can repeat the run (read 2026-11-10). | Per AWS documentation, EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three formats are available (read 2026-11-10). AWS ran TPC-DS 3 TB on 9 x r5d.4xlarge and reports $4.81 for EMR 7.12 against $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0, with repositories to repeat it. | | Limited | Per AWS documentation, Athena compaction is manual (you run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM), and Glue Iceberg compaction is off by default for new tables. Every service in this entry runs only on AWS, and Athena supports Iceberg 1.4.2 only for tables in the Glue Data Catalog (read 2026-11-10). | Per the Apache Iceberg documentation, users run table maintenance themselves: expire snapshots, remove orphan files and compact with rewriteDataFiles as jobs they schedule (read 2026-11-10). A cost comparison of open-stack Spark against managed lakehouses by a party other than AWS or a vendor: none found on 2026-11-10. | | Published speed results | AWS states its runtime executes up to 4.5x faster than open-source Spark and gives 2.7x faster Iceberg write performance; the page gives no workload, scale or configuration. | AWS ran TPC-DS 3 TB (104 SparkSQL queries, 3 rounds, 9 x r5d.4xlarge, Hadoop catalog) and reports EMR 7.12 averaged 1,349.62 s versus 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0 and 1,535.62 s for EMR 7.5. AWS notes results are not comparable to official TPC-DS. The run is by the vendor of EMR. | | Published prices and cost results | EMR on EC2 adds a per-instance uplift in addition to EC2: the AWS price list shows $0.048 per hour for EMR m5.xlarge in us-east-1 (publication 2026-11-10). EMR on EKS uplift in us-east-1 is $0.007084 per vCPU-hour and $0.000777875 per GB-hour, in addition to the EC2 or Fargate cost. The EMR pricing page worked example for EMR Serverless uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour; a job with 100 vCPU for 30 minutes plus 200 more vCPU for 15 minutes costs $9.596275 in that example. Athena SQL costs $5.00 per TB scanned in us-east-1; Athena for Apache Spark costs $0.35 per DPU-hour; provisioned capacity costs $0.30 per DPU-hour. Glue Iceberg compaction costs $0.44 per DPU-hour in us-east-1; Glue 6.0+ ETL jobs cost $0.308 per DPU-hour and older Glue ETL $0.44. S3 Tables in us-east-1: compaction is $0.005 per GB processed plus $0.002 per 1,000 objects compacted; sort or Z-order compaction is $0.01 per GB plus $0.002 per 1,000 objects; storage is $0.0265 per GB-month for the first 50 TB. | EMR on EC2 charges an EMR price in addition to the EC2 price (and EBS), billed per second with a one-minute minimum. AWS's example in us-east-1: c4.2xlarge EMR charge $0.105/hour next to EC2 $0.398/hour; one master and two core nodes for a month total $1,101.57. Spot instances are up to a 90% discount versus On-Demand. EMR on EKS adds an uplift per vCPU-hour and per GB-hour to EKS/EC2 or Fargate costs, plus $0.10 per hour for each EKS cluster. AWS's worked example uses $0.007084 per vCPU-hour and $0.000777875 per GB-hour (100 vCPU and 300 GB for 30 minutes = $0.4709 EMR uplift). The uplift applies to any pod using an EMR runtime image, including with the Spark operator. EMR Serverless is billed per vCPU-hour and GB-hour. AWS's worked example uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour for a job with 100 vCPU and 750 GB for 30 minutes, then 200 more vCPU for 15 minutes, totaling $9.596275. In the same AWS benchmark the cost was $4.81 for EMR 7.12, $5.47 for EMR 7.5 and $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0. Cost formula uses r5d.4xlarge at $1.152/hour plus $0.27/hour EMR plus EBS, On-Demand. | | Code changes and migration | The EMR runtime for Spark is described as 100% API-compatible with open-source Spark, so existing Spark jobs need no rewrite; EMR is native Spark on EC2, EKS or Serverless. Athena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client; PySpark engine version 3 offers the console notebook experience. | The EMR runtime for Spark provides 'full API compatibility with open source Apache Spark and Apache Iceberg', so existing Spark jobs run as is. Optimized runtimes are used by EMR on EC2, EMR Serverless, EMR on EKS, EMR on Outposts and AWS Glue. Self-run Spark on EKS uses the open-source Spark operator and needs no job rewrite. | | Tuning and table upkeep | EMR enables adaptive query execution, dynamic partition pruning, bloom filter join and optimized join reorder; docs say AQE adaptive join conversion is on by default and can be switched off with a Spark setting. Athena compaction is manual: run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM yourself on Iceberg tables. Glue Data Catalog Iceberg compaction is disabled by default for new tables and must be enabled per table with an IAM role. S3 Tables compaction is on by default for all tables with a 512 MB default target file size (configurable 64 to 512 MB) and also applies row-level deletes. | Iceberg docs say users must run maintenance: expire snapshots, remove old metadata files and delete orphan files are 'recommended'; compacting data files with the Spark rewriteDataFiles action (for example target-file-size-bytes of 500 MB) and rewriting manifests are 'optional' maintenance, run as jobs you schedule. Hudi merge-on-read tables need compaction to merge log files into base files and bound read latency. Compaction has two steps, scheduling (plan written to the timeline) and execution; the docs describe strategies for choosing which file slices to compact. Delta Lake OSS docs list file management features the user applies: compaction (bin-packing) with OPTIMIZE, auto compaction, data skipping, Z-ordering, multi-part checkpointing, log compactions and optimized write. | | Spark versions and table formats | EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0); the EMR Spark feature page says EMR supports Apache Iceberg v3 and Spark 4.0. Athena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only with tables in the AWS Glue Data Catalog and Glue optimistic locking; DDL is not supported on tables registered with Lake Formation. | Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. All three table formats are available on EMR. EMR on EKS supports the Kubernetes Operator for Apache Spark (Spark operator) as a job submission model with the EMR runtime on your own EKS clusters. The Kubeflow Spark Operator is Apache-2.0 licensed with Helm installation. | | Where it runs | All of these are AWS services, so they run only on AWS; EMR, Athena, Glue and S3 are priced in us-east-1 on the AWS price list. | Amazon EMR has regional endpoints in the US (us-east-1, us-east-2, us-west-1, us-west-2) and many other AWS regions per the AWS general reference endpoints table. Spark runs on EC2 instances the user chooses (the AWS benchmark used r5d.4xlarge). | | How open its benchmarks are | None found on 10 Nov 2026. | AWS publishes the setup in GitHub repositories (aws-samples/emr-on-eks-benchmark branch tpcds-v2.13_iceberg, and aws-samples/emr-spark-benchmark) with build instructions, so the TPC-DS Iceberg benchmark can be repeated. | | Compute Bill | Amazon EMR on EKS (EC2 nodes): $24.42; Amazon EMR on EC2: $25.20; Amazon EMR Serverless: $31.63 | Open stack: Spark on plain EC2: $20.16 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/ind: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/glue/latest/dg/enable-compaction.html: https://docs.aws.amazon.com/glue/latest/dg/enable-compaction.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.js: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.j: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operato: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: iceberg.apache.org/docs/latest/maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: hudi.apache.org/docs/compaction: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.delta.io/latest/optimizations-oss.html: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/general/latest/gr/emr.html: https://docs.aws.amazon.com/general/latest/gr/emr.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/aws-emr-athena-iceberg-vs-open-stack-emr-eks/ --- # Amazon EMR plus Athena on Iceberg and Snowflake: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR plus Athena on Iceberg and Snowflake are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Amazon EMR plus Athena on Iceberg | Snowflake | | --- | --- | --- | | Kind | Spark and SQL engines on AWS | Cloud data platform with PySpark support | | Does well | Per AWS documentation, the EMR runtime for Spark is 100% API-compatible with open-source Spark, so existing jobs need no rewrite (read 2026-11-10). AWS publishes the setup for its TPC-DS 3 TB Iceberg benchmark in GitHub repositories, so a reader can repeat the run (read 2026-11-10). | Per Snowflake documentation, Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect, and batch jobs are submitted with the Snowpark Submit CLI (read 2026-11-10). Per Snowflake documentation, Iceberg tables can keep data in customer-managed S3, GCS or Azure storage through external volumes and a catalog integration such as AWS Glue. | | Limited | Per AWS documentation, Athena compaction is manual (you run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM), and Glue Iceberg compaction is off by default for new tables. Every service in this entry runs only on AWS, and Athena supports Iceberg 1.4.2 only for tables in the Glue Data Catalog (read 2026-11-10). | Per Snowflake documentation, Snowpark Connect supports Apache Spark 3.5 only, and DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported (read 2026-11-10). On-demand credit prices on AWS US East (Northern Virginia) are $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical) per the Snowflake Service Consumption Table effective 2026-11-09 (read 2026-11-10); the Snowflake pricing page itself renders by script. | | Published speed results | AWS states its runtime executes up to 4.5x faster than open-source Spark and gives 2.7x faster Iceberg write performance; the page gives no workload, scale or configuration. | None found on 10 Nov 2026. | | Published prices and cost results | EMR on EC2 adds a per-instance uplift in addition to EC2: the AWS price list shows $0.048 per hour for EMR m5.xlarge in us-east-1 (publication 2026-11-10). EMR on EKS uplift in us-east-1 is $0.007084 per vCPU-hour and $0.000777875 per GB-hour, in addition to the EC2 or Fargate cost. The EMR pricing page worked example for EMR Serverless uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour; a job with 100 vCPU for 30 minutes plus 200 more vCPU for 15 minutes costs $9.596275 in that example. Athena SQL costs $5.00 per TB scanned in us-east-1; Athena for Apache Spark costs $0.35 per DPU-hour; provisioned capacity costs $0.30 per DPU-hour. Glue Iceberg compaction costs $0.44 per DPU-hour in us-east-1; Glue 6.0+ ETL jobs cost $0.308 per DPU-hour and older Glue ETL $0.44. S3 Tables in us-east-1: compaction is $0.005 per GB processed plus $0.002 per 1,000 objects compacted; sort or Z-order compaction is $0.01 per GB plus $0.002 per 1,000 objects; storage is $0.0265 per GB-month for the first 50 TB. | Warehouses bill credits per second with a 60-second minimum on each start; each size step roughly doubles credits per hour (Small is 2 credits per hour, Medium is 4). Snowflake's Service Consumption Table (effective 2026-11-09) lists on-demand credits on AWS US East at $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical). | | Code changes and migration | The EMR runtime for Spark is described as 100% API-compatible with open-source Spark, so existing Spark jobs need no rewrite; EMR is native Spark on EC2, EKS or Serverless. Athena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client; PySpark engine version 3 offers the console notebook experience. | Snowpark Connect for Apache Spark runs existing PySpark DataFrame code against the Snowflake engine over the Spark Connect protocol; there is no Spark cluster, all workloads run on a Snowflake warehouse. Batch jobs are submitted with the Snowpark Submit CLI. Because Snowflake rather than Spark is the engine, there are semantic differences: DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported, and ByteType, ShortType and IntegerType are returned as LongType. The docs list unsupported APIs for DataFrame and Dataset. Docs say the Spark DataFrame and Spark SQL code can be reused with minimal migration or changes; the compatibility notes say Spark Connect defers plan resolution, which changes behavior versus Spark Classic. | | Tuning and table upkeep | EMR enables adaptive query execution, dynamic partition pruning, bloom filter join and optimized join reorder; docs say AQE adaptive join conversion is on by default and can be switched off with a Spark setting. Athena compaction is manual: run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM yourself on Iceberg tables. Glue Data Catalog Iceberg compaction is disabled by default for new tables and must be enabled per table with an IAM role. S3 Tables compaction is on by default for all tables with a 512 MB default target file size (configurable 64 to 512 MB) and also applies row-level deletes. | Docs provide a separate Optimizing Snowpark Connect for Spark workloads section and a Troubleshooting section covering known limitations, so some job-level adjustment is expected after migration. | | Spark versions and table formats | EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0); the EMR Spark feature page says EMR supports Apache Iceberg v3 and Spark 4.0. Athena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only with tables in the AWS Glue Data Catalog and Glue optimistic locking; DDL is not supported on tables registered with Lake Formation. | Snowpark Connect supports Apache Spark 3.5 only; workloads written for 3.4 and below or 4.0 and above are not supported and may produce protocol errors or missing API failures. Snowflake supports Iceberg tables (Parquet) with data in customer-managed S3, GCS or Azure storage via external volumes, an external catalog integration such as AWS Glue, and Delta files read through Delta Direct; Hudi was not listed on the page. | | Where it runs | All of these are AWS services, so they run only on AWS; EMR, Athena, Glue and S3 are priced in us-east-1 on the AWS price list. | Snowflake runs on AWS, Azure and Google Cloud; AWS regions include us-east-1, us-west-2, eu-west-1 and ca-central-1. Each account is hosted in a single region. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | Amazon EMR on EKS (EC2 nodes): $24.42; Amazon EMR on EC2: $25.20; Amazon EMR Serverless: $31.63 | Snowflake (XL standard warehouse): $96.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/ind: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/glue/latest/dg/enable-compaction.html: https://docs.aws.amazon.com/glue/latest/dg/enable-compaction.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.js: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.j: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/cost-understanding-compute: https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/tables-iceberg: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/intro-regions.md: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/aws-emr-athena-iceberg-vs-snowflake/ --- # Amazon EMR plus Athena on Iceberg and Starburst: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Amazon EMR plus Athena on Iceberg and Starburst are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Amazon EMR plus Athena on Iceberg | Starburst | | --- | --- | --- | | Kind | Spark and SQL engines on AWS | Managed Trino service for Iceberg | | Does well | Per AWS documentation, the EMR runtime for Spark is 100% API-compatible with open-source Spark, so existing jobs need no rewrite (read 2026-11-10). AWS publishes the setup for its TPC-DS 3 TB Iceberg benchmark in GitHub repositories, so a reader can repeat the run (read 2026-11-10). | Per Starburst documentation, Galaxy creates and writes Iceberg tables (Parquet, ORC or Avro) and runs maintenance with ALTER TABLE ... EXECUTE optimize, expire_snapshots and remove_orphan_files (read 2026-11-10). Starburst publishes credit prices by AWS region, for example $0.50 (Pro) and $0.75 (Enterprise) in US East (N. Virginia) and $0.60 and $0.90 in Frankfurt (read 2026-11-10). | | Limited | Per AWS documentation, Athena compaction is manual (you run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM), and Glue Iceberg compaction is off by default for new tables. Every service in this entry runs only on AWS, and Athena supports Iceberg 1.4.2 only for tables in the Glue Data Catalog (read 2026-11-10). | Per Starburst documentation, Galaxy runs SQL on Trino, not Spark, so DataFrame code, UDFs and PySpark would be rewritten as Trino SQL or kept on Spark (read 2026-11-10). Per Starburst documentation, Hudi tables are read-only in Galaxy: it cannot create new Hudi tables or write to them. | | Published speed results | AWS states its runtime executes up to 4.5x faster than open-source Spark and gives 2.7x faster Iceberg write performance; the page gives no workload, scale or configuration. | Starburst reported GPU-accelerated Trino results (its proprietary distribution, not open-source Trino): TPC-H up to 6x on accelerated queries and ClickBench 3.4x overall, single-node development environment. This is a SQL query test, not Spark batch work. | | Published prices and cost results | EMR on EC2 adds a per-instance uplift in addition to EC2: the AWS price list shows $0.048 per hour for EMR m5.xlarge in us-east-1 (publication 2026-11-10). EMR on EKS uplift in us-east-1 is $0.007084 per vCPU-hour and $0.000777875 per GB-hour, in addition to the EC2 or Fargate cost. The EMR pricing page worked example for EMR Serverless uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour; a job with 100 vCPU for 30 minutes plus 200 more vCPU for 15 minutes costs $9.596275 in that example. Athena SQL costs $5.00 per TB scanned in us-east-1; Athena for Apache Spark costs $0.35 per DPU-hour; provisioned capacity costs $0.30 per DPU-hour. Glue Iceberg compaction costs $0.44 per DPU-hour in us-east-1; Glue 6.0+ ETL jobs cost $0.308 per DPU-hour and older Glue ETL $0.44. S3 Tables in us-east-1: compaction is $0.005 per GB processed plus $0.002 per 1,000 objects compacted; sort or Z-order compaction is $0.01 per GB plus $0.002 per 1,000 objects; storage is $0.0265 per GB-month for the first 50 TB. | Galaxy credit prices start at $0.50 (Pro), $0.75 (Enterprise) and $1.00 (Mission Critical) per credit. A Free tier allows up to 3 clusters. Pricing depends on compute usage, tier and the cloud provider and region. Annual commitments may get discounts. On-demand list prices per credit (Pro / Enterprise / Mission Critical) on AWS: US East N. Virginia, US East Ohio and US West Oregon $0.50 / $0.75 / $1.00; Europe Ireland $0.56 / $0.84 / $1.12; Europe Frankfurt $0.60 / $0.90 / $1.20; Asia Pacific Tokyo $0.64 / $0.97 / $1.29; Brazil (Sao Paulo) $0.80 / $1.19 / $1.59. Cluster compute credit consumption: 1 worker 6 credits per hour, 2 workers 12, 4 workers 24, 8 workers 48, 16 workers 96. Managed ingest: file ingest 0.60 credits per hour; table maintenance and data transforms 0.30 credits per hour. Billing tracks cluster compute hours, data ingestion compute and private link data transfer. | | Code changes and migration | The EMR runtime for Spark is described as 100% API-compatible with open-source Spark, so existing Spark jobs need no rewrite; EMR is native Spark on EC2, EKS or Serverless. Athena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client; PySpark engine version 3 offers the console notebook experience. | Galaxy is 'a fully-managed data lake analytics platform built in addition to Trino'. Work runs as SQL on Trino, not Spark, so Spark jobs (DataFrame code, UDFs, PySpark) would have to be rewritten as Trino SQL, or kept on Spark against the same Iceberg tables. The Galaxy docs read contain no Spark execution mode. | | Tuning and table upkeep | EMR enables adaptive query execution, dynamic partition pruning, bloom filter join and optimized join reorder; docs say AQE adaptive join conversion is on by default and can be switched off with a Spark setting. Athena compaction is manual: run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM yourself on Iceberg tables. Glue Data Catalog Iceberg compaction is disabled by default for new tables and must be enabled per table with an IAM role. S3 Tables compaction is on by default for all tables with a 512 MB default target file size (configurable 64 to 512 MB) and also applies row-level deletes. | Iceberg table maintenance in Galaxy is run with ALTER TABLE ... EXECUTE commands: optimize (compaction; files under file_size_threshold, default 100MB, are merged), drop_extended_stats, expire_snapshots, remove_orphan_files. Live table maintenance for ingest tables is a separate Galaxy page. For batch-style work Galaxy offers a fault tolerant execution mode that spools intermediate data and retries failed parts of a query. Docs say 'No other configuration is required' to enable it, but not to use it if most queries run under about a minute, and that it can be slightly slower than normal execution. FTE and Accelerated modes are not available on the Free cluster size and not all catalogs support FTE. | | Spark versions and table formats | EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0); the EMR Spark feature page says EMR supports Apache Iceberg v3 and Spark 4.0. Athena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only with tables in the AWS Glue Data Catalog and Glue optimistic locking; DDL is not supported on tables registered with Lake Formation. | Galaxy's Great Lakes connectivity lists Iceberg, Delta Lake, Hive and Hudi table format pages. Hudi tables are read-only: 'Galaxy cannot create new Hudi tables or write to them.' Iceberg tables can be created and written; the Iceberg data file format can be Parquet, ORC or Avro. | | Where it runs | All of these are AWS services, so they run only on AWS; EMR, Athena, Glue and S3 are priced in us-east-1 on the AWS price list. | Galaxy prices are listed for AWS, Azure and GCP regions. AWS regions priced include us-east-1, us-east-2, us-west-1, us-west-2, ca-central, Ireland, London, Frankfurt, Zurich, Singapore, Sydney, Tokyo, Cape Town, UAE, Tel Aviv and Sao Paulo. A cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | Amazon EMR on EKS (EC2 nodes): $24.42; Amazon EMR on EC2: $25.20; Amazon EMR Serverless: $31.63 | Starburst Galaxy (Pro tier): $60.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/features/spark: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/ind: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/glue/latest/dg/enable-compaction.html: https://docs.aws.amazon.com/glue/latest/dg/enable-compaction.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.js: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Amazon EMR plus Amazon Athena on Apache Iceberg: pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.j: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing: https://www.starburst.io/pricing/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing/pricing-details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/index.html: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/optimization-perform: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/aws-emr-athena-iceberg-vs-starburst/ --- # Databricks and Dremio: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks (Managed lakehouse built around Delta Lake) and Dremio (Lakehouse with its own SQL engine) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Databricks on AWS (Jobs Compute): $32.40; Dremio Cloud (Large engine): $102.40. ## The same rows for both | Row | Databricks | Dremio | | --- | --- | --- | | Kind | Managed lakehouse built around Delta Lake | Lakehouse with its own SQL engine | | Does well | Per Databricks documentation, Photon keeps the Spark Catalyst planner, works with existing Spark API code and falls back to the Spark runtime for unsupported operations (read 2026-11-10). Per Databricks documentation, Delta tables can generate Iceberg metadata from Runtime 14.3 LTS, so Iceberg clients read Delta data without rewriting files (read 2026-11-10). | Per Dremio documentation, Iceberg tables in the Open Catalog get automatic optimization: data file compaction, row-level delete handling, clustering, partition evolution and manifest rewriting (read 2026-11-10). Dremio publishes DCU rates by engine size, from 16 DCUs per hour for 2XSmall to 2048 for 3XLarge, with a 60-second minimum (read 2026-11-10). | | Limited | Per Databricks documentation, serverless compute supports only Spark Connect APIs, so RDD APIs, R and Maven coordinates are unsupported, and ANSI mode on by default can change results of existing jobs. Per Databricks, compute is billed in DBUs in addition to the cloud bill for the same instances, so a Jobs Compute run pays Databricks and AWS separately; the Compute Bill line adds $12.24 of DBUs to $20.16 of EC2 (read 2026-11-10). | Per Dremio documentation, queries run on its own columnar engine, not on Spark, and the pages read describe no mode that runs existing Spark jobs. Dremio Cloud documentation lists Iceberg and Delta Lake as table formats; Hudi support: none found on 2026-11-10. | | Published speed results | Docs claim Photon gives up to 5x better price/performance for data and analytics workloads versus other cloud data warehouses; no method is given on this page. Databricks announced an audited 100TB TPC-DS result of 32,941,245 QphDS for Databricks SQL, 2.2x the previous record held by Alibaba, with total system cost 10% lower on list pricing. This is a SQL warehouse result, not a Spark batch ETL result. The same Databricks blog says research from Barcelona Supercomputing Center found Databricks 2.7x faster and 12x better in price-performance than Snowflake on a TPC-DS derivative. This is a vendor-reported claim of a third-party study; the study itself was not fetched. | Dremio's founder reported a full 1 TB TPC-DS run (all 99 queries) in 22 seconds on an 8-node m7gd.4xlarge cluster, with no caching, and attributes it to Autonomous Reflections and automatic Iceberg clustering. This is a BI query benchmark run by the vendor, not a Spark batch ETL or compaction test. | | Published prices and cost results | Databricks bills compute in DBUs at per-second granularity with pay-as-you-go or committed-use discounts, and no up-front costs. The pricing pages render their rates by script; the AWS pricing data file those pages load lists Jobs Compute at $0.15 per DBU on the plan it names Premium (r6i.4xlarge: 4.08 DBU per hour). The file names no region. Jobs pricing offers serverless (Performance Optimized and Standard modes) and classic clusters; the jobs pricing page maps plan names across clouds (its Premium tier on Azure corresponds to Enterprise on AWS and GCP), while the AWS pricing data file lists the $0.15 Jobs Compute rate under a plan it names Premium. Photon instances consume DBUs at a different rate. | Dremio Cloud is priced at $0.20 per DCU (Dremio Compute Unit). Dremio Enterprise (self-managed) pricing is 'contact sales'. The free trial for Dremio Cloud is a $400 credit for 30 days. DCUs are consumed only while engines run. Usage equals engine and replica uptime in seconds (60-second minimum) times the DCU rate for the engine size. Published rates per hour: 2XSmall 16, XSmall 32, Small 64, Medium 128, Large 256, XLarge 512, 2XLarge 1024, 3XLarge 2048. Storage billed in TB-months only for Dremio-hosted storage; LLM tokens billed separately. Billable usage is DCUs (compute), LLM tokens (AI features) and storage. Compute is billed in DCUs only while engines are running. | | Code changes and migration | Photon is a native C++ vectorized engine that keeps Spark's Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime. Serverless compute for notebooks and jobs supports only Spark Connect APIs; Spark RDD APIs, R, Maven coordinates, global temp views, DBFS mounts with instance profiles and Databricks Container Services are not supported. ANSI mode is on by default and the default time zone is UTC, which can change results of existing jobs. | Dremio runs queries on its own columnar execution engine built on Apache Arrow, not on Spark. Docs describe Spark and Flink only as external engines that can use the Open Catalog (Iceberg REST) alongside Dremio. The pages read describe no mode that runs existing Spark jobs, so Spark ETL would stay on Spark or be rewritten as Dremio SQL. | | Tuning and table upkeep | Photon is on by default for classic all-purpose compute, jobs compute and classic pipelines; API users must set runtime_engine to PHOTON. Photon instance types use DBUs at a different rate than the same instance without Photon. On serverless compute Databricks allocates and manages compute so users do not provision clusters; the Spark UI and Spark logs are not available, query profile and client-side logs are used instead. Serverless compute runs workloads without provisioning compute in your cloud account, and Databricks says this reduces the need to manage compute resources. It requires Unity Catalog to reach external data sources. | For Iceberg tables in the Open Catalog, Dremio runs automatic optimization on a dedicated engine it configures: data file compaction, row-level delete handling (position deletes on v2, deletion vectors on v3), clustering, partition evolution and manifest rewriting. The same operations can be run by hand with OPTIMIZE TABLE. Optimization works in batches sized to engine resources. Engines have settings the admin chooses: size (executors per replica), min and max replicas, last-replica auto-stop (default 2 hours), enqueued time limit (default 5 minutes) and query runtime limit. Default concurrency per replica ranges from 2 (2XSmall) to 20 (3XLarge). | | Spark versions and table formats | Databricks Runtime 18 LTS ships Apache Spark 4.1.0 (released June 10, 2026); Runtime 17.3 LTS ships Spark 4.0.0; Runtime 16.4 LTS and 15.4 LTS ship Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). Delta Lake tables can generate Iceberg metadata (UniForm, Iceberg reads) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files. | The Dremio Cloud data formats page lists Apache Iceberg and Delta Lake as table formats (plus Parquet and text formats). Hudi is not listed on that page. The Open Catalog is built on Apache Polaris and supports the Iceberg REST Catalog API; the docs list AWS Glue Data Catalog as a connectable catalog. | | Where it runs | Databricks on AWS lists many AWS regions including us-east-1 (US East, Northern Virginia), us-east-2, us-west-1, eu-west-1, eu-central-1, ap-southeast-2 and us-gov-west-1; the platform is also hosted on Azure and Google Cloud. Some features are only in a subset of regions. | Dremio Cloud runs on AWS (Azure listed as coming soon). Supported regions: us-east-1, us-east-2, us-west-2, eu-west-1 and eu-central-1. Dremio Enterprise can be deployed on Kubernetes, on premises, or in AWS, Azure or GCP. | | How open its benchmarks are | None found on 10 Nov 2026. | Dremio published a 2020 TPC-DS methodology post (SF1000 and SF10000, Parquet on S3, Apache JMeter test plans, 58 of 99 queries that ran unmodified on Dremio and Presto distributions). The 2025 22-second TPC-DS post pages read give cluster type but no scripts, scale-factor files or configuration repository. | | Compute Bill | Databricks on AWS (Jobs Compute): $32.40 | Dremio Cloud (Large engine): $102.40 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless/limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless: https://docs.databricks.com/aws/en/compute/serverless/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/release-notes/runtime: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/delta/uniform: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/resources/supported-regions: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/aws-pricing: https://www.databricks.com/product/aws-pricing (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/pricing/jobs: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) - Dremio: dremio.com/pricing: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/engines: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/subscription/usage: https://docs.dremio.com/dremio-cloud/admin/subscription/usage (read 10 Nov 2026) - Dremio: docs.dremio.com/current/what-is-dremio/architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/developer/data-formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/manage-govern/optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/about/regions: https://docs.dremio.com/dremio-cloud/about/regions (read 10 Nov 2026) - Dremio: dremio.com/blog/breakthrough-announcement-dremio-is-the: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026) - Dremio: dremio.com/blog/dremio-benchmarking-methodology: https://www.dremio.com/blog/dremio-benchmarking-methodology/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/databricks-vs-dremio/ --- # Databricks and DualBird: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks (Managed lakehouse built around Delta Lake) and DualBird (FPGA Spark plugin for AWS F2) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Databricks on AWS (Jobs Compute): $32.40; DualBird on Amazon EMR (F2 instances): $49.50. ## The same rows for both | Row | Databricks | DualBird | | --- | --- | --- | | Kind | Managed lakehouse built around Delta Lake | FPGA Spark plugin for AWS F2 | | Does well | Per Databricks documentation, Photon keeps the Spark Catalyst planner, works with existing Spark API code and falls back to the Spark runtime for unsupported operations (read 2026-11-10). Per Databricks documentation, Delta tables can generate Iceberg metadata from Runtime 14.3 LTS, so Iceberg clients read Delta data without rewriting files (read 2026-11-10). | DualBird says onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin, with Spark still handling planning and scheduling (read 2026-11-10). DualBird published a 100 GB sort-based Iceberg compaction run on June 11, 2026, run by DualBird, and states about $2 per TB for itself against about $6 per TB for CPU clusters. | | Limited | Per Databricks documentation, serverless compute supports only Spark Connect APIs, so RDD APIs, R and Maven coordinates are unsupported, and ANSI mode on by default can change results of existing jobs. Per Databricks, compute is billed in DBUs in addition to the cloud bill for the same instances, so a Jobs Compute run pays Databricks and AWS separately; the Compute Bill line adds $12.24 of DBUs to $20.16 of EC2 (read 2026-11-10). | DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. A price for the plugin and a benchmark run by a party other than DualBird: none found on 2026-11-10. | | Published speed results | Docs claim Photon gives up to 5x better price/performance for data and analytics workloads versus other cloud data warehouses; no method is given on this page. Databricks announced an audited 100TB TPC-DS result of 32,941,245 QphDS for Databricks SQL, 2.2x the previous record held by Alibaba, with total system cost 10% lower on list pricing. This is a SQL warehouse result, not a Spark batch ETL result. The same Databricks blog says research from Barcelona Supercomputing Center found Databricks 2.7x faster and 12x better in price-performance than Snowflake on a TPC-DS derivative. This is a vendor-reported claim of a third-party study; the study itself was not fetched. | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | | Published prices and cost results | Databricks bills compute in DBUs at per-second granularity with pay-as-you-go or committed-use discounts, and no up-front costs. The pricing pages render their rates by script; the AWS pricing data file those pages load lists Jobs Compute at $0.15 per DBU on the plan it names Premium (r6i.4xlarge: 4.08 DBU per hour). The file names no region. Jobs pricing offers serverless (Performance Optimized and Standard modes) and classic clusters; the jobs pricing page maps plan names across clouds (its Premium tier on Azure corresponds to Enterprise on AWS and GCP), while the AWS pricing data file lists the $0.15 Jobs Compute rate under a plan it names Premium. Photon instances consume DBUs at a different rate. | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | | Code changes and migration | Photon is a native C++ vectorized engine that keeps Spark's Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime. Serverless compute for notebooks and jobs supports only Spark Connect APIs; Spark RDD APIs, R, Maven coordinates, global temp views, DBFS mounts with instance profiles and Databricks Container Services are not supported. ANSI mode is on by default and the default time zone is UTC, which can change results of existing jobs. | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | | Tuning and table upkeep | Photon is on by default for classic all-purpose compute, jobs compute and classic pipelines; API users must set runtime_engine to PHOTON. Photon instance types use DBUs at a different rate than the same instance without Photon. On serverless compute Databricks allocates and manages compute so users do not provision clusters; the Spark UI and Spark logs are not available, query profile and client-side logs are used instead. Serverless compute runs workloads without provisioning compute in your cloud account, and Databricks says this reduces the need to manage compute resources. It requires Unity Catalog to reach external data sources. | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | | Spark versions and table formats | Databricks Runtime 18 LTS ships Apache Spark 4.1.0 (released June 10, 2026); Runtime 17.3 LTS ships Spark 4.0.0; Runtime 16.4 LTS and 15.4 LTS ship Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). Delta Lake tables can generate Iceberg metadata (UniForm, Iceberg reads) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files. | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | | Where it runs | Databricks on AWS lists many AWS regions including us-east-1 (US East, Northern Virginia), us-east-2, us-west-1, eu-west-1, eu-central-1, ap-southeast-2 and us-gov-west-1; the platform is also hosted on Azure and Google Cloud. Some features are only in a subset of regions. | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | | How open its benchmarks are | None found on 10 Nov 2026. | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | | Compute Bill | Databricks on AWS (Jobs Compute): $32.40 | DualBird on Amazon EMR (F2 instances): $49.50 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless/limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless: https://docs.databricks.com/aws/en/compute/serverless/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/release-notes/runtime: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/delta/uniform: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/resources/supported-regions: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/aws-pricing: https://www.databricks.com/product/aws-pricing (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/pricing/jobs: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/databricks-vs-dualbird/ --- # Databricks and Google BigQuery and BigLake: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks (Managed lakehouse built around Delta Lake) and Google BigQuery and BigLake (Serverless Google Cloud data warehouse) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Databricks on AWS (Jobs Compute): $32.40; BigQuery (Enterprise edition slots): $19.20. ## The same rows for both | Row | Databricks | Google BigQuery and BigLake | | --- | --- | --- | | Kind | Managed lakehouse built around Delta Lake | Serverless Google Cloud data warehouse | | Does well | Per Databricks documentation, Photon keeps the Spark Catalyst planner, works with existing Spark API code and falls back to the Spark runtime for unsupported operations (read 2026-11-10). Per Databricks documentation, Delta tables can generate Iceberg metadata from Runtime 14.3 LTS, so Iceberg clients read Delta data without rewriting files (read 2026-11-10). | Per Google documentation, BigQuery Iceberg managed tables store data in buckets you own, support DML and streaming, and run automatic compaction, clustering and garbage collection (read 2026-11-10). Per Google documentation, stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud (read 2026-11-10). | | Limited | Per Databricks documentation, serverless compute supports only Spark Connect APIs, so RDD APIs, R and Maven coordinates are unsupported, and ANSI mode on by default can change results of existing jobs. Per Databricks, compute is billed in DBUs in addition to the cloud bill for the same instances, so a Jobs Compute run pays Databricks and AWS separately; the Compute Bill line adds $12.24 of DBUs to $20.16 of EC2 (read 2026-11-10). | Per Google documentation, BigQuery Omni on S3 does not support DML or CREATE TABLE, so it does not run write-heavy Spark batch ETL on S3. Google's pricing page text lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate (read 2026-11-10). Per-TiB on-demand and Spark DCU rates are not used on this site. | | Published speed results | Docs claim Photon gives up to 5x better price/performance for data and analytics workloads versus other cloud data warehouses; no method is given on this page. Databricks announced an audited 100TB TPC-DS result of 32,941,245 QphDS for Databricks SQL, 2.2x the previous record held by Alibaba, with total system cost 10% lower on list pricing. This is a SQL warehouse result, not a Spark batch ETL result. The same Databricks blog says research from Barcelona Supercomputing Center found Databricks 2.7x faster and 12x better in price-performance than Snowflake on a TPC-DS derivative. This is a vendor-reported claim of a third-party study; the study itself was not fetched. | Google states Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name; no method is given on the pricing page. | | Published prices and cost results | Databricks bills compute in DBUs at per-second granularity with pay-as-you-go or committed-use discounts, and no up-front costs. The pricing pages render their rates by script; the AWS pricing data file those pages load lists Jobs Compute at $0.15 per DBU on the plan it names Premium (r6i.4xlarge: 4.08 DBU per hour). The file names no region. Jobs pricing offers serverless (Performance Optimized and Standard modes) and classic clusters; the jobs pricing page maps plan names across clouds (its Premium tier on Azure corresponds to Enterprise on AWS and GCP), while the AWS pricing data file lists the $0.15 Jobs Compute rate under a plan it names Premium. Photon instances consume DBUs at a different rate. | BigQuery offers on-demand pricing per TiB scanned (first 1 TiB per month free) or capacity pricing per slot-hour. BigQuery Omni has its own on-demand per TiB pricing for AWS regions. The page text shows $0.06 per slot-hour for Enterprise edition pay-as-you-go compute, with no region shown beside the rate (read 2026-11-10). Google's Spark service (Managed Service for Apache Spark, formerly Dataproc) bills serverless Spark per second in Data Compute Units with a 1-minute minimum, plus shuffle storage; Standard tier is for batch ETL, Premium tier uses Lightning Engine. | | Code changes and migration | Photon is a native C++ vectorized engine that keeps Spark's Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime. Serverless compute for notebooks and jobs supports only Spark Connect APIs; Spark RDD APIs, R, Maven coordinates, global temp views, DBFS mounts with instance profiles and Databricks Container Services are not supported. ANSI mode is on by default and the default time zone is UTC, which can change results of existing jobs. | On BigQuery Omni, DML statements and CREATE TABLE are not supported, BigQuery ML statements are not supported, and only EXPORT DATA queries are allowed for destination tables, so it does not run write-heavy Spark batch ETL on S3. Standard and Enterprise Plus editions do not support Omni regions. BigQuery stored procedures for Apache Spark let you run PySpark or Scala/Java JAR code from SQL via CREATE PROCEDURE ... OPTIONS(engine = "SPARK", runtime_version = "2.2"), using a Cloud Storage file or inline code and a connection; this runs on Google Cloud, so code must be wrapped as procedures. | | Tuning and table upkeep | Photon is on by default for classic all-purpose compute, jobs compute and classic pipelines; API users must set runtime_engine to PHOTON. Photon instance types use DBUs at a different rate than the same instance without Photon. On serverless compute Databricks allocates and manages compute so users do not provision clusters; the Spark UI and Spark logs are not available, query profile and client-side logs are used instead. Serverless compute runs workloads without provisioning compute in your cloud account, and Databricks says this reduces the need to manage compute resources. It requires Unity Catalog to reach external data sources. | Iceberg managed tables run automatic storage optimization (adaptive file sizing, clustering, garbage collection), so compaction needs no job configuration on Google Cloud; storage optimization is a separate billed item. | | Spark versions and table formats | Databricks Runtime 18 LTS ships Apache Spark 4.1.0 (released June 10, 2026); Runtime 17.3 LTS ships Spark 4.0.0; Runtime 16.4 LTS and 15.4 LTS ship Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). Delta Lake tables can generate Iceberg metadata (UniForm, Iceberg reads) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files. | BigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, export Iceberg V2 snapshots for open engines, and do automatic compaction, clustering and garbage collection. BigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security. | | Where it runs | Databricks on AWS lists many AWS regions including us-east-1 (US East, Northern Virginia), us-east-2, us-west-1, eu-west-1, eu-central-1, ap-southeast-2 and us-gov-west-1; the platform is also hosted on Azure and Google Cloud. Some features are only in a subset of regions. | BigQuery Omni lets BigQuery analytics run on data in Amazon S3 through BigLake tables, in AWS regions us-east-1, us-west-2, ap-northeast-2, ap-southeast-2, eu-west-1 and eu-central-1. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | Databricks on AWS (Jobs Compute): $32.40 | BigQuery (Enterprise edition slots): $19.20 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless/limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless: https://docs.databricks.com/aws/en/compute/serverless/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/release-notes/runtime: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/delta/uniform: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/resources/supported-regions: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/aws-pricing: https://www.databricks.com/product/aws-pricing (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/pricing/jobs: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/omni-introduction: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/spark-procedures: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/iceberg-tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/biglake-intro: https://cloud.google.com/bigquery/docs/biglake-intro (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/dataproc-serverless/pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/bigquery-biglake-vs-databricks/ --- # Databricks and Microsoft Fabric: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks (Managed lakehouse built around Delta Lake) and Microsoft Fabric (SaaS analytics on Azure with Spark) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Databricks on AWS (Jobs Compute): $32.40; Microsoft Fabric (F128 capacity): $46.08. ## The same rows for both | Row | Databricks | Microsoft Fabric | | --- | --- | --- | | Kind | Managed lakehouse built around Delta Lake | SaaS analytics on Azure with Spark | | Does well | Per Databricks documentation, Photon keeps the Spark Catalyst planner, works with existing Spark API code and falls back to the Spark runtime for unsupported operations (read 2026-11-10). Per Databricks documentation, Delta tables can generate Iceberg metadata from Runtime 14.3 LTS, so Iceberg clients read Delta data without rewriting files (read 2026-11-10). | Per Microsoft Learn, the native execution engine (Apache Gluten and Velox) needs no code changes, falls back to JVM Spark for unsupported operators and supports Parquet and Delta (read 2026-11-10). Per Microsoft Learn, OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying it (read 2026-11-10). | | Limited | Per Databricks documentation, serverless compute supports only Spark Connect APIs, so RDD APIs, R and Maven coordinates are unsupported, and ANSI mode on by default can change results of existing jobs. Per Databricks, compute is billed in DBUs in addition to the cloud bill for the same instances, so a Jobs Compute run pays Databricks and AWS separately; the Compute Bill line adds $12.24 of DBUs to $20.16 of EC2 (read 2026-11-10). | Per Microsoft Learn, Fabric runs over OneLake on Azure and the region page lists Azure regions only, so Spark compute does not run inside AWS. Writes to S3 through a OneLake shortcut are not possible, and the method for Microsoft's native engine benchmark: none found on 2026-11-10. | | Published speed results | Docs claim Photon gives up to 5x better price/performance for data and analytics workloads versus other cloud data warehouses; no method is given on this page. Databricks announced an audited 100TB TPC-DS result of 32,941,245 QphDS for Databricks SQL, 2.2x the previous record held by Alibaba, with total system cost 10% lower on list pricing. This is a SQL warehouse result, not a Spark batch ETL result. The same Databricks blog says research from Barcelona Supercomputing Center found Databricks 2.7x faster and 12x better in price-performance than Snowflake on a TPC-DS derivative. This is a vendor-reported claim of a third-party study; the study itself was not fetched. | Microsoft reports that on TPC-DS scale factor 1000 with Delta the native engine was up to six times faster than open-source Spark, with about 83% compute cost savings on a fixed-size cluster. | | Published prices and cost results | Databricks bills compute in DBUs at per-second granularity with pay-as-you-go or committed-use discounts, and no up-front costs. The pricing pages render their rates by script; the AWS pricing data file those pages load lists Jobs Compute at $0.15 per DBU on the plan it names Premium (r6i.4xlarge: 4.08 DBU per hour). The file names no region. Jobs pricing offers serverless (Performance Optimized and Standard modes) and classic clusters; the jobs pricing page maps plan names across clouds (its Premium tier on Azure corresponds to Enterprise on AWS and GCP), while the AWS pricing data file lists the $0.15 Jobs Compute rate under a plan it names Premium. Photon instances consume DBUs at a different rate. | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus (Spark Memory Optimized and autoscale for Spark entries, pay-as-you-go). In Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling. Autoscale Billing for Spark is an opt-in pay-as-you-go model that runs Spark on dedicated serverless resources at standard PAYG CU-hour rates; reserved capacity discounts do not apply and a base capacity is still required. | | Code changes and migration | Photon is a native C++ vectorized engine that keeps Spark's Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime. Serverless compute for notebooks and jobs supports only Spark Connect APIs; Spark RDD APIs, R, Maven coordinates, global temp views, DBFS mounts with instance profiles and Databricks Container Services are not supported. ANSI mode is on by default and the default time zone is UTC, which can change results of existing jobs. | The Fabric native execution engine, based on Apache Gluten and Velox, needs no code changes, falls back to JVM Spark for unsupported operators, and supports Parquet and Delta. | | Tuning and table upkeep | Photon is on by default for classic all-purpose compute, jobs compute and classic pipelines; API users must set runtime_engine to PHOTON. Photon instance types use DBUs at a different rate than the same instance without Photon. On serverless compute Databricks allocates and manages compute so users do not provision clusters; the Spark UI and Spark logs are not available, query profile and client-side logs are used instead. Serverless compute runs workloads without provisioning compute in your cloud account, and Databricks says this reduces the need to manage compute resources. It requires Unity Catalog to reach external data sources. | Starter pools start in 5 to 10 seconds with no setup; custom Spark pools let you tune node size and scaling, and native execution can be toggled per application with a Spark setting. | | Spark versions and table formats | Databricks Runtime 18 LTS ships Apache Spark 4.1.0 (released June 10, 2026); Runtime 17.3 LTS ships Spark 4.0.0; Runtime 16.4 LTS and 15.4 LTS ship Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). Delta Lake tables can generate Iceberg metadata (UniForm, Iceberg reads) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files. | Fabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1; OneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and vice versa. | | Where it runs | Databricks on AWS lists many AWS regions including us-east-1 (US East, Northern Virginia), us-east-2, us-west-1, eu-west-1, eu-central-1, ap-southeast-2 and us-gov-west-1; the platform is also hosted on Azure and Google Cloud. Some features are only in a subset of regions. | Fabric workloads run over OneLake, described as built on Azure Data Lake Storage; the region availability page lists Azure regions and no AWS regions, so Spark compute does not run inside AWS. OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; file caching can reduce cross-cloud egress costs. Writes to S3 are not possible through the shortcut. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | Databricks on AWS (Jobs Compute): $32.40 | Microsoft Fabric (F128 capacity): $46.08 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless/limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless: https://docs.databricks.com/aws/en/compute/serverless/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/release-notes/runtime: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/delta/uniform: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/resources/supported-regions: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/aws-pricing: https://www.databricks.com/product/aws-pricing (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/pricing/jobs: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview: https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - Microsoft Fabric: prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption' (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Microsoft Fabric: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-compute: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/databricks-vs-microsoft-fabric/ --- # Databricks and Onehouse: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks and Onehouse are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Databricks | Onehouse | | --- | --- | --- | | Kind | Managed lakehouse built around Delta Lake | Managed lakehouse in your cloud account | | Does well | Per Databricks documentation, Photon keeps the Spark Catalyst planner, works with existing Spark API code and falls back to the Spark runtime for unsupported operations (read 2026-11-10). Per Databricks documentation, Delta tables can generate Iceberg metadata from Runtime 14.3 LTS, so Iceberg clients read Delta data without rewriting files (read 2026-11-10). | Per Onehouse documentation, its jobs are fully compatible with Apache Spark and EMR code can be reused unless it uses AWS-specific features (read 2026-11-10). Onehouse published the setup for its 10 TB TPC-DS and 1 TB LakeLoader comparison against EMR 7.12, and open-sourced LakeLoader under Apache-2.0 (read 2026-11-10). | | Limited | Per Databricks documentation, serverless compute supports only Spark Connect APIs, so RDD APIs, R and Maven coordinates are unsupported, and ANSI mode on by default can change results of existing jobs. Per Databricks, compute is billed in DBUs in addition to the cloud bill for the same instances, so a Jobs Compute run pays Databricks and AWS separately; the Compute Bill line adds $12.24 of DBUs to $20.16 of EC2 (read 2026-11-10). | Per Onehouse documentation, billing is in OCUs on a bring-your-own-cloud model, and no dollar rate per OCU was found on 2026-11-10. Onehouse's results against EMR 7.12 are its own and shown as charts; a confirmation by another party: none found on 2026-11-10. | | Published speed results | Docs claim Photon gives up to 5x better price/performance for data and analytics workloads versus other cloud data warehouses; no method is given on this page. Databricks announced an audited 100TB TPC-DS result of 32,941,245 QphDS for Databricks SQL, 2.2x the previous record held by Alibaba, with total system cost 10% lower on list pricing. This is a SQL warehouse result, not a Spark batch ETL result. The same Databricks blog says research from Barcelona Supercomputing Center found Databricks 2.7x faster and 12x better in price-performance than Snowflake on a TPC-DS derivative. This is a vendor-reported claim of a third-party study; the study itself was not fetched. | Onehouse benchmarked Quanton against open-source Spark on Iceberg: TPC-DS up to about 3.1x, TPCx-BB 2x, TPC-DI (1 TB) 2x with 3-4x on MERGE INTO, and LakeLoader 4.1x for writes. The baseline is open-source Spark, not EMR. Onehouse reran its benchmarks against EMR 7.12 after AWS announced it: it reports EMR improved 32% over earlier versions and that Onehouse 'still comes out with a 2.5x better price/performance advantage'. Setup: 10 TB TPC-DS for read/transform, 1 TB LakeLoader merges for load, latest general-purpose Graviton instances. Results are shown as charts; Onehouse also reports Hudi and Delta query performance on EMR 7.12 was very similar to Iceberg. A B2B SaaS customer (unnamed) ran its UDF-heavy pipeline on EMR and on Quanton in its own AWS account on m8gd nodes. Quanton was faster at p50, p97 and p99 tenant sizes; on the largest tenant (21 billion rows) EMR filled local disks on 180 nodes and was aborted after 3.7 hours, and finished on 300 nodes. The write-up is published by Onehouse. | | Published prices and cost results | Databricks bills compute in DBUs at per-second granularity with pay-as-you-go or committed-use discounts, and no up-front costs. The pricing pages render their rates by script; the AWS pricing data file those pages load lists Jobs Compute at $0.15 per DBU on the plan it names Premium (r6i.4xlarge: 4.08 DBU per hour). The file names no region. Jobs pricing offers serverless (Performance Optimized and Standard modes) and classic clusters; the jobs pricing page maps plan names across clouds (its Premium tier on Azure corresponds to Enterprise on AWS and GCP), while the AWS pricing data file lists the $0.15 Jobs Compute rate under a plan it names Premium. Photon instances consume DBUs at a different rate. | Onehouse bills in Onehouse Compute Units (OCU) on a bring-your-own-cloud model: 1 OCU is a 4-core instance for 1 hour in your account, plus support costs; no separate volume or storage fee, and the customer pays its own AWS or GCP infrastructure bill. The docs page does not state a dollar rate per OCU. Vendor claims at the Quanton launch: 2-3x better price-performance for ETL, and a guaranteed greater-than-50% cost reduction for existing Hudi users moving Spark jobs from AWS EMR or open-source Spark. These are guarantees stated by the vendor, not results from an independent party. | | Code changes and migration | Photon is a native C++ vectorized engine that keeps Spark's Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime. Serverless compute for notebooks and jobs supports only Spark Connect APIs; Spark RDD APIs, R, Maven coordinates, global temp views, DBFS mounts with instance profiles and Databricks Container Services are not supported. ANSI mode is on by default and the default time zone is UTC, which can change results of existing jobs. | Onehouse docs say its Jobs are fully compatible with Apache Spark and that EMR code can be reused unless it uses AWS-specific features. Migration work listed: map EMR steps to Onehouse Jobs created through its API or Airflow operators (CREATE CLUSTER, CREATE JOB, RUN JOB, DELETE CLUSTER), and configure Glue catalog sync or contact support to make Glue the primary catalog. At the May 20, 2025 launch Onehouse said users can run existing Spark and SQL jobs on new cluster types and that existing Hudi/Spark jobs can be pointed at Onehouse 'with no code changes'. The Quanton Kubernetes Operator is described as a 'drop-in replacement' for an existing Spark Kubernetes operator. | | Tuning and table upkeep | Photon is on by default for classic all-purpose compute, jobs compute and classic pipelines; API users must set runtime_engine to PHOTON. Photon instance types use DBUs at a different rate than the same instance without Photon. On serverless compute Databricks allocates and manages compute so users do not provision clusters; the Spark UI and Spark logs are not available, query profile and client-side logs are used instead. Serverless compute runs workloads without provisioning compute in your cloud account, and Databricks says this reduces the need to manage compute resources. It requires Unity Catalog to reach external data sources. | Each cluster needs min and max OCU limits. Docs warn that if usage reaches Max OCU the cluster stops scaling up, which can delay processing; Min OCU keeps a cluster warm. The Onehouse docs pages read give no per-job Spark tuning list; Onehouse claims automatic scaling and automated async table services. | | Spark versions and table formats | Databricks Runtime 18 LTS ships Apache Spark 4.1.0 (released June 10, 2026); Runtime 17.3 LTS ships Spark 4.0.0; Runtime 16.4 LTS and 15.4 LTS ship Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). Delta Lake tables can generate Iceberg metadata (UniForm, Iceberg reads) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files. | Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake. Docs: Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog (AWS Glue and Snowflake Open Catalog are documented); for Delta Lake the user installs Delta Spark as a dependency in the JAR or Python job. In the Nov 12, 2025 Iceberg post Onehouse said Delta acceleration is 'up next'. Apache XTable (formerly OneTable) translates table metadata between Hudi, Delta Lake and Iceberg without being a new format. It is incubating at the Apache Software Foundation; Onehouse says it created Hudi and XTable. | | Where it runs | Databricks on AWS lists many AWS regions including us-east-1 (US East, Northern Virginia), us-east-2, us-west-1, eu-west-1, eu-central-1, ap-southeast-2 and us-gov-west-1; the platform is also hosted on Azure and Google Cloud. Some features are only in a subset of regions. | Onehouse runs EC2 or Google Compute Engine instances inside the customer's AWS or GCP account (Azure not listed in the billing docs). Standard instance types map to Graviton m8g sizes on AWS (oh-general-4 = m8g.xlarge, oh-general-8 = m8g.2xlarge, oh-general-16 = m8g.4xlarge). A project is scoped to one region and account; adding a region needs a support ticket. | | How open its benchmarks are | None found on 10 Nov 2026. | Onehouse open-sourced LakeLoader, a tool that generates incremental update/insert loads for benchmarking write performance. The repository is Apache-2.0 licensed, last pushed 2026-10-30. The Quanton engine itself and the full Iceberg benchmark configurations were not found as a repository in the pages read. | | Compute Bill | Databricks on AWS (Jobs Compute): $32.40 | Onehouse (Quanton, bring-your-own-cloud): no total (see the Compute Bill for why) | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless/limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless: https://docs.databricks.com/aws/en/compute/serverless/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/release-notes/runtime: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/delta/uniform: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/resources/supported-regions: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/aws-pricing: https://www.databricks.com/product/aws-pricing (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/pricing/jobs: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs/emr-migration: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtim: https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/project-settings/usage-and-billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): xtable.apache.org: https://xtable.apache.org/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/compute/create-cluster: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-worklo: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark: https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): api.github.com/repos/onehouseinc/lake-loader: https://api.github.com/repos/onehouseinc/lake-loader (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/databricks-vs-onehouse/ --- # Databricks and Open stack on EMR and EKS: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks and Open stack on EMR and EKS are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Databricks | Open stack on EMR and EKS | | --- | --- | --- | | Kind | Managed lakehouse built around Delta Lake | Self-run Spark with open table formats | | Does well | Per Databricks documentation, Photon keeps the Spark Catalyst planner, works with existing Spark API code and falls back to the Spark runtime for unsupported operations (read 2026-11-10). Per Databricks documentation, Delta tables can generate Iceberg metadata from Runtime 14.3 LTS, so Iceberg clients read Delta data without rewriting files (read 2026-11-10). | Per AWS documentation, EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three formats are available (read 2026-11-10). AWS ran TPC-DS 3 TB on 9 x r5d.4xlarge and reports $4.81 for EMR 7.12 against $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0, with repositories to repeat it. | | Limited | Per Databricks documentation, serverless compute supports only Spark Connect APIs, so RDD APIs, R and Maven coordinates are unsupported, and ANSI mode on by default can change results of existing jobs. Per Databricks, compute is billed in DBUs in addition to the cloud bill for the same instances, so a Jobs Compute run pays Databricks and AWS separately; the Compute Bill line adds $12.24 of DBUs to $20.16 of EC2 (read 2026-11-10). | Per the Apache Iceberg documentation, users run table maintenance themselves: expire snapshots, remove orphan files and compact with rewriteDataFiles as jobs they schedule (read 2026-11-10). A cost comparison of open-stack Spark against managed lakehouses by a party other than AWS or a vendor: none found on 2026-11-10. | | Published speed results | Docs claim Photon gives up to 5x better price/performance for data and analytics workloads versus other cloud data warehouses; no method is given on this page. Databricks announced an audited 100TB TPC-DS result of 32,941,245 QphDS for Databricks SQL, 2.2x the previous record held by Alibaba, with total system cost 10% lower on list pricing. This is a SQL warehouse result, not a Spark batch ETL result. The same Databricks blog says research from Barcelona Supercomputing Center found Databricks 2.7x faster and 12x better in price-performance than Snowflake on a TPC-DS derivative. This is a vendor-reported claim of a third-party study; the study itself was not fetched. | AWS ran TPC-DS 3 TB (104 SparkSQL queries, 3 rounds, 9 x r5d.4xlarge, Hadoop catalog) and reports EMR 7.12 averaged 1,349.62 s versus 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0 and 1,535.62 s for EMR 7.5. AWS notes results are not comparable to official TPC-DS. The run is by the vendor of EMR. | | Published prices and cost results | Databricks bills compute in DBUs at per-second granularity with pay-as-you-go or committed-use discounts, and no up-front costs. The pricing pages render their rates by script; the AWS pricing data file those pages load lists Jobs Compute at $0.15 per DBU on the plan it names Premium (r6i.4xlarge: 4.08 DBU per hour). The file names no region. Jobs pricing offers serverless (Performance Optimized and Standard modes) and classic clusters; the jobs pricing page maps plan names across clouds (its Premium tier on Azure corresponds to Enterprise on AWS and GCP), while the AWS pricing data file lists the $0.15 Jobs Compute rate under a plan it names Premium. Photon instances consume DBUs at a different rate. | EMR on EC2 charges an EMR price in addition to the EC2 price (and EBS), billed per second with a one-minute minimum. AWS's example in us-east-1: c4.2xlarge EMR charge $0.105/hour next to EC2 $0.398/hour; one master and two core nodes for a month total $1,101.57. Spot instances are up to a 90% discount versus On-Demand. EMR on EKS adds an uplift per vCPU-hour and per GB-hour to EKS/EC2 or Fargate costs, plus $0.10 per hour for each EKS cluster. AWS's worked example uses $0.007084 per vCPU-hour and $0.000777875 per GB-hour (100 vCPU and 300 GB for 30 minutes = $0.4709 EMR uplift). The uplift applies to any pod using an EMR runtime image, including with the Spark operator. EMR Serverless is billed per vCPU-hour and GB-hour. AWS's worked example uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour for a job with 100 vCPU and 750 GB for 30 minutes, then 200 more vCPU for 15 minutes, totaling $9.596275. In the same AWS benchmark the cost was $4.81 for EMR 7.12, $5.47 for EMR 7.5 and $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0. Cost formula uses r5d.4xlarge at $1.152/hour plus $0.27/hour EMR plus EBS, On-Demand. | | Code changes and migration | Photon is a native C++ vectorized engine that keeps Spark's Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime. Serverless compute for notebooks and jobs supports only Spark Connect APIs; Spark RDD APIs, R, Maven coordinates, global temp views, DBFS mounts with instance profiles and Databricks Container Services are not supported. ANSI mode is on by default and the default time zone is UTC, which can change results of existing jobs. | The EMR runtime for Spark provides 'full API compatibility with open source Apache Spark and Apache Iceberg', so existing Spark jobs run as is. Optimized runtimes are used by EMR on EC2, EMR Serverless, EMR on EKS, EMR on Outposts and AWS Glue. Self-run Spark on EKS uses the open-source Spark operator and needs no job rewrite. | | Tuning and table upkeep | Photon is on by default for classic all-purpose compute, jobs compute and classic pipelines; API users must set runtime_engine to PHOTON. Photon instance types use DBUs at a different rate than the same instance without Photon. On serverless compute Databricks allocates and manages compute so users do not provision clusters; the Spark UI and Spark logs are not available, query profile and client-side logs are used instead. Serverless compute runs workloads without provisioning compute in your cloud account, and Databricks says this reduces the need to manage compute resources. It requires Unity Catalog to reach external data sources. | Iceberg docs say users must run maintenance: expire snapshots, remove old metadata files and delete orphan files are 'recommended'; compacting data files with the Spark rewriteDataFiles action (for example target-file-size-bytes of 500 MB) and rewriting manifests are 'optional' maintenance, run as jobs you schedule. Hudi merge-on-read tables need compaction to merge log files into base files and bound read latency. Compaction has two steps, scheduling (plan written to the timeline) and execution; the docs describe strategies for choosing which file slices to compact. Delta Lake OSS docs list file management features the user applies: compaction (bin-packing) with OPTIMIZE, auto compaction, data skipping, Z-ordering, multi-part checkpointing, log compactions and optimized write. | | Spark versions and table formats | Databricks Runtime 18 LTS ships Apache Spark 4.1.0 (released June 10, 2026); Runtime 17.3 LTS ships Spark 4.0.0; Runtime 16.4 LTS and 15.4 LTS ship Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). Delta Lake tables can generate Iceberg metadata (UniForm, Iceberg reads) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files. | Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. All three table formats are available on EMR. EMR on EKS supports the Kubernetes Operator for Apache Spark (Spark operator) as a job submission model with the EMR runtime on your own EKS clusters. The Kubeflow Spark Operator is Apache-2.0 licensed with Helm installation. | | Where it runs | Databricks on AWS lists many AWS regions including us-east-1 (US East, Northern Virginia), us-east-2, us-west-1, eu-west-1, eu-central-1, ap-southeast-2 and us-gov-west-1; the platform is also hosted on Azure and Google Cloud. Some features are only in a subset of regions. | Amazon EMR has regional endpoints in the US (us-east-1, us-east-2, us-west-1, us-west-2) and many other AWS regions per the AWS general reference endpoints table. Spark runs on EC2 instances the user chooses (the AWS benchmark used r5d.4xlarge). | | How open its benchmarks are | None found on 10 Nov 2026. | AWS publishes the setup in GitHub repositories (aws-samples/emr-on-eks-benchmark branch tpcds-v2.13_iceberg, and aws-samples/emr-spark-benchmark) with build instructions, so the TPC-DS Iceberg benchmark can be repeated. | | Compute Bill | Databricks on AWS (Jobs Compute): $32.40 | Open stack: Spark on plain EC2: $20.16 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless/limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless: https://docs.databricks.com/aws/en/compute/serverless/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/release-notes/runtime: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/delta/uniform: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/resources/supported-regions: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/aws-pricing: https://www.databricks.com/product/aws-pricing (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/pricing/jobs: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operato: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: iceberg.apache.org/docs/latest/maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: hudi.apache.org/docs/compaction: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.delta.io/latest/optimizations-oss.html: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/general/latest/gr/emr.html: https://docs.aws.amazon.com/general/latest/gr/emr.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/databricks-vs-open-stack-emr-eks/ --- # Databricks and Snowflake: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks (Managed lakehouse built around Delta Lake) and Snowflake (Cloud data platform with PySpark support) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Databricks on AWS (Jobs Compute): $32.40; Snowflake (XL standard warehouse): $96.00. ## The same rows for both | Row | Databricks | Snowflake | | --- | --- | --- | | Kind | Managed lakehouse built around Delta Lake | Cloud data platform with PySpark support | | Does well | Per Databricks documentation, Photon keeps the Spark Catalyst planner, works with existing Spark API code and falls back to the Spark runtime for unsupported operations (read 2026-11-10). Per Databricks documentation, Delta tables can generate Iceberg metadata from Runtime 14.3 LTS, so Iceberg clients read Delta data without rewriting files (read 2026-11-10). | Per Snowflake documentation, Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect, and batch jobs are submitted with the Snowpark Submit CLI (read 2026-11-10). Per Snowflake documentation, Iceberg tables can keep data in customer-managed S3, GCS or Azure storage through external volumes and a catalog integration such as AWS Glue. | | Limited | Per Databricks documentation, serverless compute supports only Spark Connect APIs, so RDD APIs, R and Maven coordinates are unsupported, and ANSI mode on by default can change results of existing jobs. Per Databricks, compute is billed in DBUs in addition to the cloud bill for the same instances, so a Jobs Compute run pays Databricks and AWS separately; the Compute Bill line adds $12.24 of DBUs to $20.16 of EC2 (read 2026-11-10). | Per Snowflake documentation, Snowpark Connect supports Apache Spark 3.5 only, and DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported (read 2026-11-10). On-demand credit prices on AWS US East (Northern Virginia) are $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical) per the Snowflake Service Consumption Table effective 2026-11-09 (read 2026-11-10); the Snowflake pricing page itself renders by script. | | Published speed results | Docs claim Photon gives up to 5x better price/performance for data and analytics workloads versus other cloud data warehouses; no method is given on this page. Databricks announced an audited 100TB TPC-DS result of 32,941,245 QphDS for Databricks SQL, 2.2x the previous record held by Alibaba, with total system cost 10% lower on list pricing. This is a SQL warehouse result, not a Spark batch ETL result. The same Databricks blog says research from Barcelona Supercomputing Center found Databricks 2.7x faster and 12x better in price-performance than Snowflake on a TPC-DS derivative. This is a vendor-reported claim of a third-party study; the study itself was not fetched. | None found on 10 Nov 2026. | | Published prices and cost results | Databricks bills compute in DBUs at per-second granularity with pay-as-you-go or committed-use discounts, and no up-front costs. The pricing pages render their rates by script; the AWS pricing data file those pages load lists Jobs Compute at $0.15 per DBU on the plan it names Premium (r6i.4xlarge: 4.08 DBU per hour). The file names no region. Jobs pricing offers serverless (Performance Optimized and Standard modes) and classic clusters; the jobs pricing page maps plan names across clouds (its Premium tier on Azure corresponds to Enterprise on AWS and GCP), while the AWS pricing data file lists the $0.15 Jobs Compute rate under a plan it names Premium. Photon instances consume DBUs at a different rate. | Warehouses bill credits per second with a 60-second minimum on each start; each size step roughly doubles credits per hour (Small is 2 credits per hour, Medium is 4). Snowflake's Service Consumption Table (effective 2026-11-09) lists on-demand credits on AWS US East at $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical). | | Code changes and migration | Photon is a native C++ vectorized engine that keeps Spark's Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime. Serverless compute for notebooks and jobs supports only Spark Connect APIs; Spark RDD APIs, R, Maven coordinates, global temp views, DBFS mounts with instance profiles and Databricks Container Services are not supported. ANSI mode is on by default and the default time zone is UTC, which can change results of existing jobs. | Snowpark Connect for Apache Spark runs existing PySpark DataFrame code against the Snowflake engine over the Spark Connect protocol; there is no Spark cluster, all workloads run on a Snowflake warehouse. Batch jobs are submitted with the Snowpark Submit CLI. Because Snowflake rather than Spark is the engine, there are semantic differences: DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported, and ByteType, ShortType and IntegerType are returned as LongType. The docs list unsupported APIs for DataFrame and Dataset. Docs say the Spark DataFrame and Spark SQL code can be reused with minimal migration or changes; the compatibility notes say Spark Connect defers plan resolution, which changes behavior versus Spark Classic. | | Tuning and table upkeep | Photon is on by default for classic all-purpose compute, jobs compute and classic pipelines; API users must set runtime_engine to PHOTON. Photon instance types use DBUs at a different rate than the same instance without Photon. On serverless compute Databricks allocates and manages compute so users do not provision clusters; the Spark UI and Spark logs are not available, query profile and client-side logs are used instead. Serverless compute runs workloads without provisioning compute in your cloud account, and Databricks says this reduces the need to manage compute resources. It requires Unity Catalog to reach external data sources. | Docs provide a separate Optimizing Snowpark Connect for Spark workloads section and a Troubleshooting section covering known limitations, so some job-level adjustment is expected after migration. | | Spark versions and table formats | Databricks Runtime 18 LTS ships Apache Spark 4.1.0 (released June 10, 2026); Runtime 17.3 LTS ships Spark 4.0.0; Runtime 16.4 LTS and 15.4 LTS ship Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). Delta Lake tables can generate Iceberg metadata (UniForm, Iceberg reads) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files. | Snowpark Connect supports Apache Spark 3.5 only; workloads written for 3.4 and below or 4.0 and above are not supported and may produce protocol errors or missing API failures. Snowflake supports Iceberg tables (Parquet) with data in customer-managed S3, GCS or Azure storage via external volumes, an external catalog integration such as AWS Glue, and Delta files read through Delta Direct; Hudi was not listed on the page. | | Where it runs | Databricks on AWS lists many AWS regions including us-east-1 (US East, Northern Virginia), us-east-2, us-west-1, eu-west-1, eu-central-1, ap-southeast-2 and us-gov-west-1; the platform is also hosted on Azure and Google Cloud. Some features are only in a subset of regions. | Snowflake runs on AWS, Azure and Google Cloud; AWS regions include us-east-1, us-west-2, eu-west-1 and ca-central-1. Each account is hosted in a single region. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | Databricks on AWS (Jobs Compute): $32.40 | Snowflake (XL standard warehouse): $96.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless/limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless: https://docs.databricks.com/aws/en/compute/serverless/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/release-notes/runtime: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/delta/uniform: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/resources/supported-regions: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/aws-pricing: https://www.databricks.com/product/aws-pricing (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/pricing/jobs: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/cost-understanding-compute: https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/tables-iceberg: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/intro-regions.md: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/databricks-vs-snowflake/ --- # Databricks and Starburst: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Databricks (Managed lakehouse built around Delta Lake) and Starburst (Managed Trino service for Iceberg) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Databricks on AWS (Jobs Compute): $32.40; Starburst Galaxy (Pro tier): $60.00. ## The same rows for both | Row | Databricks | Starburst | | --- | --- | --- | | Kind | Managed lakehouse built around Delta Lake | Managed Trino service for Iceberg | | Does well | Per Databricks documentation, Photon keeps the Spark Catalyst planner, works with existing Spark API code and falls back to the Spark runtime for unsupported operations (read 2026-11-10). Per Databricks documentation, Delta tables can generate Iceberg metadata from Runtime 14.3 LTS, so Iceberg clients read Delta data without rewriting files (read 2026-11-10). | Per Starburst documentation, Galaxy creates and writes Iceberg tables (Parquet, ORC or Avro) and runs maintenance with ALTER TABLE ... EXECUTE optimize, expire_snapshots and remove_orphan_files (read 2026-11-10). Starburst publishes credit prices by AWS region, for example $0.50 (Pro) and $0.75 (Enterprise) in US East (N. Virginia) and $0.60 and $0.90 in Frankfurt (read 2026-11-10). | | Limited | Per Databricks documentation, serverless compute supports only Spark Connect APIs, so RDD APIs, R and Maven coordinates are unsupported, and ANSI mode on by default can change results of existing jobs. Per Databricks, compute is billed in DBUs in addition to the cloud bill for the same instances, so a Jobs Compute run pays Databricks and AWS separately; the Compute Bill line adds $12.24 of DBUs to $20.16 of EC2 (read 2026-11-10). | Per Starburst documentation, Galaxy runs SQL on Trino, not Spark, so DataFrame code, UDFs and PySpark would be rewritten as Trino SQL or kept on Spark (read 2026-11-10). Per Starburst documentation, Hudi tables are read-only in Galaxy: it cannot create new Hudi tables or write to them. | | Published speed results | Docs claim Photon gives up to 5x better price/performance for data and analytics workloads versus other cloud data warehouses; no method is given on this page. Databricks announced an audited 100TB TPC-DS result of 32,941,245 QphDS for Databricks SQL, 2.2x the previous record held by Alibaba, with total system cost 10% lower on list pricing. This is a SQL warehouse result, not a Spark batch ETL result. The same Databricks blog says research from Barcelona Supercomputing Center found Databricks 2.7x faster and 12x better in price-performance than Snowflake on a TPC-DS derivative. This is a vendor-reported claim of a third-party study; the study itself was not fetched. | Starburst reported GPU-accelerated Trino results (its proprietary distribution, not open-source Trino): TPC-H up to 6x on accelerated queries and ClickBench 3.4x overall, single-node development environment. This is a SQL query test, not Spark batch work. | | Published prices and cost results | Databricks bills compute in DBUs at per-second granularity with pay-as-you-go or committed-use discounts, and no up-front costs. The pricing pages render their rates by script; the AWS pricing data file those pages load lists Jobs Compute at $0.15 per DBU on the plan it names Premium (r6i.4xlarge: 4.08 DBU per hour). The file names no region. Jobs pricing offers serverless (Performance Optimized and Standard modes) and classic clusters; the jobs pricing page maps plan names across clouds (its Premium tier on Azure corresponds to Enterprise on AWS and GCP), while the AWS pricing data file lists the $0.15 Jobs Compute rate under a plan it names Premium. Photon instances consume DBUs at a different rate. | Galaxy credit prices start at $0.50 (Pro), $0.75 (Enterprise) and $1.00 (Mission Critical) per credit. A Free tier allows up to 3 clusters. Pricing depends on compute usage, tier and the cloud provider and region. Annual commitments may get discounts. On-demand list prices per credit (Pro / Enterprise / Mission Critical) on AWS: US East N. Virginia, US East Ohio and US West Oregon $0.50 / $0.75 / $1.00; Europe Ireland $0.56 / $0.84 / $1.12; Europe Frankfurt $0.60 / $0.90 / $1.20; Asia Pacific Tokyo $0.64 / $0.97 / $1.29; Brazil (Sao Paulo) $0.80 / $1.19 / $1.59. Cluster compute credit consumption: 1 worker 6 credits per hour, 2 workers 12, 4 workers 24, 8 workers 48, 16 workers 96. Managed ingest: file ingest 0.60 credits per hour; table maintenance and data transforms 0.30 credits per hour. Billing tracks cluster compute hours, data ingestion compute and private link data transfer. | | Code changes and migration | Photon is a native C++ vectorized engine that keeps Spark's Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime. Serverless compute for notebooks and jobs supports only Spark Connect APIs; Spark RDD APIs, R, Maven coordinates, global temp views, DBFS mounts with instance profiles and Databricks Container Services are not supported. ANSI mode is on by default and the default time zone is UTC, which can change results of existing jobs. | Galaxy is 'a fully-managed data lake analytics platform built in addition to Trino'. Work runs as SQL on Trino, not Spark, so Spark jobs (DataFrame code, UDFs, PySpark) would have to be rewritten as Trino SQL, or kept on Spark against the same Iceberg tables. The Galaxy docs read contain no Spark execution mode. | | Tuning and table upkeep | Photon is on by default for classic all-purpose compute, jobs compute and classic pipelines; API users must set runtime_engine to PHOTON. Photon instance types use DBUs at a different rate than the same instance without Photon. On serverless compute Databricks allocates and manages compute so users do not provision clusters; the Spark UI and Spark logs are not available, query profile and client-side logs are used instead. Serverless compute runs workloads without provisioning compute in your cloud account, and Databricks says this reduces the need to manage compute resources. It requires Unity Catalog to reach external data sources. | Iceberg table maintenance in Galaxy is run with ALTER TABLE ... EXECUTE commands: optimize (compaction; files under file_size_threshold, default 100MB, are merged), drop_extended_stats, expire_snapshots, remove_orphan_files. Live table maintenance for ingest tables is a separate Galaxy page. For batch-style work Galaxy offers a fault tolerant execution mode that spools intermediate data and retries failed parts of a query. Docs say 'No other configuration is required' to enable it, but not to use it if most queries run under about a minute, and that it can be slightly slower than normal execution. FTE and Accelerated modes are not available on the Free cluster size and not all catalogs support FTE. | | Spark versions and table formats | Databricks Runtime 18 LTS ships Apache Spark 4.1.0 (released June 10, 2026); Runtime 17.3 LTS ships Spark 4.0.0; Runtime 16.4 LTS and 15.4 LTS ship Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). Delta Lake tables can generate Iceberg metadata (UniForm, Iceberg reads) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files. | Galaxy's Great Lakes connectivity lists Iceberg, Delta Lake, Hive and Hudi table format pages. Hudi tables are read-only: 'Galaxy cannot create new Hudi tables or write to them.' Iceberg tables can be created and written; the Iceberg data file format can be Parquet, ORC or Avro. | | Where it runs | Databricks on AWS lists many AWS regions including us-east-1 (US East, Northern Virginia), us-east-2, us-west-1, eu-west-1, eu-central-1, ap-southeast-2 and us-gov-west-1; the platform is also hosted on Azure and Google Cloud. Some features are only in a subset of regions. | Galaxy prices are listed for AWS, Azure and GCP regions. AWS regions priced include us-east-1, us-east-2, us-west-1, us-west-2, ca-central, Ireland, London, Frankfurt, Zurich, Singapore, Sydney, Tokyo, Cape Town, UAE, Tel Aviv and Sao Paulo. A cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | Databricks on AWS (Jobs Compute): $32.40 | Starburst Galaxy (Pro tier): $60.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless/limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/compute/serverless: https://docs.databricks.com/aws/en/compute/serverless/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/release-notes/runtime: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/delta/uniform: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: docs.databricks.com/aws/en/resources/supported-regions: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/aws-pricing: https://www.databricks.com/product/aws-pricing (read 10 Nov 2026) - Databricks Data Intelligence Platform on AWS: databricks.com/product/pricing/jobs: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing: https://www.starburst.io/pricing/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing/pricing-details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/index.html: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/optimization-perform: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/databricks-vs-starburst/ --- # Dremio and DualBird: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Dremio (Lakehouse with its own SQL engine) and DualBird (FPGA Spark plugin for AWS F2) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Dremio Cloud (Large engine): $102.40; DualBird on Amazon EMR (F2 instances): $49.50. ## The same rows for both | Row | Dremio | DualBird | | --- | --- | --- | | Kind | Lakehouse with its own SQL engine | FPGA Spark plugin for AWS F2 | | Does well | Per Dremio documentation, Iceberg tables in the Open Catalog get automatic optimization: data file compaction, row-level delete handling, clustering, partition evolution and manifest rewriting (read 2026-11-10). Dremio publishes DCU rates by engine size, from 16 DCUs per hour for 2XSmall to 2048 for 3XLarge, with a 60-second minimum (read 2026-11-10). | DualBird says onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin, with Spark still handling planning and scheduling (read 2026-11-10). DualBird published a 100 GB sort-based Iceberg compaction run on June 11, 2026, run by DualBird, and states about $2 per TB for itself against about $6 per TB for CPU clusters. | | Limited | Per Dremio documentation, queries run on its own columnar engine, not on Spark, and the pages read describe no mode that runs existing Spark jobs. Dremio Cloud documentation lists Iceberg and Delta Lake as table formats; Hudi support: none found on 2026-11-10. | DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. A price for the plugin and a benchmark run by a party other than DualBird: none found on 2026-11-10. | | Published speed results | Dremio's founder reported a full 1 TB TPC-DS run (all 99 queries) in 22 seconds on an 8-node m7gd.4xlarge cluster, with no caching, and attributes it to Autonomous Reflections and automatic Iceberg clustering. This is a BI query benchmark run by the vendor, not a Spark batch ETL or compaction test. | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | | Published prices and cost results | Dremio Cloud is priced at $0.20 per DCU (Dremio Compute Unit). Dremio Enterprise (self-managed) pricing is 'contact sales'. The free trial for Dremio Cloud is a $400 credit for 30 days. DCUs are consumed only while engines run. Usage equals engine and replica uptime in seconds (60-second minimum) times the DCU rate for the engine size. Published rates per hour: 2XSmall 16, XSmall 32, Small 64, Medium 128, Large 256, XLarge 512, 2XLarge 1024, 3XLarge 2048. Storage billed in TB-months only for Dremio-hosted storage; LLM tokens billed separately. Billable usage is DCUs (compute), LLM tokens (AI features) and storage. Compute is billed in DCUs only while engines are running. | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | | Code changes and migration | Dremio runs queries on its own columnar execution engine built on Apache Arrow, not on Spark. Docs describe Spark and Flink only as external engines that can use the Open Catalog (Iceberg REST) alongside Dremio. The pages read describe no mode that runs existing Spark jobs, so Spark ETL would stay on Spark or be rewritten as Dremio SQL. | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | | Tuning and table upkeep | For Iceberg tables in the Open Catalog, Dremio runs automatic optimization on a dedicated engine it configures: data file compaction, row-level delete handling (position deletes on v2, deletion vectors on v3), clustering, partition evolution and manifest rewriting. The same operations can be run by hand with OPTIMIZE TABLE. Optimization works in batches sized to engine resources. Engines have settings the admin chooses: size (executors per replica), min and max replicas, last-replica auto-stop (default 2 hours), enqueued time limit (default 5 minutes) and query runtime limit. Default concurrency per replica ranges from 2 (2XSmall) to 20 (3XLarge). | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | | Spark versions and table formats | The Dremio Cloud data formats page lists Apache Iceberg and Delta Lake as table formats (plus Parquet and text formats). Hudi is not listed on that page. The Open Catalog is built on Apache Polaris and supports the Iceberg REST Catalog API; the docs list AWS Glue Data Catalog as a connectable catalog. | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | | Where it runs | Dremio Cloud runs on AWS (Azure listed as coming soon). Supported regions: us-east-1, us-east-2, us-west-2, eu-west-1 and eu-central-1. Dremio Enterprise can be deployed on Kubernetes, on premises, or in AWS, Azure or GCP. | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | | How open its benchmarks are | Dremio published a 2020 TPC-DS methodology post (SF1000 and SF10000, Parquet on S3, Apache JMeter test plans, 58 of 99 queries that ran unmodified on Dremio and Presto distributions). The 2025 22-second TPC-DS post pages read give cluster type but no scripts, scale-factor files or configuration repository. | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | | Compute Bill | Dremio Cloud (Large engine): $102.40 | DualBird on Amazon EMR (F2 instances): $49.50 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Dremio: dremio.com/pricing: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/engines: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/subscription/usage: https://docs.dremio.com/dremio-cloud/admin/subscription/usage (read 10 Nov 2026) - Dremio: docs.dremio.com/current/what-is-dremio/architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/developer/data-formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/manage-govern/optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/about/regions: https://docs.dremio.com/dremio-cloud/about/regions (read 10 Nov 2026) - Dremio: dremio.com/blog/breakthrough-announcement-dremio-is-the: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026) - Dremio: dremio.com/blog/dremio-benchmarking-methodology: https://www.dremio.com/blog/dremio-benchmarking-methodology/ (read 10 Nov 2026) - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/dremio-vs-dualbird/ --- # Dremio and Google BigQuery and BigLake: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Dremio (Lakehouse with its own SQL engine) and Google BigQuery and BigLake (Serverless Google Cloud data warehouse) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Dremio Cloud (Large engine): $102.40; BigQuery (Enterprise edition slots): $19.20. ## The same rows for both | Row | Dremio | Google BigQuery and BigLake | | --- | --- | --- | | Kind | Lakehouse with its own SQL engine | Serverless Google Cloud data warehouse | | Does well | Per Dremio documentation, Iceberg tables in the Open Catalog get automatic optimization: data file compaction, row-level delete handling, clustering, partition evolution and manifest rewriting (read 2026-11-10). Dremio publishes DCU rates by engine size, from 16 DCUs per hour for 2XSmall to 2048 for 3XLarge, with a 60-second minimum (read 2026-11-10). | Per Google documentation, BigQuery Iceberg managed tables store data in buckets you own, support DML and streaming, and run automatic compaction, clustering and garbage collection (read 2026-11-10). Per Google documentation, stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud (read 2026-11-10). | | Limited | Per Dremio documentation, queries run on its own columnar engine, not on Spark, and the pages read describe no mode that runs existing Spark jobs. Dremio Cloud documentation lists Iceberg and Delta Lake as table formats; Hudi support: none found on 2026-11-10. | Per Google documentation, BigQuery Omni on S3 does not support DML or CREATE TABLE, so it does not run write-heavy Spark batch ETL on S3. Google's pricing page text lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate (read 2026-11-10). Per-TiB on-demand and Spark DCU rates are not used on this site. | | Published speed results | Dremio's founder reported a full 1 TB TPC-DS run (all 99 queries) in 22 seconds on an 8-node m7gd.4xlarge cluster, with no caching, and attributes it to Autonomous Reflections and automatic Iceberg clustering. This is a BI query benchmark run by the vendor, not a Spark batch ETL or compaction test. | Google states Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name; no method is given on the pricing page. | | Published prices and cost results | Dremio Cloud is priced at $0.20 per DCU (Dremio Compute Unit). Dremio Enterprise (self-managed) pricing is 'contact sales'. The free trial for Dremio Cloud is a $400 credit for 30 days. DCUs are consumed only while engines run. Usage equals engine and replica uptime in seconds (60-second minimum) times the DCU rate for the engine size. Published rates per hour: 2XSmall 16, XSmall 32, Small 64, Medium 128, Large 256, XLarge 512, 2XLarge 1024, 3XLarge 2048. Storage billed in TB-months only for Dremio-hosted storage; LLM tokens billed separately. Billable usage is DCUs (compute), LLM tokens (AI features) and storage. Compute is billed in DCUs only while engines are running. | BigQuery offers on-demand pricing per TiB scanned (first 1 TiB per month free) or capacity pricing per slot-hour. BigQuery Omni has its own on-demand per TiB pricing for AWS regions. The page text shows $0.06 per slot-hour for Enterprise edition pay-as-you-go compute, with no region shown beside the rate (read 2026-11-10). Google's Spark service (Managed Service for Apache Spark, formerly Dataproc) bills serverless Spark per second in Data Compute Units with a 1-minute minimum, plus shuffle storage; Standard tier is for batch ETL, Premium tier uses Lightning Engine. | | Code changes and migration | Dremio runs queries on its own columnar execution engine built on Apache Arrow, not on Spark. Docs describe Spark and Flink only as external engines that can use the Open Catalog (Iceberg REST) alongside Dremio. The pages read describe no mode that runs existing Spark jobs, so Spark ETL would stay on Spark or be rewritten as Dremio SQL. | On BigQuery Omni, DML statements and CREATE TABLE are not supported, BigQuery ML statements are not supported, and only EXPORT DATA queries are allowed for destination tables, so it does not run write-heavy Spark batch ETL on S3. Standard and Enterprise Plus editions do not support Omni regions. BigQuery stored procedures for Apache Spark let you run PySpark or Scala/Java JAR code from SQL via CREATE PROCEDURE ... OPTIONS(engine = "SPARK", runtime_version = "2.2"), using a Cloud Storage file or inline code and a connection; this runs on Google Cloud, so code must be wrapped as procedures. | | Tuning and table upkeep | For Iceberg tables in the Open Catalog, Dremio runs automatic optimization on a dedicated engine it configures: data file compaction, row-level delete handling (position deletes on v2, deletion vectors on v3), clustering, partition evolution and manifest rewriting. The same operations can be run by hand with OPTIMIZE TABLE. Optimization works in batches sized to engine resources. Engines have settings the admin chooses: size (executors per replica), min and max replicas, last-replica auto-stop (default 2 hours), enqueued time limit (default 5 minutes) and query runtime limit. Default concurrency per replica ranges from 2 (2XSmall) to 20 (3XLarge). | Iceberg managed tables run automatic storage optimization (adaptive file sizing, clustering, garbage collection), so compaction needs no job configuration on Google Cloud; storage optimization is a separate billed item. | | Spark versions and table formats | The Dremio Cloud data formats page lists Apache Iceberg and Delta Lake as table formats (plus Parquet and text formats). Hudi is not listed on that page. The Open Catalog is built on Apache Polaris and supports the Iceberg REST Catalog API; the docs list AWS Glue Data Catalog as a connectable catalog. | BigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, export Iceberg V2 snapshots for open engines, and do automatic compaction, clustering and garbage collection. BigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security. | | Where it runs | Dremio Cloud runs on AWS (Azure listed as coming soon). Supported regions: us-east-1, us-east-2, us-west-2, eu-west-1 and eu-central-1. Dremio Enterprise can be deployed on Kubernetes, on premises, or in AWS, Azure or GCP. | BigQuery Omni lets BigQuery analytics run on data in Amazon S3 through BigLake tables, in AWS regions us-east-1, us-west-2, ap-northeast-2, ap-southeast-2, eu-west-1 and eu-central-1. | | How open its benchmarks are | Dremio published a 2020 TPC-DS methodology post (SF1000 and SF10000, Parquet on S3, Apache JMeter test plans, 58 of 99 queries that ran unmodified on Dremio and Presto distributions). The 2025 22-second TPC-DS post pages read give cluster type but no scripts, scale-factor files or configuration repository. | None found on 10 Nov 2026. | | Compute Bill | Dremio Cloud (Large engine): $102.40 | BigQuery (Enterprise edition slots): $19.20 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Dremio: dremio.com/pricing: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/engines: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/subscription/usage: https://docs.dremio.com/dremio-cloud/admin/subscription/usage (read 10 Nov 2026) - Dremio: docs.dremio.com/current/what-is-dremio/architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/developer/data-formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/manage-govern/optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/about/regions: https://docs.dremio.com/dremio-cloud/about/regions (read 10 Nov 2026) - Dremio: dremio.com/blog/breakthrough-announcement-dremio-is-the: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026) - Dremio: dremio.com/blog/dremio-benchmarking-methodology: https://www.dremio.com/blog/dremio-benchmarking-methodology/ (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/omni-introduction: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/spark-procedures: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/iceberg-tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/biglake-intro: https://cloud.google.com/bigquery/docs/biglake-intro (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/dataproc-serverless/pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/bigquery-biglake-vs-dremio/ --- # Dremio and Microsoft Fabric: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Dremio (Lakehouse with its own SQL engine) and Microsoft Fabric (SaaS analytics on Azure with Spark) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Dremio Cloud (Large engine): $102.40; Microsoft Fabric (F128 capacity): $46.08. ## The same rows for both | Row | Dremio | Microsoft Fabric | | --- | --- | --- | | Kind | Lakehouse with its own SQL engine | SaaS analytics on Azure with Spark | | Does well | Per Dremio documentation, Iceberg tables in the Open Catalog get automatic optimization: data file compaction, row-level delete handling, clustering, partition evolution and manifest rewriting (read 2026-11-10). Dremio publishes DCU rates by engine size, from 16 DCUs per hour for 2XSmall to 2048 for 3XLarge, with a 60-second minimum (read 2026-11-10). | Per Microsoft Learn, the native execution engine (Apache Gluten and Velox) needs no code changes, falls back to JVM Spark for unsupported operators and supports Parquet and Delta (read 2026-11-10). Per Microsoft Learn, OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying it (read 2026-11-10). | | Limited | Per Dremio documentation, queries run on its own columnar engine, not on Spark, and the pages read describe no mode that runs existing Spark jobs. Dremio Cloud documentation lists Iceberg and Delta Lake as table formats; Hudi support: none found on 2026-11-10. | Per Microsoft Learn, Fabric runs over OneLake on Azure and the region page lists Azure regions only, so Spark compute does not run inside AWS. Writes to S3 through a OneLake shortcut are not possible, and the method for Microsoft's native engine benchmark: none found on 2026-11-10. | | Published speed results | Dremio's founder reported a full 1 TB TPC-DS run (all 99 queries) in 22 seconds on an 8-node m7gd.4xlarge cluster, with no caching, and attributes it to Autonomous Reflections and automatic Iceberg clustering. This is a BI query benchmark run by the vendor, not a Spark batch ETL or compaction test. | Microsoft reports that on TPC-DS scale factor 1000 with Delta the native engine was up to six times faster than open-source Spark, with about 83% compute cost savings on a fixed-size cluster. | | Published prices and cost results | Dremio Cloud is priced at $0.20 per DCU (Dremio Compute Unit). Dremio Enterprise (self-managed) pricing is 'contact sales'. The free trial for Dremio Cloud is a $400 credit for 30 days. DCUs are consumed only while engines run. Usage equals engine and replica uptime in seconds (60-second minimum) times the DCU rate for the engine size. Published rates per hour: 2XSmall 16, XSmall 32, Small 64, Medium 128, Large 256, XLarge 512, 2XLarge 1024, 3XLarge 2048. Storage billed in TB-months only for Dremio-hosted storage; LLM tokens billed separately. Billable usage is DCUs (compute), LLM tokens (AI features) and storage. Compute is billed in DCUs only while engines are running. | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus (Spark Memory Optimized and autoscale for Spark entries, pay-as-you-go). In Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling. Autoscale Billing for Spark is an opt-in pay-as-you-go model that runs Spark on dedicated serverless resources at standard PAYG CU-hour rates; reserved capacity discounts do not apply and a base capacity is still required. | | Code changes and migration | Dremio runs queries on its own columnar execution engine built on Apache Arrow, not on Spark. Docs describe Spark and Flink only as external engines that can use the Open Catalog (Iceberg REST) alongside Dremio. The pages read describe no mode that runs existing Spark jobs, so Spark ETL would stay on Spark or be rewritten as Dremio SQL. | The Fabric native execution engine, based on Apache Gluten and Velox, needs no code changes, falls back to JVM Spark for unsupported operators, and supports Parquet and Delta. | | Tuning and table upkeep | For Iceberg tables in the Open Catalog, Dremio runs automatic optimization on a dedicated engine it configures: data file compaction, row-level delete handling (position deletes on v2, deletion vectors on v3), clustering, partition evolution and manifest rewriting. The same operations can be run by hand with OPTIMIZE TABLE. Optimization works in batches sized to engine resources. Engines have settings the admin chooses: size (executors per replica), min and max replicas, last-replica auto-stop (default 2 hours), enqueued time limit (default 5 minutes) and query runtime limit. Default concurrency per replica ranges from 2 (2XSmall) to 20 (3XLarge). | Starter pools start in 5 to 10 seconds with no setup; custom Spark pools let you tune node size and scaling, and native execution can be toggled per application with a Spark setting. | | Spark versions and table formats | The Dremio Cloud data formats page lists Apache Iceberg and Delta Lake as table formats (plus Parquet and text formats). Hudi is not listed on that page. The Open Catalog is built on Apache Polaris and supports the Iceberg REST Catalog API; the docs list AWS Glue Data Catalog as a connectable catalog. | Fabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1; OneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and vice versa. | | Where it runs | Dremio Cloud runs on AWS (Azure listed as coming soon). Supported regions: us-east-1, us-east-2, us-west-2, eu-west-1 and eu-central-1. Dremio Enterprise can be deployed on Kubernetes, on premises, or in AWS, Azure or GCP. | Fabric workloads run over OneLake, described as built on Azure Data Lake Storage; the region availability page lists Azure regions and no AWS regions, so Spark compute does not run inside AWS. OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; file caching can reduce cross-cloud egress costs. Writes to S3 are not possible through the shortcut. | | How open its benchmarks are | Dremio published a 2020 TPC-DS methodology post (SF1000 and SF10000, Parquet on S3, Apache JMeter test plans, 58 of 99 queries that ran unmodified on Dremio and Presto distributions). The 2025 22-second TPC-DS post pages read give cluster type but no scripts, scale-factor files or configuration repository. | None found on 10 Nov 2026. | | Compute Bill | Dremio Cloud (Large engine): $102.40 | Microsoft Fabric (F128 capacity): $46.08 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Dremio: dremio.com/pricing: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/engines: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/subscription/usage: https://docs.dremio.com/dremio-cloud/admin/subscription/usage (read 10 Nov 2026) - Dremio: docs.dremio.com/current/what-is-dremio/architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/developer/data-formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/manage-govern/optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/about/regions: https://docs.dremio.com/dremio-cloud/about/regions (read 10 Nov 2026) - Dremio: dremio.com/blog/breakthrough-announcement-dremio-is-the: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026) - Dremio: dremio.com/blog/dremio-benchmarking-methodology: https://www.dremio.com/blog/dremio-benchmarking-methodology/ (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview: https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - Microsoft Fabric: prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption' (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Microsoft Fabric: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-compute: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/dremio-vs-microsoft-fabric/ --- # Dremio and Onehouse: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Dremio (Lakehouse with its own SQL engine) and Onehouse (Managed lakehouse in your cloud account) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Dremio Cloud (Large engine): $102.40; Onehouse (Quanton, bring-your-own-cloud): no total (see the Compute Bill for why). ## The same rows for both | Row | Dremio | Onehouse | | --- | --- | --- | | Kind | Lakehouse with its own SQL engine | Managed lakehouse in your cloud account | | Does well | Per Dremio documentation, Iceberg tables in the Open Catalog get automatic optimization: data file compaction, row-level delete handling, clustering, partition evolution and manifest rewriting (read 2026-11-10). Dremio publishes DCU rates by engine size, from 16 DCUs per hour for 2XSmall to 2048 for 3XLarge, with a 60-second minimum (read 2026-11-10). | Per Onehouse documentation, its jobs are fully compatible with Apache Spark and EMR code can be reused unless it uses AWS-specific features (read 2026-11-10). Onehouse published the setup for its 10 TB TPC-DS and 1 TB LakeLoader comparison against EMR 7.12, and open-sourced LakeLoader under Apache-2.0 (read 2026-11-10). | | Limited | Per Dremio documentation, queries run on its own columnar engine, not on Spark, and the pages read describe no mode that runs existing Spark jobs. Dremio Cloud documentation lists Iceberg and Delta Lake as table formats; Hudi support: none found on 2026-11-10. | Per Onehouse documentation, billing is in OCUs on a bring-your-own-cloud model, and no dollar rate per OCU was found on 2026-11-10. Onehouse's results against EMR 7.12 are its own and shown as charts; a confirmation by another party: none found on 2026-11-10. | | Published speed results | Dremio's founder reported a full 1 TB TPC-DS run (all 99 queries) in 22 seconds on an 8-node m7gd.4xlarge cluster, with no caching, and attributes it to Autonomous Reflections and automatic Iceberg clustering. This is a BI query benchmark run by the vendor, not a Spark batch ETL or compaction test. | Onehouse benchmarked Quanton against open-source Spark on Iceberg: TPC-DS up to about 3.1x, TPCx-BB 2x, TPC-DI (1 TB) 2x with 3-4x on MERGE INTO, and LakeLoader 4.1x for writes. The baseline is open-source Spark, not EMR. Onehouse reran its benchmarks against EMR 7.12 after AWS announced it: it reports EMR improved 32% over earlier versions and that Onehouse 'still comes out with a 2.5x better price/performance advantage'. Setup: 10 TB TPC-DS for read/transform, 1 TB LakeLoader merges for load, latest general-purpose Graviton instances. Results are shown as charts; Onehouse also reports Hudi and Delta query performance on EMR 7.12 was very similar to Iceberg. A B2B SaaS customer (unnamed) ran its UDF-heavy pipeline on EMR and on Quanton in its own AWS account on m8gd nodes. Quanton was faster at p50, p97 and p99 tenant sizes; on the largest tenant (21 billion rows) EMR filled local disks on 180 nodes and was aborted after 3.7 hours, and finished on 300 nodes. The write-up is published by Onehouse. | | Published prices and cost results | Dremio Cloud is priced at $0.20 per DCU (Dremio Compute Unit). Dremio Enterprise (self-managed) pricing is 'contact sales'. The free trial for Dremio Cloud is a $400 credit for 30 days. DCUs are consumed only while engines run. Usage equals engine and replica uptime in seconds (60-second minimum) times the DCU rate for the engine size. Published rates per hour: 2XSmall 16, XSmall 32, Small 64, Medium 128, Large 256, XLarge 512, 2XLarge 1024, 3XLarge 2048. Storage billed in TB-months only for Dremio-hosted storage; LLM tokens billed separately. Billable usage is DCUs (compute), LLM tokens (AI features) and storage. Compute is billed in DCUs only while engines are running. | Onehouse bills in Onehouse Compute Units (OCU) on a bring-your-own-cloud model: 1 OCU is a 4-core instance for 1 hour in your account, plus support costs; no separate volume or storage fee, and the customer pays its own AWS or GCP infrastructure bill. The docs page does not state a dollar rate per OCU. Vendor claims at the Quanton launch: 2-3x better price-performance for ETL, and a guaranteed greater-than-50% cost reduction for existing Hudi users moving Spark jobs from AWS EMR or open-source Spark. These are guarantees stated by the vendor, not results from an independent party. | | Code changes and migration | Dremio runs queries on its own columnar execution engine built on Apache Arrow, not on Spark. Docs describe Spark and Flink only as external engines that can use the Open Catalog (Iceberg REST) alongside Dremio. The pages read describe no mode that runs existing Spark jobs, so Spark ETL would stay on Spark or be rewritten as Dremio SQL. | Onehouse docs say its Jobs are fully compatible with Apache Spark and that EMR code can be reused unless it uses AWS-specific features. Migration work listed: map EMR steps to Onehouse Jobs created through its API or Airflow operators (CREATE CLUSTER, CREATE JOB, RUN JOB, DELETE CLUSTER), and configure Glue catalog sync or contact support to make Glue the primary catalog. At the May 20, 2025 launch Onehouse said users can run existing Spark and SQL jobs on new cluster types and that existing Hudi/Spark jobs can be pointed at Onehouse 'with no code changes'. The Quanton Kubernetes Operator is described as a 'drop-in replacement' for an existing Spark Kubernetes operator. | | Tuning and table upkeep | For Iceberg tables in the Open Catalog, Dremio runs automatic optimization on a dedicated engine it configures: data file compaction, row-level delete handling (position deletes on v2, deletion vectors on v3), clustering, partition evolution and manifest rewriting. The same operations can be run by hand with OPTIMIZE TABLE. Optimization works in batches sized to engine resources. Engines have settings the admin chooses: size (executors per replica), min and max replicas, last-replica auto-stop (default 2 hours), enqueued time limit (default 5 minutes) and query runtime limit. Default concurrency per replica ranges from 2 (2XSmall) to 20 (3XLarge). | Each cluster needs min and max OCU limits. Docs warn that if usage reaches Max OCU the cluster stops scaling up, which can delay processing; Min OCU keeps a cluster warm. The Onehouse docs pages read give no per-job Spark tuning list; Onehouse claims automatic scaling and automated async table services. | | Spark versions and table formats | The Dremio Cloud data formats page lists Apache Iceberg and Delta Lake as table formats (plus Parquet and text formats). Hudi is not listed on that page. The Open Catalog is built on Apache Polaris and supports the Iceberg REST Catalog API; the docs list AWS Glue Data Catalog as a connectable catalog. | Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake. Docs: Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog (AWS Glue and Snowflake Open Catalog are documented); for Delta Lake the user installs Delta Spark as a dependency in the JAR or Python job. In the Nov 12, 2025 Iceberg post Onehouse said Delta acceleration is 'up next'. Apache XTable (formerly OneTable) translates table metadata between Hudi, Delta Lake and Iceberg without being a new format. It is incubating at the Apache Software Foundation; Onehouse says it created Hudi and XTable. | | Where it runs | Dremio Cloud runs on AWS (Azure listed as coming soon). Supported regions: us-east-1, us-east-2, us-west-2, eu-west-1 and eu-central-1. Dremio Enterprise can be deployed on Kubernetes, on premises, or in AWS, Azure or GCP. | Onehouse runs EC2 or Google Compute Engine instances inside the customer's AWS or GCP account (Azure not listed in the billing docs). Standard instance types map to Graviton m8g sizes on AWS (oh-general-4 = m8g.xlarge, oh-general-8 = m8g.2xlarge, oh-general-16 = m8g.4xlarge). A project is scoped to one region and account; adding a region needs a support ticket. | | How open its benchmarks are | Dremio published a 2020 TPC-DS methodology post (SF1000 and SF10000, Parquet on S3, Apache JMeter test plans, 58 of 99 queries that ran unmodified on Dremio and Presto distributions). The 2025 22-second TPC-DS post pages read give cluster type but no scripts, scale-factor files or configuration repository. | Onehouse open-sourced LakeLoader, a tool that generates incremental update/insert loads for benchmarking write performance. The repository is Apache-2.0 licensed, last pushed 2026-10-30. The Quanton engine itself and the full Iceberg benchmark configurations were not found as a repository in the pages read. | | Compute Bill | Dremio Cloud (Large engine): $102.40 | Onehouse (Quanton, bring-your-own-cloud): no total (see the Compute Bill for why) | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Dremio: dremio.com/pricing: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/engines: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/subscription/usage: https://docs.dremio.com/dremio-cloud/admin/subscription/usage (read 10 Nov 2026) - Dremio: docs.dremio.com/current/what-is-dremio/architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/developer/data-formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/manage-govern/optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/about/regions: https://docs.dremio.com/dremio-cloud/about/regions (read 10 Nov 2026) - Dremio: dremio.com/blog/breakthrough-announcement-dremio-is-the: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026) - Dremio: dremio.com/blog/dremio-benchmarking-methodology: https://www.dremio.com/blog/dremio-benchmarking-methodology/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs/emr-migration: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtim: https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/project-settings/usage-and-billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): xtable.apache.org: https://xtable.apache.org/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/compute/create-cluster: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-worklo: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark: https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): api.github.com/repos/onehouseinc/lake-loader: https://api.github.com/repos/onehouseinc/lake-loader (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/dremio-vs-onehouse/ --- # Dremio and Open stack on EMR and EKS: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Dremio and Open stack on EMR and EKS are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Dremio | Open stack on EMR and EKS | | --- | --- | --- | | Kind | Lakehouse with its own SQL engine | Self-run Spark with open table formats | | Does well | Per Dremio documentation, Iceberg tables in the Open Catalog get automatic optimization: data file compaction, row-level delete handling, clustering, partition evolution and manifest rewriting (read 2026-11-10). Dremio publishes DCU rates by engine size, from 16 DCUs per hour for 2XSmall to 2048 for 3XLarge, with a 60-second minimum (read 2026-11-10). | Per AWS documentation, EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three formats are available (read 2026-11-10). AWS ran TPC-DS 3 TB on 9 x r5d.4xlarge and reports $4.81 for EMR 7.12 against $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0, with repositories to repeat it. | | Limited | Per Dremio documentation, queries run on its own columnar engine, not on Spark, and the pages read describe no mode that runs existing Spark jobs. Dremio Cloud documentation lists Iceberg and Delta Lake as table formats; Hudi support: none found on 2026-11-10. | Per the Apache Iceberg documentation, users run table maintenance themselves: expire snapshots, remove orphan files and compact with rewriteDataFiles as jobs they schedule (read 2026-11-10). A cost comparison of open-stack Spark against managed lakehouses by a party other than AWS or a vendor: none found on 2026-11-10. | | Published speed results | Dremio's founder reported a full 1 TB TPC-DS run (all 99 queries) in 22 seconds on an 8-node m7gd.4xlarge cluster, with no caching, and attributes it to Autonomous Reflections and automatic Iceberg clustering. This is a BI query benchmark run by the vendor, not a Spark batch ETL or compaction test. | AWS ran TPC-DS 3 TB (104 SparkSQL queries, 3 rounds, 9 x r5d.4xlarge, Hadoop catalog) and reports EMR 7.12 averaged 1,349.62 s versus 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0 and 1,535.62 s for EMR 7.5. AWS notes results are not comparable to official TPC-DS. The run is by the vendor of EMR. | | Published prices and cost results | Dremio Cloud is priced at $0.20 per DCU (Dremio Compute Unit). Dremio Enterprise (self-managed) pricing is 'contact sales'. The free trial for Dremio Cloud is a $400 credit for 30 days. DCUs are consumed only while engines run. Usage equals engine and replica uptime in seconds (60-second minimum) times the DCU rate for the engine size. Published rates per hour: 2XSmall 16, XSmall 32, Small 64, Medium 128, Large 256, XLarge 512, 2XLarge 1024, 3XLarge 2048. Storage billed in TB-months only for Dremio-hosted storage; LLM tokens billed separately. Billable usage is DCUs (compute), LLM tokens (AI features) and storage. Compute is billed in DCUs only while engines are running. | EMR on EC2 charges an EMR price in addition to the EC2 price (and EBS), billed per second with a one-minute minimum. AWS's example in us-east-1: c4.2xlarge EMR charge $0.105/hour next to EC2 $0.398/hour; one master and two core nodes for a month total $1,101.57. Spot instances are up to a 90% discount versus On-Demand. EMR on EKS adds an uplift per vCPU-hour and per GB-hour to EKS/EC2 or Fargate costs, plus $0.10 per hour for each EKS cluster. AWS's worked example uses $0.007084 per vCPU-hour and $0.000777875 per GB-hour (100 vCPU and 300 GB for 30 minutes = $0.4709 EMR uplift). The uplift applies to any pod using an EMR runtime image, including with the Spark operator. EMR Serverless is billed per vCPU-hour and GB-hour. AWS's worked example uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour for a job with 100 vCPU and 750 GB for 30 minutes, then 200 more vCPU for 15 minutes, totaling $9.596275. In the same AWS benchmark the cost was $4.81 for EMR 7.12, $5.47 for EMR 7.5 and $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0. Cost formula uses r5d.4xlarge at $1.152/hour plus $0.27/hour EMR plus EBS, On-Demand. | | Code changes and migration | Dremio runs queries on its own columnar execution engine built on Apache Arrow, not on Spark. Docs describe Spark and Flink only as external engines that can use the Open Catalog (Iceberg REST) alongside Dremio. The pages read describe no mode that runs existing Spark jobs, so Spark ETL would stay on Spark or be rewritten as Dremio SQL. | The EMR runtime for Spark provides 'full API compatibility with open source Apache Spark and Apache Iceberg', so existing Spark jobs run as is. Optimized runtimes are used by EMR on EC2, EMR Serverless, EMR on EKS, EMR on Outposts and AWS Glue. Self-run Spark on EKS uses the open-source Spark operator and needs no job rewrite. | | Tuning and table upkeep | For Iceberg tables in the Open Catalog, Dremio runs automatic optimization on a dedicated engine it configures: data file compaction, row-level delete handling (position deletes on v2, deletion vectors on v3), clustering, partition evolution and manifest rewriting. The same operations can be run by hand with OPTIMIZE TABLE. Optimization works in batches sized to engine resources. Engines have settings the admin chooses: size (executors per replica), min and max replicas, last-replica auto-stop (default 2 hours), enqueued time limit (default 5 minutes) and query runtime limit. Default concurrency per replica ranges from 2 (2XSmall) to 20 (3XLarge). | Iceberg docs say users must run maintenance: expire snapshots, remove old metadata files and delete orphan files are 'recommended'; compacting data files with the Spark rewriteDataFiles action (for example target-file-size-bytes of 500 MB) and rewriting manifests are 'optional' maintenance, run as jobs you schedule. Hudi merge-on-read tables need compaction to merge log files into base files and bound read latency. Compaction has two steps, scheduling (plan written to the timeline) and execution; the docs describe strategies for choosing which file slices to compact. Delta Lake OSS docs list file management features the user applies: compaction (bin-packing) with OPTIMIZE, auto compaction, data skipping, Z-ordering, multi-part checkpointing, log compactions and optimized write. | | Spark versions and table formats | The Dremio Cloud data formats page lists Apache Iceberg and Delta Lake as table formats (plus Parquet and text formats). Hudi is not listed on that page. The Open Catalog is built on Apache Polaris and supports the Iceberg REST Catalog API; the docs list AWS Glue Data Catalog as a connectable catalog. | Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. All three table formats are available on EMR. EMR on EKS supports the Kubernetes Operator for Apache Spark (Spark operator) as a job submission model with the EMR runtime on your own EKS clusters. The Kubeflow Spark Operator is Apache-2.0 licensed with Helm installation. | | Where it runs | Dremio Cloud runs on AWS (Azure listed as coming soon). Supported regions: us-east-1, us-east-2, us-west-2, eu-west-1 and eu-central-1. Dremio Enterprise can be deployed on Kubernetes, on premises, or in AWS, Azure or GCP. | Amazon EMR has regional endpoints in the US (us-east-1, us-east-2, us-west-1, us-west-2) and many other AWS regions per the AWS general reference endpoints table. Spark runs on EC2 instances the user chooses (the AWS benchmark used r5d.4xlarge). | | How open its benchmarks are | Dremio published a 2020 TPC-DS methodology post (SF1000 and SF10000, Parquet on S3, Apache JMeter test plans, 58 of 99 queries that ran unmodified on Dremio and Presto distributions). The 2025 22-second TPC-DS post pages read give cluster type but no scripts, scale-factor files or configuration repository. | AWS publishes the setup in GitHub repositories (aws-samples/emr-on-eks-benchmark branch tpcds-v2.13_iceberg, and aws-samples/emr-spark-benchmark) with build instructions, so the TPC-DS Iceberg benchmark can be repeated. | | Compute Bill | Dremio Cloud (Large engine): $102.40 | Open stack: Spark on plain EC2: $20.16 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Dremio: dremio.com/pricing: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/engines: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/subscription/usage: https://docs.dremio.com/dremio-cloud/admin/subscription/usage (read 10 Nov 2026) - Dremio: docs.dremio.com/current/what-is-dremio/architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/developer/data-formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/manage-govern/optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/about/regions: https://docs.dremio.com/dremio-cloud/about/regions (read 10 Nov 2026) - Dremio: dremio.com/blog/breakthrough-announcement-dremio-is-the: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026) - Dremio: dremio.com/blog/dremio-benchmarking-methodology: https://www.dremio.com/blog/dremio-benchmarking-methodology/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operato: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: iceberg.apache.org/docs/latest/maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: hudi.apache.org/docs/compaction: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.delta.io/latest/optimizations-oss.html: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/general/latest/gr/emr.html: https://docs.aws.amazon.com/general/latest/gr/emr.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/dremio-vs-open-stack-emr-eks/ --- # Dremio and Snowflake: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Dremio (Lakehouse with its own SQL engine) and Snowflake (Cloud data platform with PySpark support) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Dremio Cloud (Large engine): $102.40; Snowflake (XL standard warehouse): $96.00. ## The same rows for both | Row | Dremio | Snowflake | | --- | --- | --- | | Kind | Lakehouse with its own SQL engine | Cloud data platform with PySpark support | | Does well | Per Dremio documentation, Iceberg tables in the Open Catalog get automatic optimization: data file compaction, row-level delete handling, clustering, partition evolution and manifest rewriting (read 2026-11-10). Dremio publishes DCU rates by engine size, from 16 DCUs per hour for 2XSmall to 2048 for 3XLarge, with a 60-second minimum (read 2026-11-10). | Per Snowflake documentation, Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect, and batch jobs are submitted with the Snowpark Submit CLI (read 2026-11-10). Per Snowflake documentation, Iceberg tables can keep data in customer-managed S3, GCS or Azure storage through external volumes and a catalog integration such as AWS Glue. | | Limited | Per Dremio documentation, queries run on its own columnar engine, not on Spark, and the pages read describe no mode that runs existing Spark jobs. Dremio Cloud documentation lists Iceberg and Delta Lake as table formats; Hudi support: none found on 2026-11-10. | Per Snowflake documentation, Snowpark Connect supports Apache Spark 3.5 only, and DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported (read 2026-11-10). On-demand credit prices on AWS US East (Northern Virginia) are $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical) per the Snowflake Service Consumption Table effective 2026-11-09 (read 2026-11-10); the Snowflake pricing page itself renders by script. | | Published speed results | Dremio's founder reported a full 1 TB TPC-DS run (all 99 queries) in 22 seconds on an 8-node m7gd.4xlarge cluster, with no caching, and attributes it to Autonomous Reflections and automatic Iceberg clustering. This is a BI query benchmark run by the vendor, not a Spark batch ETL or compaction test. | None found on 10 Nov 2026. | | Published prices and cost results | Dremio Cloud is priced at $0.20 per DCU (Dremio Compute Unit). Dremio Enterprise (self-managed) pricing is 'contact sales'. The free trial for Dremio Cloud is a $400 credit for 30 days. DCUs are consumed only while engines run. Usage equals engine and replica uptime in seconds (60-second minimum) times the DCU rate for the engine size. Published rates per hour: 2XSmall 16, XSmall 32, Small 64, Medium 128, Large 256, XLarge 512, 2XLarge 1024, 3XLarge 2048. Storage billed in TB-months only for Dremio-hosted storage; LLM tokens billed separately. Billable usage is DCUs (compute), LLM tokens (AI features) and storage. Compute is billed in DCUs only while engines are running. | Warehouses bill credits per second with a 60-second minimum on each start; each size step roughly doubles credits per hour (Small is 2 credits per hour, Medium is 4). Snowflake's Service Consumption Table (effective 2026-11-09) lists on-demand credits on AWS US East at $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical). | | Code changes and migration | Dremio runs queries on its own columnar execution engine built on Apache Arrow, not on Spark. Docs describe Spark and Flink only as external engines that can use the Open Catalog (Iceberg REST) alongside Dremio. The pages read describe no mode that runs existing Spark jobs, so Spark ETL would stay on Spark or be rewritten as Dremio SQL. | Snowpark Connect for Apache Spark runs existing PySpark DataFrame code against the Snowflake engine over the Spark Connect protocol; there is no Spark cluster, all workloads run on a Snowflake warehouse. Batch jobs are submitted with the Snowpark Submit CLI. Because Snowflake rather than Spark is the engine, there are semantic differences: DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported, and ByteType, ShortType and IntegerType are returned as LongType. The docs list unsupported APIs for DataFrame and Dataset. Docs say the Spark DataFrame and Spark SQL code can be reused with minimal migration or changes; the compatibility notes say Spark Connect defers plan resolution, which changes behavior versus Spark Classic. | | Tuning and table upkeep | For Iceberg tables in the Open Catalog, Dremio runs automatic optimization on a dedicated engine it configures: data file compaction, row-level delete handling (position deletes on v2, deletion vectors on v3), clustering, partition evolution and manifest rewriting. The same operations can be run by hand with OPTIMIZE TABLE. Optimization works in batches sized to engine resources. Engines have settings the admin chooses: size (executors per replica), min and max replicas, last-replica auto-stop (default 2 hours), enqueued time limit (default 5 minutes) and query runtime limit. Default concurrency per replica ranges from 2 (2XSmall) to 20 (3XLarge). | Docs provide a separate Optimizing Snowpark Connect for Spark workloads section and a Troubleshooting section covering known limitations, so some job-level adjustment is expected after migration. | | Spark versions and table formats | The Dremio Cloud data formats page lists Apache Iceberg and Delta Lake as table formats (plus Parquet and text formats). Hudi is not listed on that page. The Open Catalog is built on Apache Polaris and supports the Iceberg REST Catalog API; the docs list AWS Glue Data Catalog as a connectable catalog. | Snowpark Connect supports Apache Spark 3.5 only; workloads written for 3.4 and below or 4.0 and above are not supported and may produce protocol errors or missing API failures. Snowflake supports Iceberg tables (Parquet) with data in customer-managed S3, GCS or Azure storage via external volumes, an external catalog integration such as AWS Glue, and Delta files read through Delta Direct; Hudi was not listed on the page. | | Where it runs | Dremio Cloud runs on AWS (Azure listed as coming soon). Supported regions: us-east-1, us-east-2, us-west-2, eu-west-1 and eu-central-1. Dremio Enterprise can be deployed on Kubernetes, on premises, or in AWS, Azure or GCP. | Snowflake runs on AWS, Azure and Google Cloud; AWS regions include us-east-1, us-west-2, eu-west-1 and ca-central-1. Each account is hosted in a single region. | | How open its benchmarks are | Dremio published a 2020 TPC-DS methodology post (SF1000 and SF10000, Parquet on S3, Apache JMeter test plans, 58 of 99 queries that ran unmodified on Dremio and Presto distributions). The 2025 22-second TPC-DS post pages read give cluster type but no scripts, scale-factor files or configuration repository. | None found on 10 Nov 2026. | | Compute Bill | Dremio Cloud (Large engine): $102.40 | Snowflake (XL standard warehouse): $96.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Dremio: dremio.com/pricing: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/engines: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/subscription/usage: https://docs.dremio.com/dremio-cloud/admin/subscription/usage (read 10 Nov 2026) - Dremio: docs.dremio.com/current/what-is-dremio/architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/developer/data-formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/manage-govern/optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/about/regions: https://docs.dremio.com/dremio-cloud/about/regions (read 10 Nov 2026) - Dremio: dremio.com/blog/breakthrough-announcement-dremio-is-the: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026) - Dremio: dremio.com/blog/dremio-benchmarking-methodology: https://www.dremio.com/blog/dremio-benchmarking-methodology/ (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/cost-understanding-compute: https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/tables-iceberg: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/intro-regions.md: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/dremio-vs-snowflake/ --- # Dremio and Starburst: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Dremio (Lakehouse with its own SQL engine) and Starburst (Managed Trino service for Iceberg) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Dremio Cloud (Large engine): $102.40; Starburst Galaxy (Pro tier): $60.00. ## The same rows for both | Row | Dremio | Starburst | | --- | --- | --- | | Kind | Lakehouse with its own SQL engine | Managed Trino service for Iceberg | | Does well | Per Dremio documentation, Iceberg tables in the Open Catalog get automatic optimization: data file compaction, row-level delete handling, clustering, partition evolution and manifest rewriting (read 2026-11-10). Dremio publishes DCU rates by engine size, from 16 DCUs per hour for 2XSmall to 2048 for 3XLarge, with a 60-second minimum (read 2026-11-10). | Per Starburst documentation, Galaxy creates and writes Iceberg tables (Parquet, ORC or Avro) and runs maintenance with ALTER TABLE ... EXECUTE optimize, expire_snapshots and remove_orphan_files (read 2026-11-10). Starburst publishes credit prices by AWS region, for example $0.50 (Pro) and $0.75 (Enterprise) in US East (N. Virginia) and $0.60 and $0.90 in Frankfurt (read 2026-11-10). | | Limited | Per Dremio documentation, queries run on its own columnar engine, not on Spark, and the pages read describe no mode that runs existing Spark jobs. Dremio Cloud documentation lists Iceberg and Delta Lake as table formats; Hudi support: none found on 2026-11-10. | Per Starburst documentation, Galaxy runs SQL on Trino, not Spark, so DataFrame code, UDFs and PySpark would be rewritten as Trino SQL or kept on Spark (read 2026-11-10). Per Starburst documentation, Hudi tables are read-only in Galaxy: it cannot create new Hudi tables or write to them. | | Published speed results | Dremio's founder reported a full 1 TB TPC-DS run (all 99 queries) in 22 seconds on an 8-node m7gd.4xlarge cluster, with no caching, and attributes it to Autonomous Reflections and automatic Iceberg clustering. This is a BI query benchmark run by the vendor, not a Spark batch ETL or compaction test. | Starburst reported GPU-accelerated Trino results (its proprietary distribution, not open-source Trino): TPC-H up to 6x on accelerated queries and ClickBench 3.4x overall, single-node development environment. This is a SQL query test, not Spark batch work. | | Published prices and cost results | Dremio Cloud is priced at $0.20 per DCU (Dremio Compute Unit). Dremio Enterprise (self-managed) pricing is 'contact sales'. The free trial for Dremio Cloud is a $400 credit for 30 days. DCUs are consumed only while engines run. Usage equals engine and replica uptime in seconds (60-second minimum) times the DCU rate for the engine size. Published rates per hour: 2XSmall 16, XSmall 32, Small 64, Medium 128, Large 256, XLarge 512, 2XLarge 1024, 3XLarge 2048. Storage billed in TB-months only for Dremio-hosted storage; LLM tokens billed separately. Billable usage is DCUs (compute), LLM tokens (AI features) and storage. Compute is billed in DCUs only while engines are running. | Galaxy credit prices start at $0.50 (Pro), $0.75 (Enterprise) and $1.00 (Mission Critical) per credit. A Free tier allows up to 3 clusters. Pricing depends on compute usage, tier and the cloud provider and region. Annual commitments may get discounts. On-demand list prices per credit (Pro / Enterprise / Mission Critical) on AWS: US East N. Virginia, US East Ohio and US West Oregon $0.50 / $0.75 / $1.00; Europe Ireland $0.56 / $0.84 / $1.12; Europe Frankfurt $0.60 / $0.90 / $1.20; Asia Pacific Tokyo $0.64 / $0.97 / $1.29; Brazil (Sao Paulo) $0.80 / $1.19 / $1.59. Cluster compute credit consumption: 1 worker 6 credits per hour, 2 workers 12, 4 workers 24, 8 workers 48, 16 workers 96. Managed ingest: file ingest 0.60 credits per hour; table maintenance and data transforms 0.30 credits per hour. Billing tracks cluster compute hours, data ingestion compute and private link data transfer. | | Code changes and migration | Dremio runs queries on its own columnar execution engine built on Apache Arrow, not on Spark. Docs describe Spark and Flink only as external engines that can use the Open Catalog (Iceberg REST) alongside Dremio. The pages read describe no mode that runs existing Spark jobs, so Spark ETL would stay on Spark or be rewritten as Dremio SQL. | Galaxy is 'a fully-managed data lake analytics platform built in addition to Trino'. Work runs as SQL on Trino, not Spark, so Spark jobs (DataFrame code, UDFs, PySpark) would have to be rewritten as Trino SQL, or kept on Spark against the same Iceberg tables. The Galaxy docs read contain no Spark execution mode. | | Tuning and table upkeep | For Iceberg tables in the Open Catalog, Dremio runs automatic optimization on a dedicated engine it configures: data file compaction, row-level delete handling (position deletes on v2, deletion vectors on v3), clustering, partition evolution and manifest rewriting. The same operations can be run by hand with OPTIMIZE TABLE. Optimization works in batches sized to engine resources. Engines have settings the admin chooses: size (executors per replica), min and max replicas, last-replica auto-stop (default 2 hours), enqueued time limit (default 5 minutes) and query runtime limit. Default concurrency per replica ranges from 2 (2XSmall) to 20 (3XLarge). | Iceberg table maintenance in Galaxy is run with ALTER TABLE ... EXECUTE commands: optimize (compaction; files under file_size_threshold, default 100MB, are merged), drop_extended_stats, expire_snapshots, remove_orphan_files. Live table maintenance for ingest tables is a separate Galaxy page. For batch-style work Galaxy offers a fault tolerant execution mode that spools intermediate data and retries failed parts of a query. Docs say 'No other configuration is required' to enable it, but not to use it if most queries run under about a minute, and that it can be slightly slower than normal execution. FTE and Accelerated modes are not available on the Free cluster size and not all catalogs support FTE. | | Spark versions and table formats | The Dremio Cloud data formats page lists Apache Iceberg and Delta Lake as table formats (plus Parquet and text formats). Hudi is not listed on that page. The Open Catalog is built on Apache Polaris and supports the Iceberg REST Catalog API; the docs list AWS Glue Data Catalog as a connectable catalog. | Galaxy's Great Lakes connectivity lists Iceberg, Delta Lake, Hive and Hudi table format pages. Hudi tables are read-only: 'Galaxy cannot create new Hudi tables or write to them.' Iceberg tables can be created and written; the Iceberg data file format can be Parquet, ORC or Avro. | | Where it runs | Dremio Cloud runs on AWS (Azure listed as coming soon). Supported regions: us-east-1, us-east-2, us-west-2, eu-west-1 and eu-central-1. Dremio Enterprise can be deployed on Kubernetes, on premises, or in AWS, Azure or GCP. | Galaxy prices are listed for AWS, Azure and GCP regions. AWS regions priced include us-east-1, us-east-2, us-west-1, us-west-2, ca-central, Ireland, London, Frankfurt, Zurich, Singapore, Sydney, Tokyo, Cape Town, UAE, Tel Aviv and Sao Paulo. A cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature. | | How open its benchmarks are | Dremio published a 2020 TPC-DS methodology post (SF1000 and SF10000, Parquet on S3, Apache JMeter test plans, 58 of 99 queries that ran unmodified on Dremio and Presto distributions). The 2025 22-second TPC-DS post pages read give cluster type but no scripts, scale-factor files or configuration repository. | None found on 10 Nov 2026. | | Compute Bill | Dremio Cloud (Large engine): $102.40 | Starburst Galaxy (Pro tier): $60.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Dremio: dremio.com/pricing: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/engines: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/admin/subscription/usage: https://docs.dremio.com/dremio-cloud/admin/subscription/usage (read 10 Nov 2026) - Dremio: docs.dremio.com/current/what-is-dremio/architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/developer/data-formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/manage-govern/optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Dremio: docs.dremio.com/dremio-cloud/about/regions: https://docs.dremio.com/dremio-cloud/about/regions (read 10 Nov 2026) - Dremio: dremio.com/blog/breakthrough-announcement-dremio-is-the: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026) - Dremio: dremio.com/blog/dremio-benchmarking-methodology: https://www.dremio.com/blog/dremio-benchmarking-methodology/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing: https://www.starburst.io/pricing/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing/pricing-details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/index.html: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/optimization-perform: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/dremio-vs-starburst/ --- # DualBird and Google BigQuery and BigLake: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DualBird (FPGA Spark plugin for AWS F2) and Google BigQuery and BigLake (Serverless Google Cloud data warehouse) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: DualBird on Amazon EMR (F2 instances): $49.50; BigQuery (Enterprise edition slots): $19.20. ## The same rows for both | Row | DualBird | Google BigQuery and BigLake | | --- | --- | --- | | Kind | FPGA Spark plugin for AWS F2 | Serverless Google Cloud data warehouse | | Does well | DualBird says onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin, with Spark still handling planning and scheduling (read 2026-11-10). DualBird published a 100 GB sort-based Iceberg compaction run on June 11, 2026, run by DualBird, and states about $2 per TB for itself against about $6 per TB for CPU clusters. | Per Google documentation, BigQuery Iceberg managed tables store data in buckets you own, support DML and streaming, and run automatic compaction, clustering and garbage collection (read 2026-11-10). Per Google documentation, stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud (read 2026-11-10). | | Limited | DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. A price for the plugin and a benchmark run by a party other than DualBird: none found on 2026-11-10. | Per Google documentation, BigQuery Omni on S3 does not support DML or CREATE TABLE, so it does not run write-heavy Spark batch ETL on S3. Google's pricing page text lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate (read 2026-11-10). Per-TiB on-demand and Spark DCU rates are not used on this site. | | Published speed results | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | Google states Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name; no method is given on the pricing page. | | Published prices and cost results | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | BigQuery offers on-demand pricing per TiB scanned (first 1 TiB per month free) or capacity pricing per slot-hour. BigQuery Omni has its own on-demand per TiB pricing for AWS regions. The page text shows $0.06 per slot-hour for Enterprise edition pay-as-you-go compute, with no region shown beside the rate (read 2026-11-10). Google's Spark service (Managed Service for Apache Spark, formerly Dataproc) bills serverless Spark per second in Data Compute Units with a 1-minute minimum, plus shuffle storage; Standard tier is for batch ETL, Premium tier uses Lightning Engine. | | Code changes and migration | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | On BigQuery Omni, DML statements and CREATE TABLE are not supported, BigQuery ML statements are not supported, and only EXPORT DATA queries are allowed for destination tables, so it does not run write-heavy Spark batch ETL on S3. Standard and Enterprise Plus editions do not support Omni regions. BigQuery stored procedures for Apache Spark let you run PySpark or Scala/Java JAR code from SQL via CREATE PROCEDURE ... OPTIONS(engine = "SPARK", runtime_version = "2.2"), using a Cloud Storage file or inline code and a connection; this runs on Google Cloud, so code must be wrapped as procedures. | | Tuning and table upkeep | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | Iceberg managed tables run automatic storage optimization (adaptive file sizing, clustering, garbage collection), so compaction needs no job configuration on Google Cloud; storage optimization is a separate billed item. | | Spark versions and table formats | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | BigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, export Iceberg V2 snapshots for open engines, and do automatic compaction, clustering and garbage collection. BigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security. | | Where it runs | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | BigQuery Omni lets BigQuery analytics run on data in Amazon S3 through BigLake tables, in AWS regions us-east-1, us-west-2, ap-northeast-2, ap-southeast-2, eu-west-1 and eu-central-1. | | How open its benchmarks are | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | None found on 10 Nov 2026. | | Compute Bill | DualBird on Amazon EMR (F2 instances): $49.50 | BigQuery (Enterprise edition slots): $19.20 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/omni-introduction: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/spark-procedures: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/iceberg-tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/biglake-intro: https://cloud.google.com/bigquery/docs/biglake-intro (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/dataproc-serverless/pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/bigquery-biglake-vs-dualbird/ --- # DualBird and Microsoft Fabric: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DualBird (FPGA Spark plugin for AWS F2) and Microsoft Fabric (SaaS analytics on Azure with Spark) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: DualBird on Amazon EMR (F2 instances): $49.50; Microsoft Fabric (F128 capacity): $46.08. ## The same rows for both | Row | DualBird | Microsoft Fabric | | --- | --- | --- | | Kind | FPGA Spark plugin for AWS F2 | SaaS analytics on Azure with Spark | | Does well | DualBird says onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin, with Spark still handling planning and scheduling (read 2026-11-10). DualBird published a 100 GB sort-based Iceberg compaction run on June 11, 2026, run by DualBird, and states about $2 per TB for itself against about $6 per TB for CPU clusters. | Per Microsoft Learn, the native execution engine (Apache Gluten and Velox) needs no code changes, falls back to JVM Spark for unsupported operators and supports Parquet and Delta (read 2026-11-10). Per Microsoft Learn, OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying it (read 2026-11-10). | | Limited | DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. A price for the plugin and a benchmark run by a party other than DualBird: none found on 2026-11-10. | Per Microsoft Learn, Fabric runs over OneLake on Azure and the region page lists Azure regions only, so Spark compute does not run inside AWS. Writes to S3 through a OneLake shortcut are not possible, and the method for Microsoft's native engine benchmark: none found on 2026-11-10. | | Published speed results | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | Microsoft reports that on TPC-DS scale factor 1000 with Delta the native engine was up to six times faster than open-source Spark, with about 83% compute cost savings on a fixed-size cluster. | | Published prices and cost results | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus (Spark Memory Optimized and autoscale for Spark entries, pay-as-you-go). In Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling. Autoscale Billing for Spark is an opt-in pay-as-you-go model that runs Spark on dedicated serverless resources at standard PAYG CU-hour rates; reserved capacity discounts do not apply and a base capacity is still required. | | Code changes and migration | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | The Fabric native execution engine, based on Apache Gluten and Velox, needs no code changes, falls back to JVM Spark for unsupported operators, and supports Parquet and Delta. | | Tuning and table upkeep | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | Starter pools start in 5 to 10 seconds with no setup; custom Spark pools let you tune node size and scaling, and native execution can be toggled per application with a Spark setting. | | Spark versions and table formats | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | Fabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1; OneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and vice versa. | | Where it runs | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | Fabric workloads run over OneLake, described as built on Azure Data Lake Storage; the region availability page lists Azure regions and no AWS regions, so Spark compute does not run inside AWS. OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; file caching can reduce cross-cloud egress costs. Writes to S3 are not possible through the shortcut. | | How open its benchmarks are | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | None found on 10 Nov 2026. | | Compute Bill | DualBird on Amazon EMR (F2 instances): $49.50 | Microsoft Fabric (F128 capacity): $46.08 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview: https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - Microsoft Fabric: prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption' (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Microsoft Fabric: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-compute: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/dualbird-vs-microsoft-fabric/ --- # DualBird and Onehouse: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DualBird and Onehouse are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | DualBird | Onehouse | | --- | --- | --- | | Kind | FPGA Spark plugin for AWS F2 | Managed lakehouse in your cloud account | | Does well | DualBird says onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin, with Spark still handling planning and scheduling (read 2026-11-10). DualBird published a 100 GB sort-based Iceberg compaction run on June 11, 2026, run by DualBird, and states about $2 per TB for itself against about $6 per TB for CPU clusters. | Per Onehouse documentation, its jobs are fully compatible with Apache Spark and EMR code can be reused unless it uses AWS-specific features (read 2026-11-10). Onehouse published the setup for its 10 TB TPC-DS and 1 TB LakeLoader comparison against EMR 7.12, and open-sourced LakeLoader under Apache-2.0 (read 2026-11-10). | | Limited | DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. A price for the plugin and a benchmark run by a party other than DualBird: none found on 2026-11-10. | Per Onehouse documentation, billing is in OCUs on a bring-your-own-cloud model, and no dollar rate per OCU was found on 2026-11-10. Onehouse's results against EMR 7.12 are its own and shown as charts; a confirmation by another party: none found on 2026-11-10. | | Published speed results | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | Onehouse benchmarked Quanton against open-source Spark on Iceberg: TPC-DS up to about 3.1x, TPCx-BB 2x, TPC-DI (1 TB) 2x with 3-4x on MERGE INTO, and LakeLoader 4.1x for writes. The baseline is open-source Spark, not EMR. Onehouse reran its benchmarks against EMR 7.12 after AWS announced it: it reports EMR improved 32% over earlier versions and that Onehouse 'still comes out with a 2.5x better price/performance advantage'. Setup: 10 TB TPC-DS for read/transform, 1 TB LakeLoader merges for load, latest general-purpose Graviton instances. Results are shown as charts; Onehouse also reports Hudi and Delta query performance on EMR 7.12 was very similar to Iceberg. A B2B SaaS customer (unnamed) ran its UDF-heavy pipeline on EMR and on Quanton in its own AWS account on m8gd nodes. Quanton was faster at p50, p97 and p99 tenant sizes; on the largest tenant (21 billion rows) EMR filled local disks on 180 nodes and was aborted after 3.7 hours, and finished on 300 nodes. The write-up is published by Onehouse. | | Published prices and cost results | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | Onehouse bills in Onehouse Compute Units (OCU) on a bring-your-own-cloud model: 1 OCU is a 4-core instance for 1 hour in your account, plus support costs; no separate volume or storage fee, and the customer pays its own AWS or GCP infrastructure bill. The docs page does not state a dollar rate per OCU. Vendor claims at the Quanton launch: 2-3x better price-performance for ETL, and a guaranteed greater-than-50% cost reduction for existing Hudi users moving Spark jobs from AWS EMR or open-source Spark. These are guarantees stated by the vendor, not results from an independent party. | | Code changes and migration | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | Onehouse docs say its Jobs are fully compatible with Apache Spark and that EMR code can be reused unless it uses AWS-specific features. Migration work listed: map EMR steps to Onehouse Jobs created through its API or Airflow operators (CREATE CLUSTER, CREATE JOB, RUN JOB, DELETE CLUSTER), and configure Glue catalog sync or contact support to make Glue the primary catalog. At the May 20, 2025 launch Onehouse said users can run existing Spark and SQL jobs on new cluster types and that existing Hudi/Spark jobs can be pointed at Onehouse 'with no code changes'. The Quanton Kubernetes Operator is described as a 'drop-in replacement' for an existing Spark Kubernetes operator. | | Tuning and table upkeep | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | Each cluster needs min and max OCU limits. Docs warn that if usage reaches Max OCU the cluster stops scaling up, which can delay processing; Min OCU keeps a cluster warm. The Onehouse docs pages read give no per-job Spark tuning list; Onehouse claims automatic scaling and automated async table services. | | Spark versions and table formats | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake. Docs: Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog (AWS Glue and Snowflake Open Catalog are documented); for Delta Lake the user installs Delta Spark as a dependency in the JAR or Python job. In the Nov 12, 2025 Iceberg post Onehouse said Delta acceleration is 'up next'. Apache XTable (formerly OneTable) translates table metadata between Hudi, Delta Lake and Iceberg without being a new format. It is incubating at the Apache Software Foundation; Onehouse says it created Hudi and XTable. | | Where it runs | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | Onehouse runs EC2 or Google Compute Engine instances inside the customer's AWS or GCP account (Azure not listed in the billing docs). Standard instance types map to Graviton m8g sizes on AWS (oh-general-4 = m8g.xlarge, oh-general-8 = m8g.2xlarge, oh-general-16 = m8g.4xlarge). A project is scoped to one region and account; adding a region needs a support ticket. | | How open its benchmarks are | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | Onehouse open-sourced LakeLoader, a tool that generates incremental update/insert loads for benchmarking write performance. The repository is Apache-2.0 licensed, last pushed 2026-10-30. The Quanton engine itself and the full Iceberg benchmark configurations were not found as a repository in the pages read. | | Compute Bill | DualBird on Amazon EMR (F2 instances): $49.50 | Onehouse (Quanton, bring-your-own-cloud): no total (see the Compute Bill for why) | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs/emr-migration: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtim: https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/project-settings/usage-and-billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): xtable.apache.org: https://xtable.apache.org/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/compute/create-cluster: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-worklo: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark: https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): api.github.com/repos/onehouseinc/lake-loader: https://api.github.com/repos/onehouseinc/lake-loader (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/dualbird-vs-onehouse/ --- # DualBird and Open stack on EMR and EKS: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DualBird and Open stack on EMR and EKS are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | DualBird | Open stack on EMR and EKS | | --- | --- | --- | | Kind | FPGA Spark plugin for AWS F2 | Self-run Spark with open table formats | | Does well | DualBird says onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin, with Spark still handling planning and scheduling (read 2026-11-10). DualBird published a 100 GB sort-based Iceberg compaction run on June 11, 2026, run by DualBird, and states about $2 per TB for itself against about $6 per TB for CPU clusters. | Per AWS documentation, EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three formats are available (read 2026-11-10). AWS ran TPC-DS 3 TB on 9 x r5d.4xlarge and reports $4.81 for EMR 7.12 against $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0, with repositories to repeat it. | | Limited | DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. A price for the plugin and a benchmark run by a party other than DualBird: none found on 2026-11-10. | Per the Apache Iceberg documentation, users run table maintenance themselves: expire snapshots, remove orphan files and compact with rewriteDataFiles as jobs they schedule (read 2026-11-10). A cost comparison of open-stack Spark against managed lakehouses by a party other than AWS or a vendor: none found on 2026-11-10. | | Published speed results | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | AWS ran TPC-DS 3 TB (104 SparkSQL queries, 3 rounds, 9 x r5d.4xlarge, Hadoop catalog) and reports EMR 7.12 averaged 1,349.62 s versus 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0 and 1,535.62 s for EMR 7.5. AWS notes results are not comparable to official TPC-DS. The run is by the vendor of EMR. | | Published prices and cost results | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | EMR on EC2 charges an EMR price in addition to the EC2 price (and EBS), billed per second with a one-minute minimum. AWS's example in us-east-1: c4.2xlarge EMR charge $0.105/hour next to EC2 $0.398/hour; one master and two core nodes for a month total $1,101.57. Spot instances are up to a 90% discount versus On-Demand. EMR on EKS adds an uplift per vCPU-hour and per GB-hour to EKS/EC2 or Fargate costs, plus $0.10 per hour for each EKS cluster. AWS's worked example uses $0.007084 per vCPU-hour and $0.000777875 per GB-hour (100 vCPU and 300 GB for 30 minutes = $0.4709 EMR uplift). The uplift applies to any pod using an EMR runtime image, including with the Spark operator. EMR Serverless is billed per vCPU-hour and GB-hour. AWS's worked example uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour for a job with 100 vCPU and 750 GB for 30 minutes, then 200 more vCPU for 15 minutes, totaling $9.596275. In the same AWS benchmark the cost was $4.81 for EMR 7.12, $5.47 for EMR 7.5 and $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0. Cost formula uses r5d.4xlarge at $1.152/hour plus $0.27/hour EMR plus EBS, On-Demand. | | Code changes and migration | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | The EMR runtime for Spark provides 'full API compatibility with open source Apache Spark and Apache Iceberg', so existing Spark jobs run as is. Optimized runtimes are used by EMR on EC2, EMR Serverless, EMR on EKS, EMR on Outposts and AWS Glue. Self-run Spark on EKS uses the open-source Spark operator and needs no job rewrite. | | Tuning and table upkeep | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | Iceberg docs say users must run maintenance: expire snapshots, remove old metadata files and delete orphan files are 'recommended'; compacting data files with the Spark rewriteDataFiles action (for example target-file-size-bytes of 500 MB) and rewriting manifests are 'optional' maintenance, run as jobs you schedule. Hudi merge-on-read tables need compaction to merge log files into base files and bound read latency. Compaction has two steps, scheduling (plan written to the timeline) and execution; the docs describe strategies for choosing which file slices to compact. Delta Lake OSS docs list file management features the user applies: compaction (bin-packing) with OPTIMIZE, auto compaction, data skipping, Z-ordering, multi-part checkpointing, log compactions and optimized write. | | Spark versions and table formats | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. All three table formats are available on EMR. EMR on EKS supports the Kubernetes Operator for Apache Spark (Spark operator) as a job submission model with the EMR runtime on your own EKS clusters. The Kubeflow Spark Operator is Apache-2.0 licensed with Helm installation. | | Where it runs | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | Amazon EMR has regional endpoints in the US (us-east-1, us-east-2, us-west-1, us-west-2) and many other AWS regions per the AWS general reference endpoints table. Spark runs on EC2 instances the user chooses (the AWS benchmark used r5d.4xlarge). | | How open its benchmarks are | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | AWS publishes the setup in GitHub repositories (aws-samples/emr-on-eks-benchmark branch tpcds-v2.13_iceberg, and aws-samples/emr-spark-benchmark) with build instructions, so the TPC-DS Iceberg benchmark can be repeated. | | Compute Bill | DualBird on Amazon EMR (F2 instances): $49.50 | Open stack: Spark on plain EC2: $20.16 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operato: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: iceberg.apache.org/docs/latest/maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: hudi.apache.org/docs/compaction: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.delta.io/latest/optimizations-oss.html: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/general/latest/gr/emr.html: https://docs.aws.amazon.com/general/latest/gr/emr.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/dualbird-vs-open-stack-emr-eks/ --- # DualBird and Snowflake: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DualBird (FPGA Spark plugin for AWS F2) and Snowflake (Cloud data platform with PySpark support) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: DualBird on Amazon EMR (F2 instances): $49.50; Snowflake (XL standard warehouse): $96.00. ## The same rows for both | Row | DualBird | Snowflake | | --- | --- | --- | | Kind | FPGA Spark plugin for AWS F2 | Cloud data platform with PySpark support | | Does well | DualBird says onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin, with Spark still handling planning and scheduling (read 2026-11-10). DualBird published a 100 GB sort-based Iceberg compaction run on June 11, 2026, run by DualBird, and states about $2 per TB for itself against about $6 per TB for CPU clusters. | Per Snowflake documentation, Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect, and batch jobs are submitted with the Snowpark Submit CLI (read 2026-11-10). Per Snowflake documentation, Iceberg tables can keep data in customer-managed S3, GCS or Azure storage through external volumes and a catalog integration such as AWS Glue. | | Limited | DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. A price for the plugin and a benchmark run by a party other than DualBird: none found on 2026-11-10. | Per Snowflake documentation, Snowpark Connect supports Apache Spark 3.5 only, and DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported (read 2026-11-10). On-demand credit prices on AWS US East (Northern Virginia) are $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical) per the Snowflake Service Consumption Table effective 2026-11-09 (read 2026-11-10); the Snowflake pricing page itself renders by script. | | Published speed results | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | None found on 10 Nov 2026. | | Published prices and cost results | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | Warehouses bill credits per second with a 60-second minimum on each start; each size step roughly doubles credits per hour (Small is 2 credits per hour, Medium is 4). Snowflake's Service Consumption Table (effective 2026-11-09) lists on-demand credits on AWS US East at $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical). | | Code changes and migration | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | Snowpark Connect for Apache Spark runs existing PySpark DataFrame code against the Snowflake engine over the Spark Connect protocol; there is no Spark cluster, all workloads run on a Snowflake warehouse. Batch jobs are submitted with the Snowpark Submit CLI. Because Snowflake rather than Spark is the engine, there are semantic differences: DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported, and ByteType, ShortType and IntegerType are returned as LongType. The docs list unsupported APIs for DataFrame and Dataset. Docs say the Spark DataFrame and Spark SQL code can be reused with minimal migration or changes; the compatibility notes say Spark Connect defers plan resolution, which changes behavior versus Spark Classic. | | Tuning and table upkeep | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | Docs provide a separate Optimizing Snowpark Connect for Spark workloads section and a Troubleshooting section covering known limitations, so some job-level adjustment is expected after migration. | | Spark versions and table formats | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | Snowpark Connect supports Apache Spark 3.5 only; workloads written for 3.4 and below or 4.0 and above are not supported and may produce protocol errors or missing API failures. Snowflake supports Iceberg tables (Parquet) with data in customer-managed S3, GCS or Azure storage via external volumes, an external catalog integration such as AWS Glue, and Delta files read through Delta Direct; Hudi was not listed on the page. | | Where it runs | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | Snowflake runs on AWS, Azure and Google Cloud; AWS regions include us-east-1, us-west-2, eu-west-1 and ca-central-1. Each account is hosted in a single region. | | How open its benchmarks are | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | None found on 10 Nov 2026. | | Compute Bill | DualBird on Amazon EMR (F2 instances): $49.50 | Snowflake (XL standard warehouse): $96.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/cost-understanding-compute: https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/tables-iceberg: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/intro-regions.md: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/dualbird-vs-snowflake/ --- # DualBird and Starburst: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** DualBird (FPGA Spark plugin for AWS F2) and Starburst (Managed Trino service for Iceberg) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: DualBird on Amazon EMR (F2 instances): $49.50; Starburst Galaxy (Pro tier): $60.00. ## The same rows for both | Row | DualBird | Starburst | | --- | --- | --- | | Kind | FPGA Spark plugin for AWS F2 | Managed Trino service for Iceberg | | Does well | DualBird says onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin, with Spark still handling planning and scheduling (read 2026-11-10). DualBird published a 100 GB sort-based Iceberg compaction run on June 11, 2026, run by DualBird, and states about $2 per TB for itself against about $6 per TB for CPU clusters. | Per Starburst documentation, Galaxy creates and writes Iceberg tables (Parquet, ORC or Avro) and runs maintenance with ALTER TABLE ... EXECUTE optimize, expire_snapshots and remove_orphan_files (read 2026-11-10). Starburst publishes credit prices by AWS region, for example $0.50 (Pro) and $0.75 (Enterprise) in US East (N. Virginia) and $0.60 and $0.90 in Frankfurt (read 2026-11-10). | | Limited | DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. A price for the plugin and a benchmark run by a party other than DualBird: none found on 2026-11-10. | Per Starburst documentation, Galaxy runs SQL on Trino, not Spark, so DataFrame code, UDFs and PySpark would be rewritten as Trino SQL or kept on Spark (read 2026-11-10). Per Starburst documentation, Hudi tables are read-only in Galaxy: it cannot create new Hudi tables or write to them. | | Published speed results | The home page body text states the headline claim as 10x-30x faster data processing with 50%-90% lower costs and no changes required. The same wording appears in the page's JSON-LD organization description and in the April 28, 2026 blog post. The raw HTML of the home page has a different range in its meta description and og:description (and twitter:description): 10x-100x faster. This is the meta tag, not body text. The February 5, 2026 funding blog post also says 10-100x faster and 50-90% cost savings. The body text of the home and product pages says 10x-30x, so the two ranges differ by site location. The product page repeats the 10x-30x end-to-end figure labeled 'Instance for instance' and 50%-90% cost reduction labeled 'Workload for workload'. The meta description of the product page is only 'Plug-and-play Spark acceleration.' Home and product pages show a '110GB Enrichment Pipeline (JOIN Denormalization)' benchmark: 50%-85% lower EC2 cost with 6x-18x higher performance ceiling. The pages give no hardware, baseline, dates or method, and no link to a write-up was found. Sort-based Iceberg compaction on 100 GB, compared with vanilla Spark and with an unnamed 'C++ accelerated Spark' (unnamed) with tuned EBS, memory and parallelism at each cluster size. Dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3. Results: roughly 12-20x faster single-task execution, 55-85% lower EC2 cost, stable cost across scaling, about 6x higher practical performance ceiling. Cost per terabyte compacted is stated as about $2/TB for DualBird versus about $6/TB rising past $13/TB for CPU clusters. Part of the saving is attributed to F2 local NVMe replacing EBS volumes. The home page shows 'X15-X20' single-task speedup for the same benchmark, while the June 11, 2026 blog and the white paper landing page say 12-20x. The two do not match. The 'Single Task Performance' home-page tile says X15-X20 speedup for the same cluster size while reducing cost by over 50%. The blog describes the single-task comparison as against vanilla Spark, and the April 28 post as against 'a high-end C++ columnar engine running on a single CPU core'. The baseline wording is therefore not consistent across pages. | Starburst reported GPU-accelerated Trino results (its proprietary distribution, not open-source Trino): TPC-H up to 6x on accelerated queries and ClickBench 3.4x overall, single-node development environment. This is a SQL query test, not Spark batch work. | | Published prices and cost results | No pricing, price list or marketplace listing was found on the site. Cost claims are only percentages (50%-90% lower, 55%-85% lower EC2) and the about $2/TB versus $6-13/TB compaction figure. The pricing model for the plugin itself is not stated, so users cannot tell whether F2 instance price or license fee is included in the claimed savings. | Galaxy credit prices start at $0.50 (Pro), $0.75 (Enterprise) and $1.00 (Mission Critical) per credit. A Free tier allows up to 3 clusters. Pricing depends on compute usage, tier and the cloud provider and region. Annual commitments may get discounts. On-demand list prices per credit (Pro / Enterprise / Mission Critical) on AWS: US East N. Virginia, US East Ohio and US West Oregon $0.50 / $0.75 / $1.00; Europe Ireland $0.56 / $0.84 / $1.12; Europe Frankfurt $0.60 / $0.90 / $1.20; Asia Pacific Tokyo $0.64 / $0.97 / $1.29; Brazil (Sao Paulo) $0.80 / $1.19 / $1.59. Cluster compute credit consumption: 1 worker 6 credits per hour, 2 workers 12, 4 workers 24, 8 workers 48, 16 workers 96. Managed ingest: file ingest 0.60 credits per hour; table maintenance and data transforms 0.30 credits per hour. Billing tracks cluster compute hours, data ingestion compute and private link data transfer. | | Code changes and migration | Onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. The page shows three steps: change instance type, use our Spark plugin, see the results. The about page says 'No infrastructure changes. No code changes.' The compaction blog says Spark still handles planning, scheduling and fault tolerance and no pipeline rewrite or platform migration is needed. A DualBird engineer states the FPGA path accelerates a defined set of operators (sort, join, aggregation and others). Workloads dominated by Python UDFs or unsupported operators fall back to CPU and do not see the same benefit yet. Coverage is described as expanding. | Galaxy is 'a fully-managed data lake analytics platform built in addition to Trino'. Work runs as SQL on Trino, not Spark, so Spark jobs (DataFrame code, UDFs, PySpark) would have to be rewritten as Trino SQL, or kept on Spark against the same Iceberg tables. The Galaxy docs read contain no Spark execution mode. | | Tuning and table upkeep | DualBird claims tens of Spark settings (executor sizing, CPU-memory ratios, parallelism) become irrelevant, disk spills effectively disappear and data skew stops causing spills, because only a few tasks run at once on the FPGA. The product page says the majority of common Spark performance configurations do not exist when using the solution. Cluster sizing reduces to the number of F2 nodes. These are vendor claims, not independently tested. Part 2 of the series (Sep 1, 2026) says skew still costs time, but less, and that it does not need repartitioning or salting. | Iceberg table maintenance in Galaxy is run with ALTER TABLE ... EXECUTE commands: optimize (compaction; files under file_size_threshold, default 100MB, are merged), drop_extended_stats, expire_snapshots, remove_orphan_files. Live table maintenance for ingest tables is a separate Galaxy page. For batch-style work Galaxy offers a fault tolerant execution mode that spools intermediate data and retries failed parts of a query. Docs say 'No other configuration is required' to enable it, but not to use it if most queries run under about a minute, and that it can be slightly slower than normal execution. FTE and Accelerated modes are not available on the Free cluster size and not all catalogs support FTE. | | Spark versions and table formats | The home page says the plugin is compatible with Apache Spark, Amazon EMR and Amazon EKS. It lists 'Optimized out of the box', 'Migration-free' and 'No more performance tuning and troubleshooting'. No Spark version numbers, Databricks support or Iceberg/Delta/Hudi support list are stated on the home, product, about or resources pages. Iceberg appears only in the compaction benchmark and in the meta description of the Dec 1, 2025 demo post ('Accelerate Apache Spark and Apache Iceberg'). | Galaxy's Great Lakes connectivity lists Iceberg, Delta Lake, Hive and Hudi table format pages. Hudi tables are read-only: 'Galaxy cannot create new Hudi tables or write to them.' Iceberg tables can be created and written; the Iceberg data file format can be Parquet, ORC or Avro. | | Where it runs | DualBird needs AWS EC2 F2 instances. AWS describes F2 as second-generation FPGA instances with up to 8 AMD Virtex UltraScale+ HBM VU47P FPGAs, 16 GiB HBM per FPGA, 3rd-gen AMD EPYC (Milan) CPUs up to 192 vCPU, up to 2 TiB memory and up to 7.6 TiB NVMe. Sizes: f2.6xlarge (1 FPGA, 24 vCPU, 256 GiB), f2.12xlarge (2 FPGAs, 48 vCPU, 512 GiB), f2.48xlarge (8 FPGAs, 192 vCPU, 2,048 GiB). The AWS 'instance types by Region' doc lists F2 in eight regions: us-east-1 (N. Virginia), us-west-2 (Oregon), ap-northeast-1 (Tokyo), ap-northeast-2 (Seoul), ap-southeast-2 (Sydney), ca-central-1 (Canada Central), eu-central-1 (Frankfurt) and eu-west-2 (London). F2 is not listed in other regions on that page (for example us-east-2, eu-west-1). DualBird is AWS-only: no other cloud is mentioned. The product page says deployment runs on the customer's own servers in their AWS environment and 'Everything stays under your control'. | Galaxy prices are listed for AWS, Azure and GCP regions. AWS regions priced include us-east-1, us-east-2, us-west-1, us-west-2, ca-central, Ireland, London, Frankfurt, Zurich, Singapore, Sydney, Tokyo, Cape Town, UAE, Tel Aviv and Sao Paulo. A cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature. | | How open its benchmarks are | The white paper (dated July 1, 2026) has a landing page that restates the results and says methodology, system architecture and detailed analysis are in a download. The full document is gated behind a form and was not readable. The blog gives dataset shape and baselines but not instance counts, Spark version, configs or scripts. No public repo was found (github.com/dualbird returns 404). | None found on 10 Nov 2026. | | Compute Bill | DualBird on Amazon EMR (F2 instances): $49.50 | Starburst Galaxy (Pro tier): $60.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - DualBird: dualbird.io/product: https://dualbird.io/product (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/a-simpler-spark: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-low: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird: dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird: aws.amazon.com/ec2/instance-types/f2: https://aws.amazon.com/ec2/instance-types/f2/ (read 10 Nov 2026) - DualBird: docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing: https://www.starburst.io/pricing/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing/pricing-details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/index.html: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/optimization-perform: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/dualbird-vs-starburst/ --- # Google BigQuery and BigLake and Microsoft Fabric: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Google BigQuery and BigLake (Serverless Google Cloud data warehouse) and Microsoft Fabric (SaaS analytics on Azure with Spark) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: BigQuery (Enterprise edition slots): $19.20; Microsoft Fabric (F128 capacity): $46.08. ## The same rows for both | Row | Google BigQuery and BigLake | Microsoft Fabric | | --- | --- | --- | | Kind | Serverless Google Cloud data warehouse | SaaS analytics on Azure with Spark | | Does well | Per Google documentation, BigQuery Iceberg managed tables store data in buckets you own, support DML and streaming, and run automatic compaction, clustering and garbage collection (read 2026-11-10). Per Google documentation, stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud (read 2026-11-10). | Per Microsoft Learn, the native execution engine (Apache Gluten and Velox) needs no code changes, falls back to JVM Spark for unsupported operators and supports Parquet and Delta (read 2026-11-10). Per Microsoft Learn, OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying it (read 2026-11-10). | | Limited | Per Google documentation, BigQuery Omni on S3 does not support DML or CREATE TABLE, so it does not run write-heavy Spark batch ETL on S3. Google's pricing page text lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate (read 2026-11-10). Per-TiB on-demand and Spark DCU rates are not used on this site. | Per Microsoft Learn, Fabric runs over OneLake on Azure and the region page lists Azure regions only, so Spark compute does not run inside AWS. Writes to S3 through a OneLake shortcut are not possible, and the method for Microsoft's native engine benchmark: none found on 2026-11-10. | | Published speed results | Google states Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name; no method is given on the pricing page. | Microsoft reports that on TPC-DS scale factor 1000 with Delta the native engine was up to six times faster than open-source Spark, with about 83% compute cost savings on a fixed-size cluster. | | Published prices and cost results | BigQuery offers on-demand pricing per TiB scanned (first 1 TiB per month free) or capacity pricing per slot-hour. BigQuery Omni has its own on-demand per TiB pricing for AWS regions. The page text shows $0.06 per slot-hour for Enterprise edition pay-as-you-go compute, with no region shown beside the rate (read 2026-11-10). Google's Spark service (Managed Service for Apache Spark, formerly Dataproc) bills serverless Spark per second in Data Compute Units with a 1-minute minimum, plus shuffle storage; Standard tier is for batch ETL, Premium tier uses Lightning Engine. | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus (Spark Memory Optimized and autoscale for Spark entries, pay-as-you-go). In Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling. Autoscale Billing for Spark is an opt-in pay-as-you-go model that runs Spark on dedicated serverless resources at standard PAYG CU-hour rates; reserved capacity discounts do not apply and a base capacity is still required. | | Code changes and migration | On BigQuery Omni, DML statements and CREATE TABLE are not supported, BigQuery ML statements are not supported, and only EXPORT DATA queries are allowed for destination tables, so it does not run write-heavy Spark batch ETL on S3. Standard and Enterprise Plus editions do not support Omni regions. BigQuery stored procedures for Apache Spark let you run PySpark or Scala/Java JAR code from SQL via CREATE PROCEDURE ... OPTIONS(engine = "SPARK", runtime_version = "2.2"), using a Cloud Storage file or inline code and a connection; this runs on Google Cloud, so code must be wrapped as procedures. | The Fabric native execution engine, based on Apache Gluten and Velox, needs no code changes, falls back to JVM Spark for unsupported operators, and supports Parquet and Delta. | | Tuning and table upkeep | Iceberg managed tables run automatic storage optimization (adaptive file sizing, clustering, garbage collection), so compaction needs no job configuration on Google Cloud; storage optimization is a separate billed item. | Starter pools start in 5 to 10 seconds with no setup; custom Spark pools let you tune node size and scaling, and native execution can be toggled per application with a Spark setting. | | Spark versions and table formats | BigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, export Iceberg V2 snapshots for open engines, and do automatic compaction, clustering and garbage collection. BigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security. | Fabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1; OneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and vice versa. | | Where it runs | BigQuery Omni lets BigQuery analytics run on data in Amazon S3 through BigLake tables, in AWS regions us-east-1, us-west-2, ap-northeast-2, ap-southeast-2, eu-west-1 and eu-central-1. | Fabric workloads run over OneLake, described as built on Azure Data Lake Storage; the region availability page lists Azure regions and no AWS regions, so Spark compute does not run inside AWS. OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; file caching can reduce cross-cloud egress costs. Writes to S3 are not possible through the shortcut. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | BigQuery (Enterprise edition slots): $19.20 | Microsoft Fabric (F128 capacity): $46.08 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/omni-introduction: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/spark-procedures: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/iceberg-tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/biglake-intro: https://cloud.google.com/bigquery/docs/biglake-intro (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/dataproc-serverless/pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview: https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - Microsoft Fabric: prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption' (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Microsoft Fabric: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-compute: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/bigquery-biglake-vs-microsoft-fabric/ --- # Google BigQuery and BigLake and Onehouse: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Google BigQuery and BigLake and Onehouse are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Google BigQuery and BigLake | Onehouse | | --- | --- | --- | | Kind | Serverless Google Cloud data warehouse | Managed lakehouse in your cloud account | | Does well | Per Google documentation, BigQuery Iceberg managed tables store data in buckets you own, support DML and streaming, and run automatic compaction, clustering and garbage collection (read 2026-11-10). Per Google documentation, stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud (read 2026-11-10). | Per Onehouse documentation, its jobs are fully compatible with Apache Spark and EMR code can be reused unless it uses AWS-specific features (read 2026-11-10). Onehouse published the setup for its 10 TB TPC-DS and 1 TB LakeLoader comparison against EMR 7.12, and open-sourced LakeLoader under Apache-2.0 (read 2026-11-10). | | Limited | Per Google documentation, BigQuery Omni on S3 does not support DML or CREATE TABLE, so it does not run write-heavy Spark batch ETL on S3. Google's pricing page text lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate (read 2026-11-10). Per-TiB on-demand and Spark DCU rates are not used on this site. | Per Onehouse documentation, billing is in OCUs on a bring-your-own-cloud model, and no dollar rate per OCU was found on 2026-11-10. Onehouse's results against EMR 7.12 are its own and shown as charts; a confirmation by another party: none found on 2026-11-10. | | Published speed results | Google states Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name; no method is given on the pricing page. | Onehouse benchmarked Quanton against open-source Spark on Iceberg: TPC-DS up to about 3.1x, TPCx-BB 2x, TPC-DI (1 TB) 2x with 3-4x on MERGE INTO, and LakeLoader 4.1x for writes. The baseline is open-source Spark, not EMR. Onehouse reran its benchmarks against EMR 7.12 after AWS announced it: it reports EMR improved 32% over earlier versions and that Onehouse 'still comes out with a 2.5x better price/performance advantage'. Setup: 10 TB TPC-DS for read/transform, 1 TB LakeLoader merges for load, latest general-purpose Graviton instances. Results are shown as charts; Onehouse also reports Hudi and Delta query performance on EMR 7.12 was very similar to Iceberg. A B2B SaaS customer (unnamed) ran its UDF-heavy pipeline on EMR and on Quanton in its own AWS account on m8gd nodes. Quanton was faster at p50, p97 and p99 tenant sizes; on the largest tenant (21 billion rows) EMR filled local disks on 180 nodes and was aborted after 3.7 hours, and finished on 300 nodes. The write-up is published by Onehouse. | | Published prices and cost results | BigQuery offers on-demand pricing per TiB scanned (first 1 TiB per month free) or capacity pricing per slot-hour. BigQuery Omni has its own on-demand per TiB pricing for AWS regions. The page text shows $0.06 per slot-hour for Enterprise edition pay-as-you-go compute, with no region shown beside the rate (read 2026-11-10). Google's Spark service (Managed Service for Apache Spark, formerly Dataproc) bills serverless Spark per second in Data Compute Units with a 1-minute minimum, plus shuffle storage; Standard tier is for batch ETL, Premium tier uses Lightning Engine. | Onehouse bills in Onehouse Compute Units (OCU) on a bring-your-own-cloud model: 1 OCU is a 4-core instance for 1 hour in your account, plus support costs; no separate volume or storage fee, and the customer pays its own AWS or GCP infrastructure bill. The docs page does not state a dollar rate per OCU. Vendor claims at the Quanton launch: 2-3x better price-performance for ETL, and a guaranteed greater-than-50% cost reduction for existing Hudi users moving Spark jobs from AWS EMR or open-source Spark. These are guarantees stated by the vendor, not results from an independent party. | | Code changes and migration | On BigQuery Omni, DML statements and CREATE TABLE are not supported, BigQuery ML statements are not supported, and only EXPORT DATA queries are allowed for destination tables, so it does not run write-heavy Spark batch ETL on S3. Standard and Enterprise Plus editions do not support Omni regions. BigQuery stored procedures for Apache Spark let you run PySpark or Scala/Java JAR code from SQL via CREATE PROCEDURE ... OPTIONS(engine = "SPARK", runtime_version = "2.2"), using a Cloud Storage file or inline code and a connection; this runs on Google Cloud, so code must be wrapped as procedures. | Onehouse docs say its Jobs are fully compatible with Apache Spark and that EMR code can be reused unless it uses AWS-specific features. Migration work listed: map EMR steps to Onehouse Jobs created through its API or Airflow operators (CREATE CLUSTER, CREATE JOB, RUN JOB, DELETE CLUSTER), and configure Glue catalog sync or contact support to make Glue the primary catalog. At the May 20, 2025 launch Onehouse said users can run existing Spark and SQL jobs on new cluster types and that existing Hudi/Spark jobs can be pointed at Onehouse 'with no code changes'. The Quanton Kubernetes Operator is described as a 'drop-in replacement' for an existing Spark Kubernetes operator. | | Tuning and table upkeep | Iceberg managed tables run automatic storage optimization (adaptive file sizing, clustering, garbage collection), so compaction needs no job configuration on Google Cloud; storage optimization is a separate billed item. | Each cluster needs min and max OCU limits. Docs warn that if usage reaches Max OCU the cluster stops scaling up, which can delay processing; Min OCU keeps a cluster warm. The Onehouse docs pages read give no per-job Spark tuning list; Onehouse claims automatic scaling and automated async table services. | | Spark versions and table formats | BigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, export Iceberg V2 snapshots for open engines, and do automatic compaction, clustering and garbage collection. BigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security. | Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake. Docs: Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog (AWS Glue and Snowflake Open Catalog are documented); for Delta Lake the user installs Delta Spark as a dependency in the JAR or Python job. In the Nov 12, 2025 Iceberg post Onehouse said Delta acceleration is 'up next'. Apache XTable (formerly OneTable) translates table metadata between Hudi, Delta Lake and Iceberg without being a new format. It is incubating at the Apache Software Foundation; Onehouse says it created Hudi and XTable. | | Where it runs | BigQuery Omni lets BigQuery analytics run on data in Amazon S3 through BigLake tables, in AWS regions us-east-1, us-west-2, ap-northeast-2, ap-southeast-2, eu-west-1 and eu-central-1. | Onehouse runs EC2 or Google Compute Engine instances inside the customer's AWS or GCP account (Azure not listed in the billing docs). Standard instance types map to Graviton m8g sizes on AWS (oh-general-4 = m8g.xlarge, oh-general-8 = m8g.2xlarge, oh-general-16 = m8g.4xlarge). A project is scoped to one region and account; adding a region needs a support ticket. | | How open its benchmarks are | None found on 10 Nov 2026. | Onehouse open-sourced LakeLoader, a tool that generates incremental update/insert loads for benchmarking write performance. The repository is Apache-2.0 licensed, last pushed 2026-10-30. The Quanton engine itself and the full Iceberg benchmark configurations were not found as a repository in the pages read. | | Compute Bill | BigQuery (Enterprise edition slots): $19.20 | Onehouse (Quanton, bring-your-own-cloud): no total (see the Compute Bill for why) | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/omni-introduction: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/spark-procedures: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/iceberg-tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/biglake-intro: https://cloud.google.com/bigquery/docs/biglake-intro (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/dataproc-serverless/pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs/emr-migration: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtim: https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/project-settings/usage-and-billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): xtable.apache.org: https://xtable.apache.org/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/compute/create-cluster: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-worklo: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark: https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): api.github.com/repos/onehouseinc/lake-loader: https://api.github.com/repos/onehouseinc/lake-loader (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/bigquery-biglake-vs-onehouse/ --- # Google BigQuery and BigLake and Open stack on EMR and EKS: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Google BigQuery and BigLake and Open stack on EMR and EKS are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Google BigQuery and BigLake | Open stack on EMR and EKS | | --- | --- | --- | | Kind | Serverless Google Cloud data warehouse | Self-run Spark with open table formats | | Does well | Per Google documentation, BigQuery Iceberg managed tables store data in buckets you own, support DML and streaming, and run automatic compaction, clustering and garbage collection (read 2026-11-10). Per Google documentation, stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud (read 2026-11-10). | Per AWS documentation, EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three formats are available (read 2026-11-10). AWS ran TPC-DS 3 TB on 9 x r5d.4xlarge and reports $4.81 for EMR 7.12 against $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0, with repositories to repeat it. | | Limited | Per Google documentation, BigQuery Omni on S3 does not support DML or CREATE TABLE, so it does not run write-heavy Spark batch ETL on S3. Google's pricing page text lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate (read 2026-11-10). Per-TiB on-demand and Spark DCU rates are not used on this site. | Per the Apache Iceberg documentation, users run table maintenance themselves: expire snapshots, remove orphan files and compact with rewriteDataFiles as jobs they schedule (read 2026-11-10). A cost comparison of open-stack Spark against managed lakehouses by a party other than AWS or a vendor: none found on 2026-11-10. | | Published speed results | Google states Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name; no method is given on the pricing page. | AWS ran TPC-DS 3 TB (104 SparkSQL queries, 3 rounds, 9 x r5d.4xlarge, Hadoop catalog) and reports EMR 7.12 averaged 1,349.62 s versus 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0 and 1,535.62 s for EMR 7.5. AWS notes results are not comparable to official TPC-DS. The run is by the vendor of EMR. | | Published prices and cost results | BigQuery offers on-demand pricing per TiB scanned (first 1 TiB per month free) or capacity pricing per slot-hour. BigQuery Omni has its own on-demand per TiB pricing for AWS regions. The page text shows $0.06 per slot-hour for Enterprise edition pay-as-you-go compute, with no region shown beside the rate (read 2026-11-10). Google's Spark service (Managed Service for Apache Spark, formerly Dataproc) bills serverless Spark per second in Data Compute Units with a 1-minute minimum, plus shuffle storage; Standard tier is for batch ETL, Premium tier uses Lightning Engine. | EMR on EC2 charges an EMR price in addition to the EC2 price (and EBS), billed per second with a one-minute minimum. AWS's example in us-east-1: c4.2xlarge EMR charge $0.105/hour next to EC2 $0.398/hour; one master and two core nodes for a month total $1,101.57. Spot instances are up to a 90% discount versus On-Demand. EMR on EKS adds an uplift per vCPU-hour and per GB-hour to EKS/EC2 or Fargate costs, plus $0.10 per hour for each EKS cluster. AWS's worked example uses $0.007084 per vCPU-hour and $0.000777875 per GB-hour (100 vCPU and 300 GB for 30 minutes = $0.4709 EMR uplift). The uplift applies to any pod using an EMR runtime image, including with the Spark operator. EMR Serverless is billed per vCPU-hour and GB-hour. AWS's worked example uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour for a job with 100 vCPU and 750 GB for 30 minutes, then 200 more vCPU for 15 minutes, totaling $9.596275. In the same AWS benchmark the cost was $4.81 for EMR 7.12, $5.47 for EMR 7.5 and $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0. Cost formula uses r5d.4xlarge at $1.152/hour plus $0.27/hour EMR plus EBS, On-Demand. | | Code changes and migration | On BigQuery Omni, DML statements and CREATE TABLE are not supported, BigQuery ML statements are not supported, and only EXPORT DATA queries are allowed for destination tables, so it does not run write-heavy Spark batch ETL on S3. Standard and Enterprise Plus editions do not support Omni regions. BigQuery stored procedures for Apache Spark let you run PySpark or Scala/Java JAR code from SQL via CREATE PROCEDURE ... OPTIONS(engine = "SPARK", runtime_version = "2.2"), using a Cloud Storage file or inline code and a connection; this runs on Google Cloud, so code must be wrapped as procedures. | The EMR runtime for Spark provides 'full API compatibility with open source Apache Spark and Apache Iceberg', so existing Spark jobs run as is. Optimized runtimes are used by EMR on EC2, EMR Serverless, EMR on EKS, EMR on Outposts and AWS Glue. Self-run Spark on EKS uses the open-source Spark operator and needs no job rewrite. | | Tuning and table upkeep | Iceberg managed tables run automatic storage optimization (adaptive file sizing, clustering, garbage collection), so compaction needs no job configuration on Google Cloud; storage optimization is a separate billed item. | Iceberg docs say users must run maintenance: expire snapshots, remove old metadata files and delete orphan files are 'recommended'; compacting data files with the Spark rewriteDataFiles action (for example target-file-size-bytes of 500 MB) and rewriting manifests are 'optional' maintenance, run as jobs you schedule. Hudi merge-on-read tables need compaction to merge log files into base files and bound read latency. Compaction has two steps, scheduling (plan written to the timeline) and execution; the docs describe strategies for choosing which file slices to compact. Delta Lake OSS docs list file management features the user applies: compaction (bin-packing) with OPTIMIZE, auto compaction, data skipping, Z-ordering, multi-part checkpointing, log compactions and optimized write. | | Spark versions and table formats | BigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, export Iceberg V2 snapshots for open engines, and do automatic compaction, clustering and garbage collection. BigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security. | Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. All three table formats are available on EMR. EMR on EKS supports the Kubernetes Operator for Apache Spark (Spark operator) as a job submission model with the EMR runtime on your own EKS clusters. The Kubeflow Spark Operator is Apache-2.0 licensed with Helm installation. | | Where it runs | BigQuery Omni lets BigQuery analytics run on data in Amazon S3 through BigLake tables, in AWS regions us-east-1, us-west-2, ap-northeast-2, ap-southeast-2, eu-west-1 and eu-central-1. | Amazon EMR has regional endpoints in the US (us-east-1, us-east-2, us-west-1, us-west-2) and many other AWS regions per the AWS general reference endpoints table. Spark runs on EC2 instances the user chooses (the AWS benchmark used r5d.4xlarge). | | How open its benchmarks are | None found on 10 Nov 2026. | AWS publishes the setup in GitHub repositories (aws-samples/emr-on-eks-benchmark branch tpcds-v2.13_iceberg, and aws-samples/emr-spark-benchmark) with build instructions, so the TPC-DS Iceberg benchmark can be repeated. | | Compute Bill | BigQuery (Enterprise edition slots): $19.20 | Open stack: Spark on plain EC2: $20.16 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/omni-introduction: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/spark-procedures: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/iceberg-tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/biglake-intro: https://cloud.google.com/bigquery/docs/biglake-intro (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/dataproc-serverless/pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operato: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: iceberg.apache.org/docs/latest/maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: hudi.apache.org/docs/compaction: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.delta.io/latest/optimizations-oss.html: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/general/latest/gr/emr.html: https://docs.aws.amazon.com/general/latest/gr/emr.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/bigquery-biglake-vs-open-stack-emr-eks/ --- # Google BigQuery and BigLake and Snowflake: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Google BigQuery and BigLake (Serverless Google Cloud data warehouse) and Snowflake (Cloud data platform with PySpark support) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: BigQuery (Enterprise edition slots): $19.20; Snowflake (XL standard warehouse): $96.00. ## The same rows for both | Row | Google BigQuery and BigLake | Snowflake | | --- | --- | --- | | Kind | Serverless Google Cloud data warehouse | Cloud data platform with PySpark support | | Does well | Per Google documentation, BigQuery Iceberg managed tables store data in buckets you own, support DML and streaming, and run automatic compaction, clustering and garbage collection (read 2026-11-10). Per Google documentation, stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud (read 2026-11-10). | Per Snowflake documentation, Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect, and batch jobs are submitted with the Snowpark Submit CLI (read 2026-11-10). Per Snowflake documentation, Iceberg tables can keep data in customer-managed S3, GCS or Azure storage through external volumes and a catalog integration such as AWS Glue. | | Limited | Per Google documentation, BigQuery Omni on S3 does not support DML or CREATE TABLE, so it does not run write-heavy Spark batch ETL on S3. Google's pricing page text lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate (read 2026-11-10). Per-TiB on-demand and Spark DCU rates are not used on this site. | Per Snowflake documentation, Snowpark Connect supports Apache Spark 3.5 only, and DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported (read 2026-11-10). On-demand credit prices on AWS US East (Northern Virginia) are $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical) per the Snowflake Service Consumption Table effective 2026-11-09 (read 2026-11-10); the Snowflake pricing page itself renders by script. | | Published speed results | Google states Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name; no method is given on the pricing page. | None found on 10 Nov 2026. | | Published prices and cost results | BigQuery offers on-demand pricing per TiB scanned (first 1 TiB per month free) or capacity pricing per slot-hour. BigQuery Omni has its own on-demand per TiB pricing for AWS regions. The page text shows $0.06 per slot-hour for Enterprise edition pay-as-you-go compute, with no region shown beside the rate (read 2026-11-10). Google's Spark service (Managed Service for Apache Spark, formerly Dataproc) bills serverless Spark per second in Data Compute Units with a 1-minute minimum, plus shuffle storage; Standard tier is for batch ETL, Premium tier uses Lightning Engine. | Warehouses bill credits per second with a 60-second minimum on each start; each size step roughly doubles credits per hour (Small is 2 credits per hour, Medium is 4). Snowflake's Service Consumption Table (effective 2026-11-09) lists on-demand credits on AWS US East at $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical). | | Code changes and migration | On BigQuery Omni, DML statements and CREATE TABLE are not supported, BigQuery ML statements are not supported, and only EXPORT DATA queries are allowed for destination tables, so it does not run write-heavy Spark batch ETL on S3. Standard and Enterprise Plus editions do not support Omni regions. BigQuery stored procedures for Apache Spark let you run PySpark or Scala/Java JAR code from SQL via CREATE PROCEDURE ... OPTIONS(engine = "SPARK", runtime_version = "2.2"), using a Cloud Storage file or inline code and a connection; this runs on Google Cloud, so code must be wrapped as procedures. | Snowpark Connect for Apache Spark runs existing PySpark DataFrame code against the Snowflake engine over the Spark Connect protocol; there is no Spark cluster, all workloads run on a Snowflake warehouse. Batch jobs are submitted with the Snowpark Submit CLI. Because Snowflake rather than Spark is the engine, there are semantic differences: DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported, and ByteType, ShortType and IntegerType are returned as LongType. The docs list unsupported APIs for DataFrame and Dataset. Docs say the Spark DataFrame and Spark SQL code can be reused with minimal migration or changes; the compatibility notes say Spark Connect defers plan resolution, which changes behavior versus Spark Classic. | | Tuning and table upkeep | Iceberg managed tables run automatic storage optimization (adaptive file sizing, clustering, garbage collection), so compaction needs no job configuration on Google Cloud; storage optimization is a separate billed item. | Docs provide a separate Optimizing Snowpark Connect for Spark workloads section and a Troubleshooting section covering known limitations, so some job-level adjustment is expected after migration. | | Spark versions and table formats | BigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, export Iceberg V2 snapshots for open engines, and do automatic compaction, clustering and garbage collection. BigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security. | Snowpark Connect supports Apache Spark 3.5 only; workloads written for 3.4 and below or 4.0 and above are not supported and may produce protocol errors or missing API failures. Snowflake supports Iceberg tables (Parquet) with data in customer-managed S3, GCS or Azure storage via external volumes, an external catalog integration such as AWS Glue, and Delta files read through Delta Direct; Hudi was not listed on the page. | | Where it runs | BigQuery Omni lets BigQuery analytics run on data in Amazon S3 through BigLake tables, in AWS regions us-east-1, us-west-2, ap-northeast-2, ap-southeast-2, eu-west-1 and eu-central-1. | Snowflake runs on AWS, Azure and Google Cloud; AWS regions include us-east-1, us-west-2, eu-west-1 and ca-central-1. Each account is hosted in a single region. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | BigQuery (Enterprise edition slots): $19.20 | Snowflake (XL standard warehouse): $96.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/omni-introduction: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/spark-procedures: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/iceberg-tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/biglake-intro: https://cloud.google.com/bigquery/docs/biglake-intro (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/dataproc-serverless/pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/cost-understanding-compute: https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/tables-iceberg: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/intro-regions.md: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/bigquery-biglake-vs-snowflake/ --- # Google BigQuery and BigLake and Starburst: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Google BigQuery and BigLake (Serverless Google Cloud data warehouse) and Starburst (Managed Trino service for Iceberg) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: BigQuery (Enterprise edition slots): $19.20; Starburst Galaxy (Pro tier): $60.00. ## The same rows for both | Row | Google BigQuery and BigLake | Starburst | | --- | --- | --- | | Kind | Serverless Google Cloud data warehouse | Managed Trino service for Iceberg | | Does well | Per Google documentation, BigQuery Iceberg managed tables store data in buckets you own, support DML and streaming, and run automatic compaction, clustering and garbage collection (read 2026-11-10). Per Google documentation, stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud (read 2026-11-10). | Per Starburst documentation, Galaxy creates and writes Iceberg tables (Parquet, ORC or Avro) and runs maintenance with ALTER TABLE ... EXECUTE optimize, expire_snapshots and remove_orphan_files (read 2026-11-10). Starburst publishes credit prices by AWS region, for example $0.50 (Pro) and $0.75 (Enterprise) in US East (N. Virginia) and $0.60 and $0.90 in Frankfurt (read 2026-11-10). | | Limited | Per Google documentation, BigQuery Omni on S3 does not support DML or CREATE TABLE, so it does not run write-heavy Spark batch ETL on S3. Google's pricing page text lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate (read 2026-11-10). Per-TiB on-demand and Spark DCU rates are not used on this site. | Per Starburst documentation, Galaxy runs SQL on Trino, not Spark, so DataFrame code, UDFs and PySpark would be rewritten as Trino SQL or kept on Spark (read 2026-11-10). Per Starburst documentation, Hudi tables are read-only in Galaxy: it cannot create new Hudi tables or write to them. | | Published speed results | Google states Lightning Engine delivers up to 4.9x faster performance than open source Spark and up to 2.5x the price-performance over a high-speed Spark offering Google does not name; no method is given on the pricing page. | Starburst reported GPU-accelerated Trino results (its proprietary distribution, not open-source Trino): TPC-H up to 6x on accelerated queries and ClickBench 3.4x overall, single-node development environment. This is a SQL query test, not Spark batch work. | | Published prices and cost results | BigQuery offers on-demand pricing per TiB scanned (first 1 TiB per month free) or capacity pricing per slot-hour. BigQuery Omni has its own on-demand per TiB pricing for AWS regions. The page text shows $0.06 per slot-hour for Enterprise edition pay-as-you-go compute, with no region shown beside the rate (read 2026-11-10). Google's Spark service (Managed Service for Apache Spark, formerly Dataproc) bills serverless Spark per second in Data Compute Units with a 1-minute minimum, plus shuffle storage; Standard tier is for batch ETL, Premium tier uses Lightning Engine. | Galaxy credit prices start at $0.50 (Pro), $0.75 (Enterprise) and $1.00 (Mission Critical) per credit. A Free tier allows up to 3 clusters. Pricing depends on compute usage, tier and the cloud provider and region. Annual commitments may get discounts. On-demand list prices per credit (Pro / Enterprise / Mission Critical) on AWS: US East N. Virginia, US East Ohio and US West Oregon $0.50 / $0.75 / $1.00; Europe Ireland $0.56 / $0.84 / $1.12; Europe Frankfurt $0.60 / $0.90 / $1.20; Asia Pacific Tokyo $0.64 / $0.97 / $1.29; Brazil (Sao Paulo) $0.80 / $1.19 / $1.59. Cluster compute credit consumption: 1 worker 6 credits per hour, 2 workers 12, 4 workers 24, 8 workers 48, 16 workers 96. Managed ingest: file ingest 0.60 credits per hour; table maintenance and data transforms 0.30 credits per hour. Billing tracks cluster compute hours, data ingestion compute and private link data transfer. | | Code changes and migration | On BigQuery Omni, DML statements and CREATE TABLE are not supported, BigQuery ML statements are not supported, and only EXPORT DATA queries are allowed for destination tables, so it does not run write-heavy Spark batch ETL on S3. Standard and Enterprise Plus editions do not support Omni regions. BigQuery stored procedures for Apache Spark let you run PySpark or Scala/Java JAR code from SQL via CREATE PROCEDURE ... OPTIONS(engine = "SPARK", runtime_version = "2.2"), using a Cloud Storage file or inline code and a connection; this runs on Google Cloud, so code must be wrapped as procedures. | Galaxy is 'a fully-managed data lake analytics platform built in addition to Trino'. Work runs as SQL on Trino, not Spark, so Spark jobs (DataFrame code, UDFs, PySpark) would have to be rewritten as Trino SQL, or kept on Spark against the same Iceberg tables. The Galaxy docs read contain no Spark execution mode. | | Tuning and table upkeep | Iceberg managed tables run automatic storage optimization (adaptive file sizing, clustering, garbage collection), so compaction needs no job configuration on Google Cloud; storage optimization is a separate billed item. | Iceberg table maintenance in Galaxy is run with ALTER TABLE ... EXECUTE commands: optimize (compaction; files under file_size_threshold, default 100MB, are merged), drop_extended_stats, expire_snapshots, remove_orphan_files. Live table maintenance for ingest tables is a separate Galaxy page. For batch-style work Galaxy offers a fault tolerant execution mode that spools intermediate data and retries failed parts of a query. Docs say 'No other configuration is required' to enable it, but not to use it if most queries run under about a minute, and that it can be slightly slower than normal execution. FTE and Accelerated modes are not available on the Free cluster size and not all catalogs support FTE. | | Spark versions and table formats | BigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, export Iceberg V2 snapshots for open engines, and do automatic compaction, clustering and garbage collection. BigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security. | Galaxy's Great Lakes connectivity lists Iceberg, Delta Lake, Hive and Hudi table format pages. Hudi tables are read-only: 'Galaxy cannot create new Hudi tables or write to them.' Iceberg tables can be created and written; the Iceberg data file format can be Parquet, ORC or Avro. | | Where it runs | BigQuery Omni lets BigQuery analytics run on data in Amazon S3 through BigLake tables, in AWS regions us-east-1, us-west-2, ap-northeast-2, ap-southeast-2, eu-west-1 and eu-central-1. | Galaxy prices are listed for AWS, Azure and GCP regions. AWS regions priced include us-east-1, us-east-2, us-west-1, us-west-2, ca-central, Ireland, London, Frankfurt, Zurich, Singapore, Sydney, Tokyo, Cape Town, UAE, Tel Aviv and Sao Paulo. A cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | BigQuery (Enterprise edition slots): $19.20 | Starburst Galaxy (Pro tier): $60.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/omni-introduction: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/spark-procedures: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/iceberg-tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/docs/biglake-intro: https://cloud.google.com/bigquery/docs/biglake-intro (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery/pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/dataproc-serverless/pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing: https://www.starburst.io/pricing/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing/pricing-details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/index.html: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/optimization-perform: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/bigquery-biglake-vs-starburst/ --- # Microsoft Fabric and Onehouse: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Microsoft Fabric and Onehouse are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Microsoft Fabric | Onehouse | | --- | --- | --- | | Kind | SaaS analytics on Azure with Spark | Managed lakehouse in your cloud account | | Does well | Per Microsoft Learn, the native execution engine (Apache Gluten and Velox) needs no code changes, falls back to JVM Spark for unsupported operators and supports Parquet and Delta (read 2026-11-10). Per Microsoft Learn, OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying it (read 2026-11-10). | Per Onehouse documentation, its jobs are fully compatible with Apache Spark and EMR code can be reused unless it uses AWS-specific features (read 2026-11-10). Onehouse published the setup for its 10 TB TPC-DS and 1 TB LakeLoader comparison against EMR 7.12, and open-sourced LakeLoader under Apache-2.0 (read 2026-11-10). | | Limited | Per Microsoft Learn, Fabric runs over OneLake on Azure and the region page lists Azure regions only, so Spark compute does not run inside AWS. Writes to S3 through a OneLake shortcut are not possible, and the method for Microsoft's native engine benchmark: none found on 2026-11-10. | Per Onehouse documentation, billing is in OCUs on a bring-your-own-cloud model, and no dollar rate per OCU was found on 2026-11-10. Onehouse's results against EMR 7.12 are its own and shown as charts; a confirmation by another party: none found on 2026-11-10. | | Published speed results | Microsoft reports that on TPC-DS scale factor 1000 with Delta the native engine was up to six times faster than open-source Spark, with about 83% compute cost savings on a fixed-size cluster. | Onehouse benchmarked Quanton against open-source Spark on Iceberg: TPC-DS up to about 3.1x, TPCx-BB 2x, TPC-DI (1 TB) 2x with 3-4x on MERGE INTO, and LakeLoader 4.1x for writes. The baseline is open-source Spark, not EMR. Onehouse reran its benchmarks against EMR 7.12 after AWS announced it: it reports EMR improved 32% over earlier versions and that Onehouse 'still comes out with a 2.5x better price/performance advantage'. Setup: 10 TB TPC-DS for read/transform, 1 TB LakeLoader merges for load, latest general-purpose Graviton instances. Results are shown as charts; Onehouse also reports Hudi and Delta query performance on EMR 7.12 was very similar to Iceberg. A B2B SaaS customer (unnamed) ran its UDF-heavy pipeline on EMR and on Quanton in its own AWS account on m8gd nodes. Quanton was faster at p50, p97 and p99 tenant sizes; on the largest tenant (21 billion rows) EMR filled local disks on 180 nodes and was aborted after 3.7 hours, and finished on 300 nodes. The write-up is published by Onehouse. | | Published prices and cost results | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus (Spark Memory Optimized and autoscale for Spark entries, pay-as-you-go). In Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling. Autoscale Billing for Spark is an opt-in pay-as-you-go model that runs Spark on dedicated serverless resources at standard PAYG CU-hour rates; reserved capacity discounts do not apply and a base capacity is still required. | Onehouse bills in Onehouse Compute Units (OCU) on a bring-your-own-cloud model: 1 OCU is a 4-core instance for 1 hour in your account, plus support costs; no separate volume or storage fee, and the customer pays its own AWS or GCP infrastructure bill. The docs page does not state a dollar rate per OCU. Vendor claims at the Quanton launch: 2-3x better price-performance for ETL, and a guaranteed greater-than-50% cost reduction for existing Hudi users moving Spark jobs from AWS EMR or open-source Spark. These are guarantees stated by the vendor, not results from an independent party. | | Code changes and migration | The Fabric native execution engine, based on Apache Gluten and Velox, needs no code changes, falls back to JVM Spark for unsupported operators, and supports Parquet and Delta. | Onehouse docs say its Jobs are fully compatible with Apache Spark and that EMR code can be reused unless it uses AWS-specific features. Migration work listed: map EMR steps to Onehouse Jobs created through its API or Airflow operators (CREATE CLUSTER, CREATE JOB, RUN JOB, DELETE CLUSTER), and configure Glue catalog sync or contact support to make Glue the primary catalog. At the May 20, 2025 launch Onehouse said users can run existing Spark and SQL jobs on new cluster types and that existing Hudi/Spark jobs can be pointed at Onehouse 'with no code changes'. The Quanton Kubernetes Operator is described as a 'drop-in replacement' for an existing Spark Kubernetes operator. | | Tuning and table upkeep | Starter pools start in 5 to 10 seconds with no setup; custom Spark pools let you tune node size and scaling, and native execution can be toggled per application with a Spark setting. | Each cluster needs min and max OCU limits. Docs warn that if usage reaches Max OCU the cluster stops scaling up, which can delay processing; Min OCU keeps a cluster warm. The Onehouse docs pages read give no per-job Spark tuning list; Onehouse claims automatic scaling and automated async table services. | | Spark versions and table formats | Fabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1; OneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and vice versa. | Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake. Docs: Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog (AWS Glue and Snowflake Open Catalog are documented); for Delta Lake the user installs Delta Spark as a dependency in the JAR or Python job. In the Nov 12, 2025 Iceberg post Onehouse said Delta acceleration is 'up next'. Apache XTable (formerly OneTable) translates table metadata between Hudi, Delta Lake and Iceberg without being a new format. It is incubating at the Apache Software Foundation; Onehouse says it created Hudi and XTable. | | Where it runs | Fabric workloads run over OneLake, described as built on Azure Data Lake Storage; the region availability page lists Azure regions and no AWS regions, so Spark compute does not run inside AWS. OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; file caching can reduce cross-cloud egress costs. Writes to S3 are not possible through the shortcut. | Onehouse runs EC2 or Google Compute Engine instances inside the customer's AWS or GCP account (Azure not listed in the billing docs). Standard instance types map to Graviton m8g sizes on AWS (oh-general-4 = m8g.xlarge, oh-general-8 = m8g.2xlarge, oh-general-16 = m8g.4xlarge). A project is scoped to one region and account; adding a region needs a support ticket. | | How open its benchmarks are | None found on 10 Nov 2026. | Onehouse open-sourced LakeLoader, a tool that generates incremental update/insert loads for benchmarking write performance. The repository is Apache-2.0 licensed, last pushed 2026-10-30. The Quanton engine itself and the full Iceberg benchmark configurations were not found as a repository in the pages read. | | Compute Bill | Microsoft Fabric (F128 capacity): $46.08 | Onehouse (Quanton, bring-your-own-cloud): no total (see the Compute Bill for why) | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Microsoft Fabric: learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview: https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - Microsoft Fabric: prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption' (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Microsoft Fabric: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-compute: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs/emr-migration: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtim: https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/project-settings/usage-and-billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): xtable.apache.org: https://xtable.apache.org/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/compute/create-cluster: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-worklo: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark: https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): api.github.com/repos/onehouseinc/lake-loader: https://api.github.com/repos/onehouseinc/lake-loader (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/microsoft-fabric-vs-onehouse/ --- # Microsoft Fabric and Open stack on EMR and EKS: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Microsoft Fabric and Open stack on EMR and EKS are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Microsoft Fabric | Open stack on EMR and EKS | | --- | --- | --- | | Kind | SaaS analytics on Azure with Spark | Self-run Spark with open table formats | | Does well | Per Microsoft Learn, the native execution engine (Apache Gluten and Velox) needs no code changes, falls back to JVM Spark for unsupported operators and supports Parquet and Delta (read 2026-11-10). Per Microsoft Learn, OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying it (read 2026-11-10). | Per AWS documentation, EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three formats are available (read 2026-11-10). AWS ran TPC-DS 3 TB on 9 x r5d.4xlarge and reports $4.81 for EMR 7.12 against $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0, with repositories to repeat it. | | Limited | Per Microsoft Learn, Fabric runs over OneLake on Azure and the region page lists Azure regions only, so Spark compute does not run inside AWS. Writes to S3 through a OneLake shortcut are not possible, and the method for Microsoft's native engine benchmark: none found on 2026-11-10. | Per the Apache Iceberg documentation, users run table maintenance themselves: expire snapshots, remove orphan files and compact with rewriteDataFiles as jobs they schedule (read 2026-11-10). A cost comparison of open-stack Spark against managed lakehouses by a party other than AWS or a vendor: none found on 2026-11-10. | | Published speed results | Microsoft reports that on TPC-DS scale factor 1000 with Delta the native engine was up to six times faster than open-source Spark, with about 83% compute cost savings on a fixed-size cluster. | AWS ran TPC-DS 3 TB (104 SparkSQL queries, 3 rounds, 9 x r5d.4xlarge, Hadoop catalog) and reports EMR 7.12 averaged 1,349.62 s versus 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0 and 1,535.62 s for EMR 7.5. AWS notes results are not comparable to official TPC-DS. The run is by the vendor of EMR. | | Published prices and cost results | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus (Spark Memory Optimized and autoscale for Spark entries, pay-as-you-go). In Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling. Autoscale Billing for Spark is an opt-in pay-as-you-go model that runs Spark on dedicated serverless resources at standard PAYG CU-hour rates; reserved capacity discounts do not apply and a base capacity is still required. | EMR on EC2 charges an EMR price in addition to the EC2 price (and EBS), billed per second with a one-minute minimum. AWS's example in us-east-1: c4.2xlarge EMR charge $0.105/hour next to EC2 $0.398/hour; one master and two core nodes for a month total $1,101.57. Spot instances are up to a 90% discount versus On-Demand. EMR on EKS adds an uplift per vCPU-hour and per GB-hour to EKS/EC2 or Fargate costs, plus $0.10 per hour for each EKS cluster. AWS's worked example uses $0.007084 per vCPU-hour and $0.000777875 per GB-hour (100 vCPU and 300 GB for 30 minutes = $0.4709 EMR uplift). The uplift applies to any pod using an EMR runtime image, including with the Spark operator. EMR Serverless is billed per vCPU-hour and GB-hour. AWS's worked example uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour for a job with 100 vCPU and 750 GB for 30 minutes, then 200 more vCPU for 15 minutes, totaling $9.596275. In the same AWS benchmark the cost was $4.81 for EMR 7.12, $5.47 for EMR 7.5 and $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0. Cost formula uses r5d.4xlarge at $1.152/hour plus $0.27/hour EMR plus EBS, On-Demand. | | Code changes and migration | The Fabric native execution engine, based on Apache Gluten and Velox, needs no code changes, falls back to JVM Spark for unsupported operators, and supports Parquet and Delta. | The EMR runtime for Spark provides 'full API compatibility with open source Apache Spark and Apache Iceberg', so existing Spark jobs run as is. Optimized runtimes are used by EMR on EC2, EMR Serverless, EMR on EKS, EMR on Outposts and AWS Glue. Self-run Spark on EKS uses the open-source Spark operator and needs no job rewrite. | | Tuning and table upkeep | Starter pools start in 5 to 10 seconds with no setup; custom Spark pools let you tune node size and scaling, and native execution can be toggled per application with a Spark setting. | Iceberg docs say users must run maintenance: expire snapshots, remove old metadata files and delete orphan files are 'recommended'; compacting data files with the Spark rewriteDataFiles action (for example target-file-size-bytes of 500 MB) and rewriting manifests are 'optional' maintenance, run as jobs you schedule. Hudi merge-on-read tables need compaction to merge log files into base files and bound read latency. Compaction has two steps, scheduling (plan written to the timeline) and execution; the docs describe strategies for choosing which file slices to compact. Delta Lake OSS docs list file management features the user applies: compaction (bin-packing) with OPTIMIZE, auto compaction, data skipping, Z-ordering, multi-part checkpointing, log compactions and optimized write. | | Spark versions and table formats | Fabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1; OneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and vice versa. | Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. All three table formats are available on EMR. EMR on EKS supports the Kubernetes Operator for Apache Spark (Spark operator) as a job submission model with the EMR runtime on your own EKS clusters. The Kubeflow Spark Operator is Apache-2.0 licensed with Helm installation. | | Where it runs | Fabric workloads run over OneLake, described as built on Azure Data Lake Storage; the region availability page lists Azure regions and no AWS regions, so Spark compute does not run inside AWS. OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; file caching can reduce cross-cloud egress costs. Writes to S3 are not possible through the shortcut. | Amazon EMR has regional endpoints in the US (us-east-1, us-east-2, us-west-1, us-west-2) and many other AWS regions per the AWS general reference endpoints table. Spark runs on EC2 instances the user chooses (the AWS benchmark used r5d.4xlarge). | | How open its benchmarks are | None found on 10 Nov 2026. | AWS publishes the setup in GitHub repositories (aws-samples/emr-on-eks-benchmark branch tpcds-v2.13_iceberg, and aws-samples/emr-spark-benchmark) with build instructions, so the TPC-DS Iceberg benchmark can be repeated. | | Compute Bill | Microsoft Fabric (F128 capacity): $46.08 | Open stack: Spark on plain EC2: $20.16 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Microsoft Fabric: learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview: https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - Microsoft Fabric: prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption' (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Microsoft Fabric: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-compute: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operato: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: iceberg.apache.org/docs/latest/maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: hudi.apache.org/docs/compaction: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.delta.io/latest/optimizations-oss.html: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/general/latest/gr/emr.html: https://docs.aws.amazon.com/general/latest/gr/emr.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/microsoft-fabric-vs-open-stack-emr-eks/ --- # Microsoft Fabric and Snowflake: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Microsoft Fabric (SaaS analytics on Azure with Spark) and Snowflake (Cloud data platform with PySpark support) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Microsoft Fabric (F128 capacity): $46.08; Snowflake (XL standard warehouse): $96.00. ## The same rows for both | Row | Microsoft Fabric | Snowflake | | --- | --- | --- | | Kind | SaaS analytics on Azure with Spark | Cloud data platform with PySpark support | | Does well | Per Microsoft Learn, the native execution engine (Apache Gluten and Velox) needs no code changes, falls back to JVM Spark for unsupported operators and supports Parquet and Delta (read 2026-11-10). Per Microsoft Learn, OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying it (read 2026-11-10). | Per Snowflake documentation, Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect, and batch jobs are submitted with the Snowpark Submit CLI (read 2026-11-10). Per Snowflake documentation, Iceberg tables can keep data in customer-managed S3, GCS or Azure storage through external volumes and a catalog integration such as AWS Glue. | | Limited | Per Microsoft Learn, Fabric runs over OneLake on Azure and the region page lists Azure regions only, so Spark compute does not run inside AWS. Writes to S3 through a OneLake shortcut are not possible, and the method for Microsoft's native engine benchmark: none found on 2026-11-10. | Per Snowflake documentation, Snowpark Connect supports Apache Spark 3.5 only, and DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported (read 2026-11-10). On-demand credit prices on AWS US East (Northern Virginia) are $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical) per the Snowflake Service Consumption Table effective 2026-11-09 (read 2026-11-10); the Snowflake pricing page itself renders by script. | | Published speed results | Microsoft reports that on TPC-DS scale factor 1000 with Delta the native engine was up to six times faster than open-source Spark, with about 83% compute cost savings on a fixed-size cluster. | None found on 10 Nov 2026. | | Published prices and cost results | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus (Spark Memory Optimized and autoscale for Spark entries, pay-as-you-go). In Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling. Autoscale Billing for Spark is an opt-in pay-as-you-go model that runs Spark on dedicated serverless resources at standard PAYG CU-hour rates; reserved capacity discounts do not apply and a base capacity is still required. | Warehouses bill credits per second with a 60-second minimum on each start; each size step roughly doubles credits per hour (Small is 2 credits per hour, Medium is 4). Snowflake's Service Consumption Table (effective 2026-11-09) lists on-demand credits on AWS US East at $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical). | | Code changes and migration | The Fabric native execution engine, based on Apache Gluten and Velox, needs no code changes, falls back to JVM Spark for unsupported operators, and supports Parquet and Delta. | Snowpark Connect for Apache Spark runs existing PySpark DataFrame code against the Snowflake engine over the Spark Connect protocol; there is no Spark cluster, all workloads run on a Snowflake warehouse. Batch jobs are submitted with the Snowpark Submit CLI. Because Snowflake rather than Spark is the engine, there are semantic differences: DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported, and ByteType, ShortType and IntegerType are returned as LongType. The docs list unsupported APIs for DataFrame and Dataset. Docs say the Spark DataFrame and Spark SQL code can be reused with minimal migration or changes; the compatibility notes say Spark Connect defers plan resolution, which changes behavior versus Spark Classic. | | Tuning and table upkeep | Starter pools start in 5 to 10 seconds with no setup; custom Spark pools let you tune node size and scaling, and native execution can be toggled per application with a Spark setting. | Docs provide a separate Optimizing Snowpark Connect for Spark workloads section and a Troubleshooting section covering known limitations, so some job-level adjustment is expected after migration. | | Spark versions and table formats | Fabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1; OneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and vice versa. | Snowpark Connect supports Apache Spark 3.5 only; workloads written for 3.4 and below or 4.0 and above are not supported and may produce protocol errors or missing API failures. Snowflake supports Iceberg tables (Parquet) with data in customer-managed S3, GCS or Azure storage via external volumes, an external catalog integration such as AWS Glue, and Delta files read through Delta Direct; Hudi was not listed on the page. | | Where it runs | Fabric workloads run over OneLake, described as built on Azure Data Lake Storage; the region availability page lists Azure regions and no AWS regions, so Spark compute does not run inside AWS. OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; file caching can reduce cross-cloud egress costs. Writes to S3 are not possible through the shortcut. | Snowflake runs on AWS, Azure and Google Cloud; AWS regions include us-east-1, us-west-2, eu-west-1 and ca-central-1. Each account is hosted in a single region. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | Microsoft Fabric (F128 capacity): $46.08 | Snowflake (XL standard warehouse): $96.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Microsoft Fabric: learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview: https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - Microsoft Fabric: prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption' (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Microsoft Fabric: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-compute: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/cost-understanding-compute: https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/tables-iceberg: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/intro-regions.md: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/microsoft-fabric-vs-snowflake/ --- # Microsoft Fabric and Starburst: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Microsoft Fabric (SaaS analytics on Azure with Spark) and Starburst (Managed Trino service for Iceberg) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Microsoft Fabric (F128 capacity): $46.08; Starburst Galaxy (Pro tier): $60.00. ## The same rows for both | Row | Microsoft Fabric | Starburst | | --- | --- | --- | | Kind | SaaS analytics on Azure with Spark | Managed Trino service for Iceberg | | Does well | Per Microsoft Learn, the native execution engine (Apache Gluten and Velox) needs no code changes, falls back to JVM Spark for unsupported operators and supports Parquet and Delta (read 2026-11-10). Per Microsoft Learn, OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying it (read 2026-11-10). | Per Starburst documentation, Galaxy creates and writes Iceberg tables (Parquet, ORC or Avro) and runs maintenance with ALTER TABLE ... EXECUTE optimize, expire_snapshots and remove_orphan_files (read 2026-11-10). Starburst publishes credit prices by AWS region, for example $0.50 (Pro) and $0.75 (Enterprise) in US East (N. Virginia) and $0.60 and $0.90 in Frankfurt (read 2026-11-10). | | Limited | Per Microsoft Learn, Fabric runs over OneLake on Azure and the region page lists Azure regions only, so Spark compute does not run inside AWS. Writes to S3 through a OneLake shortcut are not possible, and the method for Microsoft's native engine benchmark: none found on 2026-11-10. | Per Starburst documentation, Galaxy runs SQL on Trino, not Spark, so DataFrame code, UDFs and PySpark would be rewritten as Trino SQL or kept on Spark (read 2026-11-10). Per Starburst documentation, Hudi tables are read-only in Galaxy: it cannot create new Hudi tables or write to them. | | Published speed results | Microsoft reports that on TPC-DS scale factor 1000 with Delta the native engine was up to six times faster than open-source Spark, with about 83% compute cost savings on a fixed-size cluster. | Starburst reported GPU-accelerated Trino results (its proprietary distribution, not open-source Trino): TPC-H up to 6x on accelerated queries and ClickBench 3.4x overall, single-node development environment. This is a SQL query test, not Spark batch work. | | Published prices and cost results | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus (Spark Memory Optimized and autoscale for Spark entries, pay-as-you-go). In Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling. Autoscale Billing for Spark is an opt-in pay-as-you-go model that runs Spark on dedicated serverless resources at standard PAYG CU-hour rates; reserved capacity discounts do not apply and a base capacity is still required. | Galaxy credit prices start at $0.50 (Pro), $0.75 (Enterprise) and $1.00 (Mission Critical) per credit. A Free tier allows up to 3 clusters. Pricing depends on compute usage, tier and the cloud provider and region. Annual commitments may get discounts. On-demand list prices per credit (Pro / Enterprise / Mission Critical) on AWS: US East N. Virginia, US East Ohio and US West Oregon $0.50 / $0.75 / $1.00; Europe Ireland $0.56 / $0.84 / $1.12; Europe Frankfurt $0.60 / $0.90 / $1.20; Asia Pacific Tokyo $0.64 / $0.97 / $1.29; Brazil (Sao Paulo) $0.80 / $1.19 / $1.59. Cluster compute credit consumption: 1 worker 6 credits per hour, 2 workers 12, 4 workers 24, 8 workers 48, 16 workers 96. Managed ingest: file ingest 0.60 credits per hour; table maintenance and data transforms 0.30 credits per hour. Billing tracks cluster compute hours, data ingestion compute and private link data transfer. | | Code changes and migration | The Fabric native execution engine, based on Apache Gluten and Velox, needs no code changes, falls back to JVM Spark for unsupported operators, and supports Parquet and Delta. | Galaxy is 'a fully-managed data lake analytics platform built in addition to Trino'. Work runs as SQL on Trino, not Spark, so Spark jobs (DataFrame code, UDFs, PySpark) would have to be rewritten as Trino SQL, or kept on Spark against the same Iceberg tables. The Galaxy docs read contain no Spark execution mode. | | Tuning and table upkeep | Starter pools start in 5 to 10 seconds with no setup; custom Spark pools let you tune node size and scaling, and native execution can be toggled per application with a Spark setting. | Iceberg table maintenance in Galaxy is run with ALTER TABLE ... EXECUTE commands: optimize (compaction; files under file_size_threshold, default 100MB, are merged), drop_extended_stats, expire_snapshots, remove_orphan_files. Live table maintenance for ingest tables is a separate Galaxy page. For batch-style work Galaxy offers a fault tolerant execution mode that spools intermediate data and retries failed parts of a query. Docs say 'No other configuration is required' to enable it, but not to use it if most queries run under about a minute, and that it can be slightly slower than normal execution. FTE and Accelerated modes are not available on the Free cluster size and not all catalogs support FTE. | | Spark versions and table formats | Fabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1; OneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and vice versa. | Galaxy's Great Lakes connectivity lists Iceberg, Delta Lake, Hive and Hudi table format pages. Hudi tables are read-only: 'Galaxy cannot create new Hudi tables or write to them.' Iceberg tables can be created and written; the Iceberg data file format can be Parquet, ORC or Avro. | | Where it runs | Fabric workloads run over OneLake, described as built on Azure Data Lake Storage; the region availability page lists Azure regions and no AWS regions, so Spark compute does not run inside AWS. OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; file caching can reduce cross-cloud egress costs. Writes to S3 are not possible through the shortcut. | Galaxy prices are listed for AWS, Azure and GCP regions. AWS regions priced include us-east-1, us-east-2, us-west-1, us-west-2, ca-central, Ireland, London, Frankfurt, Zurich, Singapore, Sydney, Tokyo, Cape Town, UAE, Tel Aviv and Sao Paulo. A cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | Microsoft Fabric (F128 capacity): $46.08 | Starburst Galaxy (Pro tier): $60.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Microsoft Fabric: learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview: https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - Microsoft Fabric: prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption' (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Microsoft Fabric: azure.microsoft.com/en-us/pricing/details/microsoft-fabric: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Microsoft Fabric: learn.microsoft.com/en-us/fabric/data-engineering/spark-compute: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing: https://www.starburst.io/pricing/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing/pricing-details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/index.html: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/optimization-perform: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/microsoft-fabric-vs-starburst/ --- # Onehouse and Open stack on EMR and EKS: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Onehouse and Open stack on EMR and EKS are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Onehouse | Open stack on EMR and EKS | | --- | --- | --- | | Kind | Managed lakehouse in your cloud account | Self-run Spark with open table formats | | Does well | Per Onehouse documentation, its jobs are fully compatible with Apache Spark and EMR code can be reused unless it uses AWS-specific features (read 2026-11-10). Onehouse published the setup for its 10 TB TPC-DS and 1 TB LakeLoader comparison against EMR 7.12, and open-sourced LakeLoader under Apache-2.0 (read 2026-11-10). | Per AWS documentation, EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three formats are available (read 2026-11-10). AWS ran TPC-DS 3 TB on 9 x r5d.4xlarge and reports $4.81 for EMR 7.12 against $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0, with repositories to repeat it. | | Limited | Per Onehouse documentation, billing is in OCUs on a bring-your-own-cloud model, and no dollar rate per OCU was found on 2026-11-10. Onehouse's results against EMR 7.12 are its own and shown as charts; a confirmation by another party: none found on 2026-11-10. | Per the Apache Iceberg documentation, users run table maintenance themselves: expire snapshots, remove orphan files and compact with rewriteDataFiles as jobs they schedule (read 2026-11-10). A cost comparison of open-stack Spark against managed lakehouses by a party other than AWS or a vendor: none found on 2026-11-10. | | Published speed results | Onehouse benchmarked Quanton against open-source Spark on Iceberg: TPC-DS up to about 3.1x, TPCx-BB 2x, TPC-DI (1 TB) 2x with 3-4x on MERGE INTO, and LakeLoader 4.1x for writes. The baseline is open-source Spark, not EMR. Onehouse reran its benchmarks against EMR 7.12 after AWS announced it: it reports EMR improved 32% over earlier versions and that Onehouse 'still comes out with a 2.5x better price/performance advantage'. Setup: 10 TB TPC-DS for read/transform, 1 TB LakeLoader merges for load, latest general-purpose Graviton instances. Results are shown as charts; Onehouse also reports Hudi and Delta query performance on EMR 7.12 was very similar to Iceberg. A B2B SaaS customer (unnamed) ran its UDF-heavy pipeline on EMR and on Quanton in its own AWS account on m8gd nodes. Quanton was faster at p50, p97 and p99 tenant sizes; on the largest tenant (21 billion rows) EMR filled local disks on 180 nodes and was aborted after 3.7 hours, and finished on 300 nodes. The write-up is published by Onehouse. | AWS ran TPC-DS 3 TB (104 SparkSQL queries, 3 rounds, 9 x r5d.4xlarge, Hadoop catalog) and reports EMR 7.12 averaged 1,349.62 s versus 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0 and 1,535.62 s for EMR 7.5. AWS notes results are not comparable to official TPC-DS. The run is by the vendor of EMR. | | Published prices and cost results | Onehouse bills in Onehouse Compute Units (OCU) on a bring-your-own-cloud model: 1 OCU is a 4-core instance for 1 hour in your account, plus support costs; no separate volume or storage fee, and the customer pays its own AWS or GCP infrastructure bill. The docs page does not state a dollar rate per OCU. Vendor claims at the Quanton launch: 2-3x better price-performance for ETL, and a guaranteed greater-than-50% cost reduction for existing Hudi users moving Spark jobs from AWS EMR or open-source Spark. These are guarantees stated by the vendor, not results from an independent party. | EMR on EC2 charges an EMR price in addition to the EC2 price (and EBS), billed per second with a one-minute minimum. AWS's example in us-east-1: c4.2xlarge EMR charge $0.105/hour next to EC2 $0.398/hour; one master and two core nodes for a month total $1,101.57. Spot instances are up to a 90% discount versus On-Demand. EMR on EKS adds an uplift per vCPU-hour and per GB-hour to EKS/EC2 or Fargate costs, plus $0.10 per hour for each EKS cluster. AWS's worked example uses $0.007084 per vCPU-hour and $0.000777875 per GB-hour (100 vCPU and 300 GB for 30 minutes = $0.4709 EMR uplift). The uplift applies to any pod using an EMR runtime image, including with the Spark operator. EMR Serverless is billed per vCPU-hour and GB-hour. AWS's worked example uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour for a job with 100 vCPU and 750 GB for 30 minutes, then 200 more vCPU for 15 minutes, totaling $9.596275. In the same AWS benchmark the cost was $4.81 for EMR 7.12, $5.47 for EMR 7.5 and $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0. Cost formula uses r5d.4xlarge at $1.152/hour plus $0.27/hour EMR plus EBS, On-Demand. | | Code changes and migration | Onehouse docs say its Jobs are fully compatible with Apache Spark and that EMR code can be reused unless it uses AWS-specific features. Migration work listed: map EMR steps to Onehouse Jobs created through its API or Airflow operators (CREATE CLUSTER, CREATE JOB, RUN JOB, DELETE CLUSTER), and configure Glue catalog sync or contact support to make Glue the primary catalog. At the May 20, 2025 launch Onehouse said users can run existing Spark and SQL jobs on new cluster types and that existing Hudi/Spark jobs can be pointed at Onehouse 'with no code changes'. The Quanton Kubernetes Operator is described as a 'drop-in replacement' for an existing Spark Kubernetes operator. | The EMR runtime for Spark provides 'full API compatibility with open source Apache Spark and Apache Iceberg', so existing Spark jobs run as is. Optimized runtimes are used by EMR on EC2, EMR Serverless, EMR on EKS, EMR on Outposts and AWS Glue. Self-run Spark on EKS uses the open-source Spark operator and needs no job rewrite. | | Tuning and table upkeep | Each cluster needs min and max OCU limits. Docs warn that if usage reaches Max OCU the cluster stops scaling up, which can delay processing; Min OCU keeps a cluster warm. The Onehouse docs pages read give no per-job Spark tuning list; Onehouse claims automatic scaling and automated async table services. | Iceberg docs say users must run maintenance: expire snapshots, remove old metadata files and delete orphan files are 'recommended'; compacting data files with the Spark rewriteDataFiles action (for example target-file-size-bytes of 500 MB) and rewriting manifests are 'optional' maintenance, run as jobs you schedule. Hudi merge-on-read tables need compaction to merge log files into base files and bound read latency. Compaction has two steps, scheduling (plan written to the timeline) and execution; the docs describe strategies for choosing which file slices to compact. Delta Lake OSS docs list file management features the user applies: compaction (bin-packing) with OPTIMIZE, auto compaction, data skipping, Z-ordering, multi-part checkpointing, log compactions and optimized write. | | Spark versions and table formats | Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake. Docs: Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog (AWS Glue and Snowflake Open Catalog are documented); for Delta Lake the user installs Delta Spark as a dependency in the JAR or Python job. In the Nov 12, 2025 Iceberg post Onehouse said Delta acceleration is 'up next'. Apache XTable (formerly OneTable) translates table metadata between Hudi, Delta Lake and Iceberg without being a new format. It is incubating at the Apache Software Foundation; Onehouse says it created Hudi and XTable. | Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. All three table formats are available on EMR. EMR on EKS supports the Kubernetes Operator for Apache Spark (Spark operator) as a job submission model with the EMR runtime on your own EKS clusters. The Kubeflow Spark Operator is Apache-2.0 licensed with Helm installation. | | Where it runs | Onehouse runs EC2 or Google Compute Engine instances inside the customer's AWS or GCP account (Azure not listed in the billing docs). Standard instance types map to Graviton m8g sizes on AWS (oh-general-4 = m8g.xlarge, oh-general-8 = m8g.2xlarge, oh-general-16 = m8g.4xlarge). A project is scoped to one region and account; adding a region needs a support ticket. | Amazon EMR has regional endpoints in the US (us-east-1, us-east-2, us-west-1, us-west-2) and many other AWS regions per the AWS general reference endpoints table. Spark runs on EC2 instances the user chooses (the AWS benchmark used r5d.4xlarge). | | How open its benchmarks are | Onehouse open-sourced LakeLoader, a tool that generates incremental update/insert loads for benchmarking write performance. The repository is Apache-2.0 licensed, last pushed 2026-10-30. The Quanton engine itself and the full Iceberg benchmark configurations were not found as a repository in the pages read. | AWS publishes the setup in GitHub repositories (aws-samples/emr-on-eks-benchmark branch tpcds-v2.13_iceberg, and aws-samples/emr-spark-benchmark) with build instructions, so the TPC-DS Iceberg benchmark can be repeated. | | Compute Bill | Onehouse (Quanton, bring-your-own-cloud): no total (see the Compute Bill for why) | Open stack: Spark on plain EC2: $20.16 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs/emr-migration: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtim: https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/project-settings/usage-and-billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): xtable.apache.org: https://xtable.apache.org/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/compute/create-cluster: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-worklo: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark: https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): api.github.com/repos/onehouseinc/lake-loader: https://api.github.com/repos/onehouseinc/lake-loader (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operato: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: iceberg.apache.org/docs/latest/maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: hudi.apache.org/docs/compaction: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.delta.io/latest/optimizations-oss.html: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/general/latest/gr/emr.html: https://docs.aws.amazon.com/general/latest/gr/emr.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/onehouse-vs-open-stack-emr-eks/ --- # Onehouse and Snowflake: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Onehouse (Managed lakehouse in your cloud account) and Snowflake (Cloud data platform with PySpark support) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Onehouse (Quanton, bring-your-own-cloud): no total (see the Compute Bill for why); Snowflake (XL standard warehouse): $96.00. ## The same rows for both | Row | Onehouse | Snowflake | | --- | --- | --- | | Kind | Managed lakehouse in your cloud account | Cloud data platform with PySpark support | | Does well | Per Onehouse documentation, its jobs are fully compatible with Apache Spark and EMR code can be reused unless it uses AWS-specific features (read 2026-11-10). Onehouse published the setup for its 10 TB TPC-DS and 1 TB LakeLoader comparison against EMR 7.12, and open-sourced LakeLoader under Apache-2.0 (read 2026-11-10). | Per Snowflake documentation, Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect, and batch jobs are submitted with the Snowpark Submit CLI (read 2026-11-10). Per Snowflake documentation, Iceberg tables can keep data in customer-managed S3, GCS or Azure storage through external volumes and a catalog integration such as AWS Glue. | | Limited | Per Onehouse documentation, billing is in OCUs on a bring-your-own-cloud model, and no dollar rate per OCU was found on 2026-11-10. Onehouse's results against EMR 7.12 are its own and shown as charts; a confirmation by another party: none found on 2026-11-10. | Per Snowflake documentation, Snowpark Connect supports Apache Spark 3.5 only, and DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported (read 2026-11-10). On-demand credit prices on AWS US East (Northern Virginia) are $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical) per the Snowflake Service Consumption Table effective 2026-11-09 (read 2026-11-10); the Snowflake pricing page itself renders by script. | | Published speed results | Onehouse benchmarked Quanton against open-source Spark on Iceberg: TPC-DS up to about 3.1x, TPCx-BB 2x, TPC-DI (1 TB) 2x with 3-4x on MERGE INTO, and LakeLoader 4.1x for writes. The baseline is open-source Spark, not EMR. Onehouse reran its benchmarks against EMR 7.12 after AWS announced it: it reports EMR improved 32% over earlier versions and that Onehouse 'still comes out with a 2.5x better price/performance advantage'. Setup: 10 TB TPC-DS for read/transform, 1 TB LakeLoader merges for load, latest general-purpose Graviton instances. Results are shown as charts; Onehouse also reports Hudi and Delta query performance on EMR 7.12 was very similar to Iceberg. A B2B SaaS customer (unnamed) ran its UDF-heavy pipeline on EMR and on Quanton in its own AWS account on m8gd nodes. Quanton was faster at p50, p97 and p99 tenant sizes; on the largest tenant (21 billion rows) EMR filled local disks on 180 nodes and was aborted after 3.7 hours, and finished on 300 nodes. The write-up is published by Onehouse. | None found on 10 Nov 2026. | | Published prices and cost results | Onehouse bills in Onehouse Compute Units (OCU) on a bring-your-own-cloud model: 1 OCU is a 4-core instance for 1 hour in your account, plus support costs; no separate volume or storage fee, and the customer pays its own AWS or GCP infrastructure bill. The docs page does not state a dollar rate per OCU. Vendor claims at the Quanton launch: 2-3x better price-performance for ETL, and a guaranteed greater-than-50% cost reduction for existing Hudi users moving Spark jobs from AWS EMR or open-source Spark. These are guarantees stated by the vendor, not results from an independent party. | Warehouses bill credits per second with a 60-second minimum on each start; each size step roughly doubles credits per hour (Small is 2 credits per hour, Medium is 4). Snowflake's Service Consumption Table (effective 2026-11-09) lists on-demand credits on AWS US East at $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical). | | Code changes and migration | Onehouse docs say its Jobs are fully compatible with Apache Spark and that EMR code can be reused unless it uses AWS-specific features. Migration work listed: map EMR steps to Onehouse Jobs created through its API or Airflow operators (CREATE CLUSTER, CREATE JOB, RUN JOB, DELETE CLUSTER), and configure Glue catalog sync or contact support to make Glue the primary catalog. At the May 20, 2025 launch Onehouse said users can run existing Spark and SQL jobs on new cluster types and that existing Hudi/Spark jobs can be pointed at Onehouse 'with no code changes'. The Quanton Kubernetes Operator is described as a 'drop-in replacement' for an existing Spark Kubernetes operator. | Snowpark Connect for Apache Spark runs existing PySpark DataFrame code against the Snowflake engine over the Spark Connect protocol; there is no Spark cluster, all workloads run on a Snowflake warehouse. Batch jobs are submitted with the Snowpark Submit CLI. Because Snowflake rather than Spark is the engine, there are semantic differences: DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported, and ByteType, ShortType and IntegerType are returned as LongType. The docs list unsupported APIs for DataFrame and Dataset. Docs say the Spark DataFrame and Spark SQL code can be reused with minimal migration or changes; the compatibility notes say Spark Connect defers plan resolution, which changes behavior versus Spark Classic. | | Tuning and table upkeep | Each cluster needs min and max OCU limits. Docs warn that if usage reaches Max OCU the cluster stops scaling up, which can delay processing; Min OCU keeps a cluster warm. The Onehouse docs pages read give no per-job Spark tuning list; Onehouse claims automatic scaling and automated async table services. | Docs provide a separate Optimizing Snowpark Connect for Spark workloads section and a Troubleshooting section covering known limitations, so some job-level adjustment is expected after migration. | | Spark versions and table formats | Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake. Docs: Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog (AWS Glue and Snowflake Open Catalog are documented); for Delta Lake the user installs Delta Spark as a dependency in the JAR or Python job. In the Nov 12, 2025 Iceberg post Onehouse said Delta acceleration is 'up next'. Apache XTable (formerly OneTable) translates table metadata between Hudi, Delta Lake and Iceberg without being a new format. It is incubating at the Apache Software Foundation; Onehouse says it created Hudi and XTable. | Snowpark Connect supports Apache Spark 3.5 only; workloads written for 3.4 and below or 4.0 and above are not supported and may produce protocol errors or missing API failures. Snowflake supports Iceberg tables (Parquet) with data in customer-managed S3, GCS or Azure storage via external volumes, an external catalog integration such as AWS Glue, and Delta files read through Delta Direct; Hudi was not listed on the page. | | Where it runs | Onehouse runs EC2 or Google Compute Engine instances inside the customer's AWS or GCP account (Azure not listed in the billing docs). Standard instance types map to Graviton m8g sizes on AWS (oh-general-4 = m8g.xlarge, oh-general-8 = m8g.2xlarge, oh-general-16 = m8g.4xlarge). A project is scoped to one region and account; adding a region needs a support ticket. | Snowflake runs on AWS, Azure and Google Cloud; AWS regions include us-east-1, us-west-2, eu-west-1 and ca-central-1. Each account is hosted in a single region. | | How open its benchmarks are | Onehouse open-sourced LakeLoader, a tool that generates incremental update/insert loads for benchmarking write performance. The repository is Apache-2.0 licensed, last pushed 2026-10-30. The Quanton engine itself and the full Iceberg benchmark configurations were not found as a repository in the pages read. | None found on 10 Nov 2026. | | Compute Bill | Onehouse (Quanton, bring-your-own-cloud): no total (see the Compute Bill for why) | Snowflake (XL standard warehouse): $96.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs/emr-migration: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtim: https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/project-settings/usage-and-billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): xtable.apache.org: https://xtable.apache.org/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/compute/create-cluster: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-worklo: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark: https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): api.github.com/repos/onehouseinc/lake-loader: https://api.github.com/repos/onehouseinc/lake-loader (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/cost-understanding-compute: https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/tables-iceberg: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/intro-regions.md: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/onehouse-vs-snowflake/ --- # Onehouse and Starburst: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Onehouse (Managed lakehouse in your cloud account) and Starburst (Managed Trino service for Iceberg) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Onehouse (Quanton, bring-your-own-cloud): no total (see the Compute Bill for why); Starburst Galaxy (Pro tier): $60.00. ## The same rows for both | Row | Onehouse | Starburst | | --- | --- | --- | | Kind | Managed lakehouse in your cloud account | Managed Trino service for Iceberg | | Does well | Per Onehouse documentation, its jobs are fully compatible with Apache Spark and EMR code can be reused unless it uses AWS-specific features (read 2026-11-10). Onehouse published the setup for its 10 TB TPC-DS and 1 TB LakeLoader comparison against EMR 7.12, and open-sourced LakeLoader under Apache-2.0 (read 2026-11-10). | Per Starburst documentation, Galaxy creates and writes Iceberg tables (Parquet, ORC or Avro) and runs maintenance with ALTER TABLE ... EXECUTE optimize, expire_snapshots and remove_orphan_files (read 2026-11-10). Starburst publishes credit prices by AWS region, for example $0.50 (Pro) and $0.75 (Enterprise) in US East (N. Virginia) and $0.60 and $0.90 in Frankfurt (read 2026-11-10). | | Limited | Per Onehouse documentation, billing is in OCUs on a bring-your-own-cloud model, and no dollar rate per OCU was found on 2026-11-10. Onehouse's results against EMR 7.12 are its own and shown as charts; a confirmation by another party: none found on 2026-11-10. | Per Starburst documentation, Galaxy runs SQL on Trino, not Spark, so DataFrame code, UDFs and PySpark would be rewritten as Trino SQL or kept on Spark (read 2026-11-10). Per Starburst documentation, Hudi tables are read-only in Galaxy: it cannot create new Hudi tables or write to them. | | Published speed results | Onehouse benchmarked Quanton against open-source Spark on Iceberg: TPC-DS up to about 3.1x, TPCx-BB 2x, TPC-DI (1 TB) 2x with 3-4x on MERGE INTO, and LakeLoader 4.1x for writes. The baseline is open-source Spark, not EMR. Onehouse reran its benchmarks against EMR 7.12 after AWS announced it: it reports EMR improved 32% over earlier versions and that Onehouse 'still comes out with a 2.5x better price/performance advantage'. Setup: 10 TB TPC-DS for read/transform, 1 TB LakeLoader merges for load, latest general-purpose Graviton instances. Results are shown as charts; Onehouse also reports Hudi and Delta query performance on EMR 7.12 was very similar to Iceberg. A B2B SaaS customer (unnamed) ran its UDF-heavy pipeline on EMR and on Quanton in its own AWS account on m8gd nodes. Quanton was faster at p50, p97 and p99 tenant sizes; on the largest tenant (21 billion rows) EMR filled local disks on 180 nodes and was aborted after 3.7 hours, and finished on 300 nodes. The write-up is published by Onehouse. | Starburst reported GPU-accelerated Trino results (its proprietary distribution, not open-source Trino): TPC-H up to 6x on accelerated queries and ClickBench 3.4x overall, single-node development environment. This is a SQL query test, not Spark batch work. | | Published prices and cost results | Onehouse bills in Onehouse Compute Units (OCU) on a bring-your-own-cloud model: 1 OCU is a 4-core instance for 1 hour in your account, plus support costs; no separate volume or storage fee, and the customer pays its own AWS or GCP infrastructure bill. The docs page does not state a dollar rate per OCU. Vendor claims at the Quanton launch: 2-3x better price-performance for ETL, and a guaranteed greater-than-50% cost reduction for existing Hudi users moving Spark jobs from AWS EMR or open-source Spark. These are guarantees stated by the vendor, not results from an independent party. | Galaxy credit prices start at $0.50 (Pro), $0.75 (Enterprise) and $1.00 (Mission Critical) per credit. A Free tier allows up to 3 clusters. Pricing depends on compute usage, tier and the cloud provider and region. Annual commitments may get discounts. On-demand list prices per credit (Pro / Enterprise / Mission Critical) on AWS: US East N. Virginia, US East Ohio and US West Oregon $0.50 / $0.75 / $1.00; Europe Ireland $0.56 / $0.84 / $1.12; Europe Frankfurt $0.60 / $0.90 / $1.20; Asia Pacific Tokyo $0.64 / $0.97 / $1.29; Brazil (Sao Paulo) $0.80 / $1.19 / $1.59. Cluster compute credit consumption: 1 worker 6 credits per hour, 2 workers 12, 4 workers 24, 8 workers 48, 16 workers 96. Managed ingest: file ingest 0.60 credits per hour; table maintenance and data transforms 0.30 credits per hour. Billing tracks cluster compute hours, data ingestion compute and private link data transfer. | | Code changes and migration | Onehouse docs say its Jobs are fully compatible with Apache Spark and that EMR code can be reused unless it uses AWS-specific features. Migration work listed: map EMR steps to Onehouse Jobs created through its API or Airflow operators (CREATE CLUSTER, CREATE JOB, RUN JOB, DELETE CLUSTER), and configure Glue catalog sync or contact support to make Glue the primary catalog. At the May 20, 2025 launch Onehouse said users can run existing Spark and SQL jobs on new cluster types and that existing Hudi/Spark jobs can be pointed at Onehouse 'with no code changes'. The Quanton Kubernetes Operator is described as a 'drop-in replacement' for an existing Spark Kubernetes operator. | Galaxy is 'a fully-managed data lake analytics platform built in addition to Trino'. Work runs as SQL on Trino, not Spark, so Spark jobs (DataFrame code, UDFs, PySpark) would have to be rewritten as Trino SQL, or kept on Spark against the same Iceberg tables. The Galaxy docs read contain no Spark execution mode. | | Tuning and table upkeep | Each cluster needs min and max OCU limits. Docs warn that if usage reaches Max OCU the cluster stops scaling up, which can delay processing; Min OCU keeps a cluster warm. The Onehouse docs pages read give no per-job Spark tuning list; Onehouse claims automatic scaling and automated async table services. | Iceberg table maintenance in Galaxy is run with ALTER TABLE ... EXECUTE commands: optimize (compaction; files under file_size_threshold, default 100MB, are merged), drop_extended_stats, expire_snapshots, remove_orphan_files. Live table maintenance for ingest tables is a separate Galaxy page. For batch-style work Galaxy offers a fault tolerant execution mode that spools intermediate data and retries failed parts of a query. Docs say 'No other configuration is required' to enable it, but not to use it if most queries run under about a minute, and that it can be slightly slower than normal execution. FTE and Accelerated modes are not available on the Free cluster size and not all catalogs support FTE. | | Spark versions and table formats | Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake. Docs: Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog (AWS Glue and Snowflake Open Catalog are documented); for Delta Lake the user installs Delta Spark as a dependency in the JAR or Python job. In the Nov 12, 2025 Iceberg post Onehouse said Delta acceleration is 'up next'. Apache XTable (formerly OneTable) translates table metadata between Hudi, Delta Lake and Iceberg without being a new format. It is incubating at the Apache Software Foundation; Onehouse says it created Hudi and XTable. | Galaxy's Great Lakes connectivity lists Iceberg, Delta Lake, Hive and Hudi table format pages. Hudi tables are read-only: 'Galaxy cannot create new Hudi tables or write to them.' Iceberg tables can be created and written; the Iceberg data file format can be Parquet, ORC or Avro. | | Where it runs | Onehouse runs EC2 or Google Compute Engine instances inside the customer's AWS or GCP account (Azure not listed in the billing docs). Standard instance types map to Graviton m8g sizes on AWS (oh-general-4 = m8g.xlarge, oh-general-8 = m8g.2xlarge, oh-general-16 = m8g.4xlarge). A project is scoped to one region and account; adding a region needs a support ticket. | Galaxy prices are listed for AWS, Azure and GCP regions. AWS regions priced include us-east-1, us-east-2, us-west-1, us-west-2, ca-central, Ireland, London, Frankfurt, Zurich, Singapore, Sydney, Tokyo, Cape Town, UAE, Tel Aviv and Sao Paulo. A cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature. | | How open its benchmarks are | Onehouse open-sourced LakeLoader, a tool that generates incremental update/insert loads for benchmarking write performance. The repository is Apache-2.0 licensed, last pushed 2026-10-30. The Quanton engine itself and the full Iceberg benchmark configurations were not found as a repository in the pages read. | None found on 10 Nov 2026. | | Compute Bill | Onehouse (Quanton, bring-your-own-cloud): no total (see the Compute Bill for why) | Starburst Galaxy (Pro tier): $60.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs/emr-migration: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtim: https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/project-settings/usage-and-billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/jobs: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): xtable.apache.org: https://xtable.apache.org/ (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): docs.onehouse.ai/product/compute/create-cluster: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-worklo: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark: https://www.onehouse.ai/blog/onehouse-quanton-vs-the-latest-aws-emr-for-apache-spark-workloads (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - Onehouse (Onehouse Compute Runtime with the Quanton engine): api.github.com/repos/onehouseinc/lake-loader: https://api.github.com/repos/onehouseinc/lake-loader (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing: https://www.starburst.io/pricing/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing/pricing-details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/index.html: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/optimization-perform: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/onehouse-vs-starburst/ --- # Open stack on EMR and EKS and Snowflake: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Open stack on EMR and EKS and Snowflake are set out here in the same rows: published speed and cost results, code changes and migration, table upkeep, Spark versions and table formats, and where each runs. Each cell carries its source and read date, and the Compute Bill line for each is in the table. ## The same rows for both | Row | Open stack on EMR and EKS | Snowflake | | --- | --- | --- | | Kind | Self-run Spark with open table formats | Cloud data platform with PySpark support | | Does well | Per AWS documentation, EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three formats are available (read 2026-11-10). AWS ran TPC-DS 3 TB on 9 x r5d.4xlarge and reports $4.81 for EMR 7.12 against $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0, with repositories to repeat it. | Per Snowflake documentation, Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect, and batch jobs are submitted with the Snowpark Submit CLI (read 2026-11-10). Per Snowflake documentation, Iceberg tables can keep data in customer-managed S3, GCS or Azure storage through external volumes and a catalog integration such as AWS Glue. | | Limited | Per the Apache Iceberg documentation, users run table maintenance themselves: expire snapshots, remove orphan files and compact with rewriteDataFiles as jobs they schedule (read 2026-11-10). A cost comparison of open-stack Spark against managed lakehouses by a party other than AWS or a vendor: none found on 2026-11-10. | Per Snowflake documentation, Snowpark Connect supports Apache Spark 3.5 only, and DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported (read 2026-11-10). On-demand credit prices on AWS US East (Northern Virginia) are $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical) per the Snowflake Service Consumption Table effective 2026-11-09 (read 2026-11-10); the Snowflake pricing page itself renders by script. | | Published speed results | AWS ran TPC-DS 3 TB (104 SparkSQL queries, 3 rounds, 9 x r5d.4xlarge, Hadoop catalog) and reports EMR 7.12 averaged 1,349.62 s versus 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0 and 1,535.62 s for EMR 7.5. AWS notes results are not comparable to official TPC-DS. The run is by the vendor of EMR. | None found on 10 Nov 2026. | | Published prices and cost results | EMR on EC2 charges an EMR price in addition to the EC2 price (and EBS), billed per second with a one-minute minimum. AWS's example in us-east-1: c4.2xlarge EMR charge $0.105/hour next to EC2 $0.398/hour; one master and two core nodes for a month total $1,101.57. Spot instances are up to a 90% discount versus On-Demand. EMR on EKS adds an uplift per vCPU-hour and per GB-hour to EKS/EC2 or Fargate costs, plus $0.10 per hour for each EKS cluster. AWS's worked example uses $0.007084 per vCPU-hour and $0.000777875 per GB-hour (100 vCPU and 300 GB for 30 minutes = $0.4709 EMR uplift). The uplift applies to any pod using an EMR runtime image, including with the Spark operator. EMR Serverless is billed per vCPU-hour and GB-hour. AWS's worked example uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour for a job with 100 vCPU and 750 GB for 30 minutes, then 200 more vCPU for 15 minutes, totaling $9.596275. In the same AWS benchmark the cost was $4.81 for EMR 7.12, $5.47 for EMR 7.5 and $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0. Cost formula uses r5d.4xlarge at $1.152/hour plus $0.27/hour EMR plus EBS, On-Demand. | Warehouses bill credits per second with a 60-second minimum on each start; each size step roughly doubles credits per hour (Small is 2 credits per hour, Medium is 4). Snowflake's Service Consumption Table (effective 2026-11-09) lists on-demand credits on AWS US East at $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical). | | Code changes and migration | The EMR runtime for Spark provides 'full API compatibility with open source Apache Spark and Apache Iceberg', so existing Spark jobs run as is. Optimized runtimes are used by EMR on EC2, EMR Serverless, EMR on EKS, EMR on Outposts and AWS Glue. Self-run Spark on EKS uses the open-source Spark operator and needs no job rewrite. | Snowpark Connect for Apache Spark runs existing PySpark DataFrame code against the Snowflake engine over the Spark Connect protocol; there is no Spark cluster, all workloads run on a Snowflake warehouse. Batch jobs are submitted with the Snowpark Submit CLI. Because Snowflake rather than Spark is the engine, there are semantic differences: DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported, and ByteType, ShortType and IntegerType are returned as LongType. The docs list unsupported APIs for DataFrame and Dataset. Docs say the Spark DataFrame and Spark SQL code can be reused with minimal migration or changes; the compatibility notes say Spark Connect defers plan resolution, which changes behavior versus Spark Classic. | | Tuning and table upkeep | Iceberg docs say users must run maintenance: expire snapshots, remove old metadata files and delete orphan files are 'recommended'; compacting data files with the Spark rewriteDataFiles action (for example target-file-size-bytes of 500 MB) and rewriting manifests are 'optional' maintenance, run as jobs you schedule. Hudi merge-on-read tables need compaction to merge log files into base files and bound read latency. Compaction has two steps, scheduling (plan written to the timeline) and execution; the docs describe strategies for choosing which file slices to compact. Delta Lake OSS docs list file management features the user applies: compaction (bin-packing) with OPTIMIZE, auto compaction, data skipping, Z-ordering, multi-part checkpointing, log compactions and optimized write. | Docs provide a separate Optimizing Snowpark Connect for Spark workloads section and a Troubleshooting section covering known limitations, so some job-level adjustment is expected after migration. | | Spark versions and table formats | Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. All three table formats are available on EMR. EMR on EKS supports the Kubernetes Operator for Apache Spark (Spark operator) as a job submission model with the EMR runtime on your own EKS clusters. The Kubeflow Spark Operator is Apache-2.0 licensed with Helm installation. | Snowpark Connect supports Apache Spark 3.5 only; workloads written for 3.4 and below or 4.0 and above are not supported and may produce protocol errors or missing API failures. Snowflake supports Iceberg tables (Parquet) with data in customer-managed S3, GCS or Azure storage via external volumes, an external catalog integration such as AWS Glue, and Delta files read through Delta Direct; Hudi was not listed on the page. | | Where it runs | Amazon EMR has regional endpoints in the US (us-east-1, us-east-2, us-west-1, us-west-2) and many other AWS regions per the AWS general reference endpoints table. Spark runs on EC2 instances the user chooses (the AWS benchmark used r5d.4xlarge). | Snowflake runs on AWS, Azure and Google Cloud; AWS regions include us-east-1, us-west-2, eu-west-1 and ca-central-1. Each account is hosted in a single region. | | How open its benchmarks are | AWS publishes the setup in GitHub repositories (aws-samples/emr-on-eks-benchmark branch tpcds-v2.13_iceberg, and aws-samples/emr-spark-benchmark) with build instructions, so the TPC-DS Iceberg benchmark can be repeated. | None found on 10 Nov 2026. | | Compute Bill | Open stack: Spark on plain EC2: $20.16 | Snowflake (XL standard warehouse): $96.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operato: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: iceberg.apache.org/docs/latest/maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: hudi.apache.org/docs/compaction: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.delta.io/latest/optimizations-oss.html: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/general/latest/gr/emr.html: https://docs.aws.amazon.com/general/latest/gr/emr.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/cost-understanding-compute: https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/tables-iceberg: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/intro-regions.md: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/open-stack-emr-eks-vs-snowflake/ --- # Open stack on EMR and EKS and Starburst: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Open stack on EMR and EKS (Self-run Spark with open table formats) and Starburst (Managed Trino service for Iceberg) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Open stack: Spark on plain EC2: $20.16; Starburst Galaxy (Pro tier): $60.00. ## The same rows for both | Row | Open stack on EMR and EKS | Starburst | | --- | --- | --- | | Kind | Self-run Spark with open table formats | Managed Trino service for Iceberg | | Does well | Per AWS documentation, EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three formats are available (read 2026-11-10). AWS ran TPC-DS 3 TB on 9 x r5d.4xlarge and reports $4.81 for EMR 7.12 against $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0, with repositories to repeat it. | Per Starburst documentation, Galaxy creates and writes Iceberg tables (Parquet, ORC or Avro) and runs maintenance with ALTER TABLE ... EXECUTE optimize, expire_snapshots and remove_orphan_files (read 2026-11-10). Starburst publishes credit prices by AWS region, for example $0.50 (Pro) and $0.75 (Enterprise) in US East (N. Virginia) and $0.60 and $0.90 in Frankfurt (read 2026-11-10). | | Limited | Per the Apache Iceberg documentation, users run table maintenance themselves: expire snapshots, remove orphan files and compact with rewriteDataFiles as jobs they schedule (read 2026-11-10). A cost comparison of open-stack Spark against managed lakehouses by a party other than AWS or a vendor: none found on 2026-11-10. | Per Starburst documentation, Galaxy runs SQL on Trino, not Spark, so DataFrame code, UDFs and PySpark would be rewritten as Trino SQL or kept on Spark (read 2026-11-10). Per Starburst documentation, Hudi tables are read-only in Galaxy: it cannot create new Hudi tables or write to them. | | Published speed results | AWS ran TPC-DS 3 TB (104 SparkSQL queries, 3 rounds, 9 x r5d.4xlarge, Hadoop catalog) and reports EMR 7.12 averaged 1,349.62 s versus 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0 and 1,535.62 s for EMR 7.5. AWS notes results are not comparable to official TPC-DS. The run is by the vendor of EMR. | Starburst reported GPU-accelerated Trino results (its proprietary distribution, not open-source Trino): TPC-H up to 6x on accelerated queries and ClickBench 3.4x overall, single-node development environment. This is a SQL query test, not Spark batch work. | | Published prices and cost results | EMR on EC2 charges an EMR price in addition to the EC2 price (and EBS), billed per second with a one-minute minimum. AWS's example in us-east-1: c4.2xlarge EMR charge $0.105/hour next to EC2 $0.398/hour; one master and two core nodes for a month total $1,101.57. Spot instances are up to a 90% discount versus On-Demand. EMR on EKS adds an uplift per vCPU-hour and per GB-hour to EKS/EC2 or Fargate costs, plus $0.10 per hour for each EKS cluster. AWS's worked example uses $0.007084 per vCPU-hour and $0.000777875 per GB-hour (100 vCPU and 300 GB for 30 minutes = $0.4709 EMR uplift). The uplift applies to any pod using an EMR runtime image, including with the Spark operator. EMR Serverless is billed per vCPU-hour and GB-hour. AWS's worked example uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour for a job with 100 vCPU and 750 GB for 30 minutes, then 200 more vCPU for 15 minutes, totaling $9.596275. In the same AWS benchmark the cost was $4.81 for EMR 7.12, $5.47 for EMR 7.5 and $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0. Cost formula uses r5d.4xlarge at $1.152/hour plus $0.27/hour EMR plus EBS, On-Demand. | Galaxy credit prices start at $0.50 (Pro), $0.75 (Enterprise) and $1.00 (Mission Critical) per credit. A Free tier allows up to 3 clusters. Pricing depends on compute usage, tier and the cloud provider and region. Annual commitments may get discounts. On-demand list prices per credit (Pro / Enterprise / Mission Critical) on AWS: US East N. Virginia, US East Ohio and US West Oregon $0.50 / $0.75 / $1.00; Europe Ireland $0.56 / $0.84 / $1.12; Europe Frankfurt $0.60 / $0.90 / $1.20; Asia Pacific Tokyo $0.64 / $0.97 / $1.29; Brazil (Sao Paulo) $0.80 / $1.19 / $1.59. Cluster compute credit consumption: 1 worker 6 credits per hour, 2 workers 12, 4 workers 24, 8 workers 48, 16 workers 96. Managed ingest: file ingest 0.60 credits per hour; table maintenance and data transforms 0.30 credits per hour. Billing tracks cluster compute hours, data ingestion compute and private link data transfer. | | Code changes and migration | The EMR runtime for Spark provides 'full API compatibility with open source Apache Spark and Apache Iceberg', so existing Spark jobs run as is. Optimized runtimes are used by EMR on EC2, EMR Serverless, EMR on EKS, EMR on Outposts and AWS Glue. Self-run Spark on EKS uses the open-source Spark operator and needs no job rewrite. | Galaxy is 'a fully-managed data lake analytics platform built in addition to Trino'. Work runs as SQL on Trino, not Spark, so Spark jobs (DataFrame code, UDFs, PySpark) would have to be rewritten as Trino SQL, or kept on Spark against the same Iceberg tables. The Galaxy docs read contain no Spark execution mode. | | Tuning and table upkeep | Iceberg docs say users must run maintenance: expire snapshots, remove old metadata files and delete orphan files are 'recommended'; compacting data files with the Spark rewriteDataFiles action (for example target-file-size-bytes of 500 MB) and rewriting manifests are 'optional' maintenance, run as jobs you schedule. Hudi merge-on-read tables need compaction to merge log files into base files and bound read latency. Compaction has two steps, scheduling (plan written to the timeline) and execution; the docs describe strategies for choosing which file slices to compact. Delta Lake OSS docs list file management features the user applies: compaction (bin-packing) with OPTIMIZE, auto compaction, data skipping, Z-ordering, multi-part checkpointing, log compactions and optimized write. | Iceberg table maintenance in Galaxy is run with ALTER TABLE ... EXECUTE commands: optimize (compaction; files under file_size_threshold, default 100MB, are merged), drop_extended_stats, expire_snapshots, remove_orphan_files. Live table maintenance for ingest tables is a separate Galaxy page. For batch-style work Galaxy offers a fault tolerant execution mode that spools intermediate data and retries failed parts of a query. Docs say 'No other configuration is required' to enable it, but not to use it if most queries run under about a minute, and that it can be slightly slower than normal execution. FTE and Accelerated modes are not available on the Free cluster size and not all catalogs support FTE. | | Spark versions and table formats | Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. All three table formats are available on EMR. EMR on EKS supports the Kubernetes Operator for Apache Spark (Spark operator) as a job submission model with the EMR runtime on your own EKS clusters. The Kubeflow Spark Operator is Apache-2.0 licensed with Helm installation. | Galaxy's Great Lakes connectivity lists Iceberg, Delta Lake, Hive and Hudi table format pages. Hudi tables are read-only: 'Galaxy cannot create new Hudi tables or write to them.' Iceberg tables can be created and written; the Iceberg data file format can be Parquet, ORC or Avro. | | Where it runs | Amazon EMR has regional endpoints in the US (us-east-1, us-east-2, us-west-1, us-west-2) and many other AWS regions per the AWS general reference endpoints table. Spark runs on EC2 instances the user chooses (the AWS benchmark used r5d.4xlarge). | Galaxy prices are listed for AWS, Azure and GCP regions. AWS regions priced include us-east-1, us-east-2, us-west-1, us-west-2, ca-central, Ireland, London, Frankfurt, Zurich, Singapore, Sydney, Tokyo, Cape Town, UAE, Tel Aviv and Sao Paulo. A cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature. | | How open its benchmarks are | AWS publishes the setup in GitHub repositories (aws-samples/emr-on-eks-benchmark branch tpcds-v2.13_iceberg, and aws-samples/emr-spark-benchmark) with build instructions, so the TPC-DS Iceberg benchmark can be repeated. | None found on 10 Nov 2026. | | Compute Bill | Open stack: Spark on plain EC2: $20.16 | Starburst Galaxy (Pro tier): $60.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/emr/pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-t: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operato: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: iceberg.apache.org/docs/latest/maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: hudi.apache.org/docs/compaction: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.delta.io/latest/optimizations-oss.html: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: docs.aws.amazon.com/general/latest/gr/emr.html: https://docs.aws.amazon.com/general/latest/gr/emr.html (read 10 Nov 2026) - Open stack: Spark on EMR or EKS with Iceberg, Delta or Hudi: github.com/aws-samples/emr-on-eks-benchmark: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing: https://www.starburst.io/pricing/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing/pricing-details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/index.html: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/optimization-perform: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/open-stack-emr-eks-vs-starburst/ --- # Snowflake and Starburst: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Snowflake (Cloud data platform with PySpark support) and Starburst (Managed Trino service for Iceberg) are set out here in the same rows: published speed and cost results, code changes, table upkeep, table formats and where each runs. On the Compute Bill's reference job: Snowflake (XL standard warehouse): $96.00; Starburst Galaxy (Pro tier): $60.00. ## The same rows for both | Row | Snowflake | Starburst | | --- | --- | --- | | Kind | Cloud data platform with PySpark support | Managed Trino service for Iceberg | | Does well | Per Snowflake documentation, Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect, and batch jobs are submitted with the Snowpark Submit CLI (read 2026-11-10). Per Snowflake documentation, Iceberg tables can keep data in customer-managed S3, GCS or Azure storage through external volumes and a catalog integration such as AWS Glue. | Per Starburst documentation, Galaxy creates and writes Iceberg tables (Parquet, ORC or Avro) and runs maintenance with ALTER TABLE ... EXECUTE optimize, expire_snapshots and remove_orphan_files (read 2026-11-10). Starburst publishes credit prices by AWS region, for example $0.50 (Pro) and $0.75 (Enterprise) in US East (N. Virginia) and $0.60 and $0.90 in Frankfurt (read 2026-11-10). | | Limited | Per Snowflake documentation, Snowpark Connect supports Apache Spark 3.5 only, and DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported (read 2026-11-10). On-demand credit prices on AWS US East (Northern Virginia) are $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical) per the Snowflake Service Consumption Table effective 2026-11-09 (read 2026-11-10); the Snowflake pricing page itself renders by script. | Per Starburst documentation, Galaxy runs SQL on Trino, not Spark, so DataFrame code, UDFs and PySpark would be rewritten as Trino SQL or kept on Spark (read 2026-11-10). Per Starburst documentation, Hudi tables are read-only in Galaxy: it cannot create new Hudi tables or write to them. | | Published speed results | None found on 10 Nov 2026. | Starburst reported GPU-accelerated Trino results (its proprietary distribution, not open-source Trino): TPC-H up to 6x on accelerated queries and ClickBench 3.4x overall, single-node development environment. This is a SQL query test, not Spark batch work. | | Published prices and cost results | Warehouses bill credits per second with a 60-second minimum on each start; each size step roughly doubles credits per hour (Small is 2 credits per hour, Medium is 4). Snowflake's Service Consumption Table (effective 2026-11-09) lists on-demand credits on AWS US East at $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical). | Galaxy credit prices start at $0.50 (Pro), $0.75 (Enterprise) and $1.00 (Mission Critical) per credit. A Free tier allows up to 3 clusters. Pricing depends on compute usage, tier and the cloud provider and region. Annual commitments may get discounts. On-demand list prices per credit (Pro / Enterprise / Mission Critical) on AWS: US East N. Virginia, US East Ohio and US West Oregon $0.50 / $0.75 / $1.00; Europe Ireland $0.56 / $0.84 / $1.12; Europe Frankfurt $0.60 / $0.90 / $1.20; Asia Pacific Tokyo $0.64 / $0.97 / $1.29; Brazil (Sao Paulo) $0.80 / $1.19 / $1.59. Cluster compute credit consumption: 1 worker 6 credits per hour, 2 workers 12, 4 workers 24, 8 workers 48, 16 workers 96. Managed ingest: file ingest 0.60 credits per hour; table maintenance and data transforms 0.30 credits per hour. Billing tracks cluster compute hours, data ingestion compute and private link data transfer. | | Code changes and migration | Snowpark Connect for Apache Spark runs existing PySpark DataFrame code against the Snowflake engine over the Spark Connect protocol; there is no Spark cluster, all workloads run on a Snowflake warehouse. Batch jobs are submitted with the Snowpark Submit CLI. Because Snowflake rather than Spark is the engine, there are semantic differences: DayTimeIntervalType, YearMonthIntervalType and user-defined types are unsupported, and ByteType, ShortType and IntegerType are returned as LongType. The docs list unsupported APIs for DataFrame and Dataset. Docs say the Spark DataFrame and Spark SQL code can be reused with minimal migration or changes; the compatibility notes say Spark Connect defers plan resolution, which changes behavior versus Spark Classic. | Galaxy is 'a fully-managed data lake analytics platform built in addition to Trino'. Work runs as SQL on Trino, not Spark, so Spark jobs (DataFrame code, UDFs, PySpark) would have to be rewritten as Trino SQL, or kept on Spark against the same Iceberg tables. The Galaxy docs read contain no Spark execution mode. | | Tuning and table upkeep | Docs provide a separate Optimizing Snowpark Connect for Spark workloads section and a Troubleshooting section covering known limitations, so some job-level adjustment is expected after migration. | Iceberg table maintenance in Galaxy is run with ALTER TABLE ... EXECUTE commands: optimize (compaction; files under file_size_threshold, default 100MB, are merged), drop_extended_stats, expire_snapshots, remove_orphan_files. Live table maintenance for ingest tables is a separate Galaxy page. For batch-style work Galaxy offers a fault tolerant execution mode that spools intermediate data and retries failed parts of a query. Docs say 'No other configuration is required' to enable it, but not to use it if most queries run under about a minute, and that it can be slightly slower than normal execution. FTE and Accelerated modes are not available on the Free cluster size and not all catalogs support FTE. | | Spark versions and table formats | Snowpark Connect supports Apache Spark 3.5 only; workloads written for 3.4 and below or 4.0 and above are not supported and may produce protocol errors or missing API failures. Snowflake supports Iceberg tables (Parquet) with data in customer-managed S3, GCS or Azure storage via external volumes, an external catalog integration such as AWS Glue, and Delta files read through Delta Direct; Hudi was not listed on the page. | Galaxy's Great Lakes connectivity lists Iceberg, Delta Lake, Hive and Hudi table format pages. Hudi tables are read-only: 'Galaxy cannot create new Hudi tables or write to them.' Iceberg tables can be created and written; the Iceberg data file format can be Parquet, ORC or Avro. | | Where it runs | Snowflake runs on AWS, Azure and Google Cloud; AWS regions include us-east-1, us-west-2, eu-west-1 and ca-central-1. Each account is hosted in a single region. | Galaxy prices are listed for AWS, Azure and GCP regions. AWS regions priced include us-east-1, us-east-2, us-west-1, us-west-2, ca-central, Ireland, London, Frankfurt, Zurich, Singapore, Sydney, Tokyo, Cape Town, UAE, Tel Aviv and Sao Paulo. A cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature. | | How open its benchmarks are | None found on 10 Nov 2026. | None found on 10 Nov 2026. | | Compute Bill | Snowflake (XL standard warehouse): $96.00 | Starburst Galaxy (Pro tier): $60.00 | Alphabetical order. Vendor statements are the vendor's own. Read 10 Nov 2026. ## How to read this page The rows hold what each vendor or a named third party published, with no conclusion drawn. Benchmarks in the two columns ran on different hardware, data and baselines, so their numbers are not comparable across columns. The [Compute Bill method](/compute-bill/method/) explains the cost lines. ## Sources - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/cost-understanding-compute: https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/tables-iceberg: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/user-guide/intro-regions.md: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) - Snowflake: docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing: https://www.starburst.io/pricing/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/pricing/pricing-details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/index.html: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-la: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): docs.starburst.io/starburst-galaxy/data-engineering/optimization-perform: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026) - Starburst (Starburst Galaxy, Starburst Enterprise): starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/compare/snowflake-vs-starburst/ --- # Iceberg, Delta Lake and Hudi in the same eight rows Table formats, read 10 Nov 2026 **The short answer.** Apache Hudi, Apache Iceberg and Delta Lake are the three open table formats behind most lakehouses. This page sets them out in eight rows: origin and governance, the latest release read on 10 Nov 2026, engines that write each, compaction, partitioning, row-level changes, catalogs and interoperability. ## The eight rows | Row | Apache Hudi | Apache Iceberg | Delta Lake | | --- | --- | --- | --- | | Origin and governance | Apache Hudi is a trademark of the Apache Software Foundation, per its site footer, and its home page describes it as an open data lakehouse platform built on an open table format. | Apache Iceberg is an Apache Software Foundation project, per its site footer, with its community run through dev, commits and issues mailing lists and a private PMC list. | Delta Lake says it is an independent open-source project not controlled by any single company, and that it joined the Delta Lake Project, a sub-project of the Linux Foundation Projects, in 2019. The site footer reads Delta Lake, a series of LF Projects, LLC. | | Latest release read | Apache Hudi 1.2.1, listed on GitHub on 24 Sep 2026, keeps table version 9 as in 1.2.0 (23 May 2026). A 1.1.2 patch release on the older 1.1 line was listed on 17 Oct 2026. Hudi 1.2.0 added VECTOR, VARIANT and BLOB types and Lance file format support. | Apache Iceberg 1.12.0, which the project releases page says was released on September 29, 2026 (GitHub lists the release on 2026-09-30). It removes Spark 3.4 support and Flink 2.0 support. The previous minor release, 1.11.0, was listed on GitHub on 2026-05-20. | Delta Lake 4.4.1, listed on GitHub on 05 Oct 2026 as a patch release on Delta Lake 4.4.0 (20 Aug 2026). It is built for Apache Spark 4.2.0, 4.1.0 and 4.0.1. The delta.io home page announces 4.4.0 on Spark 4.2.0. | | Engines that write it | The Hudi home page lists Apache Spark, Apache Flink, Databricks, AWS EMR, Azure HDInsight, Onehouse, Ray and Daft under data processing, and Kafka, Debezium and Flink CDC under data streaming. | The Iceberg home page names Spark, Trino, Flink, Presto, Hive and Impala as engines that work with the same tables at the same time, and shows MERGE INTO as a SQL command for writes. | The Delta Lake home page names Spark, PrestoDB, Flink, Trino, Hive, Snowflake, Google BigQuery, Athena, Redshift, Databricks and Azure Fabric as compute engines, with APIs for Scala, Java, Rust and Python. It does not separate read from write support on that page. | | Compaction | Compaction in Hudi applies only to Merge-on-Read tables and merges row-based log files into the columnar base file to produce a new base file. It has two steps, scheduling and execution. Clustering is a separate service that rewrites small files into larger ones. | Iceberg compacts data files in parallel with Spark through the rewriteDataFiles action (the documentation example sets target-file-size-bytes to 500 MB). Rewriting manifests is also documented, and the jobs are run by the user. | Delta Lake documents compaction (bin-packing) with OPTIMIZE, available from Delta Lake 1.2.0, with an optional WHERE predicate on partitions. The same page lists auto compaction, Z-ordering, data skipping and optimized write. | | Partitioning and clustering | Hudi clustering batches small files into larger files and can cluster data by sort key for locality, using a replace commit. A hidden-partitioning feature like the one in Iceberg: none found on 2026-11-10 in the pages read. | Iceberg uses hidden partitioning: it produces partition values by transforming a column (for example day(event_time)), and consumers need no extra filters. Partition layout and sort order can evolve in place without rewriting table data. | Delta Lake documents liquid clustering (CLUSTER BY), available from 3.1.0, which lets clustering columns be redefined without rewriting existing data. Clustering is not compatible with partitioning or ZORDER, and OPTIMIZE clusters data incrementally. | | Row-level updates and deletes | Hudi is designed for mutable change streams: Merge-on-Read tables write updates and deletes to log files and merge them with base files during queries, and Copy-on-Write rewrites base files. The Record Level Index, added in 0.14.0, speeds key lookups. | Iceberg documents write.delete.mode, write.update.mode and write.merge.mode, each defaulting to copy-on-write, with merge-on-read available for table format v2 and above. Spark supports MERGE INTO, UPDATE and DELETE FROM. | Delta Lake supports DELETE, UPDATE and MERGE INTO statements. Deletion vectors, available from 2.3.0 and labeled experimental on the page, mark rows as removed without rewriting the Parquet file until a rewrite such as OPTIMIZE applies them. | | Catalogs | The Hudi home page lists AWS Glue Data Catalog, Google BigQuery, Apache Hive Metastore and DataHub as catalogs, with Apache XTable (incubating) for sync. The documentation describes syncing table metadata to a Hive metastore. | Spark catalog configuration documents Hive metastore, REST and directory-based Hadoop catalog types, and the documentation also lists AWS Glue, DynamoDB, JDBC and Nessie catalogs and a REST catalog protocol. | Delta Lake 4.4.0, dated 20 Aug 2026, adds Unity Catalog Delta API integration for catalog-managed tables in Delta Kernel and the experimental Flink connector, and generated columns for tables created through Unity Catalog. | | Interoperability | The Hudi home page lists Apache XTable (incubating) for metadata sync, and queries through Presto, Trino, Hive, AWS Athena, Google BigQuery, Apache Doris, StarRocks and Apache Impala. Delta UniForm documentation says Hudi clients can read Delta tables. | The multi-engine support page lists Spark, Flink, Hive, Trino, Presto, Dremio, Amazon Athena, Amazon EMR, Amazon Redshift, Google BigQuery, Snowflake, Impala and others, and the documentation includes migration guides for Hive and Delta Lake tables. | Delta Universal Format (UniForm) generates Iceberg metadata asynchronously, so Iceberg clients read Delta tables as if they were Iceberg, and the page says the same for Hudi clients. A single copy of the data files is kept. | Alphabetical by format. Sources on each format page. ## Each format - [Apache Hudi](/formats/hudi/): Open lakehouse platform and table format under the Apache Software Foundation, with Merge-on-Read compaction, clustering and indexes; release 1.2.1 dated 2026-09-24. - [Apache Iceberg](/formats/iceberg/): Open table format under the Apache Software Foundation, with hidden partitioning, in-place evolution and a Spark compaction action; release 1.12.0 dated 2026-09-29. - [Delta Lake](/formats/delta-lake/): Open storage framework under the Linux Foundation, with OPTIMIZE compaction, liquid clustering, deletion vectors and UniForm; release 4.4.1 dated 2026-10-05. ## Sources - Apache Hudi home page: https://hudi.apache.org/ (read 10 Nov 2026) - Apache Hudi 1.2.1 GitHub release: https://github.com/apache/hudi/releases/tag/release-1.2.1 (read 10 Nov 2026) - Apache Hudi compaction documentation: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Apache Hudi clustering documentation: https://hudi.apache.org/docs/clustering (read 10 Nov 2026) - Apache Hudi table types documentation: https://hudi.apache.org/docs/table_types (read 10 Nov 2026) - Apache Iceberg community page: https://iceberg.apache.org/community/ (read 10 Nov 2026) - Apache Iceberg releases page: https://iceberg.apache.org/releases/ (read 10 Nov 2026) - Apache Iceberg home page: https://iceberg.apache.org/ (read 10 Nov 2026) - Apache Iceberg maintenance documentation: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Apache Iceberg partitioning documentation: https://iceberg.apache.org/docs/latest/partitioning/ (read 10 Nov 2026) - Apache Iceberg configuration documentation: https://iceberg.apache.org/docs/latest/configuration/ (read 10 Nov 2026) - Apache Iceberg Spark configuration documentation: https://iceberg.apache.org/docs/latest/spark-configuration/ (read 10 Nov 2026) - Apache Iceberg multi-engine support page: https://iceberg.apache.org/multi-engine-support/ (read 10 Nov 2026) - Delta Lake home page: https://delta.io/ (read 10 Nov 2026) - Delta Lake 4.4.1 GitHub release: https://github.com/delta-io/delta/releases/tag/v4.4.1 (read 10 Nov 2026) - Delta Lake optimizations documentation: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Delta Lake liquid clustering documentation: https://docs.delta.io/latest/delta-clustering.html (read 10 Nov 2026) - Delta Lake deletion vectors documentation: https://docs.delta.io/latest/delta-deletion-vectors.html (read 10 Nov 2026) - Delta Lake 4.4.0 GitHub release: https://github.com/delta-io/delta/releases/tag/v4.4.0 (read 10 Nov 2026) - Delta Lake UniForm documentation: https://docs.delta.io/latest/delta-uniform.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/formats/ --- # Apache Hudi: what it is, what it does well, where it is limited Table format, read 10 Nov 2026 **The short answer.** Apache Hudi is an open data lakehouse platform and table format under the Apache Software Foundation. Its latest release is 1.2.1, listed on GitHub on September 24, 2026, and a 1.1.2 patch followed on October 17, 2026. Hudi offers Merge-on-Read tables with compaction, clustering by sort key and a Record Level Index. - **Website:** [hudi.apache.org](https://hudi.apache.org/) - **Read:** 10 Nov 2026 ## What it does well - Per the Hudi documentation, Merge-on-Read tables write updates and deletes to log files and merge them with base files at query time, which supports near real-time data availability (read 2026-11-10). - Per the Hudi documentation, a Record Level Index, added in 0.14.0, avoids the operational overhead of an external index system for large tables. ## Where it is limited - Per the Hudi documentation, query performance on Merge-on-Read tables varies with whether log files have been compacted, and compaction does not apply to Copy-on-Write tables (read 2026-11-10). - Per Starburst documentation, Hudi tables are read-only in Galaxy. Hudi support on the Dremio Cloud data formats page: none found on 2026-11-10. ## The eight rows | Row | Fact | Source, read | | --- | --- | --- | | Origin and governance | Apache Hudi is a trademark of the Apache Software Foundation, per its site footer, and its home page describes it as an open data lakehouse platform built on an open table format. | [Apache Hudi home page](https://hudi.apache.org/), 10 Nov 2026 | | Latest release read | Apache Hudi 1.2.1, listed on GitHub on 24 Sep 2026, keeps table version 9 as in 1.2.0 (23 May 2026). A 1.1.2 patch release on the older 1.1 line was listed on 17 Oct 2026. Hudi 1.2.0 added VECTOR, VARIANT and BLOB types and Lance file format support. | [Apache Hudi 1.2.1 GitHub release](https://github.com/apache/hudi/releases/tag/release-1.2.1), 10 Nov 2026 | | Engines that write it | The Hudi home page lists Apache Spark, Apache Flink, Databricks, AWS EMR, Azure HDInsight, Onehouse, Ray and Daft under data processing, and Kafka, Debezium and Flink CDC under data streaming. | [Apache Hudi home page](https://hudi.apache.org/), 10 Nov 2026 | | Compaction | Compaction in Hudi applies only to Merge-on-Read tables and merges row-based log files into the columnar base file to produce a new base file. It has two steps, scheduling and execution. Clustering is a separate service that rewrites small files into larger ones. | [Apache Hudi compaction documentation](https://hudi.apache.org/docs/compaction), 10 Nov 2026 | | Partitioning and clustering | Hudi clustering batches small files into larger files and can cluster data by sort key for locality, using a replace commit. A hidden-partitioning feature like the one in Iceberg: none found on 2026-11-10 in the pages read. | [Apache Hudi clustering documentation](https://hudi.apache.org/docs/clustering), 10 Nov 2026 | | Row-level updates and deletes | Hudi is designed for mutable change streams: Merge-on-Read tables write updates and deletes to log files and merge them with base files during queries, and Copy-on-Write rewrites base files. The Record Level Index, added in 0.14.0, speeds key lookups. | [Apache Hudi table types documentation](https://hudi.apache.org/docs/table_types), 10 Nov 2026 | | Catalogs | The Hudi home page lists AWS Glue Data Catalog, Google BigQuery, Apache Hive Metastore and DataHub as catalogs, with Apache XTable (incubating) for sync. The documentation describes syncing table metadata to a Hive metastore. | [Apache Hudi home page](https://hudi.apache.org/), 10 Nov 2026 | | Interoperability | The Hudi home page lists Apache XTable (incubating) for metadata sync, and queries through Presto, Trino, Hive, AWS Athena, Google BigQuery, Apache Doris, StarRocks and Apache Impala. Delta UniForm documentation says Hudi clients can read Delta tables. | [Apache Hudi home page](https://hudi.apache.org/), 10 Nov 2026 | ## Apache Hudi side by side - [Apache Hudi and Apache Iceberg](/formats/hudi-vs-iceberg/) - [Apache Hudi and Delta Lake](/formats/delta-lake-vs-hudi/) ## Sources - Apache Hudi home page: https://hudi.apache.org/ (read 10 Nov 2026) - Apache Hudi 1.2.1 GitHub release: https://github.com/apache/hudi/releases/tag/release-1.2.1 (read 10 Nov 2026) - Apache Hudi compaction documentation: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Apache Hudi clustering documentation: https://hudi.apache.org/docs/clustering (read 10 Nov 2026) - Apache Hudi table types documentation: https://hudi.apache.org/docs/table_types (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/formats/hudi/ --- # Apache Iceberg: what it is, what it does well, where it is limited Table format, read 10 Nov 2026 **The short answer.** Apache Iceberg is an open table format for large analytic tables, run as an Apache Software Foundation project. Its latest release is 1.12.0, dated September 29, 2026 on its releases page. Iceberg offers hidden partitioning, in-place partition evolution and Spark compaction through rewriteDataFiles, which teams schedule themselves. - **Website:** [iceberg.apache.org](https://iceberg.apache.org/) - **Read:** 10 Nov 2026 ## What it does well - Per the Iceberg documentation, hidden partitioning produces partition values from column transforms such as day(event_time), and partition layout can change without rewriting table data (read 2026-11-10). - The Iceberg home page says engines such as Spark, Trino, Flink, Presto, Hive and Impala can work with the same tables at the same time. ## Where it is limited - Per the Iceberg maintenance documentation, compaction is an action you schedule: Spark rewriteDataFiles, with expiring snapshots and removing orphan files as separate recommended jobs (read 2026-11-10). - Per the Iceberg configuration documentation, update, delete and merge modes default to copy-on-write, and merge-on-read applies only to table format v2 and above. ## The eight rows | Row | Fact | Source, read | | --- | --- | --- | | Origin and governance | Apache Iceberg is an Apache Software Foundation project, per its site footer, with its community run through dev, commits and issues mailing lists and a private PMC list. | [Apache Iceberg community page](https://iceberg.apache.org/community/), 10 Nov 2026 | | Latest release read | Apache Iceberg 1.12.0, which the project releases page says was released on September 29, 2026 (GitHub lists the release on 2026-09-30). It removes Spark 3.4 support and Flink 2.0 support. The previous minor release, 1.11.0, was listed on GitHub on 2026-05-20. | [Apache Iceberg releases page](https://iceberg.apache.org/releases/), 10 Nov 2026 | | Engines that write it | The Iceberg home page names Spark, Trino, Flink, Presto, Hive and Impala as engines that work with the same tables at the same time, and shows MERGE INTO as a SQL command for writes. | [Apache Iceberg home page](https://iceberg.apache.org/), 10 Nov 2026 | | Compaction | Iceberg compacts data files in parallel with Spark through the rewriteDataFiles action (the documentation example sets target-file-size-bytes to 500 MB). Rewriting manifests is also documented, and the jobs are run by the user. | [Apache Iceberg maintenance documentation](https://iceberg.apache.org/docs/latest/maintenance/), 10 Nov 2026 | | Partitioning and clustering | Iceberg uses hidden partitioning: it produces partition values by transforming a column (for example day(event_time)), and consumers need no extra filters. Partition layout and sort order can evolve in place without rewriting table data. | [Apache Iceberg partitioning documentation](https://iceberg.apache.org/docs/latest/partitioning/), 10 Nov 2026 | | Row-level updates and deletes | Iceberg documents write.delete.mode, write.update.mode and write.merge.mode, each defaulting to copy-on-write, with merge-on-read available for table format v2 and above. Spark supports MERGE INTO, UPDATE and DELETE FROM. | [Apache Iceberg configuration documentation](https://iceberg.apache.org/docs/latest/configuration/), 10 Nov 2026 | | Catalogs | Spark catalog configuration documents Hive metastore, REST and directory-based Hadoop catalog types, and the documentation also lists AWS Glue, DynamoDB, JDBC and Nessie catalogs and a REST catalog protocol. | [Apache Iceberg Spark configuration documentation](https://iceberg.apache.org/docs/latest/spark-configuration/), 10 Nov 2026 | | Interoperability | The multi-engine support page lists Spark, Flink, Hive, Trino, Presto, Dremio, Amazon Athena, Amazon EMR, Amazon Redshift, Google BigQuery, Snowflake, Impala and others, and the documentation includes migration guides for Hive and Delta Lake tables. | [Apache Iceberg multi-engine support page](https://iceberg.apache.org/multi-engine-support/), 10 Nov 2026 | ## Apache Iceberg side by side - [Apache Iceberg and Apache Hudi](/formats/hudi-vs-iceberg/) - [Apache Iceberg and Delta Lake](/formats/delta-lake-vs-iceberg/) ## Sources - Apache Iceberg community page: https://iceberg.apache.org/community/ (read 10 Nov 2026) - Apache Iceberg releases page: https://iceberg.apache.org/releases/ (read 10 Nov 2026) - Apache Iceberg home page: https://iceberg.apache.org/ (read 10 Nov 2026) - Apache Iceberg maintenance documentation: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Apache Iceberg partitioning documentation: https://iceberg.apache.org/docs/latest/partitioning/ (read 10 Nov 2026) - Apache Iceberg configuration documentation: https://iceberg.apache.org/docs/latest/configuration/ (read 10 Nov 2026) - Apache Iceberg Spark configuration documentation: https://iceberg.apache.org/docs/latest/spark-configuration/ (read 10 Nov 2026) - Apache Iceberg multi-engine support page: https://iceberg.apache.org/multi-engine-support/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/formats/iceberg/ --- # Delta Lake: what it is, what it does well, where it is limited Table format, read 10 Nov 2026 **The short answer.** Delta Lake is an open-source storage framework hosted by the Linux Foundation. Its latest release is 4.4.1, dated October 5, 2026 on GitHub, a patch on 4.4.0 of August 20, 2026. Delta offers OPTIMIZE compaction, liquid clustering from 3.1.0, deletion vectors and UniForm, which lets Iceberg and Hudi clients read Delta tables. - **Website:** [delta.io](https://delta.io/) - **Read:** 10 Nov 2026 ## What it does well - Per the Delta Lake documentation, UniForm generates Iceberg metadata asynchronously so Iceberg clients can read Delta tables, with a single copy of the data files (read 2026-11-10). - The Delta Lake home page lists Spark, PrestoDB, Flink, Trino, Hive, Snowflake, Google BigQuery, Athena, Redshift, Databricks and Azure Fabric as compute engines, with Rust and Python APIs. ## Where it is limited - Per the Delta Lake documentation, liquid clustering is not compatible with partitioning or ZORDER on the same table, and it needs Delta Lake 3.1.0 or later (read 2026-11-10). - Per the Delta Lake documentation, deletion vectors are labeled experimental and the page notes UPDATE, MERGE and OPTIMIZE give no strict guarantee of applying every recorded deletion. ## The eight rows | Row | Fact | Source, read | | --- | --- | --- | | Origin and governance | Delta Lake says it is an independent open-source project not controlled by any single company, and that it joined the Delta Lake Project, a sub-project of the Linux Foundation Projects, in 2019. The site footer reads Delta Lake, a series of LF Projects, LLC. | [Delta Lake home page](https://delta.io/), 10 Nov 2026 | | Latest release read | Delta Lake 4.4.1, listed on GitHub on 05 Oct 2026 as a patch release on Delta Lake 4.4.0 (20 Aug 2026). It is built for Apache Spark 4.2.0, 4.1.0 and 4.0.1. The delta.io home page announces 4.4.0 on Spark 4.2.0. | [Delta Lake 4.4.1 GitHub release](https://github.com/delta-io/delta/releases/tag/v4.4.1), 10 Nov 2026 | | Engines that write it | The Delta Lake home page names Spark, PrestoDB, Flink, Trino, Hive, Snowflake, Google BigQuery, Athena, Redshift, Databricks and Azure Fabric as compute engines, with APIs for Scala, Java, Rust and Python. It does not separate read from write support on that page. | [Delta Lake home page](https://delta.io/), 10 Nov 2026 | | Compaction | Delta Lake documents compaction (bin-packing) with OPTIMIZE, available from Delta Lake 1.2.0, with an optional WHERE predicate on partitions. The same page lists auto compaction, Z-ordering, data skipping and optimized write. | [Delta Lake optimizations documentation](https://docs.delta.io/latest/optimizations-oss.html), 10 Nov 2026 | | Partitioning and clustering | Delta Lake documents liquid clustering (CLUSTER BY), available from 3.1.0, which lets clustering columns be redefined without rewriting existing data. Clustering is not compatible with partitioning or ZORDER, and OPTIMIZE clusters data incrementally. | [Delta Lake liquid clustering documentation](https://docs.delta.io/latest/delta-clustering.html), 10 Nov 2026 | | Row-level updates and deletes | Delta Lake supports DELETE, UPDATE and MERGE INTO statements. Deletion vectors, available from 2.3.0 and labeled experimental on the page, mark rows as removed without rewriting the Parquet file until a rewrite such as OPTIMIZE applies them. | [Delta Lake deletion vectors documentation](https://docs.delta.io/latest/delta-deletion-vectors.html), 10 Nov 2026 | | Catalogs | Delta Lake 4.4.0, dated 20 Aug 2026, adds Unity Catalog Delta API integration for catalog-managed tables in Delta Kernel and the experimental Flink connector, and generated columns for tables created through Unity Catalog. | [Delta Lake 4.4.0 GitHub release](https://github.com/delta-io/delta/releases/tag/v4.4.0), 10 Nov 2026 | | Interoperability | Delta Universal Format (UniForm) generates Iceberg metadata asynchronously, so Iceberg clients read Delta tables as if they were Iceberg, and the page says the same for Hudi clients. A single copy of the data files is kept. | [Delta Lake UniForm documentation](https://docs.delta.io/latest/delta-uniform.html), 10 Nov 2026 | ## Delta Lake side by side - [Delta Lake and Apache Hudi](/formats/delta-lake-vs-hudi/) - [Delta Lake and Apache Iceberg](/formats/delta-lake-vs-iceberg/) ## Sources - Delta Lake home page: https://delta.io/ (read 10 Nov 2026) - Delta Lake 4.4.1 GitHub release: https://github.com/delta-io/delta/releases/tag/v4.4.1 (read 10 Nov 2026) - Delta Lake optimizations documentation: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Delta Lake liquid clustering documentation: https://docs.delta.io/latest/delta-clustering.html (read 10 Nov 2026) - Delta Lake deletion vectors documentation: https://docs.delta.io/latest/delta-deletion-vectors.html (read 10 Nov 2026) - Delta Lake 4.4.0 GitHub release: https://github.com/delta-io/delta/releases/tag/v4.4.0 (read 10 Nov 2026) - Delta Lake UniForm documentation: https://docs.delta.io/latest/delta-uniform.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/formats/delta-lake/ --- # Delta Lake and Apache Hudi: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Delta Lake and Apache Hudi are set out here in the same eight rows: origin and governance, the latest release read on 10 Nov 2026, engines that write each, compaction, partitioning and clustering, row-level updates and deletes, catalogs, and interoperability. Each cell carries its source, and the page draws no conclusion. ## The same rows for both | Row | Delta Lake | Apache Hudi | | --- | --- | --- | | Does well | Per the Delta Lake documentation, UniForm generates Iceberg metadata asynchronously so Iceberg clients can read Delta tables, with a single copy of the data files (read 2026-11-10). The Delta Lake home page lists Spark, PrestoDB, Flink, Trino, Hive, Snowflake, Google BigQuery, Athena, Redshift, Databricks and Azure Fabric as compute engines, with Rust and Python APIs. | Per the Hudi documentation, Merge-on-Read tables write updates and deletes to log files and merge them with base files at query time, which supports near real-time data availability (read 2026-11-10). Per the Hudi documentation, a Record Level Index, added in 0.14.0, avoids the operational overhead of an external index system for large tables. | | Limited | Per the Delta Lake documentation, liquid clustering is not compatible with partitioning or ZORDER on the same table, and it needs Delta Lake 3.1.0 or later (read 2026-11-10). Per the Delta Lake documentation, deletion vectors are labeled experimental and the page notes UPDATE, MERGE and OPTIMIZE give no strict guarantee of applying every recorded deletion. | Per the Hudi documentation, query performance on Merge-on-Read tables varies with whether log files have been compacted, and compaction does not apply to Copy-on-Write tables (read 2026-11-10). Per Starburst documentation, Hudi tables are read-only in Galaxy. Hudi support on the Dremio Cloud data formats page: none found on 2026-11-10. | | Origin and governance | Delta Lake says it is an independent open-source project not controlled by any single company, and that it joined the Delta Lake Project, a sub-project of the Linux Foundation Projects, in 2019. The site footer reads Delta Lake, a series of LF Projects, LLC. | Apache Hudi is a trademark of the Apache Software Foundation, per its site footer, and its home page describes it as an open data lakehouse platform built on an open table format. | | Latest release read | Delta Lake 4.4.1, listed on GitHub on 05 Oct 2026 as a patch release on Delta Lake 4.4.0 (20 Aug 2026). It is built for Apache Spark 4.2.0, 4.1.0 and 4.0.1. The delta.io home page announces 4.4.0 on Spark 4.2.0. | Apache Hudi 1.2.1, listed on GitHub on 24 Sep 2026, keeps table version 9 as in 1.2.0 (23 May 2026). A 1.1.2 patch release on the older 1.1 line was listed on 17 Oct 2026. Hudi 1.2.0 added VECTOR, VARIANT and BLOB types and Lance file format support. | | Engines that write it | The Delta Lake home page names Spark, PrestoDB, Flink, Trino, Hive, Snowflake, Google BigQuery, Athena, Redshift, Databricks and Azure Fabric as compute engines, with APIs for Scala, Java, Rust and Python. It does not separate read from write support on that page. | The Hudi home page lists Apache Spark, Apache Flink, Databricks, AWS EMR, Azure HDInsight, Onehouse, Ray and Daft under data processing, and Kafka, Debezium and Flink CDC under data streaming. | | Compaction | Delta Lake documents compaction (bin-packing) with OPTIMIZE, available from Delta Lake 1.2.0, with an optional WHERE predicate on partitions. The same page lists auto compaction, Z-ordering, data skipping and optimized write. | Compaction in Hudi applies only to Merge-on-Read tables and merges row-based log files into the columnar base file to produce a new base file. It has two steps, scheduling and execution. Clustering is a separate service that rewrites small files into larger ones. | | Partitioning and clustering | Delta Lake documents liquid clustering (CLUSTER BY), available from 3.1.0, which lets clustering columns be redefined without rewriting existing data. Clustering is not compatible with partitioning or ZORDER, and OPTIMIZE clusters data incrementally. | Hudi clustering batches small files into larger files and can cluster data by sort key for locality, using a replace commit. A hidden-partitioning feature like the one in Iceberg: none found on 2026-11-10 in the pages read. | | Row-level updates and deletes | Delta Lake supports DELETE, UPDATE and MERGE INTO statements. Deletion vectors, available from 2.3.0 and labeled experimental on the page, mark rows as removed without rewriting the Parquet file until a rewrite such as OPTIMIZE applies them. | Hudi is designed for mutable change streams: Merge-on-Read tables write updates and deletes to log files and merge them with base files during queries, and Copy-on-Write rewrites base files. The Record Level Index, added in 0.14.0, speeds key lookups. | | Catalogs | Delta Lake 4.4.0, dated 20 Aug 2026, adds Unity Catalog Delta API integration for catalog-managed tables in Delta Kernel and the experimental Flink connector, and generated columns for tables created through Unity Catalog. | The Hudi home page lists AWS Glue Data Catalog, Google BigQuery, Apache Hive Metastore and DataHub as catalogs, with Apache XTable (incubating) for sync. The documentation describes syncing table metadata to a Hive metastore. | | Interoperability | Delta Universal Format (UniForm) generates Iceberg metadata asynchronously, so Iceberg clients read Delta tables as if they were Iceberg, and the page says the same for Hudi clients. A single copy of the data files is kept. | The Hudi home page lists Apache XTable (incubating) for metadata sync, and queries through Presto, Trino, Hive, AWS Athena, Google BigQuery, Apache Doris, StarRocks and Apache Impala. Delta UniForm documentation says Hudi clients can read Delta tables. | Alphabetical order. ## Sources - Delta Lake home page: https://delta.io/ (read 10 Nov 2026) - Delta Lake 4.4.1 GitHub release: https://github.com/delta-io/delta/releases/tag/v4.4.1 (read 10 Nov 2026) - Delta Lake optimizations documentation: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Delta Lake liquid clustering documentation: https://docs.delta.io/latest/delta-clustering.html (read 10 Nov 2026) - Delta Lake deletion vectors documentation: https://docs.delta.io/latest/delta-deletion-vectors.html (read 10 Nov 2026) - Delta Lake 4.4.0 GitHub release: https://github.com/delta-io/delta/releases/tag/v4.4.0 (read 10 Nov 2026) - Delta Lake UniForm documentation: https://docs.delta.io/latest/delta-uniform.html (read 10 Nov 2026) - Apache Hudi home page: https://hudi.apache.org/ (read 10 Nov 2026) - Apache Hudi 1.2.1 GitHub release: https://github.com/apache/hudi/releases/tag/release-1.2.1 (read 10 Nov 2026) - Apache Hudi compaction documentation: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Apache Hudi clustering documentation: https://hudi.apache.org/docs/clustering (read 10 Nov 2026) - Apache Hudi table types documentation: https://hudi.apache.org/docs/table_types (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/formats/delta-lake-vs-hudi/ --- # Delta Lake and Apache Iceberg: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Delta Lake and Apache Iceberg are set out here in the same eight rows: origin and governance, the latest release read on 10 Nov 2026, engines that write each, compaction, partitioning and clustering, row-level updates and deletes, catalogs, and interoperability. Each cell carries its source, and the page draws no conclusion. ## The same rows for both | Row | Delta Lake | Apache Iceberg | | --- | --- | --- | | Does well | Per the Delta Lake documentation, UniForm generates Iceberg metadata asynchronously so Iceberg clients can read Delta tables, with a single copy of the data files (read 2026-11-10). The Delta Lake home page lists Spark, PrestoDB, Flink, Trino, Hive, Snowflake, Google BigQuery, Athena, Redshift, Databricks and Azure Fabric as compute engines, with Rust and Python APIs. | Per the Iceberg documentation, hidden partitioning produces partition values from column transforms such as day(event_time), and partition layout can change without rewriting table data (read 2026-11-10). The Iceberg home page says engines such as Spark, Trino, Flink, Presto, Hive and Impala can work with the same tables at the same time. | | Limited | Per the Delta Lake documentation, liquid clustering is not compatible with partitioning or ZORDER on the same table, and it needs Delta Lake 3.1.0 or later (read 2026-11-10). Per the Delta Lake documentation, deletion vectors are labeled experimental and the page notes UPDATE, MERGE and OPTIMIZE give no strict guarantee of applying every recorded deletion. | Per the Iceberg maintenance documentation, compaction is an action you schedule: Spark rewriteDataFiles, with expiring snapshots and removing orphan files as separate recommended jobs (read 2026-11-10). Per the Iceberg configuration documentation, update, delete and merge modes default to copy-on-write, and merge-on-read applies only to table format v2 and above. | | Origin and governance | Delta Lake says it is an independent open-source project not controlled by any single company, and that it joined the Delta Lake Project, a sub-project of the Linux Foundation Projects, in 2019. The site footer reads Delta Lake, a series of LF Projects, LLC. | Apache Iceberg is an Apache Software Foundation project, per its site footer, with its community run through dev, commits and issues mailing lists and a private PMC list. | | Latest release read | Delta Lake 4.4.1, listed on GitHub on 05 Oct 2026 as a patch release on Delta Lake 4.4.0 (20 Aug 2026). It is built for Apache Spark 4.2.0, 4.1.0 and 4.0.1. The delta.io home page announces 4.4.0 on Spark 4.2.0. | Apache Iceberg 1.12.0, which the project releases page says was released on September 29, 2026 (GitHub lists the release on 2026-09-30). It removes Spark 3.4 support and Flink 2.0 support. The previous minor release, 1.11.0, was listed on GitHub on 2026-05-20. | | Engines that write it | The Delta Lake home page names Spark, PrestoDB, Flink, Trino, Hive, Snowflake, Google BigQuery, Athena, Redshift, Databricks and Azure Fabric as compute engines, with APIs for Scala, Java, Rust and Python. It does not separate read from write support on that page. | The Iceberg home page names Spark, Trino, Flink, Presto, Hive and Impala as engines that work with the same tables at the same time, and shows MERGE INTO as a SQL command for writes. | | Compaction | Delta Lake documents compaction (bin-packing) with OPTIMIZE, available from Delta Lake 1.2.0, with an optional WHERE predicate on partitions. The same page lists auto compaction, Z-ordering, data skipping and optimized write. | Iceberg compacts data files in parallel with Spark through the rewriteDataFiles action (the documentation example sets target-file-size-bytes to 500 MB). Rewriting manifests is also documented, and the jobs are run by the user. | | Partitioning and clustering | Delta Lake documents liquid clustering (CLUSTER BY), available from 3.1.0, which lets clustering columns be redefined without rewriting existing data. Clustering is not compatible with partitioning or ZORDER, and OPTIMIZE clusters data incrementally. | Iceberg uses hidden partitioning: it produces partition values by transforming a column (for example day(event_time)), and consumers need no extra filters. Partition layout and sort order can evolve in place without rewriting table data. | | Row-level updates and deletes | Delta Lake supports DELETE, UPDATE and MERGE INTO statements. Deletion vectors, available from 2.3.0 and labeled experimental on the page, mark rows as removed without rewriting the Parquet file until a rewrite such as OPTIMIZE applies them. | Iceberg documents write.delete.mode, write.update.mode and write.merge.mode, each defaulting to copy-on-write, with merge-on-read available for table format v2 and above. Spark supports MERGE INTO, UPDATE and DELETE FROM. | | Catalogs | Delta Lake 4.4.0, dated 20 Aug 2026, adds Unity Catalog Delta API integration for catalog-managed tables in Delta Kernel and the experimental Flink connector, and generated columns for tables created through Unity Catalog. | Spark catalog configuration documents Hive metastore, REST and directory-based Hadoop catalog types, and the documentation also lists AWS Glue, DynamoDB, JDBC and Nessie catalogs and a REST catalog protocol. | | Interoperability | Delta Universal Format (UniForm) generates Iceberg metadata asynchronously, so Iceberg clients read Delta tables as if they were Iceberg, and the page says the same for Hudi clients. A single copy of the data files is kept. | The multi-engine support page lists Spark, Flink, Hive, Trino, Presto, Dremio, Amazon Athena, Amazon EMR, Amazon Redshift, Google BigQuery, Snowflake, Impala and others, and the documentation includes migration guides for Hive and Delta Lake tables. | Alphabetical order. ## Sources - Delta Lake home page: https://delta.io/ (read 10 Nov 2026) - Delta Lake 4.4.1 GitHub release: https://github.com/delta-io/delta/releases/tag/v4.4.1 (read 10 Nov 2026) - Delta Lake optimizations documentation: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Delta Lake liquid clustering documentation: https://docs.delta.io/latest/delta-clustering.html (read 10 Nov 2026) - Delta Lake deletion vectors documentation: https://docs.delta.io/latest/delta-deletion-vectors.html (read 10 Nov 2026) - Delta Lake 4.4.0 GitHub release: https://github.com/delta-io/delta/releases/tag/v4.4.0 (read 10 Nov 2026) - Delta Lake UniForm documentation: https://docs.delta.io/latest/delta-uniform.html (read 10 Nov 2026) - Apache Iceberg community page: https://iceberg.apache.org/community/ (read 10 Nov 2026) - Apache Iceberg releases page: https://iceberg.apache.org/releases/ (read 10 Nov 2026) - Apache Iceberg home page: https://iceberg.apache.org/ (read 10 Nov 2026) - Apache Iceberg maintenance documentation: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Apache Iceberg partitioning documentation: https://iceberg.apache.org/docs/latest/partitioning/ (read 10 Nov 2026) - Apache Iceberg configuration documentation: https://iceberg.apache.org/docs/latest/configuration/ (read 10 Nov 2026) - Apache Iceberg Spark configuration documentation: https://iceberg.apache.org/docs/latest/spark-configuration/ (read 10 Nov 2026) - Apache Iceberg multi-engine support page: https://iceberg.apache.org/multi-engine-support/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/formats/delta-lake-vs-iceberg/ --- # Apache Hudi and Apache Iceberg: the published facts in the same rows Side by side, read 10 Nov 2026 **The short answer.** Apache Hudi and Apache Iceberg are set out here in the same eight rows: origin and governance, the latest release read on 10 Nov 2026, engines that write each, compaction, partitioning and clustering, row-level updates and deletes, catalogs, and interoperability. Each cell carries its source, and the page draws no conclusion. ## The same rows for both | Row | Apache Hudi | Apache Iceberg | | --- | --- | --- | | Does well | Per the Hudi documentation, Merge-on-Read tables write updates and deletes to log files and merge them with base files at query time, which supports near real-time data availability (read 2026-11-10). Per the Hudi documentation, a Record Level Index, added in 0.14.0, avoids the operational overhead of an external index system for large tables. | Per the Iceberg documentation, hidden partitioning produces partition values from column transforms such as day(event_time), and partition layout can change without rewriting table data (read 2026-11-10). The Iceberg home page says engines such as Spark, Trino, Flink, Presto, Hive and Impala can work with the same tables at the same time. | | Limited | Per the Hudi documentation, query performance on Merge-on-Read tables varies with whether log files have been compacted, and compaction does not apply to Copy-on-Write tables (read 2026-11-10). Per Starburst documentation, Hudi tables are read-only in Galaxy. Hudi support on the Dremio Cloud data formats page: none found on 2026-11-10. | Per the Iceberg maintenance documentation, compaction is an action you schedule: Spark rewriteDataFiles, with expiring snapshots and removing orphan files as separate recommended jobs (read 2026-11-10). Per the Iceberg configuration documentation, update, delete and merge modes default to copy-on-write, and merge-on-read applies only to table format v2 and above. | | Origin and governance | Apache Hudi is a trademark of the Apache Software Foundation, per its site footer, and its home page describes it as an open data lakehouse platform built on an open table format. | Apache Iceberg is an Apache Software Foundation project, per its site footer, with its community run through dev, commits and issues mailing lists and a private PMC list. | | Latest release read | Apache Hudi 1.2.1, listed on GitHub on 24 Sep 2026, keeps table version 9 as in 1.2.0 (23 May 2026). A 1.1.2 patch release on the older 1.1 line was listed on 17 Oct 2026. Hudi 1.2.0 added VECTOR, VARIANT and BLOB types and Lance file format support. | Apache Iceberg 1.12.0, which the project releases page says was released on September 29, 2026 (GitHub lists the release on 2026-09-30). It removes Spark 3.4 support and Flink 2.0 support. The previous minor release, 1.11.0, was listed on GitHub on 2026-05-20. | | Engines that write it | The Hudi home page lists Apache Spark, Apache Flink, Databricks, AWS EMR, Azure HDInsight, Onehouse, Ray and Daft under data processing, and Kafka, Debezium and Flink CDC under data streaming. | The Iceberg home page names Spark, Trino, Flink, Presto, Hive and Impala as engines that work with the same tables at the same time, and shows MERGE INTO as a SQL command for writes. | | Compaction | Compaction in Hudi applies only to Merge-on-Read tables and merges row-based log files into the columnar base file to produce a new base file. It has two steps, scheduling and execution. Clustering is a separate service that rewrites small files into larger ones. | Iceberg compacts data files in parallel with Spark through the rewriteDataFiles action (the documentation example sets target-file-size-bytes to 500 MB). Rewriting manifests is also documented, and the jobs are run by the user. | | Partitioning and clustering | Hudi clustering batches small files into larger files and can cluster data by sort key for locality, using a replace commit. A hidden-partitioning feature like the one in Iceberg: none found on 2026-11-10 in the pages read. | Iceberg uses hidden partitioning: it produces partition values by transforming a column (for example day(event_time)), and consumers need no extra filters. Partition layout and sort order can evolve in place without rewriting table data. | | Row-level updates and deletes | Hudi is designed for mutable change streams: Merge-on-Read tables write updates and deletes to log files and merge them with base files during queries, and Copy-on-Write rewrites base files. The Record Level Index, added in 0.14.0, speeds key lookups. | Iceberg documents write.delete.mode, write.update.mode and write.merge.mode, each defaulting to copy-on-write, with merge-on-read available for table format v2 and above. Spark supports MERGE INTO, UPDATE and DELETE FROM. | | Catalogs | The Hudi home page lists AWS Glue Data Catalog, Google BigQuery, Apache Hive Metastore and DataHub as catalogs, with Apache XTable (incubating) for sync. The documentation describes syncing table metadata to a Hive metastore. | Spark catalog configuration documents Hive metastore, REST and directory-based Hadoop catalog types, and the documentation also lists AWS Glue, DynamoDB, JDBC and Nessie catalogs and a REST catalog protocol. | | Interoperability | The Hudi home page lists Apache XTable (incubating) for metadata sync, and queries through Presto, Trino, Hive, AWS Athena, Google BigQuery, Apache Doris, StarRocks and Apache Impala. Delta UniForm documentation says Hudi clients can read Delta tables. | The multi-engine support page lists Spark, Flink, Hive, Trino, Presto, Dremio, Amazon Athena, Amazon EMR, Amazon Redshift, Google BigQuery, Snowflake, Impala and others, and the documentation includes migration guides for Hive and Delta Lake tables. | Alphabetical order. ## Sources - Apache Hudi home page: https://hudi.apache.org/ (read 10 Nov 2026) - Apache Hudi 1.2.1 GitHub release: https://github.com/apache/hudi/releases/tag/release-1.2.1 (read 10 Nov 2026) - Apache Hudi compaction documentation: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Apache Hudi clustering documentation: https://hudi.apache.org/docs/clustering (read 10 Nov 2026) - Apache Hudi table types documentation: https://hudi.apache.org/docs/table_types (read 10 Nov 2026) - Apache Iceberg community page: https://iceberg.apache.org/community/ (read 10 Nov 2026) - Apache Iceberg releases page: https://iceberg.apache.org/releases/ (read 10 Nov 2026) - Apache Iceberg home page: https://iceberg.apache.org/ (read 10 Nov 2026) - Apache Iceberg maintenance documentation: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Apache Iceberg partitioning documentation: https://iceberg.apache.org/docs/latest/partitioning/ (read 10 Nov 2026) - Apache Iceberg configuration documentation: https://iceberg.apache.org/docs/latest/configuration/ (read 10 Nov 2026) - Apache Iceberg Spark configuration documentation: https://iceberg.apache.org/docs/latest/spark-configuration/ (read 10 Nov 2026) - Apache Iceberg multi-engine support page: https://iceberg.apache.org/multi-engine-support/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/formats/hudi-vs-iceberg/ --- # Iceberg compaction: rewrite_data_files, strategies and what platforms automate Table maintenance, read 10 Nov 2026 **The short answer.** Iceberg compaction rewrites many small data files into fewer files near a 512 MB target, using the rewrite_data_files procedure with the binpack, sort or z-order strategy. Open-source Iceberg leaves scheduling to you; AWS, Databricks, Dremio, Snowflake and BigQuery document managed versions. ## The procedure and the action Iceberg tracks each data file in manifest files, so more small files means more metadata and more file-open cost. The Spark procedure rewrite_data_files combines small files into larger ones in parallel. In SQL: CALL catalog_name.system.rewrite_data_files(table => 'db.sample'). The Java and Scala equivalent is SparkActions.rewriteDataFiles, which accepts a filter and options such as target-file-size-bytes. The where argument limits the rewrite to files that may contain matching data. Iceberg's maintenance page lists compaction under optional maintenance, next to rewriting manifests, and notes that streaming queries may produce small data files. The related procedures are rewrite_manifests and rewrite_position_delete_files. ## Strategies | Strategy | How it is set | What it does | | --- | --- | --- | | binpack | strategy => 'binpack' (the default) | Combines small files into files near the target size without reordering rows | | sort | strategy => 'sort', sort_order => 'id DESC NULLS LAST' | Rewrites files with rows sorted by the listed columns; defaults to the table's sort order | | z-order | strategy => 'sort', sort_order => 'zorder(c1,c2)' | Interleaves several columns into one sort key | | hilbert | strategy => 'sort', sort_order => 'hilbert(c1,c2)' | Hilbert curve ordering; every column contributes 8 bytes; cannot be combined with zorder() or plain columns | Source: Apache Iceberg Spark procedures page, read 2026-11-10. ## Defaults that decide when files are rewritten The target file size is 512 MB, the default of write.target-file-size-bytes. A file under 75% of the target (384 MB) or over 180% of it (about 922 MB) is selected for rewriting regardless of other criteria. Any file group with 5 or more files (min-input-files) is rewritten, and a single group is capped at 100 GB of input (max-file-group-size-bytes). Up to 5 file groups are rewritten at once (max-concurrent-file-group-rewrites). Deletes count too. A data file with a deletion ratio of 0.3 or more (delete-ratio-threshold) is considered for rewriting, and remove-dangling-deletes, off by default, adds a commit that drops delete files that no longer apply. Set partial-progress.enabled to commit groups as they finish, with 10 commits allowed by default, so a long rewrite on a big table survives a failure part way. Rewrite-all forces every file to be rewritten, which is what a sort or z-order run over the whole table needs. AWS states that sort and z-order compaction may cost more than binpack in S3 Tables, and prices them higher, as the last section shows. ## What each platform automates for Iceberg Vendors alphabetical, with the AWS services grouped first. Details and sources are on [compaction on each platform](/maintenance/compaction-on-each-platform/). AWS Glue Data Catalog compaction is off for new tables and enabled per table; it starts when a table or partition has more than 100 files, each smaller than 75% of the target. S3 Tables compaction is on by default and offers auto, binpack, sort and z-order, with a target of 64 to 512 MB. Athena runs compaction only when you issue OPTIMIZE ... REWRITE DATA USING BIN_PACK. BigQuery Iceberg managed tables compact automatically. Databricks predictive optimization covers Unity Catalog managed Iceberg tables. Dremio optimizes Iceberg tables in its Open Catalog automatically. Snowflake compacts Snowflake-managed Iceberg tables by default and does nothing for externally managed ones. Starburst Galaxy runs optimize when you issue ALTER TABLE ... EXECUTE. ## Cost of running it yourself On EMR the rewrite is a Spark job billed as EC2 hours plus the EMR uplift; see [EMR cost optimization](/guides/emr-cost-optimization-lakehouse/). The managed AWS options bill separately: Glue at $0.44 per DPU-hour, S3 Tables at $0.005 per GB processed plus $0.002 per 1,000 objects, or $0.01 per GB for sort and z-order. No published benchmark comparing these on one table was found on 2026-11-10; the one compaction benchmark with a stated cost is DualBird's own and is described on the platforms page. ## Other maintenance pages - [Iceberg snapshot expiry and orphan file removal](/maintenance/snapshot-expiry-and-orphan-files/) - [Delta Lake OPTIMIZE and VACUUM: defaults and platform automation](/maintenance/delta-optimize-and-vacuum/) - [Apache Hudi table services: compaction, clustering, cleaning](/maintenance/hudi-table-services/) - [Which platforms automate Iceberg compaction](/maintenance/compaction-on-each-platform/) ## Sources - Apache Iceberg Spark procedures: https://iceberg.apache.org/docs/latest/spark-procedures/ (read 10 Nov 2026) - Apache Iceberg maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Apache Iceberg configuration: https://iceberg.apache.org/docs/latest/configuration/ (read 10 Nov 2026) - Amazon S3 Tables maintenance: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - AWS Glue compaction optimization: https://docs.aws.amazon.com/glue/latest/dg/compaction-management.html (read 10 Nov 2026) - Athena Iceberg optimization: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - BigQuery Iceberg tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Databricks predictive optimization: https://docs.databricks.com/aws/en/optimizations/predictive-optimization (read 10 Nov 2026) - Dremio Cloud table optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Snowflake Iceberg table management: https://docs.snowflake.com/en/user-guide/tables-iceberg-manage (read 10 Nov 2026) - Starburst Galaxy Iceberg: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - AWS price list, AWS Glue, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - AWS price list, Amazon S3, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/maintenance/iceberg-compaction/ --- # Snapshot expiry and orphan file removal on Iceberg tables Table maintenance, read 10 Nov 2026 **The short answer.** Every Iceberg write creates a snapshot, and old snapshots keep data files alive until expire_snapshots removes them. Orphan files, never referenced by metadata, need remove_orphan_files with a retention longer than any write, 3 days by default. AWS, Snowflake and Athena document managed versions. ## expire_snapshots Each write, update, delete, upsert or compaction in Iceberg produces a new snapshot while keeping the old data and metadata for snapshot isolation and time travel. The expire_snapshots procedure removes old snapshots and the data files that only those snapshots need. It never removes files that a non-expired snapshot still requires. Arguments: older_than (default 5 days ago), retain_last (default 1), snapshot_ids, max_concurrent_deletes, stream_results and clean_expired_metadata. The docs advise setting stream_results to true to avoid Spark driver out-of-memory errors on large file counts. If older_than and retain_last are omitted, the table properties history.expire.max-snapshot-age-ms (432,000,000 ms, 5 days) and history.expire.min-snapshots-to-keep (1) apply. Snapshots still referenced by branches or tags are not removed, and the main branch never expires. ## remove_orphan_files Orphan files are files in the table location that no metadata file references, often left by failed jobs. remove_orphan_files deletes files older than older_than, which defaults to 3 days ago. It supports dry_run, a location argument, file_list_view for a precomputed listing, and prefix_mismatch_mode for storage schemes that differ in spelling. Iceberg's maintenance page warns that removing orphan files with a retention interval shorter than the time any write takes to complete is dangerous, because an in-progress file can be treated as orphaned and deleted, which can corrupt the table. Run dry_run first on a table you have not cleaned before. ## Old metadata files Metadata JSON files accumulate as well. write.metadata.delete-after-commit.enabled is false by default and write.metadata.previous-versions-max is 100. The docs give an example: with delete-after-commit false and previous-versions-max 10, after 100 commits there are 10 tracked metadata files and 90 orphaned ones, and those 90 can only be removed by orphan file deletion. ## What the managed services do | Service | Snapshot expiry | Orphan files | Notes from the documentation | | --- | --- | --- | --- | | Amazon Athena | VACUUM expires snapshots | VACUUM also removes orphan files | Retention set with vacuum_max_snapshot_age_seconds; run by you | | AWS Glue Data Catalog | Snapshot retention optimizer: retention in days and maximum snapshots to keep | Orphan file deletion optimizer | Only deletes orphans created after the optimizer's creation date; table location must not overlap other data | | Amazon S3 Tables | Snapshot management on by default: MinimumSnapshots 1, MaximumSnapshotAge 120 hours | Unreferenced file removal policy with NoncurrentDays | Fails for the whole table if user tags, branches or history.expire properties are set | | Snowflake | Automatic for Snowflake-managed tables, no cost | Not supported for Snowflake-managed tables; contact Support | Externally managed tables: you run maintenance in your own engine | | Starburst Galaxy | ALTER TABLE ... EXECUTE expire_snapshots | remove_orphan_files | Run by you | Sources are listed below; all read 2026-11-10. ## Order of operations Expire snapshots first, then remove orphans, then compact on a schedule that does not overlap. Compaction creates new files and a new snapshot; the replaced files are only deleted once the snapshot that referenced them expires. A retention window shorter than your longest reader or time-travel need cuts off queries that rely on it. The [compaction page](/maintenance/iceberg-compaction/) covers the rewrite side. ## Other maintenance pages - [Iceberg compaction: rewrite_data_files, strategies, platforms](/maintenance/iceberg-compaction/) - [Delta Lake OPTIMIZE and VACUUM: defaults and platform automation](/maintenance/delta-optimize-and-vacuum/) - [Apache Hudi table services: compaction, clustering, cleaning](/maintenance/hudi-table-services/) - [Which platforms automate Iceberg compaction](/maintenance/compaction-on-each-platform/) ## Sources - Apache Iceberg Spark procedures: https://iceberg.apache.org/docs/latest/spark-procedures/ (read 10 Nov 2026) - Apache Iceberg maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Apache Iceberg configuration: https://iceberg.apache.org/docs/latest/configuration/ (read 10 Nov 2026) - Athena Iceberg optimization: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - AWS Glue snapshot retention: https://docs.aws.amazon.com/glue/latest/dg/snapshot-retention-management.html (read 10 Nov 2026) - AWS Glue orphan file deletion: https://docs.aws.amazon.com/glue/latest/dg/orphan-file-deletion.html (read 10 Nov 2026) - Amazon S3 Tables maintenance: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Snowflake Iceberg table management: https://docs.snowflake.com/en/user-guide/tables-iceberg-manage (read 10 Nov 2026) - Starburst Galaxy Iceberg: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/maintenance/snapshot-expiry-and-orphan-files/ --- # Delta Lake OPTIMIZE and VACUUM Table maintenance, read 10 Nov 2026 **The short answer.** Delta Lake compacts small files with OPTIMIZE, whose bin-packing is idempotent, and deletes unreferenced files with VACUUM, which never runs on its own and keeps 7 days by default. Databricks predictive optimization schedules both on Unity Catalog managed tables and skips ZORDER. ## OPTIMIZE OPTIMIZE compacts a Delta table by rewriting small files into larger ones. It accepts a partition predicate with WHERE so you can limit the work, and is available in SQL and through deltaTable.optimize().executeCompaction() in Python and Scala. Bin-packing is idempotent: running it twice on the same data makes no second change. Readers keep snapshot isolation, so OPTIMIZE does not interrupt them, and it makes no data changes. Z-ordering is OPTIMIZE ... ZORDER BY (col) and colocates related rows to help data skipping. It is not idempotent: each run tries to recluster all files in the partition. The docs say locality drops with each extra column and that columns without collected statistics make Z-ordering ineffective. They also note it balances tuple counts, not bytes, so wide recent rows can skew task times and output file sizes. ## Auto compaction and optimized writes Auto compaction is enabled with the delta.autoOptimize.autoCompact table property or the spark.databricks.delta.autoCompact.enabled session setting, and by default uses 128 MB as the target file size. It triggers only when a partition has at least a minimum number of small files, set with spark.databricks.delta.autoCompact.minNumFiles. Optimized writes shuffle data before writing so fewer, larger files are produced. They are disabled by default, can be set at table, session or writer level, and use a bin size of 512 MiB by default. The docs note the shuffle can increase write latency. Fabric documents the same auto compaction settings and, from Fabric Spark runtime 2.0 (Delta 4.2), an onCheckpointOnly mode that defers the check to checkpoints, typically every 10 commits. ## VACUUM VACUUM removes files no longer referenced by a table that are older than the retention threshold, 7 days by default. It is not triggered automatically. It deletes data files only; log files are removed automatically after checkpoints and kept 30 days by default (delta.logRetentionDuration). Time travel to a version older than the retention period is lost after VACUUM. Modes and options: the default FULL mode lists the table directory; LITE reads the transaction log instead and runs faster, failing with DELTA_CANNOT_VACUUM_LITE if the log was pruned. DRY RUN lists files to delete. USING INVENTORY takes a table or query of file paths to avoid a full listing. Deleting in parallel needs spark.databricks.delta.vacuum.parallelDelete.enabled set to true. The docs advise a retention interval of at least 7 days and include a safety check that blocks shorter intervals unless you turn it off. ## Who runs it for you Databricks predictive optimization runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables, is on by default, and bills a serverless jobs SKU. It does not run ZORDER; on tables that use Z-order it ignores Z-ordered files. Its VACUUM retention follows delta.deletedFileRetentionDuration, default 7 days, and never goes below 7 days when it runs VACUUM FULL. On Databricks Runtime 18 LTS and above, DESCRIBE TABLE EXTENDED ... AS JSON shows why an operation was skipped. Microsoft Fabric documents OPTIMIZE, Z-order and V-order on Delta tables and notes that liquid-clustered tables are only clustered when OPTIMIZE runs, so a compaction schedule or auto compaction is needed. Amazon EMR 7.14.0 bundles Delta 3.3.2-amzn-3; none of the EMR pages read describes a scheduled OPTIMIZE or VACUUM. See [compaction on each platform](/maintenance/compaction-on-each-platform/). ## Deletion vectors With deletion vectors enabled, deleting a row records it separately instead of rewriting the whole Parquet file, and a later OPTIMIZE can apply them. The Delta docs include a table of which operations each Delta version supports with deletion vectors, so check your runtime version against it. ## Other maintenance pages - [Iceberg compaction: rewrite_data_files, strategies, platforms](/maintenance/iceberg-compaction/) - [Iceberg snapshot expiry and orphan file removal](/maintenance/snapshot-expiry-and-orphan-files/) - [Apache Hudi table services: compaction, clustering, cleaning](/maintenance/hudi-table-services/) - [Which platforms automate Iceberg compaction](/maintenance/compaction-on-each-platform/) ## Sources - Delta Lake file management: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Delta Lake utility commands (VACUUM): https://docs.delta.io/latest/delta-utility.html (read 10 Nov 2026) - Delta Lake deletion vectors: https://docs.delta.io/latest/delta-deletion-vectors.html (read 10 Nov 2026) - Databricks predictive optimization: https://docs.databricks.com/aws/en/optimizations/predictive-optimization (read 10 Nov 2026) - Fabric table compaction: https://learn.microsoft.com/en-us/fabric/data-engineering/table-compaction (read 10 Nov 2026) - Delta Lake on Amazon EMR: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-delta.html (read 10 Nov 2026) - Delta Lake latest release (GitHub API): https://api.github.com/repos/delta-io/delta/releases/latest (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/maintenance/delta-optimize-and-vacuum/ --- # Apache Hudi table services: compaction, clustering, cleaning and archival Table maintenance, read 10 Nov 2026 **The short answer.** Hudi runs four table services: compaction folds merge-on-read log files into base files, clustering rewrites file layout, cleaning removes old file versions after every commit by default, and archival trims the timeline. The Hudi documentation lists a default for each service, and Hudi 1.2.1 was released September 24, 2026. ## Compaction Merge-on-read tables write updates to log files and need compaction to merge them into base files and bound read latency. Compaction has a scheduling step, which writes a plan to the timeline, and an execution step. The trigger strategy decides how often to schedule: NUM_COMMITS, NUM_COMMITS_AFTER_LAST_REQUEST, TIME_ELAPSED, NUM_AND_TIME or NUM_OR_TIME. The compaction strategy decides which file groups to compact; the default, LogFileSizeBasedCompactionStrategy, picks file groups with the most accumulated unmerged log data. Others order by log file count, bound the IO, or limit by partition or day. By default compaction runs asynchronously, so ingestion latency is not blocked. Set hoodie.compact.inline to true to compact in the same job as the write. Hudi Streamer continuous mode, Flink and Spark Streaming run async compaction alongside ingestion, and a standalone HoodieCompactor can run it as a separate job. Incremental scheduling is on by default through hoodie.table.services.incremental.enabled and scans only partitions changed since the last compaction, which the docs say helps tables with many partitions. The Hudi metadata table has its own compaction, scheduled after 10 delta commits by default. ## Clustering Clustering is a layout service. It stitches small files into larger ones and can sort data by chosen columns so query predicates skip more files. It runs async or sync, and records a REPLACE action on the timeline. It builds on Hudi's MVCC design so writers can continue while clustering runs. A plan strategy chooses files, for example SparkSizeBasedClusteringPlanStrategy, and an execution strategy rewrites them, by default SparkSortAndSizeExecutionStrategy, with hoodie.clustering.plan.strategy.sort.columns naming the sort columns. SparkSingleFileSortPlanStrategy and a binary file-stitching strategy are documented for cases where records need not be reread. Inline clustering uses hoodie.clustering.inline and hoodie.clustering.inline.max.commits, and blocks the next ingestion round until it finishes. ## Cleaning In Spark writers, cleaning runs inline after every commit by default (hoodie.clean.automatic is true) under the KEEP_LATEST_COMMITS policy, retaining 10 commits (hoodie.clean.commits.retained). Other policies are KEEP_LATEST_FILE_VERSIONS and KEEP_LATEST_BY_HOURS. hoodie.clean.max.commits schedules the cleaner every few commits, and hoodie.clean.async runs it alongside writing. Hudi 1.2.0 added hoodie.prewrite.cleaner.policy, off by default, which can clean or roll back failed writes before each write. The number of commits retained also sets how far back incremental queries can read, so a shorter window saves storage and shortens incremental history. ## Timeline archival Completed actions move from the active timeline to an LSM-structured archived timeline. hoodie.keep.max.commits defaults to 30 and hoodie.keep.min.commits to 20 instants in the active timeline. Archival removes side effects of completed or pending actions, such as uncommitted files, from storage before archiving them. ## Who runs it Amazon EMR installs Hudi components by default when Spark, Hive, Presto or Flink is installed, and release 7.14.0 packages Hudi 1.0.2-amzn-3; you schedule or enable the services. Onehouse says its platform automates asynchronous table services for Hudi, Iceberg and Delta, and that claim is the vendor's. Starburst Galaxy treats Hudi tables as read-only, so it cannot run these services on them. Compare with [Delta OPTIMIZE and VACUUM](/maintenance/delta-optimize-and-vacuum/) and [Iceberg compaction](/maintenance/iceberg-compaction/). ## Other maintenance pages - [Iceberg compaction: rewrite_data_files, strategies, platforms](/maintenance/iceberg-compaction/) - [Iceberg snapshot expiry and orphan file removal](/maintenance/snapshot-expiry-and-orphan-files/) - [Delta Lake OPTIMIZE and VACUUM: defaults and platform automation](/maintenance/delta-optimize-and-vacuum/) - [Which platforms automate Iceberg compaction](/maintenance/compaction-on-each-platform/) ## Sources - Apache Hudi compaction: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Apache Hudi clustering: https://hudi.apache.org/docs/clustering (read 10 Nov 2026) - Apache Hudi cleaning: https://hudi.apache.org/docs/cleaning (read 10 Nov 2026) - Apache Hudi timeline and archival: https://hudi.apache.org/docs/timeline (read 10 Nov 2026) - Apache Hudi latest release (GitHub API): https://api.github.com/repos/apache/hudi/releases/latest (read 10 Nov 2026) - Apache Hudi on Amazon EMR: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi.html (read 10 Nov 2026) - Amazon EMR 7.x release versions: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Onehouse about: https://www.onehouse.ai/about-us (read 10 Nov 2026) - Starburst Galaxy Hudi: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/maintenance/hudi-table-services/ --- # Compaction on each platform: what is automated, what is billed Table maintenance, read 10 Nov 2026 **The short answer.** AWS S3 Tables compacts on by default and Glue on request; Databricks, Dremio, Snowflake and BigQuery document managed compaction; Athena and Starburst Galaxy run it only when you issue a command. DualBird reports its own compaction benchmark, shown here as DualBird's. ## The table | Platform | Automatic compaction | How it is billed or triggered | Source | | --- | --- | --- | --- | | Amazon Athena | No: manual OPTIMIZE ... REWRITE DATA USING BIN_PACK, plus VACUUM | You issue the statements | Athena docs | | Amazon S3 Tables | Yes, on by default for every table; 512 MB default target (64 to 512 MB); auto, binpack, sort, z-order | $0.005 per GB plus $0.002 per 1,000 objects; sort or z-order $0.01 per GB plus $0.002 per 1,000 objects (us-east-1) | S3 docs and price list | | AWS Glue Data Catalog | Yes, but off by default; enabled per table with an IAM role; starts above 100 files each under 75% of target | $0.44 per DPU-hour (us-east-1); Parquet tables only | Glue docs and price list | | BigQuery | Yes, for Iceberg managed tables: compaction, clustering and garbage collection | DCUs billed per second; load and export use Enterprise pay-as-you-go slots | Google Cloud docs | | Databricks | Yes, predictive optimization on Unity Catalog managed Delta and Iceberg tables; on by default | Serverless jobs SKU; does not run ZORDER | Databricks docs | | Dremio | Yes, for Iceberg tables in the Open Catalog, on a dedicated engine it configures; OPTIMIZE TABLE for manual runs | Engines bill DCUs; Dremio says it prefers automatic over manual runs | Dremio docs | | Onehouse | Vendor claims automated asynchronous table services | Billed in OCUs; no dollar rate found | Onehouse docs | | Snowflake | Yes for Snowflake-managed Iceberg tables, on by default; none for externally managed tables | Billable credits; bundled when only Snowflake writes to Snowflake storage; Automatic Clustering billed separately | Snowflake docs | | Starburst Galaxy | No: ALTER TABLE ... EXECUTE optimize; files under 100 MB by default are merged | You issue the command; managed ingest table maintenance 0.30 credits per hour | Starburst docs | Vendors alphabetical, AWS services first. Facts read 2026-11-10 from each vendor's documentation. This table lists automation and does not compare quality or cost. ## Open-source engines and EMR Open-source Iceberg, Delta Lake and Hudi leave compaction to a job you schedule, with settings such as Delta auto compaction (128 MB target) and Hudi async compaction as partial exceptions. Amazon EMR runs those jobs as Spark on EC2, EKS or Serverless and does not schedule them. The format pages are [Iceberg compaction](/maintenance/iceberg-compaction/), [Delta OPTIMIZE and VACUUM](/maintenance/delta-optimize-and-vacuum/) and [Hudi table services](/maintenance/hudi-table-services/). Microsoft Fabric documents OPTIMIZE and an auto compaction setting for Delta tables; managed Iceberg compaction for Fabric: none found on 2026-11-10 in the pages read, and Fabric does not run inside AWS. ## What the managed options leave to you Snapshot expiry and orphan file removal are separate services on some platforms and none were found on others in the pages read; see [snapshot expiry and orphan files](/maintenance/snapshot-expiry-and-orphan-files/). S3 Tables snapshot management fails on tables with user-defined tags or branches. Glue deletes only orphans created after the optimizer was created. Snowflake says it does not support orphan file deletion for Snowflake-managed Iceberg tables. Databricks predictive optimization skips tables outside Unity Catalog management. ## DualBird's own compaction benchmark DualBird, a Spark plugin vendor, published a sort-based Iceberg compaction benchmark in a blog post dated June 11, 2026, and a white paper landing page dated July 1, 2026. DualBird ran it, and no third-party run was found on 2026-11-10. The dataset is synthetic: about 370 million rows, 64 mixed-type columns, ZSTD level 3, 100 GB. The baselines are vanilla Spark and an unnamed C++ accelerated Spark. DualBird says it measured roughly 12 to 20 times faster single-task execution, 55% to 85% lower EC2 cost, and a compaction cost of about $2 per TB against about $6 per TB rising past $13 per TB on CPU clusters. It attributes part of the saving to F2 local NVMe replacing EBS volumes. The DualBird home page shows a tile reading X15-X20 for what appears to be the same test, so the two published ranges differ. The white paper with the method is behind a sign-up form, and the blog gives no instance counts, Spark version, configuration or scripts. DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. ## Other maintenance pages - [Iceberg compaction: rewrite_data_files, strategies, platforms](/maintenance/iceberg-compaction/) - [Iceberg snapshot expiry and orphan file removal](/maintenance/snapshot-expiry-and-orphan-files/) - [Delta Lake OPTIMIZE and VACUUM: defaults and platform automation](/maintenance/delta-optimize-and-vacuum/) - [Apache Hudi table services: compaction, clustering, cleaning](/maintenance/hudi-table-services/) ## Sources - Athena Iceberg optimization: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Amazon S3 Tables maintenance: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - AWS price list, Amazon S3, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - AWS Glue compaction optimization: https://docs.aws.amazon.com/glue/latest/dg/compaction-management.html (read 10 Nov 2026) - AWS price list, AWS Glue, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - BigQuery Iceberg tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Databricks predictive optimization: https://docs.databricks.com/aws/en/optimizations/predictive-optimization (read 10 Nov 2026) - Dremio Cloud table optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Snowflake Iceberg table management: https://docs.snowflake.com/en/user-guide/tables-iceberg-manage (read 10 Nov 2026) - Snowflake Iceberg storage: https://docs.snowflake.com/en/user-guide/tables-iceberg-storage (read 10 Nov 2026) - Starburst Galaxy Iceberg: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - Starburst Galaxy pricing details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Onehouse about: https://www.onehouse.ai/about-us (read 10 Nov 2026) - Fabric table compaction: https://learn.microsoft.com/en-us/fabric/data-engineering/table-compaction (read 10 Nov 2026) - DualBird, Iceberg compaction benchmark blog, 2026-06-11: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird, compaction white paper landing page: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/maintenance/compaction-on-each-platform/ --- # How Lakehouse Compare works Method, read 10 Nov 2026 **The short answer.** Lakehouse Compare lists ten lakehouse platforms and three table formats and puts none of them in an order of merit. Every fact carries a source link and a read date, claims are labelled with whoever made them, and the Compute Bill prices one reference workload on 12 lines in two groups, each alphabetical. ## What the site does and does not do The site lists facts about ten ways to run Spark batch work on a lakehouse and about Apache Iceberg, Delta Lake and Apache Hudi. It does not recommend an option and does not say which one a reader should choose. Pages end with facts, not with a pick. A buyer situation page says what a team in that situation should check and which published facts answer each check. ## How facts are sourced, dated and labelled Each fact has a source URL and a read date in the form YYYY-MM-DD. The read date is the day the page was fetched, and facts are re-read when a page is updated. A claim made by a vendor is written as the vendor's claim ("DualBird says", "per Databricks documentation", "AWS states"). Every vendor, DualBird included, is labelled the same way. A claim stays a claim until a named party other than the vendor has repeated it, and then the page names that party. ## The none-found rule When a fact was looked for in the sources named and was not there, the page says "none found on 2026-11-10" (or the date of the search). The page does not write that a vendor has no such feature, because the absence applies to the pages read, not to the product. ## The benchmark rule A benchmark appears only as a number that a named party published, with the link, the publication date and who ran the test. Numbers are not derived, averaged across sources or estimated. A result run by the vendor of the product tested is marked as such, together with whether the method is published. ## Order rules - Platforms are listed alphabetically by name, and so are vendors inside every situation check. - Table formats are listed alphabetically. - The Compute Bill has two groups: lines that price the same ten EC2 nodes (the open stack on EC2, EMR on EKS, EMR on EC2, EMR Serverless and Databricks on AWS) and lines whose billing unit is mapped to the job by a stated assumption (the other seven). Both groups are alphabetical, and a line with no total is shown in place with the missing price stated. Totals in the mapped group are read against their own assumptions, not against each other. - Guides, glossary entries and news items are ordered by date or alphabetically, as each page states. ## Corrections A reader who finds a wrong fact, a broken source link or an out-of-date price can write to editor@datalakehousecompare.com. A correction is made on the page, with the date of the change, and the correction page lists it. ## Sources - AWS EMR pricing page: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/method/ --- # About Lakehouse Compare About, read 10 Nov 2026 **The short answer.** Lakehouse Compare is a reference publication for teams choosing or re-choosing a lakehouse platform. It lists ten ways to run Spark batch work, the Iceberg, Delta Lake and Hudi table formats, and the Lakehouse Compute Bill, which prices one reference workload from published prices with every assumption stated. ## What the publication covers Lakehouse Compare covers lakehouse platforms and table formats for a staff data engineer who owns the Spark cluster bill. The platforms are Amazon EMR plus Athena on Iceberg, Databricks, Dremio, DualBird, Google BigQuery and BigLake, Microsoft Fabric, Onehouse, an open stack on EMR and EKS, Snowflake and Starburst. The formats are Apache Iceberg, Delta Lake and Apache Hudi. Maintenance pages cover compaction and table upkeep. ## Its method Facts come from the vendors' and projects' own documentation, price lists and release pages, each cited with a read date. The [method page](/method/) sets out how claims are labelled, what "none found" means and how the Compute Bill is ordered. The [Compute Bill](/compute-bill/) prices one Spark batch workload on each way of running it, using published prices only. ## Sources - Amazon EMR plus Athena on Iceberg: aws.amazon.com/emr: https://aws.amazon.com/emr/ (read 10 Nov 2026) - Databricks: databricks.com: https://www.databricks.com/ (read 10 Nov 2026) - Dremio: dremio.com: https://www.dremio.com/ (read 10 Nov 2026) - DualBird: dualbird.io: https://dualbird.io/ (read 10 Nov 2026) - Google BigQuery and BigLake: cloud.google.com/bigquery: https://cloud.google.com/bigquery (read 10 Nov 2026) - Microsoft Fabric: microsoft.com/en-us/microsoft-fabric: https://www.microsoft.com/en-us/microsoft-fabric (read 10 Nov 2026) - Onehouse: onehouse.ai: https://www.onehouse.ai/ (read 10 Nov 2026) - Open stack on EMR and EKS: spark.apache.org: https://spark.apache.org/ (read 10 Nov 2026) - Snowflake: snowflake.com: https://www.snowflake.com/ (read 10 Nov 2026) - Starburst: starburst.io: https://www.starburst.io/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/about/ --- # Editorial standards Editorial policy, read 10 Nov 2026 **The short answer.** Pages on Lakehouse Compare are written from primary sources: vendor documentation, price lists, project release pages and published benchmarks. Each fact is dated, claims are labelled with whoever made them, numbers are never derived or averaged, and no page tells a reader which option to choose. Corrections are published with a date. ## How a page is written A page starts from a source fetched on a stated date. The writer records the URL and the read date, then writes the fact in plain sentences with the number, the unit and the party that published it. Vendor claims are written as claims. A figure is not restated in other units, summed or averaged. ## What a page never does - Put options in an order of merit, or name a pick. - Estimate a number that a source did not publish. - Describe a gap as a missing feature when the fact is only that nothing was found in the pages read. ## Scope The site covers lakehouse platforms, table formats, compute pricing and table maintenance. Data catalogs in the governance sense, lineage, semantic layers and MCP servers for data are outside the subject. Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/editorial-policy/ --- # Corrections Corrections, read 10 Nov 2026 **The short answer.** Send a correction to editor@datalakehousecompare.com with the page address, the sentence that is wrong and a link to the source that shows the right fact. Corrections are made on the page itself, dated, and recorded here. A price or benchmark changes only when its published source changes or was misread. ## How to report an error Write to editor@datalakehousecompare.com. Include the page address, the sentence in question and a source link. A reply says whether the page was changed. ## What happens next A confirmed error is fixed on the page and the fix is dated. When the fix changes a number in the Compute Bill, the formula and the assumptions on the Compute Bill page are updated with it. No corrections have been recorded as of 2026-11-10. Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/corrections/ --- # Contact Contact, read 10 Nov 2026 **The short answer.** Write to editor@datalakehousecompare.com for corrections, questions about a source, a missing platform or an out-of-date price. Include the page address and, for a correction, a link to the source. Messages are read by the editors of Lakehouse Compare, and a reply states whether a page was changed. ## Email editor@datalakehousecompare.com is the one address for corrections, source questions and suggestions for platforms or formats to add. ## What to include - The page address. - The sentence or figure in question. - A link to the source that shows the right fact, with the date you read it. Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/contact/ --- # Privacy Privacy, read 10 Nov 2026 **The short answer.** Lakehouse Compare uses Google Analytics 4, which sets cookies, to count visits. The site is served through Amazon CloudFront, whose standard access logs are used only to count page views and crawler fetches. An email sent to editor@datalakehousecompare.com is used only to answer it and to handle a correction. ## Cookies and scripts The site loads one third-party script, the Google Analytics 4 tag (gtag.js), which sets cookies. It loads no advertising tag and no third-party widget. ## Google Analytics Lakehouse Compare uses Google Analytics 4 to count visits. Through cookies, it records the pages viewed, the page that sent the visitor (the referrer), the device and browser, and an approximate location worked out from the IP address. Google processes this data; Google's page [How Google uses information from sites or apps that use our services](https://policies.google.com/technologies/partner-sites) describes how. To opt out, install the [Google Analytics opt-out browser add-on](https://tools.google.com/dlpage/gaoptout) (https://tools.google.com/dlpage/gaoptout), available for Chrome, Safari, Firefox and Microsoft Edge, or block cookies for this site in the browser's settings. ## Access logs CloudFront standard access logs record each request, including the time, the requested path, the response status and the visitor's IP address and user agent. They are used only to count page views and crawler fetches. ## Email A message sent to editor@datalakehousecompare.com is used to answer it and to check and record a correction. ## Sources - Google, How Google uses information from sites or apps that use our services: https://policies.google.com/technologies/partner-sites (read 21 Nov 2026) - Google Analytics opt-out browser add-on: https://tools.google.com/dlpage/gaoptout (read 21 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/privacy/ --- # Terms of use Terms, read 10 Nov 2026 **The short answer.** Lakehouse Compare is a reference publication. Each page states the date its facts were read, and prices, versions and features change after that date, so the source link on each fact is the place to confirm it. Short quotations with a link back to the page are welcome; product names belong to their owners. ## Use of the content Pages may be read, linked and quoted in part with a link back to the page. Bulk copying of the pages or the data files is not permitted without written agreement from the editors. ## Dates and sources Every fact carries a read date. Prices, release numbers and feature lists change after that date, and the source link on the fact shows the current state. ## Names Product, company and project names are used to identify what a fact is about and belong to their owners. Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/terms/ --- # Guides to choosing and running a lakehouse Guides, read 10 Nov 2026 **The short answer.** Ten guides cover what a lakehouse is, Databricks and Snowflake facts side by side, Iceberg and Delta Lake, EMR cost levers, choosing a table format, the open stack on AWS, leaving a managed platform, pricing models, table catalogs and where Spark batch work runs. Each guide cites sources with read dates and ends with checks, never a pick. ## All 10 guides, A to Z - [Choosing a table format: what to check for Iceberg, Delta, Hudi](/guides/choosing-a-table-format/): Checks before choosing Iceberg, Delta Lake or Hudi: writers, maintenance, platform support and exit cost, each with published facts and sources. - [Databricks vs Snowflake: published facts side by side](/guides/databricks-vs-snowflake-facts/): Documented facts for Databricks and Snowflake side by side: Spark code, billing units, table formats, maintenance and regions, with dates and sources. - [EMR cost optimization for an Iceberg lakehouse](/guides/emr-cost-optimization-lakehouse/): The EMR cost levers for Iceberg tables: instance choice, Spot, Graviton, managed scaling and compaction cost, with AWS list prices read 2026-11-10. - [Iceberg vs Delta Lake: documented facts, no pick](/guides/iceberg-vs-delta-lake/): Apache Iceberg and Delta Lake side by side: latest releases, spec features, compaction defaults, Spark compatibility and cross-reading, all from project docs. - [Lakehouse pricing models: DBUs, credits, slots, CUs and EMR uplift](/guides/lakehouse-pricing-models/): How each lakehouse platform bills (DBU, credit, slot, capacity unit, DCU, OCU, EC2 plus EMR uplift) with the dollar rates that could be read on 2026-11-10. - [Leaving Databricks or Snowflake: what moves and what does not](/guides/leaving-a-managed-platform/): What moves when leaving Databricks or Snowflake for an open stack: table files, Spark code, maintenance, catalogs and billing, with documentation facts. - [Table catalogs for lakehouse engines: Glue, Unity, Polaris, Nessie](/guides/lakehouse-catalogs-for-tables/): Table catalogs for Spark and other engines: AWS Glue Data Catalog, S3 Tables, Unity Catalog open source, Apache Polaris and Nessie, with documented interfaces. - [The open stack on AWS: Spark, table formats and catalogs](/guides/open-stack-on-aws/): The parts of an open lakehouse stack on AWS: EMR and EKS Spark, Iceberg, Delta and Hudi bundles, the Glue Data Catalog, S3 Tables and what you operate. - [What is a data lakehouse: definition, origins, 2021 paper](/guides/what-is-a-data-lakehouse/): Definition of a data lakehouse, where the term came from (the CIDR 2021 paper), and how warehouse, lake and lakehouse differ, with each claim tied to its source. - [Where Spark batch work runs on each lakehouse platform](/guides/batch-spark-on-a-lakehouse/): Where Spark batch ETL runs on AWS, BigQuery, Databricks, Dremio, Fabric, Onehouse, Snowflake and Starburst, with versions, billing units and limits. Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/guides/ --- # What is a data lakehouse Guide, read 10 Nov 2026 **The short answer.** A data lakehouse keeps data in open files on cheap object storage and adds a transactional table layer so warehouse-style SQL and machine learning read the same copy. The term comes from a CIDR paper presented January 11 to 15, 2021 by four Databricks-affiliated authors. ## The definition the term started with The word lakehouse entered the research literature in "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics" by Michael Armbrust, Ali Ghodsi, Reynold Xin and Matei Zaharia. It appeared at the 11th Annual Conference on Innovative Data Systems Research (CIDR), held online January 11 to 15, 2021. The authors list Databricks, UC Berkeley and Stanford as their affiliations. The paper defines the pattern by three traits: open direct-access data formats such as Apache Parquet and ORC, first-class support for machine learning and data science workloads, and performance the authors describe as competitive with cloud data warehouses. That last trait is the authors' own claim, and the paper treats SQL performance on open files as its largest open technical question. Databricks's own glossary page gives a looser, vendor-written version: a lakehouse combines the flexibility, cost-efficiency and scale of data lakes with the data management and ACID transactions of data warehouses. Both definitions agree on the mechanism. The data stays in files you can read without the vendor's engine, and a metadata layer decides which files make up a table. ## Where the idea came from The paper describes three generations. In the first, data was loaded into a warehouse with schema-on-write, and compute and storage were coupled in on-premises appliances. In the second, raw data was offloaded into data lakes holding open files, starting with Hadoop and HDFS, with a small subset later copied into a warehouse for business intelligence. From 2015, the authors write, cloud object stores such as S3, ADLS and GCS replaced HDFS. They call the resulting pair of a cloud lake plus a downstream warehouse the two-tier architecture and say it was dominant in their experience. The paper cites a survey by Dimensional Research and Fivetran stating that 86% of analysts use out-of-date data. That survey figure is the paper's citation, not something this site measured. The lakehouse is the proposal to collapse the two tiers: keep one copy on the object store, and give it the management features that made warehouses useful, which the paper lists as ACID transactions, versioning and indexing. ## Warehouse, lake and lakehouse side by side | Trait | Data warehouse (paper's first generation) | Data lake (second generation) | Lakehouse (paper's proposal) | | --- | --- | --- | --- | | Write model | Schema-on-write | Schema-on-read | Files plus a transactional metadata layer | | Storage | Compute and storage coupled in on-premises appliances | HDFS, then cloud object stores | Cloud object store, open formats such as Parquet | | Transactions and versioning | Built in | Left to the user, per the paper | Provided by the table layer (Delta Lake, Iceberg, Hudi) | | Machine learning access | Export files or read over ODBC or JDBC | Direct file reads | Direct file reads with table semantics | | Copies of the data | One | Lake plus a warehouse copy | One, per the proposal | Characterizations are the CIDR 2021 paper's, not measurements. Read 2026-11-10. ## What a table format adds The paper's proposed implementation is a transactional metadata layer over an object store that defines which objects belong to a table version. It names Delta Lake, Apache Iceberg and Apache Hudi as systems in this area. It says Iceberg started at Netflix and Hudi at Uber, and it states that Delta Lake was then used in about half of Databricks's workload by thousands of customers. The paper also lists limits that still shape operations today: object stores have high latency, which caps the transaction rate, and Delta Lake, Iceberg and Hudi support transactions on one table at a time. Separately, data files on an object store are written once, so every write and compaction leaves older files behind until a cleanup job removes them. That is why the [maintenance pages](/maintenance/iceberg-compaction/) exist on this site. ## Performance evidence in the paper The paper reports one experiment: the Databricks Delta Engine, a SQL engine over Parquet, against popular cloud data warehouses on AWS, Azure and Google Cloud, using TPC-DS at scale factor 30K, with time and cost shown in its Figure 3. The authors ran it and they work at Databricks, so it is a vendor-run result. It is a SQL query benchmark, not a Spark batch ETL test, and this site does not restate its numbers. Read it as the origin of the claim that open-format files can match warehouse speed, not as a comparison of today's platforms. Later vendor results are listed on each [platform page](/platforms/) with who ran them. ## What the term leaves open Vendors now use the word for different products. Databricks applies it to its platform, Snowflake and BigQuery offer Iceberg tables on cloud storage, and AWS assembles Iceberg or Delta on S3 with EMR and Glue. The [formats pages](/formats/iceberg/) separate the table format from the platform that runs it. Questions that the definition does not answer, and that published facts can: which engines can write the table, who runs compaction, where the catalog lives, and what moving the data out would involve. The guides on [choosing a table format](/guides/choosing-a-table-format/) and [leaving a managed platform](/guides/leaving-a-managed-platform/) take those in turn. ## Sources - Armbrust, Ghodsi, Xin, Zaharia, "Lakehouse", CIDR 2021: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf (read 10 Nov 2026) - Databricks, What is a data lakehouse: https://www.databricks.com/glossary/data-lakehouse (read 10 Nov 2026) - Apache Iceberg home page: https://iceberg.apache.org/ (read 10 Nov 2026) - Delta Lake home page: https://delta.io/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/guides/what-is-a-data-lakehouse/ --- # Databricks vs Snowflake: the published facts, by situation Guide, read 10 Nov 2026 **The short answer.** Databricks bills DBUs per second and ships Spark 3.5 to 4.2 runtimes; Snowflake bills credits per second with a 60-second minimum and runs PySpark through Snowpark Connect on Spark 3.5 only. This page sets out each vendor's documented facts by situation and names no pick. ## Running existing Spark batch code Databricks runs Apache Spark in its own runtime. Its release notes list Databricks Runtime 18 LTS with Apache Spark 4.1.0 (released June 10, 2026), Runtime 17.3 LTS with Spark 4.0.0, and Runtime 16.4 LTS and 15.4 LTS with Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). Photon, its native C++ engine, keeps the Catalyst planner and falls back to the Spark runtime for unsupported operations. Snowflake does not run a Spark cluster. Snowpark Connect for Apache Spark sends PySpark DataFrame code over the Spark Connect protocol to a Snowflake warehouse, and batch jobs go in through the Snowpark Submit CLI. Snowflake documents support for Spark 3.5 only; code written for 3.4 or earlier, or for 4.0 and later, is listed as unsupported. The compatibility page lists DayTimeIntervalType, YearMonthIntervalType and user-defined types as unsupported. On Databricks serverless compute, only Spark Connect APIs are supported. RDD APIs, R, Maven coordinates and Databricks Container Services are listed as unsupported, ANSI mode is on by default and the default time zone is UTC. Either vendor can change results of an existing job; test a copy first. ## Billing units Databricks bills in DBUs, a normalized unit of processing power, per second, with pay-as-you-go or committed-use discounts. Photon instance types use DBUs at a different rate than the same instance without Photon. The pricing pages render their rates by script; the data file behind the pricing calculator lists Jobs Compute on AWS at $0.15 per DBU on the Premium plan and $0.20 on Enterprise, and 4.08 DBU per node-hour for r6i.4xlarge (read 2026-11-10). Snowflake bills credits per second with a 60-second minimum each time a warehouse starts, and each size step roughly doubles the credits per hour (an XL standard warehouse is 16). Snowflake's [Service Consumption Table](https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf), effective 2026-11-09, lists $2.00, $3.00 and $4.00 per credit for Standard, Enterprise and Business Critical on AWS US East (Northern Virginia). A per-DBU rate for Databricks: none found on 2026-11-10, so the two vendors' dollar rates are not set against each other here; see [lakehouse pricing models](/guides/lakehouse-pricing-models/). ## Table formats and open storage Databricks documents Delta Lake as its native format. UniForm lets Delta tables generate Iceberg metadata from Databricks Runtime 14.3 LTS so Iceberg clients can read them without rewriting data files. Snowflake supports Iceberg tables in Parquet with data in customer-managed S3, Google Cloud Storage or Azure storage through external volumes, with an external catalog integration such as AWS Glue. Delta files are read through Delta Direct. Hudi was not listed on the Snowflake page read. Compare the formats themselves on the [Iceberg vs Delta Lake page](/formats/delta-lake-vs-iceberg/). ## Table maintenance Databricks predictive optimization runs OPTIMIZE, VACUUM and ANALYZE automatically on Unity Catalog managed tables, Delta and Iceberg, and is on by default. It uses serverless compute and is billed through a serverless jobs SKU. It does not apply to tables outside Unity Catalog management. For Snowflake-managed Iceberg tables, Snowflake documents data compaction as on by default and billable, manifest compaction and snapshot expiry as automatic with no cost, and Automatic Clustering as off by default and billed separately. For Iceberg tables managed by an external catalog, Snowflake states it performs no maintenance and you must run compaction and snapshot expiry in your own engine. ## Cloud and region coverage Databricks runs on AWS, Azure and Google Cloud. The AWS region page lists regions including us-east-1, us-east-2, us-west-1, eu-west-1, eu-central-1, ap-southeast-2 and us-gov-west-1, and notes some features exist only in a subset of regions. Snowflake also runs on all three clouds. Its AWS regions include us-east-1, us-west-2, eu-west-1 and ca-central-1, and each account is hosted in a single region. ## Published performance claims Databricks announced an audited 100 TB TPC-DS result of 32,941,245 QphDS for Databricks SQL on November 2, 2021, which it says is 2.2 times the previous record held by Alibaba, with total system cost 10% lower on list pricing. It is a SQL warehouse result, not Spark batch ETL, and Databricks ran it. The same post cites research it attributes to the Barcelona Supercomputing Center; that study was not fetched. For Spark batch jobs on Snowpark Connect, no published benchmark with a stated method was found on 2026-11-10. For Databricks, none was found for Spark batch ETL with a method either, apart from the vendor claim that Photon gives up to 5 times better price/performance than other cloud data warehouses, which the documentation states without a method. ## Sources - Databricks Runtime release notes: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Databricks Photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks serverless compute limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks UniForm: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Databricks predictive optimization: https://docs.databricks.com/aws/en/optimizations/predictive-optimization (read 10 Nov 2026) - Databricks blog, TPC-DS record, 2021-11-02: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Databricks supported regions: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Databricks pricing data file (AWS), loaded by its pricing calculator: https://www.databricks.com/en-pricing-assets/data/pricing/AWS.json (read 10 Nov 2026) - Snowpark Connect for Apache Spark: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowpark Connect limitations: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Snowpark Connect compatibility: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - Snowflake compute cost: https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowflake Iceberg tables: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Snowflake Iceberg table management: https://docs.snowflake.com/en/user-guide/tables-iceberg-manage (read 10 Nov 2026) - Snowflake regions: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) - Snowflake Service Consumption Table: https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/guides/databricks-vs-snowflake-facts/ --- # Iceberg vs Delta Lake: documented facts Guide, read 10 Nov 2026 **The short answer.** Apache Iceberg 1.12.0 was released September 29, 2026 per its releases page, and Delta Lake 4.4.1 on October 5, 2026. Both are open table formats on Parquet files with snapshot isolation. This page sets out what each project documents about row-level deletes, compaction defaults, Spark versions and interoperability. ## Projects and latest releases Apache Iceberg describes itself as an open table format for analytic datasets that lets engines such as Spark, Trino, Flink, Presto, Hive and Impala work on the same tables at the same time. The latest release read through the GitHub API on 2026-11-10 was apache-iceberg-1.12.0, listed September 30, 2026 (the Iceberg releases page says September 29). Delta Lake's site says the project joined the Linux Foundation Projects in 2019 as a sub-project. The latest release read through the GitHub API was Delta Lake 4.4.1, published October 5, 2026. The project home page announces Delta Lake 4.4.0 on Apache Spark 4.2.0. The releases page lists Delta 4.3.x with Spark 4.0.x and 4.1.x, and Delta 3.3.x with Spark 3.5.x. ## Row-level deletes Iceberg spec version 3 adds binary deletion vectors, row lineage tracking, default column values and new types including variant, geometry and geography. Version 4 is described in the specification as restructuring metadata into an adaptive tree. Iceberg's home page says it can eagerly rewrite data files for read performance, or use delete deltas for faster updates. Delta Lake documents deletion vectors as an opt-in storage optimization. Without them, deleting one row rewrites the whole Parquet file that holds it. With them, the delete is recorded separately and the file is rewritten later. The page includes a table of which operations each Delta version supports. ## Compaction defaults | Item | Apache Iceberg | Delta Lake | | --- | --- | --- | | Compaction command | rewrite_data_files procedure or rewriteDataFiles action | OPTIMIZE | | Strategies documented | binpack (default), sort, and z-order or hilbert inside sort | Bin-packing; ZORDER BY; auto compaction | | Default target file size | 512 MB (write.target-file-size-bytes) | Auto compaction targets 128 MB by default | | Idempotent | Files already near target size are skipped unless the rewrite-all option is set | Bin-packing is idempotent; Z-ordering is not | | Runs automatically in the open-source project | No, you schedule it | Auto compaction is a setting you enable | Sources: Iceberg Spark procedures and configuration pages; Delta Lake file management page. Read 2026-11-10. ## Partitioning and clustering Iceberg's home page lists hidden partitioning as a feature, so queries need not name partition columns. For layout inside partitions, rewrite_data_files accepts a sort strategy with a sort_order such as zorder(c1,c2) or hilbert(c1,c2). The z-order option var-length-contribution defaults to 8 bytes taken from string and binary columns, and the Hilbert order gives every column its full primitive width of 8 bytes. Delta Lake tables are partitioned with PARTITIONED BY at creation. OPTIMIZE ... ZORDER BY colocates related rows, and the docs say the locality benefit drops with each extra column and that Z-ordering is not idempotent, so a repeat run reworks the partition's files again. Databricks adds liquid clustering on its platform, and its OPTIMIZE page says Iceberg tables there use clustering and sorting strategies. ## Cleanup and retention Iceberg defaults to keeping snapshots for 5 days (history.expire.max-snapshot-age-ms is 432,000,000 ms) and a minimum of 1 snapshot. Expiring snapshots is a step you run with expire_snapshots; the docs list it as recommended maintenance. Delta VACUUM is not triggered automatically. Its default retention threshold is 7 days, and the docs advise a retention interval of at least 7 days so that readers and writers are not disturbed. Delta log files are cleaned automatically after checkpoints, with a 30-day default log retention. The details sit on the pages for [Iceberg compaction](/maintenance/iceberg-compaction/), [snapshot expiry and orphan files](/maintenance/snapshot-expiry-and-orphan-files/) and [Delta OPTIMIZE and VACUUM](/maintenance/delta-optimize-and-vacuum/). ## Reading one format as the other Delta UniForm generates Iceberg and Hudi metadata for a Delta table asynchronously, so Iceberg and Hudi clients can read it. Databricks documents this from Runtime 14.3 LTS. Microsoft documents that OneLake in Fabric exposes Iceberg tables as Delta and Delta as Iceberg through metadata virtualization. Apache XTable, an incubating Apache project, translates table metadata between Hudi, Delta Lake and Iceberg without introducing a new format. Onehouse says it created XTable and Hudi. ## Engine and platform support to verify On Amazon EMR 7.14.0, AWS lists Iceberg 1.10.1-amzn-1, Delta 3.3.2-amzn-3 and Hudi 1.0.2-amzn-3 with Spark 3.5.8-amzn-0. That means the bundled Delta version is behind the project's latest release, and the bundled Iceberg version is behind 1.12.0. Check the version your platform bundles, not the project's newest. Starburst Galaxy reads and writes Iceberg, and treats Hudi as read-only. Dremio Cloud lists Iceberg and Delta Lake, and its page read does not list Hudi. The [platform pages](/platforms/) carry each vendor's entries. ## Sources - Apache Iceberg home page: https://iceberg.apache.org/ (read 10 Nov 2026) - Apache Iceberg latest release (GitHub API): https://api.github.com/repos/apache/iceberg/releases/latest (read 10 Nov 2026) - Apache Iceberg table specification: https://iceberg.apache.org/spec/ (read 10 Nov 2026) - Apache Iceberg Spark procedures: https://iceberg.apache.org/docs/latest/spark-procedures/ (read 10 Nov 2026) - Apache Iceberg configuration: https://iceberg.apache.org/docs/latest/configuration/ (read 10 Nov 2026) - Delta Lake home page: https://delta.io/ (read 10 Nov 2026) - Delta Lake latest release (GitHub API): https://api.github.com/repos/delta-io/delta/releases/latest (read 10 Nov 2026) - Delta Lake releases and Spark compatibility: https://docs.delta.io/latest/releases.html (read 10 Nov 2026) - Delta Lake file management: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Delta Lake utility commands (VACUUM): https://docs.delta.io/latest/delta-utility.html (read 10 Nov 2026) - Delta Lake deletion vectors: https://docs.delta.io/latest/delta-deletion-vectors.html (read 10 Nov 2026) - Delta Lake UniForm: https://docs.delta.io/latest/delta-uniform.html (read 10 Nov 2026) - Amazon EMR 7.x release versions: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Apache XTable: https://xtable.apache.org/ (read 10 Nov 2026) - Starburst Galaxy Hudi: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - Dremio Cloud data formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Databricks UniForm: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Fabric OneLake Iceberg tables: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Databricks predictive optimization: https://docs.databricks.com/aws/en/optimizations/predictive-optimization (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/guides/iceberg-vs-delta-lake/ --- # EMR cost optimization when the tables are Iceberg Guide, read 10 Nov 2026 **The short answer.** On Amazon EMR on EC2 the bill is EC2 plus an EMR uplift per instance plus EBS, billed per second. For an Iceberg lakehouse the levers AWS documents are instance family, Spot, Graviton, managed scaling and how compaction is run. This page lists the published prices and the docs behind each lever. ## What the EMR bill is made of AWS states that on EMR on EC2 the EMR price is added to the EC2 price and to the EBS price if volumes are attached, all billed per second with a one-minute minimum. The EMR uplift is a separate line per instance type. On the AWS price list for us-east-1 (publication 2026-11-10), it is $0.192 per hour for m5.4xlarge, $0.1632 for m7g.4xlarge, $0.2142 for r7g.4xlarge and $0.27 for r5d.4xlarge. EMR on EKS replaces the per-instance uplift with a charge per vCPU-hour ($0.007084) and per GB-hour ($0.000777875) added to EC2 or Fargate, plus $0.10 per hour per EKS cluster. EMR Serverless bills vCPU-hours and GB-hours of workers; AWS's worked example uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour. The [compute bill](/compute-bill/) page carries these in a priced workload. ## Instance choice | Instance (us-east-1) | vCPU / memory | EC2 On-Demand per hour | EMR uplift per hour | | --- | --- | --- | --- | | m5.4xlarge | 16 / 64 GiB | $0.768 | $0.192 | | m6g.4xlarge | 16 / 64 GiB | $0.616 | $0.154 | | m7g.4xlarge | 16 / 64 GiB | $0.6528 | $0.1632 | | r5d.4xlarge | 16 / 128 GiB | $1.152 | $0.27 | | r6g.4xlarge | 16 / 128 GiB | $0.8064 | $0.2016 | | r7g.4xlarge | 16 / 128 GiB | $0.8568 | $0.2142 | Source: AWS price lists for Amazon EC2 and Amazon EMR, us-east-1, publication 2026-11-10. Prices only; no performance is implied. ## Spot and Graviton AWS says Spot Instances are available at a discount of up to 90% off On-Demand, and the Spot price moves, so the Spot Instance Advisor is the place to read a current figure. The EMR docs advise caution on which nodes go on Spot: you cannot change a running cluster's purchasing option, and by default EMR keeps YARN application masters off task nodes so that losing Spot task nodes does not fail running jobs. Core nodes on Spot risk partial HDFS data loss. AWS states Graviton-based instances cost up to 20% less than comparable x86 instances. The price list above shows the EC2 gap for the same 16 vCPU and 64 GiB shape: $0.6528 for m7g.4xlarge against $0.768 for m5.4xlarge. Whether your Spark jobs and libraries run the same on arm64 is a test you have to run; no AWS benchmark for Iceberg compaction on Graviton was found on 2026-11-10. ## Managed scaling and instance fleets EMR managed scaling (EMR 5.30.0 and later, except 6.0.0) raises and lowers core and task capacity from workload metrics. You set a minimum, a maximum, an optional On-Demand limit that sends the rest to Spot, and an optional maximum for core nodes. It works with YARN applications such as Spark. AWS advises keeping Spark dynamic resource allocation on, because turning it off can scale a cluster up to its maximum, and keeping EBS use under 90%. Instance fleets let one fleet name up to five EC2 instance types and an allocation strategy such as capacity-optimized, lowest-price or diversified. That widens the Spot pools a fleet can draw from. ## Compaction is a cost line too Compaction on EMR is a Spark job you schedule, so it consumes the same EC2 plus uplift hours. Apache Iceberg's defaults give a file group a rewrite when it holds 5 or more files, and a rewritten group is capped at 100 GB. The AWS-managed alternatives bill by their own meters: Glue Data Catalog Iceberg compaction costs $0.44 per DPU-hour in us-east-1, and S3 Tables compaction costs $0.005 per GB processed plus $0.002 per 1,000 objects compacted, or $0.01 per GB for sort and z-order. Which of those is lower for a given table depends on file counts, sort strategy and cluster shape. No published benchmark of Iceberg compaction cost on EMR against these services was found on 2026-11-10. The [maintenance pages](/maintenance/compaction-on-each-platform/) compare what each platform automates. ## Plugins and hardware options on EMR AWS publishes its own runtime figures: on TPC-DS at 3 TB with 9 r5d.4xlarge nodes, AWS reports EMR 7.12 running in 1,349.62 seconds on average against 6,113.92 for open-source Spark 3.5.6 with Iceberg 1.10.0, and a run cost of $4.81 against $17.65. AWS ran it, notes it is not comparable to official TPC-DS, and publishes the setup in GitHub repositories. DualBird says its Spark plugin runs on AWS F2 FPGA instances and reports a 100 GB sort-based Iceberg compaction test in a June 11, 2026 blog post, with about $2 per TB against about $6 to over $13 per TB on CPU clusters. DualBird ran that test and has not published its method outside a sign-up form. DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. ## Sources - Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - AWS price list, Amazon EMR, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - AWS price list, Amazon EC2, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json (read 10 Nov 2026) - Amazon EC2 Spot pricing: https://aws.amazon.com/ec2/spot/pricing/ (read 10 Nov 2026) - AWS Graviton: https://aws.amazon.com/ec2/graviton/ (read 10 Nov 2026) - Amazon EMR managed scaling: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-scaling.html (read 10 Nov 2026) - Amazon EMR instance types and Spot guidance: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-instances-guidelines.html (read 10 Nov 2026) - Amazon EMR instance fleets: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-instance-fleet.html (read 10 Nov 2026) - Apache Iceberg Spark procedures: https://iceberg.apache.org/docs/latest/spark-procedures/ (read 10 Nov 2026) - AWS price list, AWS Glue, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - AWS price list, Amazon S3, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - AWS Big Data Blog, EMR 7.12 and Iceberg benchmark: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - DualBird, Iceberg compaction benchmark blog, 2026-06-11: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/guides/emr-cost-optimization-lakehouse/ --- # Choosing a table format: the checks and the published facts Guide, read 10 Nov 2026 **The short answer.** Choosing among Apache Iceberg, Delta Lake and Apache Hudi comes down to which engines must write, who runs maintenance, and what each platform you rely on supports. This page lists those checks with the documented facts for each format, and names no pick. ## Check 1: which engines must write the table List the engines that write, not only the ones that read. Iceberg's home page names Spark, Trino, Flink, Presto, Hive and Impala as engines that can work on the same tables. Delta Lake's home page names Spark, PrestoDB, Flink, Trino, Hive, Snowflake, Google BigQuery, Athena, Redshift, Databricks and Fabric, with Scala, Java, Rust and Python APIs. Platform support differs by format. Starburst Galaxy creates and writes Iceberg tables but states that it cannot create or write Hudi tables. Dremio Cloud's data-formats page lists Iceberg and Delta Lake and does not list Hudi. Snowflake's Iceberg page did not list Hudi. Fabric's OneLake exposes Iceberg tables as Delta and Delta tables as Iceberg through metadata virtualization. ## Check 2: who runs maintenance Every format needs file compaction and cleanup, and who does it differs by platform. Open-source Iceberg leaves compaction to a job you schedule, with expiring snapshots listed as recommended and rewriting data files as optional. Hudi merge-on-read tables need compaction to fold log files into base files, and Hudi runs cleaning inline after each write by default, keeping the last 10 commits. Delta needs OPTIMIZE and VACUUM, and VACUUM is not triggered automatically. Managed options exist for each. Databricks predictive optimization covers Delta and Iceberg tables under Unity Catalog. S3 Tables and the Glue Data Catalog cover Iceberg only. Onehouse offers managed table services for Hudi, Iceberg and Delta. See [compaction on each platform](/maintenance/compaction-on-each-platform/). ## Check 3: what your AWS runtime bundles The version in your runtime is the version you get. Amazon EMR 7.14.0 packages Iceberg 1.10.1-amzn-1, Delta 3.3.2-amzn-3 and Hudi 1.0.2-amzn-3 with Spark 3.5.8-amzn-0. Project releases read on 2026-11-10 were Iceberg 1.12.0 (September 29, 2026 on the project releases page), Delta Lake 4.4.1 (October 5, 2026) and Hudi 1.2.1 (September 24, 2026). Delta's compatibility table pairs Delta 4.x releases with Spark 4.0 and 4.1 and the 3.x line with Spark 3.5. A Spark 3.5 runtime therefore constrains which Delta line you can run. The same version check applies to Hudi and Iceberg bundles. ## Check 4: whether you need to read across formats Delta UniForm generates Iceberg and Hudi metadata for Delta tables, and Databricks documents it from Runtime 14.3 LTS. Apache XTable translates metadata between Hudi, Delta and Iceberg and is incubating at the Apache Software Foundation. Both leave the Parquet data files in place. Cross-reading has costs: Snowflake states that for Iceberg tables managed by an external catalog it does no maintenance, so metadata refresh has to be lined up with whichever engine maintains the table. Read the [Delta Lake vs Iceberg](/formats/delta-lake-vs-iceberg/) page for the feature rows. ## Check 5: what leaving would involve Open formats keep data in files you own, but exit cost sits in the catalog, the maintenance jobs and any proprietary features layered on the format. Databricks predictive optimization works only on Unity Catalog managed tables. Snowflake storage-managed Iceberg tables can be moved to your own storage with CTAS at standard rates. Walk through the specifics in [leaving a managed platform](/guides/leaving-a-managed-platform/). ## The maintenance load per format | Format | Compaction | Cleanup | Layout | | --- | --- | --- | --- | | Apache Hudi | Compaction folds log files into base files; runs asynchronously by default | Cleaner keeps the last 10 commits by default; timeline archival keeps up to 30 commits in the active timeline | Clustering rewrites files and records a replace commit | | Apache Iceberg | rewrite_data_files, binpack by default | expire_snapshots (5 days by default); remove_orphan_files (3 days by default) | Sort strategy with sort_order, z-order or hilbert | | Delta Lake | OPTIMIZE; auto compaction targets 128 MB | VACUUM, 7 days by default | OPTIMIZE ... ZORDER BY | Sources: Hudi, Iceberg and Delta Lake project documentation, read 2026-11-10. Defaults are the documented project defaults, not any platform's. ## Check 6: row-level change patterns If the table takes frequent updates or deletes, check the delete mechanism. Iceberg spec version 3 adds binary deletion vectors; Delta documents deletion vectors as opt-in; Hudi offers copy-on-write and merge-on-read table types, with merge-on-read the one that needs compaction. Hudi's default compaction strategy picks the file groups with the most accumulated unmerged log data, and its trigger can be a count of delta commits, elapsed seconds, or both. Starburst's Iceberg optimize merges files under a 100 MB threshold by default. Iceberg's own rewrite starts a group at 5 input files. Thresholds like these decide how often compaction runs, so write them down per format before you compare cost. ## Sources - Apache Iceberg home page: https://iceberg.apache.org/ (read 10 Nov 2026) - Delta Lake home page: https://delta.io/ (read 10 Nov 2026) - Apache Hudi table types: https://hudi.apache.org/docs/table_types (read 10 Nov 2026) - Apache Hudi compaction: https://hudi.apache.org/docs/compaction (read 10 Nov 2026) - Apache Hudi cleaning: https://hudi.apache.org/docs/cleaning (read 10 Nov 2026) - Amazon EMR 7.x release versions: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Apache Iceberg latest release (GitHub API): https://api.github.com/repos/apache/iceberg/releases/latest (read 10 Nov 2026) - Delta Lake latest release (GitHub API): https://api.github.com/repos/delta-io/delta/releases/latest (read 10 Nov 2026) - Apache Hudi latest release (GitHub API): https://api.github.com/repos/apache/hudi/releases/latest (read 10 Nov 2026) - Delta Lake releases and Spark compatibility: https://docs.delta.io/latest/releases.html (read 10 Nov 2026) - Starburst Galaxy Hudi: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - Starburst Galaxy Iceberg: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - Dremio Cloud data formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Snowflake Iceberg tables: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Snowflake Iceberg table management: https://docs.snowflake.com/en/user-guide/tables-iceberg-manage (read 10 Nov 2026) - Snowflake Iceberg storage: https://docs.snowflake.com/en/user-guide/tables-iceberg-storage (read 10 Nov 2026) - Databricks predictive optimization: https://docs.databricks.com/aws/en/optimizations/predictive-optimization (read 10 Nov 2026) - Databricks UniForm: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Apache XTable: https://xtable.apache.org/ (read 10 Nov 2026) - Apache Iceberg table specification: https://iceberg.apache.org/spec/ (read 10 Nov 2026) - Delta Lake deletion vectors: https://docs.delta.io/latest/delta-deletion-vectors.html (read 10 Nov 2026) - Fabric OneLake Iceberg tables: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Onehouse about: https://www.onehouse.ai/about-us (read 10 Nov 2026) - Apache Hudi timeline and archival: https://hudi.apache.org/docs/timeline (read 10 Nov 2026) - Apache Hudi clustering: https://hudi.apache.org/docs/clustering (read 10 Nov 2026) - Apache Iceberg Spark procedures: https://iceberg.apache.org/docs/latest/spark-procedures/ (read 10 Nov 2026) - Delta Lake utility commands (VACUUM): https://docs.delta.io/latest/delta-utility.html (read 10 Nov 2026) - Delta Lake file management: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/guides/choosing-a-table-format/ --- # The open stack on AWS: Spark on EMR or EKS with Iceberg, Delta or Hudi Guide, read 10 Nov 2026 **The short answer.** An open stack on AWS runs unmodified Apache Spark on EMR or EKS over Iceberg, Delta Lake or Hudi tables in S3, with the Glue Data Catalog or S3 Tables as catalog. AWS documents each part. This page lists the components, versions and who operates what. ## The compute layer: EMR on EC2, EKS or Serverless AWS describes the EMR runtime for Spark as 100% API-compatible with open-source Spark, and EMR runs on EC2, on EKS or as Serverless. EMR on EKS lets teams run EMR-based applications beside other workloads on one EKS cluster, with Airflow or MWAA for scheduling. It supports the Kubernetes Operator for Apache Spark as a job submission model on your own clusters. EMR Serverless needs no cluster configuration: it determines the resources a job needs and releases them afterward. Pricing for all three modes is on the [EMR pricing page](https://aws.amazon.com/emr/pricing/) and summarized in [EMR cost optimization](/guides/emr-cost-optimization-lakehouse/). ## The table layer: what EMR bundles Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0 with Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. The EMR Spark feature page says EMR supports Apache Iceberg v3 and Spark 4.0. All three formats are available; you are not forced into one by the platform. For Hudi, EMR installs the Hudi components by default when Spark, Hive, Presto or Flink is installed, from release 5.28.0, and Hudi is integrated with Trino from 6.1.0. The Delta Lake page lists Delta on EMR from release 6.9.0. ## The catalog layer: Glue Data Catalog and S3 Tables The AWS Glue Data Catalog stores table metadata and integrates with Athena, Lake Formation, EMR and SageMaker. Its Iceberg REST endpoint implements the Apache Iceberg REST specification, supports table specifications v1 and v2 with v2 the default, and for us-east-1 is the endpoint `glue.us-east-1.amazonaws.com/iceberg` (an API address, not a web page). S3 Tables stores tables in the Iceberg format in table buckets. Glue ETL jobs can reach S3 Tables through the analytics-services integration, through the S3 Tables Iceberg REST endpoint, or through a Java catalog library. S3 Tables needs Glue version 5.0 or higher, Iceberg v3 needs Glue 5.1 or higher, and the v3 data types need Glue 6.0 or higher. The [catalogs guide](/guides/lakehouse-catalogs-for-tables/) covers the alternatives. ## What you operate In an open stack you own the maintenance. Iceberg's documentation lists expiring snapshots, removing old metadata files and deleting orphan files as recommended, and compaction and manifest rewriting as optional, all as jobs you schedule. AWS offers managed versions: Glue table optimizers for Iceberg compaction, snapshot retention and orphan file deletion, and S3 Tables maintenance on by default. Athena offers manual OPTIMIZE and VACUUM statements. You also own Spark tuning, upgrades and the choice of purchase options. EMR enables adaptive query execution, dynamic partition pruning and bloom filter joins, and the docs say adaptive join conversion is on by default. ## Published performance and cost for this stack The one published run of this stack is AWS's own: a TPC-DS 3 TB test on 9 r5d.4xlarge nodes with a Hadoop catalog, in which EMR 7.12 averaged 1,349.62 seconds and open-source Spark 3.5.6 with Iceberg 1.10.0 averaged 6,113.92. AWS's stated costs were $4.81 and $17.65 on On-Demand prices. The setup is in the aws-samples/emr-on-eks-benchmark and aws-samples/emr-spark-benchmark repositories, and AWS notes the results are not comparable to official TPC-DS. No third-party run of the open stack against managed lakehouses was found on 2026-11-10, so cost comparisons on this site stay in the [compute bill](/compute-bill/), which uses published prices and states every assumption. ## Kubernetes and the Spark operator EMR on EKS applies its uplift to any pod that uses an EMR runtime image, including pods outside job submission. The Kubeflow Spark Operator that EMR on EKS supports is Apache-2.0 licensed and installs with Helm. Self-run Spark on EKS with the open-source operator needs no job rewrite, but you then run the Spark build yourself and carry none of the EMR uplift. EMR Serverless bills only the vCPU and memory of workers while they run, so a job that scales in and out is billed for less than its peak size held for the whole time. The [compute bill](/compute-bill/) line for EMR Serverless does not model that. ## Glue as another place to run Spark AWS Glue ETL also runs Spark. A DPU is 4 vCPU and 16 GB of memory, Spark jobs need at least 2 DPUs and default to 10, and billing is per second. The us-east-1 price list shows $0.308 per DPU-hour for Glue 6.0 and later and $0.44 for older versions. Glue sits outside EMR's uplift model, so a job moved between them changes the pricing unit, not only the price. ## Region and cloud limits All of these services run only on AWS. EMR has regional endpoints across the US regions (us-east-1, us-east-2, us-west-1, us-west-2) and many others. Glue Iceberg compaction pricing, S3 Tables pricing and EMR uplifts differ by region; the figures on this site are us-east-1 list prices read 2026-11-10. ## Sources - AWS Big Data Blog, EMR 7.12 and Iceberg benchmark: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Amazon EMR on EKS: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks.html (read 10 Nov 2026) - EMR on EKS Spark operator: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) - Amazon EMR Serverless: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html (read 10 Nov 2026) - Amazon EMR 7.x release versions: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Delta Lake on Amazon EMR: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-delta.html (read 10 Nov 2026) - Apache Hudi on Amazon EMR: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi.html (read 10 Nov 2026) - AWS Glue Data Catalog: https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html (read 10 Nov 2026) - AWS Glue Iceberg REST endpoint: https://docs.aws.amazon.com/glue/latest/dg/connect-glu-iceberg-rest.html (read 10 Nov 2026) - Amazon S3 Tables overview: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html (read 10 Nov 2026) - S3 Tables with AWS Glue: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-glue.html (read 10 Nov 2026) - Apache Iceberg maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - AWS Glue optimizing Iceberg tables: https://docs.aws.amazon.com/glue/latest/dg/table-optimizers.html (read 10 Nov 2026) - Athena Iceberg optimization: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Amazon S3 Tables maintenance: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - AWS Glue pricing: https://aws.amazon.com/glue/pricing/ (read 10 Nov 2026) - AWS price list, AWS Glue, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/guides/open-stack-on-aws/ --- # Leaving a managed platform: what moves and what does not Guide, read 10 Nov 2026 **The short answer.** Moving Spark batch work off Databricks or Snowflake splits into parts that travel as files and code, parts that need edits, and parts that stay behind, such as automatic maintenance and billing commitments. This page lists each part with the vendor documentation that says so. ## What moves as files Open table formats keep data in Parquet files. Databricks documents that Delta tables can generate Iceberg metadata through UniForm from Databricks Runtime 14.3 LTS, so Iceberg clients read them without rewriting data files. Delta Lake's own docs describe UniForm as generating the metadata asynchronously for Iceberg and Hudi clients. On Snowflake, Iceberg tables with an external volume sit in customer-managed S3, Google Cloud Storage or Azure storage, so the data files are already yours. Snowflake also documents that you can move Iceberg data from Snowflake storage to your own cloud storage at any time with CTAS at standard rates. ## A published exit incentive Snowflake states that it will reimburse the eligible Snowflake compute costs of a qualifying one-time migration of Iceberg data to customer-owned storage in the same cloud platform and region as the Snowflake account. The conditions on the page: the data must have been in Snowflake storage for at least 30 days, and the total size must be at least 1 TB. Contact is through Snowflake Support. No equivalent offer appears in the Databricks pages read on 2026-11-10. ## What moves with edits Spark code moves in proportion to how much of it is standard Spark. Snowpark Connect runs PySpark DataFrame code on Snowflake warehouses, so that code is already written to the Spark Connect protocol, but Snowflake documents semantic differences: ByteType, ShortType and IntegerType come back as LongType, and several interval and user-defined types are unsupported. Code tested only on Snowflake needs a check against real Spark behavior. Snowflake supports Spark 3.5 only, which is the line Amazon EMR 7.14.0 ships as Spark 3.5.8-amzn-0. Databricks code that uses platform features does not move unchanged. The serverless compute page lists Databricks Container Services, DBFS mounts with instance profiles and global temp views as unsupported on serverless, which tells you what the other direction would rely on. Photon is a Databricks runtime feature; EMR has its own runtime, and the two are separate products with separate published claims. ## What stays behind Automatic maintenance does not travel. Databricks predictive optimization runs OPTIMIZE, VACUUM and ANALYZE only on Unity Catalog managed tables. Snowflake runs compaction, manifest compaction and snapshot expiry on Snowflake-managed Iceberg tables, and states it performs no maintenance on tables managed by an external catalog, where you must expire snapshots, remove old metadata and compact files in your own engine. In an open stack that work becomes jobs you schedule or services you turn on. Iceberg's documentation lists expiring snapshots, removing old metadata files and deleting orphan files as recommended maintenance. AWS offers Glue table optimizers and S3 Tables maintenance for Iceberg. See [compaction on each platform](/maintenance/compaction-on-each-platform/). Billing arrangements also stay: Databricks offers committed-use discounts on DBUs, so check the contract end date against the migration plan. ## Catalog and permissions The catalog decides who can find and write a table. Snowflake Open Catalog is a managed service for Apache Polaris, and Snowflake states that customers who had not already created an Open Catalog account cannot sign up for a first one. Unity Catalog has an Apache-2.0 open-source project hosted under LF AI & Data, with Delta Lake and an Iceberg REST Catalog API. The [catalogs guide](/guides/lakehouse-catalogs-for-tables/) lists the options. Databricks serverless compute needs Unity Catalog to reach external data sources, so the catalog is part of the platform, not an add-on you can leave behind. ## Where the compute can land Vendors in alphabetical order, only for Spark batch work. Amazon EMR on EC2, EKS or Serverless runs Spark code with AWS describing the runtime as 100% API-compatible. Dremio runs its own Arrow-based engine, and its documentation describes no mode that runs existing Spark jobs. Onehouse says its jobs are fully compatible with Apache Spark and that EMR code can be reused unless it uses AWS-specific features; its Quanton clusters run in your AWS account. Starburst Galaxy runs SQL on Trino, so Spark DataFrame code and UDFs would be rewritten or kept on Spark. Microsoft Fabric lists Azure regions only, so its Spark compute does not run inside AWS, and its OneLake S3 shortcuts give read-only access to S3 data. ## A short list of checks - Which tables are Snowflake-managed or Unity Catalog managed, and which sit under an external catalog. - Which Spark version the code runs on today and which versions the target runtime ships. - Which notebooks or jobs use serverless-only or warehouse-only behavior. - Who runs compaction, snapshot expiry and orphan-file cleanup after the move. - Contract dates for committed spend, and whether the Snowflake 1 TB and 30-day migration conditions apply. ## Sources - Databricks UniForm: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Delta Lake UniForm: https://docs.delta.io/latest/delta-uniform.html (read 10 Nov 2026) - Snowflake Iceberg storage: https://docs.snowflake.com/en/user-guide/tables-iceberg-storage (read 10 Nov 2026) - Snowflake Iceberg table management: https://docs.snowflake.com/en/user-guide/tables-iceberg-manage (read 10 Nov 2026) - Snowpark Connect for Apache Spark: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowpark Connect compatibility: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - Snowpark Connect limitations: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Databricks serverless compute limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks predictive optimization: https://docs.databricks.com/aws/en/optimizations/predictive-optimization (read 10 Nov 2026) - Databricks pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Amazon EMR 7.x release versions: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Apache Iceberg maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Snowflake Open Catalog overview: https://docs.snowflake.com/en/user-guide/opencatalog/overview (read 10 Nov 2026) - Unity Catalog open source: https://www.unitycatalog.io/ (read 10 Nov 2026) - Dremio architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Onehouse EMR migration: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Starburst Galaxy documentation: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - Fabric OneLake S3 shortcuts: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - AWS Big Data Blog, EMR 7.12 and Iceberg benchmark: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/guides/leaving-a-managed-platform/ --- # Lakehouse pricing models and the published prices Guide, read 10 Nov 2026 **The short answer.** Lakehouse platforms bill in different units: Databricks DBUs, Snowflake credits, BigQuery slots, Fabric capacity units, Dremio DCUs, Starburst credits and Onehouse OCUs, while AWS bills EC2 plus an EMR uplift. This page lists each unit and every dollar rate that was readable on 2026-11-10. ## The units side by side | Vendor | Billing unit | Granularity | Dollar rate read 2026-11-10 | | --- | --- | --- | --- | | Amazon EMR on EC2 | EC2 instance-hours plus an EMR uplift plus EBS | Per second, 1-minute minimum | EMR uplift $0.048 per hour for m5.xlarge, us-east-1 | | Amazon EMR on EKS | Uplift per vCPU-hour and per GB-hour, plus EKS and EC2 or Fargate | Per second, 1-minute minimum | $0.007084 per vCPU-hour; $0.000777875 per GB-hour; $0.10 per hour per EKS cluster | | Amazon EMR Serverless | vCPU-hours and GB-hours of workers | Per second, 1-minute minimum | $0.052624 per vCPU-hour; $0.0057785 per GB-hour (AWS worked example) | | Amazon Athena | TB scanned for SQL; DPU-hours for Spark | Per query or per DPU-hour | $5.00 per TB scanned; $0.35 per DPU-hour for Spark; $0.30 per DPU-hour provisioned | | BigQuery | TiB scanned (on-demand) or slot-hours (capacity) | On-demand: per TiB; capacity: per slot-hour | $0.06 per slot-hour, Enterprise edition (pricing page text; no region shown beside the rate); other rates not used | | Databricks | DBU | Per second | none found on 2026-11-10 (rates render by script) | | Dremio Cloud | DCU | Per second of engine uptime, 60-second minimum | $0.20 per DCU | | Microsoft Fabric | Capacity unit (CU) | Per CU-hour; Spark autoscale billing is pay-as-you-go | $0.18 per CU-hour, eastus, Spark entries | | Onehouse | OCU: a 4-core instance for 1 hour in your account | OCU-hours of cluster usage | none found on 2026-11-10 | | Snowflake | Credit | Per second, 60-second minimum per start | $3.00 per credit, Enterprise edition, AWS US East (Service Consumption Table, effective 2026-11-09); Standard $2.00, Business Critical $4.00 | | Starburst Galaxy | Credit | Cluster compute hours | $0.50 (Pro) and $0.75 (Enterprise) per credit on AWS us-east-1 | Vendors in alphabetical order, with the AWS services grouped first. Sources are on each platform page and listed below. Units are not interchangeable. ## EC2 plus an EMR uplift The AWS model is two prices stacked. On the AWS price list, EMR on EC2 charges an uplift per instance type (m5.4xlarge $0.192 per hour, r5d.4xlarge $0.27) added to the EC2 price and EBS. AWS's own example in its pricing page puts a c4.2xlarge EMR charge at $0.105 per hour next to $0.398 for EC2, and one primary plus two core nodes for a month at $1,101.57. Spot instances are described as up to 90% below On-Demand. ## DBUs and credits A DBU is, per Databricks, a normalized unit of processing power whose consumption depends on compute resources used and data processed. Photon instance types use DBUs at a different rate than the same instance without Photon, and the jobs page offers serverless in Performance Optimized and Standard modes as well as classic clusters. Databricks also bills predictive optimization on a serverless jobs SKU. For classic Jobs Compute on AWS, the data file behind its pricing calculator lists $0.15 per DBU on the Premium plan and 4.08 DBU per node-hour for r6i.4xlarge (read 2026-11-10). Snowflake credits are charged by the number of virtual warehouses, how long they run and their size. Each size step roughly doubles the credits per hour, and the consumption table puts an XL standard warehouse at 16 credits per hour and an Enterprise credit on AWS US East at $3.00. Serverless features bill credits per compute-hour, and Snowflake states that Iceberg data compaction on external volume storage is an additional charge that varies with the workload. ## Slots, capacity units and DCUs BigQuery offers on-demand pricing per TiB scanned, with the first 1 TiB per month free, or capacity pricing per slot-hour, which the pricing page text puts at $0.06 for Enterprise edition compute. Iceberg managed tables run storage optimization billed in Data Compute Units in per-second increments. Google's Spark service bills serverless Spark in DCUs per second with a one-minute minimum. In Microsoft Fabric each capacity unit maps to two Spark VCores, so a Spark job is admitted by available VCores. At $0.18 per CU-hour, one CU running for an hour costs $0.18 before any reserved discount; reserved capacity discounts do not apply to Autoscale Billing for Spark. Dremio Cloud engines consume DCUs per hour by size: 16 for 2XSmall, 32 for XSmall, 64 for Small, up to 2048 for 3XLarge. At $0.20 per DCU, a 2XSmall engine for one hour is 16 times $0.20, or $3.20 (arithmetic on the two published figures). Dremio's free trial is a $400 credit for 30 days. ## Credits and OCUs on bring-your-own-cloud Starburst Galaxy lists one worker at 6 credits per hour, 2 workers at 12 and 16 workers at 96, so one worker on the Pro tier at $0.50 is $3.00 per hour (arithmetic on the published figures). Prices differ by region: Europe Ireland is $0.56 and Frankfurt $0.60 for Pro. Managed table maintenance and data transforms are metered at 0.30 credits per hour. Onehouse bills OCUs and the customer pays its own AWS or GCP infrastructure bill. The docs page reads no dollar rate per OCU, and no price list was found at the vendor's pricing URL. Standard instance types map to Graviton m8g sizes, for example oh-general-4 is m8g.xlarge. ## What to check before comparing Compare on the same job, not the same unit. The [compute bill](/compute-bill/) prices one Spark batch workload across the ways of running it on AWS and states each assumption. Check minimum billing periods, since a 60-second minimum matters for short jobs, check what is metered separately (maintenance, storage requests, shuffle storage) and check whether committed discounts apply. ## Sources - Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - AWS price list, Amazon EMR, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - AWS price list, Amazon Athena, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026) - BigQuery pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - BigQuery slots: https://cloud.google.com/bigquery/docs/slots (read 10 Nov 2026) - BigQuery Iceberg tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Google serverless Spark pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) - Databricks pricing: https://www.databricks.com/product/pricing (read 10 Nov 2026) - Databricks on AWS pricing: https://www.databricks.com/product/aws-pricing (read 10 Nov 2026) - Databricks pricing data file (AWS), loaded by its pricing calculator: https://www.databricks.com/en-pricing-assets/data/pricing/AWS.json (read 10 Nov 2026) - Databricks jobs pricing: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) - Databricks Photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks predictive optimization: https://docs.databricks.com/aws/en/optimizations/predictive-optimization (read 10 Nov 2026) - Dremio pricing: https://www.dremio.com/pricing (read 10 Nov 2026) - Dremio Cloud engines: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - Azure retail prices API, Microsoft Fabric, eastus: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20%27Microsoft%20Fabric%27%20and%20armRegionName%20eq%20%27eastus%27%20and%20priceType%20eq%20%27Consumption%27 (read 10 Nov 2026) - Fabric Spark concurrency and queueing: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Microsoft Fabric licenses and SKUs: https://learn.microsoft.com/en-us/fabric/enterprise/licenses (read 10 Nov 2026) - Onehouse usage and billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Snowflake compute cost: https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowflake Iceberg table management: https://docs.snowflake.com/en/user-guide/tables-iceberg-manage (read 10 Nov 2026) - Snowflake Iceberg storage: https://docs.snowflake.com/en/user-guide/tables-iceberg-storage (read 10 Nov 2026) - Starburst Galaxy pricing details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Snowflake Service Consumption Table: https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/guides/lakehouse-pricing-models/ --- # Table catalogs for lakehouse engines: Glue, Unity Catalog, Polaris and Nessie Guide, read 10 Nov 2026 **The short answer.** A table catalog tells query engines where an Iceberg or Delta table's current metadata lives. AWS Glue, Unity Catalog open source, Apache Polaris and Project Nessie each expose a REST interface; this page lists what each documents, its license and its release signals. ## What an engine needs from a table catalog In Snowflake's description of the Iceberg design, a catalog forms the first layer of the table specification and must store the current metadata pointer for each table, mapping a table name to the location of its current metadata file. Apache Iceberg publishes a REST catalog specification so that engines can talk to different catalogs through one interface. For a Spark batch team the practical questions are which catalogs the engine can write through, who is allowed to commit, and what other engines can read the same tables. Data catalogs in the governance sense, with search and lineage, are a separate product category that this site does not cover. ## AWS Glue Data Catalog The Glue Data Catalog is a central metadata repository that integrates with Athena, Lake Formation, EMR and SageMaker. Its Iceberg REST endpoint supports API operations from the Iceberg REST specification, supports table specifications v1 and v2 (v2 by default), and for us-east-1 is the endpoint `glue.us-east-1.amazonaws.com/iceberg` (an API address, not a web page). Athena documents that its Iceberg support works only with tables in the Glue Data Catalog and uses Glue optimistic locking. Glue also hosts table optimizers for compaction, snapshot retention and orphan file deletion. Compaction is off by default for new tables and is enabled per table with an IAM role, at $0.44 per DPU-hour in us-east-1. ## Amazon S3 Tables S3 Tables stores tables in the Iceberg format in table buckets, and AWS exposes an Iceberg REST endpoint for engines outside the AWS analytics integration. Maintenance is built in and on by default for every table: compaction with a 512 MB default target and snapshot management with a 120-hour default maximum snapshot age. The service fails snapshot management for a table that has user-defined tags or branches, a restriction to check if you use Iceberg branches. ## Unity Catalog open source The Unity Catalog project describes itself as open governance for tables, multimodal data, functions and models, licensed Apache 2.0 and an LF AI & Data project, with Delta Lake tables and an Iceberg REST Catalog API. Its 0.7 announcement is titled as adding native Apache Iceberg tables. The GitHub repository was created June 13, 2024 and last pushed November 10, 2026. Databricks documents the commercial Unity Catalog separately, and several Databricks features, such as predictive optimization and serverless access to external data, depend on it. The open-source project and the Databricks product are different deployments. ## Apache Polaris Apache Polaris is described as an open-source, fully featured catalog for Apache Iceberg that implements Iceberg's REST API. Its site names Apache Doris, Apache Flink, Apache Spark, Dremio OSS, StarRocks and Trino as engines that work with it. The repository is Apache-2.0 licensed, was created May 29, 2024 and was last pushed November 10, 2026. Two vendors build on it. Dremio's Open Catalog is built on Apache Polaris. Snowflake Open Catalog is a managed service for Apache Polaris, but new customers who had not already created an account are directed to Snowflake Horizon Catalog. ## Project Nessie Project Nessie describes itself as a transactional catalog for data lakes with Git-like semantics, offering cross-table transactions and working with Iceberg tables. The main Nessie server serves the repository through the Iceberg REST API and its own native REST API. The documentation read on 2026-11-10 is for version 0.110.0, and the repository, Apache-2.0 licensed, was created April 9, 2020. ## Catalogs that vendors host Vendors in alphabetical order. Dremio's Open Catalog is built on Apache Polaris, supports the Iceberg REST Catalog API, and lists the Glue Data Catalog as a connectable catalog; for Iceberg tables in it, Dremio runs automatic optimization on a dedicated engine it configures. Onehouse documents Glue catalog sync as part of migrating from EMR. Snowflake supports an external catalog integration such as AWS Glue for Iceberg tables, and its own Open Catalog is the Polaris service described above. Hosting a catalog adds a dependency that table files alone never carry. If the catalog is vendor-hosted, ask what export or replication the vendor documents before relying on it as the only copy of table pointers. ## What to check - Which engines on your runtime can commit through the catalog, not only read. - Whether the catalog runs maintenance for you (Glue and S3 Tables do for Iceberg) or leaves it to your jobs. - Whether tags and branches are used, given the S3 Tables restriction above. - Who hosts it: AWS-managed, vendor-managed or a service you run. - How a second engine authenticates against the REST endpoint. ## Sources - Apache Iceberg REST catalog specification: https://iceberg.apache.org/rest-catalog-spec/ (read 10 Nov 2026) - Snowflake Open Catalog overview: https://docs.snowflake.com/en/user-guide/opencatalog/overview (read 10 Nov 2026) - AWS Glue Data Catalog: https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html (read 10 Nov 2026) - AWS Glue Iceberg REST endpoint: https://docs.aws.amazon.com/glue/latest/dg/connect-glu-iceberg-rest.html (read 10 Nov 2026) - AWS Glue compaction optimization: https://docs.aws.amazon.com/glue/latest/dg/compaction-management.html (read 10 Nov 2026) - Amazon S3 Tables overview: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html (read 10 Nov 2026) - Amazon S3 Tables maintenance: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Unity Catalog open source: https://www.unitycatalog.io/ (read 10 Nov 2026) - Unity Catalog repository (GitHub API): https://api.github.com/repos/unitycatalog/unitycatalog (read 10 Nov 2026) - Databricks Unity Catalog: https://docs.databricks.com/aws/en/data-governance/unity-catalog/ (read 10 Nov 2026) - Apache Polaris: https://polaris.apache.org/ (read 10 Nov 2026) - Apache Polaris repository (GitHub API): https://api.github.com/repos/apache/polaris (read 10 Nov 2026) - Project Nessie: https://projectnessie.org/ (read 10 Nov 2026) - Project Nessie 0.110.0 documentation: https://projectnessie.org/nessie-latest/ (read 10 Nov 2026) - Project Nessie repository (GitHub API): https://api.github.com/repos/projectnessie/nessie (read 10 Nov 2026) - Dremio Cloud data formats: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Dremio Cloud table optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - S3 Tables with AWS Glue: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-glue.html (read 10 Nov 2026) - Athena Iceberg optimization: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Onehouse EMR migration: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Snowflake Iceberg tables: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/guides/lakehouse-catalogs-for-tables/ --- # Where Spark batch work runs on each platform Guide, read 10 Nov 2026 **The short answer.** Spark batch jobs run natively on Amazon EMR, AWS Glue, Athena and Databricks, through a Spark Connect layer on Snowflake, as stored procedures in BigQuery and on Azure in Fabric. Dremio and Starburst Galaxy run SQL engines instead. This page lists each documented path. ## The paths side by side | Platform | Where Spark code runs | Spark version documented | Code change to expect | | --- | --- | --- | --- | | Amazon EMR, AWS Glue and Athena | EMR on EC2, EKS or Serverless; Glue ETL; Athena for Apache Spark | EMR 7.14.0: 3.5.8-amzn-0; Athena: 3.5 | None stated for EMR; AWS describes 100% API compatibility | | BigQuery and Google Cloud | Stored procedures for Apache Spark, run on Google Cloud | runtime_version 2.2 in the example | Wrap code as a procedure; none found for full job migration | | Databricks | Classic or serverless jobs compute | Runtime 18 LTS: 4.1.0; 17.3 LTS: 4.0.0; 16.4 and 15.4 LTS: 3.5.x | Serverless supports only Spark Connect APIs | | Dremio | Not Spark; Dremio's Arrow-based engine | Not applicable | Rewrite as Dremio SQL or keep Spark on the same Iceberg tables | | Microsoft Fabric | Spark pools on Azure | Runtime 1.3: 3.5; Runtime 2.0: 4.1 | Compute does not run inside AWS | | Onehouse | Quanton clusters in your AWS account | Spark versions: none found on 2026-11-10 | Jobs created through its API or Airflow operators | | Snowflake | Snowflake warehouse via Snowpark Connect | 3.5 only | Semantic differences listed in the compatibility notes | | Starburst Galaxy | Not Spark; managed Trino | Not applicable | Rewrite as Trino SQL or keep Spark | Vendors alphabetical after the AWS row. Versions and notes from each vendor's documentation, read 2026-11-10. ## AWS: three places to run the same job Athena for Apache Spark runs Spark 3.5 and accepts code from SageMaker Unified Studio notebooks or any Spark Connect compatible client. It costs $0.35 per DPU-hour in us-east-1. AWS Glue ETL bills DPUs at $0.308 per hour for Glue 6.0 and later, with a minimum of 2 DPUs per Spark job. EMR on EC2 bills instance-hours plus an uplift, and EMR Serverless bills vCPU-hours and GB-hours. All three accept ordinary Spark code; what differs is the unit of billing and who manages the cluster. ## Databricks Databricks jobs run on classic clusters or serverless. Photon is on by default for classic all-purpose and jobs compute, and falls back to the Spark runtime for unsupported operations. Serverless compute does not give you a Spark UI or Spark logs; query profile and client-side logs replace them. Jobs pricing offers serverless Performance Optimized and Standard modes next to classic clusters. ## Snowflake, BigQuery and Fabric Snowflake runs PySpark DataFrame code on a warehouse through the Spark Connect protocol, with no Spark cluster. Batch jobs are submitted with the Snowpark Submit CLI, and Snowflake lists a separate optimization and troubleshooting section for workloads after migration. BigQuery creates a Spark procedure with CREATE PROCEDURE and engine = "SPARK", using PySpark or Scala and Java code from Cloud Storage or inline. It runs on Google Cloud. BigQuery Omni can run analytics on S3 data through BigLake tables, but DML and CREATE TABLE are not supported there, so it does not run write-heavy Spark batch ETL on S3. Google's Spark service has a Standard tier for batch ETL and a Premium tier on Lightning Engine. In Fabric, starter pools start in 5 to 10 seconds with no setup, and custom pools tune node size and scaling. Fabric Spark runs on Azure capacity, and OneLake S3 shortcuts are read-only. ## Onehouse, Dremio and Starburst Onehouse documents that Quanton runs Spark and SQL jobs reading and writing Hudi, Iceberg and Delta Lake in your AWS account. Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog such as AWS Glue; for Delta Lake you add Delta Spark as a dependency. Each cluster needs minimum and maximum OCU limits, and at the maximum the cluster stops scaling up, which can delay processing. Dremio and Starburst Galaxy are query engines for SQL. Dremio's docs describe Spark and Flink only as external engines that use its Open Catalog. Galaxy offers a fault tolerant execution mode that spools intermediate data for batch-style SQL; the docs say it can be slightly slower than normal execution. ## What to check for a batch job - The Spark major version the code needs, against the table above. - Whether the job uses RDD APIs, UDFs, custom containers or mounts that a serverless path rejects. - The billing unit and minimum, since nightly jobs with many short stages meet minimums often. - Which table formats and catalogs the engine can write to. - Whether a published benchmark names its method. AWS's EMR 7.12 run publishes its setup on GitHub, Onehouse describes its setup in blog posts with results as charts, and each was run by the vendor. ## Sources - Amazon EMR 7.x release versions: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Athena for Apache Spark: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026) - AWS Glue pricing: https://aws.amazon.com/glue/pricing/ (read 10 Nov 2026) - AWS price list, AWS Glue, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - Amazon EMR Serverless: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html (read 10 Nov 2026) - Databricks Runtime release notes: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Databricks Photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks serverless compute limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Databricks jobs pricing: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026) - Snowpark Connect for Apache Spark: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Snowpark Connect limitations: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - BigQuery stored procedures for Apache Spark: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - BigQuery Omni: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Google serverless Spark pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) - Fabric runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Fabric OneLake S3 shortcuts: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - Onehouse jobs: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - Onehouse create cluster: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Dremio architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Starburst Galaxy documentation: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/guides/batch-spark-on-a-lakehouse/ --- # Lakehouse terms, each with a sourced example Glossary, read 10 Nov 2026 **The short answer.** Thirty-five short definitions of terms that appear across the platform and format pages, such as compaction, copy-on-write, merge-on-read, deletion vectors, hidden partitioning, UniForm and DBU. Each entry gives a plain definition, a sourced example with numbers where the source has them, and the source link with its read date. ## 35 terms, A to Z - [ACID transactions on object storage](/glossary/acid-transactions/) - [Amazon S3 Tables](/glossary/amazon-s3-tables/) - [Apache Hudi](/glossary/apache-hudi/) - [Apache Iceberg](/glossary/apache-iceberg/) - [Apache Polaris](/glossary/apache-polaris/) - [Apache XTable](/glossary/apache-xtable/) - [AWS Glue Data Catalog](/glossary/aws-glue-data-catalog/) - [BigQuery slot](/glossary/bigquery-slot/) - [Compaction](/glossary/compaction/) - [Copy-on-write](/glossary/copy-on-write/) - [Data lakehouse](/glossary/data-lakehouse/) - [DBU](/glossary/dbu/) - [Deletion vectors](/glossary/deletion-vectors/) - [Delta Lake](/glossary/delta-lake/) - [Delta UniForm](/glossary/delta-uniform/) - [Fabric capacity unit](/glossary/fabric-capacity-unit/) - [Hidden partitioning](/glossary/hidden-partitioning/) - [Liquid clustering](/glossary/liquid-clustering/) - [Manifest file](/glossary/manifest-file/) - [Medallion architecture](/glossary/medallion-architecture/) - [Merge-on-read](/glossary/merge-on-read/) - [OneLake shortcut](/glossary/onelake-shortcut/) - [Open table format](/glossary/open-table-format/) - [Orphan files](/glossary/orphan-files/) - [Partition evolution](/glossary/partition-evolution/) - [Project Nessie](/glossary/project-nessie/) - [REST catalog](/glossary/rest-catalog/) - [Schema evolution](/glossary/schema-evolution/) - [Small files problem](/glossary/small-files-problem/) - [Snapshot](/glossary/snapshot/) - [Snapshot expiration](/glossary/snapshot-expiration/) - [Snowflake credit](/glossary/snowflake-credit/) - [Time travel](/glossary/time-travel/) - [Unity Catalog](/glossary/unity-catalog/) - [Z-ordering](/glossary/z-ordering/) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/ --- # Data lakehouse Glossary, read 10 Nov 2026 **The short answer.** A data lakehouse is a data management system that combines the benefits of a data lake and a data warehouse, per Databricks's documentation. Table data stays in files on object storage such as Amazon S3, and a metadata layer adds transactions and schema enforcement so SQL, BI and machine learning jobs share one copy. Databricks's documentation says a lakehouse is meant to avoid separate systems for workloads such as machine learning and business intelligence, and often organizes data in staged layers it calls a medallion architecture. The pattern is not tied to one vendor. Apache Iceberg, Delta Lake and Apache Hudi each supply the table layer, and the query engine above it can be Spark, Trino, Flink or a warehouse. ## A sourced example Databricks's page, last updated Oct 29, 2026, names two components of its own lakehouse: Delta Lake for the storage layer with ACID transactions and schema enforcement, and Unity Catalog for governance. ## Sources - What is a data lakehouse? (Databricks on AWS): https://docs.databricks.com/aws/en/lakehouse/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/data-lakehouse/ --- # Open table format Glossary, read 10 Nov 2026 **The short answer.** An open table format is a published specification for tracking which data files make up a table, along with its schema and partitioning, so that more than one engine can read and write the same table. Apache Iceberg, Delta Lake and Apache Hudi are the three in common use on AWS. The files themselves are usually Parquet on object storage. The format adds a metadata layer that records file lists, statistics and commits, so an engine need not list directories to find the table's contents. Because the specification is public, a team can change query engines without exporting the data. Which engines can actually write a given format differs by engine and version, so check the support list of each one. ## A sourced example The Apache Iceberg site says Iceberg lets Spark, Trino, Flink, Presto, Hive and Impala work with the same tables at the same time, and that it supports merge, update and targeted delete commands in SQL. ## Sources - Apache Iceberg home page: https://iceberg.apache.org/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/open-table-format/ --- # Apache Iceberg Glossary, read 10 Nov 2026 **The short answer.** Apache Iceberg is an open table format for huge analytic tables, specified in a versioned document. Its metadata tree runs from a table metadata file through snapshots and manifest lists to manifests that point at data files. Spark, Trino, Flink, Presto, Hive and Impala can all read the same table. Every commit writes a new metadata file and swaps it in atomically, which is how Iceberg gets safe concurrent commits on object storage. Maintenance is not automatic in the open source project. Expiring snapshots, removing orphan files and compacting data files are actions a team schedules, unless a platform or catalog runs them. ## A sourced example The Iceberg specification says row-level deletes are stored as deletion vectors from format version 3 onward, while position delete files from the older representation can be created only in v2 tables. ## Sources - Iceberg table spec: https://iceberg.apache.org/spec/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/apache-iceberg/ --- # Delta Lake Glossary, read 10 Nov 2026 **The short answer.** Delta Lake is an open source storage layer that adds ACID transactions, scalable metadata handling and time travel to data on S3, ADLS, GCS and HDFS. Its transaction log lives beside the Parquet files, and Databricks uses it as the default table format for all reads and writes. The project documentation describes serializable isolation, schema enforcement, merge, update and delete operations, and a table that works as both a batch table and a streaming source and sink. Check the open source release notes for the version you plan to run, since the Databricks runtime and the open source project do not ship on the same schedule. ## A sourced example The Delta Lake documentation lists connectors that read and write Delta tables from Apache Spark, Apache Flink, Apache Hive, Trino and AWS Athena. ## Sources - Delta Lake documentation: introduction: https://docs.delta.io/latest/delta-intro.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/delta-lake/ --- # Apache Hudi Glossary, read 10 Nov 2026 **The short answer.** Apache Hudi, pronounced hoodie, is an open table format and lakehouse platform built for high-performance writes on incremental pipelines. It offers upserts and deletes, indexes, clustering and compaction, and two table types, copy-on-write and merge-on-read, with a timeline that logs every action on the table. Hudi describes itself as the project that pioneered the transactional data lake, the idea now usually called a lakehouse. It runs on any cloud storage and is queried by Spark, Flink, Presto, Trino and Hive. The documentation also notes a native Rust implementation, Hudi-rs, with Python bindings for non-JVM use. ## A sourced example The Hudi 1.2.1 overview names Uber, Amazon, ByteDance and Robinhood among the organizations running production data lakes on Hudi. ## Sources - Apache Hudi overview (version 1.2.1): https://hudi.apache.org/docs/overview (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/apache-hudi/ --- # Apache XTable Glossary, read 10 Nov 2026 **The short answer.** Apache XTable, formerly OneTable and still incubating at the Apache Software Foundation, translates table metadata between Hudi, Delta Lake and Iceberg without being a new format. It reads a table's existing metadata and writes metadata for the other formats beside the same data files, in either direction. XTable does not copy or rewrite the Parquet data. It adds a second metadata directory under the table's base path, so an engine that understands only one format can still read the table. Its documentation contrasts it with Delta UniForm, which it describes as a one-directional conversion from Delta Lake to Hudi or Iceberg. ## A sourced example XTable's FAQ says the metadata is stored in _delta_log for Delta, metadata for Iceberg and .hoodie for Hudi, so a Spark reader can load the same path with format delta, hudi or iceberg. ## Sources - Apache XTable (Incubating): https://xtable.apache.org/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/apache-xtable/ --- # Delta UniForm Glossary, read 10 Nov 2026 **The short answer.** Delta Universal Format, or UniForm, lets a Delta Lake table generate Iceberg metadata so Iceberg clients can read it. Databricks documents it from Runtime 14.3 LTS. The Parquet files are not rewritten, so one copy of the data serves both Delta and Iceberg readers. On Databricks the table must be registered in Unity Catalog and have column mapping enabled. Metadata generation runs asynchronously on the compute that writes the table, which can raise driver load. UniForm is one-directional, Delta to Iceberg or Hudi. Apache XTable is the project that translates in either direction. ## A sourced example Per the Databricks page, last updated Sep 11, 2026, a Delta table with Iceberg reads enabled uses Zstandard instead of Snappy to compress its Parquet data files. ## Sources - Read Delta Lake tables with Iceberg clients (Databricks on AWS): https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/delta-uniform/ --- # Manifest file Glossary, read 10 Nov 2026 **The short answer.** In Apache Iceberg, a manifest is a file with one row per data file, holding the file's partition values and column metrics. Manifests make up the lower layer of a table's metadata tree, and a manifest list names the manifests that belong to each snapshot so planners can skip irrelevant ones. Query planning reads manifests, not directory listings. The metadata tree works as an index over the table, which is why the number and layout of manifests affects planning time. Iceberg compacts manifests automatically in the order they were added, and a separate rewrite manifests action can regroup them when write order does not match read filters. ## A sourced example The Iceberg spec says manifests in format v3 and earlier store files for a single partition spec, while starting in v4 a manifest may store files from any spec. ## Sources - Iceberg table spec: https://iceberg.apache.org/spec/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/manifest-file/ --- # Snapshot Glossary, read 10 Nov 2026 **The short answer.** A snapshot is the state of an Iceberg table at one point in time, giving the full set of data files in the table. Each write creates a new snapshot, and the table metadata file keeps the list, which supports time travel, rollback and incremental reads of what changed between two commits. Snapshots share unchanged data files, so keeping many costs metadata and any files that only old snapshots reference. That is why snapshot expiration is a recommended maintenance task. Delta Lake and Hudi keep comparable history in a transaction log and a timeline, with different names and retention settings. ## A sourced example The Iceberg spec says that in format versions 1 to 3 the manifests of a snapshot are stored in a manifest list file that records partition stats and data file counts for each manifest. ## Sources - Iceberg table spec: https://iceberg.apache.org/spec/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/snapshot/ --- # Time travel Glossary, read 10 Nov 2026 **The short answer.** Time travel is querying a table as it was at an earlier version or timestamp. Delta Lake exposes it through version numbers and timestamp strings, and Iceberg through snapshot history. It supports audits, reproducible machine learning runs and recovery from a bad write, until the old data is cleaned up. Retention limits the window. Once snapshots are expired or old files are vacuumed, the earlier state can no longer be read, so retention settings are a cost and recovery decision. Delta Lake documents the feature for debugging, auditing in regulated industries, fixing mistakes and giving a set of queries snapshot isolation. ## A sourced example The Delta Lake batch documentation shows reading a table with versionAsOf, and with a timestamp string such as 2019-01-01T00:00:00.000Z, against the sample path /tmp/delta/people10m. ## Sources - Delta Lake documentation: table batch reads and writes: https://docs.delta.io/latest/delta-batch.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/time-travel/ --- # Schema evolution Glossary, read 10 Nov 2026 **The short answer.** Schema evolution is changing a table's columns without rewriting its data. Iceberg supports add, drop, rename, update (type widening) and reorder, including inside nested structs, as metadata-only changes that never make an existing column read another column's values. Engines differ in which changes they expose in SQL. Iceberg says schema changes are independent and free of side effects: an added column never reads another column's values, and dropping or updating a column changes no other column. Delta Lake and Hudi support schema changes too, with their own rules for column mapping and type changes. Check the exact operation against the engine and version you run. ## A sourced example The Iceberg evolution page says that moving a Hive table from a daily to an hourly layout needs a new table, and that even renaming a column is unsupported or unsafe in some older table formats. ## Sources - Iceberg documentation: evolution: https://iceberg.apache.org/docs/latest/evolution/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/schema-evolution/ --- # Partition evolution Glossary, read 10 Nov 2026 **The short answer.** Partition evolution changes how an existing table is partitioned without rewriting old data. In Iceberg, files written under an earlier partition spec stay as they are, new data uses the new layout, and the planner splits each query across the layouts. A team can move from daily to hourly partitions in place. Because queries do not reference partition values directly, no query has to be rewritten when the spec changes. The cost is that old and new layouts coexist until you rewrite old files. Databricks liquid clustering addresses the same need by letting you redefine clustering keys without rewriting existing data. ## A sourced example The Iceberg evolution page says metadata for each partition version is kept separately, so a query against a table with two specs gets split planning, with each layout planning its files using the filter it derives. ## Sources - Iceberg documentation: evolution: https://iceberg.apache.org/docs/latest/evolution/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/partition-evolution/ --- # Hidden partitioning Glossary, read 10 Nov 2026 **The short answer.** Hidden partitioning is an Iceberg feature where the table produces partition values by applying a transform, such as day(event_time), to a column. Users filter on the original column and Iceberg prunes partitions. Nobody maintains a separate partition column or has to know the layout when writing queries. In Hive-style tables, partitions appear as explicit columns that writers must fill and readers must filter on. A wrong or missing filter scans everything, and the layout cannot change without breaking queries. With hidden partitioning the relationship between column and partition is stored in table metadata, so partition values are produced the same way every time. ## A sourced example The Iceberg partitioning page walks through a logs table partitioned by day(event_time) and level, where Iceberg converts event_time into event_date and records the relationship itself. ## Sources - Iceberg documentation: partitioning: https://iceberg.apache.org/docs/latest/partitioning/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/hidden-partitioning/ --- # Copy-on-write Glossary, read 10 Nov 2026 **The short answer.** Copy-on-write is a table type where an update or delete rewrites the whole base file that holds the changed rows. Reads stay simple because queries touch only base files. Writes cost more, since a small change rewrites a file. Apache Hudi names the type CoW and documents it as tuned for read-heavy work. Hudi's table types page says CoW creates new base files in a file group on update or delete and writes no log files, so a query never has to merge changes at read time. The trade is write amplification: changing a few records in a large file rewrites the file. Iceberg and Delta Lake can behave the same way when deletion vectors or delete files are not in use. ## A sourced example Hudi's documentation for version 1.2.1 says CoW write operations can be slower because of base-file rewriting, even if only a small percentage of records change in each file. ## Sources - Apache Hudi: table and query types: https://hudi.apache.org/docs/table_types (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/copy-on-write/ --- # Merge-on-read Glossary, read 10 Nov 2026 **The short answer.** Merge-on-read is a table type where updates and deletes go into small log files and are merged with base files when a query runs or when compaction rewrites them. Writes are fast and data is fresh sooner. Reads pay the merge cost until compaction runs. Hudi calls it MoR. Hudi's MoR tables write changes as row-based Avro or columnar log files and reconcile them with the base file on a schedule. Without that compaction, read latency grows with the log. Iceberg's delete files and deletion vectors follow the same idea: record the change cheaply now, fold it into data files later. ## A sourced example Hudi's table types page walks through a timeline in which compaction at 10:05 produces a new base file version, so that looking at base files alone shows a layout like a copy-on-write table. ## Sources - Apache Hudi: table and query types: https://hudi.apache.org/docs/table_types (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/merge-on-read/ --- # Deletion vectors Glossary, read 10 Nov 2026 **The short answer.** Deletion vectors mark rows as deleted or modified in table metadata instead of rewriting the Parquet file that holds them. Readers apply the marks at query time. Databricks documents them for Delta Lake and Iceberg tables, and the Iceberg specification adopted them as its row-level delete representation in format v3. Without them, changing one row means rewriting the whole file containing it. With them, a DELETE, UPDATE or MERGE writes a small bitmap, and a later OPTIMIZE or compaction rewrites the file. Support is by engine and version, so a table written with deletion vectors may not be readable by an older client. ## A sourced example Databricks's page, last updated Nov 6, 2026, says Iceberg v3 tables include deletion vectors by default, Delta tables need them enabled explicitly, and reading them needs Runtime 12.2 LTS or above. ## Sources - Deletion vectors in Databricks: https://docs.databricks.com/aws/en/delta/deletion-vectors (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/deletion-vectors/ --- # Compaction Glossary, read 10 Nov 2026 **The short answer.** Compaction rewrites many small data files into fewer large ones so queries open fewer files and manifests stay small. Every format needs it: Iceberg through rewrite data files, Delta Lake through OPTIMIZE, Hudi through merge-on-read compaction, and managed services such as Amazon S3 Tables run it for the user on a schedule. Compaction is a recurring compute cost. Who runs it, on what hardware and how often differs by platform, which is why the maintenance pages on this site compare it directly. Some services also apply pending row-level deletes during compaction, so it doubles as the step that makes a delete physical. ## A sourced example Amazon S3 Tables enables compaction by default on every table, with a 512 MB default target file size that you can set anywhere from 64 MB to 512 MB, and it applies row-level deletes as it goes. ## Sources - Maintenance for tables (Amazon S3 User Guide): https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/compaction/ --- # Small files problem Glossary, read 10 Nov 2026 **The short answer.** The small files problem is the slowdown and metadata growth that follow when a table holds many data files far below the size an engine reads efficiently. Streaming and frequent micro-batch writes create them. Each file adds a metadata row and a file-open cost to every query that touches it. The fix is compaction into larger files, run on a schedule or by a managed service. The target size is a setting, and managed services publish their own defaults. Left alone, it raises both query time and the cost of metadata operations. ## A sourced example The Iceberg maintenance page says more data files means more metadata in manifest files, and its sample rewriteDataFiles call combines small files into 500 MB files for the partition with date 2020-08-18. ## Sources - Iceberg documentation: maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/small-files-problem/ --- # Z-ordering Glossary, read 10 Nov 2026 **The short answer.** Z-ordering is a technique that colocates related values from several columns in the same set of files, so data skipping can ignore more files when a query filters on those columns. Delta Lake applies it with OPTIMIZE ... ZORDER BY, available from Delta Lake 2.0.0, as an extra pass during compaction. It is a multi-dimensional clustering: unlike a single-column sort, it keeps files useful for filters on any of the chosen columns, within limits. Z-order has to be rerun as data arrives. Databricks now recommends liquid clustering instead, and says clustering cannot be combined with ZORDER. ## A sourced example The Delta Lake optimizations page gives the command OPTIMIZE events ZORDER BY (eventType) and notes that statistics on long string or binary columns are costly to collect. ## Sources - Delta Lake documentation: optimizations: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/z-ordering/ --- # Liquid clustering Glossary, read 10 Nov 2026 **The short answer.** Liquid clustering is a Databricks data layout feature that replaces table partitioning and Z-ordering. The table is declared with CLUSTER BY keys, OPTIMIZE clusters data incrementally, and the keys can be changed later without rewriting existing data. It is not compatible with partitioning or ZORDER on the same table. Databricks recommends letting the platform manage layout and optimization for data in the table, and applies clustering when you run OPTIMIZE. Liquid clustering is a Databricks feature, so portability to other engines should be checked table by table. ## A sourced example The Databricks page shows ALTER TABLE t1 REPLACE PARTITIONED BY WITH CLUSTER BY (day, id) followed by OPTIMIZE t1, and says Runtime 16.0 and above can run OPTIMIZE FULL to force reclustering. ## Sources - Use liquid clustering for tables (Databricks on AWS): https://docs.databricks.com/aws/en/delta/clustering (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/liquid-clustering/ --- # Snapshot expiration Glossary, read 10 Nov 2026 **The short answer.** Snapshot expiration removes old snapshots from an Iceberg table and deletes data files only those snapshots reference. Each write creates a snapshot, so without expiration the metadata and storage grow, and time travel reaches back ever further. The Iceberg project lists regular expiration as a recommended task. Expiring is also what physically reclaims space after deletes and compaction, because the old files stay readable until the last snapshot pointing at them is gone. It sets your time travel and rollback window. Choose the retention with that in mind, then schedule the action. ## A sourced example The Iceberg maintenance page gives a sample that expires snapshots older than 1 day, and says metadata files tracked in the log default to 100 previous versions (write.metadata.previous-versions-max). ## Sources - Iceberg documentation: maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/snapshot-expiration/ --- # Orphan files Glossary, read 10 Nov 2026 **The short answer.** Orphan files are files under a table's location that no table metadata references, usually left behind by failed tasks or jobs. Normal snapshot expiration may not be able to tell they are unneeded. Iceberg provides a delete orphan files action to find and remove them, and it is meant to be run periodically. Removing them wrongly is risky. A file that an in-progress write has created but not yet committed looks orphaned, so a short retention interval can corrupt the table. On the open stack on EMR or EKS it is another job to schedule. ## A sourced example The Iceberg maintenance page warns that removing orphan files with a retention interval shorter than the time any write needs to complete is dangerous, and gives a default interval of 3 days. ## Sources - Iceberg documentation: maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/orphan-files/ --- # REST catalog Glossary, read 10 Nov 2026 **The short answer.** The Iceberg REST catalog is a common HTTP API, defined with an OpenAPI specification, for managing table metadata and catalog operations. Any engine with one REST client can talk to any compliant catalog. It was created because language-specific pluggable catalogs were hard for vendors to support everywhere. The protocol uses change-based commits so the server can resolve conflicts and retry, and supports multi-table commits, caching and secure sharing with credential vending or remote signing. Apache Polaris implements it, and Dremio documents its Open Catalog as built on Polaris. ## A sourced example The Iceberg site points to an iceberg-rest-fixture Docker image maintained by the community for trying the REST protocol, and to a REST Catalog OpenAPI YAML file as the contract. ## Sources - Iceberg REST Catalog Spec: https://iceberg.apache.org/rest-catalog-spec/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/rest-catalog/ --- # AWS Glue Data Catalog Glossary, read 10 Nov 2026 **The short answer.** The AWS Glue Data Catalog is a centralized metadata repository on AWS that stores table locations, schemas and runtime metrics for your data sets. EMR, Athena, Redshift and Glue jobs read it. A crawler can fill it automatically, and it can hold Apache Iceberg tables with optional managed compaction. Catalog choice affects what else can see the table and whether maintenance is automatic, so the catalog is a choice to settle before a move to an open stack on AWS. ## A sourced example AWS's documentation says managed compaction for Iceberg tables in the Data Catalog is switched on or off per table, through the Glue console, the Lake Formation console, the AWS CLI or the API. ## Sources - Data Catalog and crawlers in AWS Glue: https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/aws-glue-data-catalog/ --- # Amazon S3 Tables Glossary, read 10 Nov 2026 **The short answer.** Amazon S3 Tables is S3 storage built for tabular data. Tables are stored as Apache Iceberg in a table bucket, which holds tables as subresources, and can be queried by Amazon Athena, Amazon Redshift and Apache Spark. AWS runs compaction and snapshot management for them by default. The service removes the schedule-your-own-maintenance step for tables that live in a table bucket. You can edit or disable the maintenance settings per table, using a maintenance configuration file. ## A sourced example The S3 Tables user guide has pages on table namespaces, replicating S3 tables, cost optimization with Intelligent-Tiering and working with Apache Iceberg V3. ## Sources - Working with Amazon S3 Tables and table buckets: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/amazon-s3-tables/ --- # Apache Polaris Glossary, read 10 Nov 2026 **The short answer.** Apache Polaris is an open source catalog for Apache Iceberg that implements Iceberg's REST API. The project says it gives multi-engine interoperability across Apache Doris, Apache Flink, Apache Spark, Dremio OSS, StarRocks and Trino. Dremio's Open Catalog is built on it. A REST catalog sits between engines and storage, so it is the shared point when a team runs more than one engine against the same tables. ## A sourced example The Polaris site lists six named engines that work against its REST API: Apache Doris, Apache Flink, Apache Spark, Dremio OSS, StarRocks and Trino. ## Sources - Apache Polaris: https://polaris.apache.org/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/apache-polaris/ --- # Project Nessie Glossary, read 10 Nov 2026 **The short answer.** Project Nessie is a transactional catalog for data lakes with Git-like semantics. It brings Git-inspired data version control, cross-table transactions and visibility to Apache Iceberg tables, and runs as a Docker image or on Kubernetes. Hive, Spark, Dremio and Trino can use it as an open data lake catalog. Cross-table transactions mean a change spanning several tables is visible together or not at all. Check engine support for the Nessie version you pick. ## A sourced example The Nessie home page lists four traits: transactional catalog, Git-inspired version control, cross-table transactions, and support for Hive, Spark, Dremio and Trino. ## Sources - Project Nessie: https://projectnessie.org/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/project-nessie/ --- # Unity Catalog Glossary, read 10 Nov 2026 **The short answer.** Unity Catalog is the governance layer built into Databricks. It enforces access control, tracks lineage and logs activity for data and AI assets. Databricks enables it automatically in workspaces created after November 8, 2023, and uses it as the base for serverless access, UniForm and managed tables. Governance catalogs are outside the scope of these pages beyond this entry. ## A sourced example Databricks's documentation says Unity Catalog operates beneath every data and AI interaction in a workspace and is accessed through Catalog Explorer, SQL, the CLI and REST APIs. ## Sources - Unity Catalog (Databricks on AWS): https://docs.databricks.com/aws/en/data-governance/unity-catalog/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/unity-catalog/ --- # DBU Glossary, read 10 Nov 2026 **The short answer.** A Databricks Unit, or DBU, is a normalized unit of processing power on the Databricks platform used for measurement and pricing. The number consumed depends on compute resources and data processed. Databricks bills per second, and the dollar price per DBU depends on cloud, tier and workload type. Photon instance types use DBUs at a different rate than the same instance without Photon, so the Photon setting changes the count. The pricing pages draw their rates by script. The data file behind the pricing calculator lists Jobs Compute on AWS at $0.15 per DBU on the Premium plan (read 2026-11-10). ## A sourced example Databricks's pricing page defines a second unit beside the DBU, the Databricks Storage Unit (DSU), which varies by region, storage type, volume stored and transactions. ## Sources - Pricing (Databricks on AWS documentation): https://docs.databricks.com/aws/en/resources/pricing (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/dbu/ --- # Snowflake credit Glossary, read 10 Nov 2026 **The short answer.** A Snowflake credit is the unit in which Snowflake meters compute: virtual warehouses, serverless features, compute pools and cloud services all consume credits. A warehouse bills per second with a 60-second minimum each time it starts, and each size step roughly doubles credits per hour. A credit count is not a cost until a dollar rate is applied. Snowflake's [Service Consumption Table](https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf), effective 2026-11-09, lists $3.00 per credit for the Enterprise edition on AWS US East (Northern Virginia), and other editions, clouds and regions are priced separately. Because compute runs only while a warehouse runs, suspend settings drive the bill as much as size does. ## A sourced example Snowflake's documentation uses Small at 2 credits per hour and Medium at 4, and says resizing a Small warehouse to Medium bills one minute of the 2 additional credits. ## Sources - Understanding compute cost (Snowflake documentation): https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/snowflake-credit/ --- # BigQuery slot Glossary, read 10 Nov 2026 **The short answer.** A BigQuery slot is a virtual compute unit BigQuery uses to run SQL queries, Python code and other jobs. BigQuery decides how many slots a query uses. You cannot set that number per query, but you can buy slot capacity in reservations, with baseline or autoscaled slots, or pay on demand per TiB processed. By default a project is charged on demand for the TiB of data each query processes. Capacity pricing charges for slots instead; Google's pricing page text lists Enterprise edition compute at $0.06 per slot-hour. BigQuery can read Iceberg managed tables and BigLake tables, so slots are the unit when BigQuery is the engine over an open table. ## A sourced example Google's slot page walks through a reservation of 1,000 slots facing 2,000 units of work: 1,000 run, 1,000 queue, and when 100 slots finish they pick up 100 queued units, leaving 900. ## Sources - Understand slots (BigQuery documentation): https://cloud.google.com/bigquery/docs/slots (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/bigquery-slot/ --- # Fabric capacity unit Glossary, read 10 Nov 2026 **The short answer.** A capacity unit, or CU, measures compute power in Microsoft Fabric. A capacity is bought as an F SKU such as F2 (2 CUs), F64 (64 CUs) or F8192 (8,192 CUs), and Spark, SQL and Power BI workloads share it. Spark jobs are admitted by the VCores that a capacity supplies. Microsoft maps each CU to two Spark VCores, and jobs beyond the available VCores queue or are throttled. Capacity is pay-as-you-go or reserved. The dollar rate per CU-hour comes from the Azure price list for a region; the Azure retail price API lists Spark usage at $0.18 per CU-hour in eastus. ## A sourced example Microsoft's licensing table shows F64 as 64 CUs matching Power BI P1 and 8 Power BI v-cores, and shows the Trial capacity also at 64 CUs. ## Sources - Microsoft Fabric concepts and licenses: https://learn.microsoft.com/en-us/fabric/enterprise/licenses (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/fabric-capacity-unit/ --- # OneLake shortcut Glossary, read 10 Nov 2026 **The short answer.** A OneLake shortcut is a pointer in Microsoft Fabric to data stored elsewhere, so Fabric engines read it in place instead of copying it. Sources include Amazon S3, Google Cloud Storage, Azure Data Lake Storage Gen2 and Iceberg tables. OneLake manages permissions and credentials for the connection. Shortcuts remove edge copies and some pipeline processing, which is the usual reason a team on AWS storage adds Fabric. Cross-cloud reads may incur egress, and Microsoft documents shortcut caching to reduce those charges. ## A sourced example Microsoft's shortcut page lists eight source types: Amazon S3, S3 compatible, ADLS Gen2, Azure Blob Storage, Dataverse, Google Cloud Storage, Iceberg, and OneDrive and SharePoint. ## Sources - OneLake shortcuts (Microsoft Learn): https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/onelake-shortcut/ --- # Medallion architecture Glossary, read 10 Nov 2026 **The short answer.** A medallion architecture is a data design pattern that organizes lakehouse tables in three layers, bronze for raw data, silver for validated data and gold for enriched data. Each hop improves structure and quality. Databricks calls it a recommended practice, not a requirement, and also names it multi-hop. Databricks's page lists the intended users of each layer in a table, from data engineers at bronze to executives and decision makers at gold. The pattern is independent of table format. It affects compute because every hop is a job that reads and writes a table. ## A sourced example Databricks's page gives a table of questions by layer: bronze is raw ingestion, silver is cleaning and validation, and gold is dimensional modeling and aggregation. ## Sources - What is the medallion lakehouse architecture? (Databricks on AWS): https://docs.databricks.com/aws/en/lakehouse/medallion (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/medallion-architecture/ --- # ACID transactions on object storage Glossary, read 10 Nov 2026 **The short answer.** ACID means atomicity, consistency, isolation and durability. Object stores such as S3 do not offer multi-file transactions, so table formats supply them: each commit atomically publishes a new metadata version that readers either see completely or not at all. Delta Lake, Iceberg and Hudi each do this differently. Concurrent writers are managed with optimistic concurrency control: no locks are held, writers check for conflicts at commit and retry or fail. Transactions across several tables are a separate feature that needs catalog support on some platforms. ## A sourced example Databricks's page says it provides snapshot isolation on reads and write-serializable isolation on writes by default, and that multi-table transactions require catalog commits enabled on the participating tables. ## Sources - What are ACID guarantees on Databricks?: https://docs.databricks.com/aws/en/lakehouse/acid (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/glossary/acid-transactions/ --- # Lakehouse and table-format news, dated and sourced News, read 10 Nov 2026 **The short answer.** Fifteen dated news items on lakehouse platforms and table formats: Iceberg 1.12.0, Delta Lake 4.4.1 and Hudi 1.2.1 releases, Polaris and Unity Catalog releases, and announcements from AWS, Snowflake, Databricks and Google. Each item names the party that published it and shows its read date. ## 15 items, newest first - [Amazon Athena system tables store usage data as Iceberg tables on S3 Tables](/news/athena-system-tables-on-s3-tables/): 9 Nov 2026. AWS announced on 2026-11-09 that Amazon Athena system tables capture usage data within minutes of query completion. - [AWS Glue Data Catalog supports vended credentials for federated Iceberg REST catalogs](/news/glue-catalog-vended-credentials-iceberg-rest/): 6 Nov 2026. AWS announced on 2026-11-06 that Glue Data Catalog supports an external catalog's vended credentials in Iceberg REST catalog federation. - [Amazon S3 Tables adds managed migration from Hive to Iceberg](/news/s3-tables-hive-to-iceberg-migration/): 4 Nov 2026. AWS announced on 2026-11-04 that Amazon S3 Tables can manage migration of Apache Hive tables to Apache Iceberg. - [Snowflake opens externally managed Iceberg tables to outside engines via Horizon](/news/snowflake-horizon-external-iceberg-ga/): 29 Oct 2026. Snowflake's release notes dated 2026-10-29 say access to externally managed Iceberg tables through Snowflake Horizon Catalog is generally available. - [Snowflake makes in-place Iceberg v2 to v3 upgrade generally available](/news/snowflake-iceberg-v3-upgrade-ga/): 16 Oct 2026. Snowflake's release notes dated 2026-10-16 say in-place upgrade of Apache Iceberg tables from v2 to v3 is generally available. - [Databricks: multi-table transactions generally available for managed Delta and Iceberg](/news/databricks-transactions-delta-iceberg-ga/): 14 Oct 2026. Databricks's release notes dated 2026-10-14 say transactions are generally available for Unity Catalog managed Delta Lake and managed Apache Iceberg tables. - [Unity Catalog OSS 0.7.0 enables native Iceberg tables by default](/news/unity-catalog-0-7-0-iceberg-rest/): 13 Oct 2026. Unity Catalog 0.7.0 was published on 2026-10-13. - [Delta Lake 4.4.1 patch fixes S3A fast listing snapshots](/news/delta-lake-4-4-1-patch/): 5 Oct 2026. Delta Lake 4.4.1 was published on 2026-10-05 as a patch on 4.4.0. - [Apache Iceberg 1.12.0 released, drops Spark 3.4 support](/news/iceberg-1-12-0-released/): 30 Sep 2026. Apache Iceberg 1.12.0 was published on GitHub on 2026-09-30. - [Apache Polaris 1.8.0 released with concurrent-commit retry fix](/news/polaris-1-8-0-released/): 28 Sep 2026. Apache Polaris 1.8.0 was published on 2026-09-28. - [BigQuery continuous queries can write to Iceberg managed tables](/news/bigquery-continuous-queries-to-iceberg/): 28 Sep 2026. Google's BigQuery release notes dated 2026-09-28 say output rows from continuous queries can be written directly into Apache Iceberg managed tables with an INSERT statement. - [Apache Hudi 1.2.1 moves Trino connector, raises heartbeat tolerance](/news/hudi-1-2-1-trino-packaging/): 24 Sep 2026. Apache Hudi 1.2.1 was published on 2026-09-24. - [Apache XTable 0.4.0 adds Paimon and Parquet sources, Delta Kernel target](/news/xtable-0-4-0-incubating/): 24 Aug 2026. Apache XTable 0.4.0-incubating was published on 2026-08-24. - [Delta Lake 4.4.0 adds Apache Spark 4.2 support and identity columns](/news/delta-lake-4-4-0-released/): 20 Aug 2026. Delta Lake 4.4.0 was released on 2026-08-20. - [Unity Catalog OSS 0.6.0 adds Spark 4.2 connector and SQL views](/news/unity-catalog-0-6-0-released/): 20 Aug 2026. Unity Catalog 0.6.0 was published on 2026-08-20. Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/ --- # Apache Iceberg 1.12.0 released, drops Spark 3.4 support News, 30 Sep 2026 **The short answer.** Apache Iceberg 1.12.0 was published on GitHub on 2026-09-30. Per its release notes, it removes Spark 3.4 support, upgrades the Spark 4.1 module to Spark 4.1.2, adds a REST spec endpoint to unregister a table, and continues work on writing v4 manifests and a compaction-only Spark benchmark. The 1.12.0 release page lists merged pull requests rather than a summary. Among them: removal of Spark 3.4 support (#14122), an upgrade of the Spark 4.1 module to Spark 4.1.2 (#16365), and a new `ignore_missing_files` option on the Spark `migrate` and `snapshot` procedures. For table maintenance, the notes include a Spark benchmark that measures only the rewrite-data-files compaction step (#16219) and a fix for a null-pointer error in `RemoveOrphanFiles` when prefix listing is used on a root table location (#16351). On the format side, the notes list work toward Iceberg v4: Parquet and Avro manifests can now be written in v4 (#15634) and the v4 spec adds relative paths (#15630). A team on Spark 3.4 has to move engine versions before taking this release. ## Sources - Apache Iceberg 1.12.0 release notes, GitHub: https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.12.0 (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/iceberg-1-12-0-released/ --- # Delta Lake 4.4.0 adds Apache Spark 4.2 support and identity columns News, 20 Aug 2026 **The short answer.** Delta Lake 4.4.0 was released on 2026-08-20. According to the project's release notes, Delta Spark, Delta Connect and Delta Sharing now support Apache Spark 4.2.0, CREATE TABLE accepts identity columns, and Delta Kernel and the experimental Flink connector can use the Unity Catalog Delta API. The release notes list four highlights. Spark 4.2.0 support covers Delta Spark, Delta Connect and Delta Sharing. Delta Spark 4.4.0 is built for Spark 4.2.0, 4.1.0 and 4.0.1, with Maven artifacts published for Scala 2.13. In SQL, `CREATE TABLE` now supports `GENERATED ALWAYS AS IDENTITY` and `GENERATED BY DEFAULT AS IDENTITY`. Delta tables created through Unity Catalog can also define generated columns. Delta Kernel and the experimental Flink connector can use the UC Delta API for catalog-managed tables. A team running Delta on Spark 4.0 or 4.1 can take 4.4.0 without changing Spark versions. ## Sources - Delta Lake v4.4.0 release notes, GitHub: https://github.com/delta-io/delta/releases/tag/v4.4.0 (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/delta-lake-4-4-0-released/ --- # Delta Lake 4.4.1 patch fixes S3A fast listing snapshots News, 5 Oct 2026 **The short answer.** Delta Lake 4.4.1 was published on 2026-10-05 as a patch on 4.4.0. Its notes say it skips automatic maintenance on catalog-managed tables so writes complete, and fixes S3A fast listing that could produce incorrect snapshots, so delta.enableFastS3AListFrom can be enabled again. The patch has two bug-fix entries. The first extends existing safeguards on catalog-managed tables to prevent dangerous data and metadata cleanup. Per the notes, automatic maintenance operations are skipped so writes can complete without failure. The second fixes an issue in S3A fast listing that could produce incorrect snapshots. The notes say users can enable fast listing with `delta.enableFastS3AListFrom` again. Delta Spark 4.4.1 is built for Apache Spark 4.2.0, 4.1.0 and 4.0.1. A team that turned fast listing off after the earlier issue can read the release page before turning it back on. ## Sources - Delta Lake v4.4.1 release notes, GitHub: https://github.com/delta-io/delta/releases/tag/v4.4.1 (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/delta-lake-4-4-1-patch/ --- # Apache Hudi 1.2.1 moves Trino connector, raises heartbeat tolerance News, 24 Sep 2026 **The short answer.** Apache Hudi 1.2.1 was published on 2026-09-24. Its notes say the Trino-Hudi connector now lives in the Hudi repo as hudi-trino, built against Trino 483 and JDK 25, and that the hoodie.client.heartbeat.tolerable.misses writer setting now defaults to 10 instead of the previous value of 2. The release page leads with packaging changes. `org.apache.hudi:hudi-trino-bundle` is no longer published, and 1.2.0 is its last version. The connector is now `org.apache.hudi:hudi-trino`, a regular non-shaded jar. Three Trino test Docker images are no longer published. One default changed. `hoodie.client.heartbeat.tolerable.misses` is now 10 instead of 2. With the default 60-second heartbeat interval, the notes say a writer is deemed failed after 10 minutes without a heartbeat, up from 2. The notes describe the tradeoff: a healthy writer is no longer aborted after a transient pause, but pending writes from a crashed writer are detected and rolled back later. Setting the value back to 2 keeps the old behavior. ## Sources - Apache Hudi release-1.2.1 notes, GitHub: https://github.com/apache/hudi/releases/tag/release-1.2.1 (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/hudi-1-2-1-trino-packaging/ --- # Apache XTable 0.4.0 adds Paimon and Parquet sources, Delta Kernel target News, 24 Aug 2026 **The short answer.** Apache XTable 0.4.0-incubating was published on 2026-08-24. Per its release notes, it adds a Delta Kernel conversion target, Apache Paimon as a source, Apache Parquet as a source, a thin xtable-spark-runtime bundle, fixes for Delta, Hudi, Iceberg and Parquet conversions, and an upgrade to Apache Iceberg 1.9.2. The project describes XTable as a cross-table converter for Hudi, Iceberg, Delta Lake and, as a source, Paimon. The 0.4.0 notes add a Delta Kernel conversion target, plus Paimon and Parquet sources with incremental sync. Sync changes include extraction of nested field statistics from Delta sources, nested-column partition support for Hudi, and a fallback to Parquet footers when metadata-table column stats are missing. The notes also list fixes for Delta, Hudi, Iceberg and Parquet conversions. On packaging, the notes list a new xtable-spark-runtime bundle for Spark 3.4 and 3.5 and a fix that makes the bundled xtable-utilities jar runnable again. ## Sources - Apache XTable 0.4.0-incubating release notes, GitHub: https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/xtable-0-4-0-incubating/ --- # Apache Polaris 1.8.0 released with concurrent-commit retry fix News, 28 Sep 2026 **The short answer.** Apache Polaris 1.8.0 was published on 2026-09-28. Per its release notes, concurrent table commits that failed with a fatal 400 now return a retryable 409, and the Python CLI gains GCP external catalog authentication plus --no-sts and --no-kms options for catalog updates. The 1.8.0 release page lists source and binary distributions, a Helm chart, Docker images `apache/polaris:1.8.0` and `apache/polaris-admin:1.8.0`, and Maven artifacts. The first listed fix is for concurrent table commits, which used to fail with a fatal 400 and now return a retryable 409 (#5138). A writer that retries on 409 can now recover from a commit race instead of failing the job. The CLI gets GCP external catalog authentication (#5178), `--no-sts` and `--no-kms` in catalog updates (#5166), and export of nested namespaces in setup (#5147). ## Sources - Apache Polaris 1.8.0 release notes, GitHub: https://github.com/apache/polaris/releases/tag/apache-polaris-1.8.0 (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/polaris-1-8-0-released/ --- # Unity Catalog OSS 0.6.0 adds Spark 4.2 connector and SQL views News, 20 Aug 2026 **The short answer.** Unity Catalog 0.6.0 was published on 2026-08-20. Its release notes add a Spark 4.2.x connector artifact, SQL views as a catalog object, and end-to-end metric view support on Spark 4.2. The connector works with Delta Lake 4.4.0 and uses the UC Delta API for managed tables by default. The open source Unity Catalog 0.6.0 adds `unitycatalog-spark_4.2_2.13` alongside the Spark 4.0.x and 4.1.x artifacts. The notes say the connector works with Delta Lake 4.4.0 and uses the UC Delta API to access UC managed tables by default. SQL views become a catalog object. The server stores the view's SQL text, column list and source tables. Per the notes, Spark 4.0 and 4.1 can read a view but not create, replace, rename or drop one; creating a view needs Spark 4.2. Metric views, which define named measures and dimensions once in the catalog, are fully supported with Spark 4.2 in this release. The notes say 0.5.0 had introduced the catalog side as an experimental preview. ## Sources - Unity Catalog v0.6.0 release notes, GitHub: https://github.com/unitycatalog/unitycatalog/releases/tag/v0.6.0 (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/unity-catalog-0-6-0-released/ --- # Unity Catalog OSS 0.7.0 enables native Iceberg tables by default News, 13 Oct 2026 **The short answer.** Unity Catalog 0.7.0 was published on 2026-10-13. Per its notes, any engine speaking the Iceberg REST protocol can create, write, read, rename and drop Iceberg tables, native Iceberg tables are enabled by default, and dropped managed tables are deleted after a default 7-day delay. The release notes lead with Iceberg REST catalog APIs for table operations. Their Spark example points Iceberg's own REST catalog at the server's `/api/2.1/unity-catalog/iceberg` path with Iceberg 1.11.0 runtime packages, with no UC connector needed. Two behavior changes need a look before upgrading. Servers that never set `server.iceberg-table.enabled` now accept Iceberg creates and commits; setting it to false keeps the old behavior. Dropping a managed table, volume or model now reclaims its storage after `server.storage-cleanup.initial-delay`, default 7 days. Other items: health and metrics endpoints on a dedicated `--obs-port`, multi-instance authorization refresh, and a loadTable signal telling clients which maintenance operations the catalog permits. ## Sources - Unity Catalog v0.7.0 release notes, GitHub: https://github.com/unitycatalog/unitycatalog/releases/tag/v0.7.0 (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/unity-catalog-0-7-0-iceberg-rest/ --- # Amazon S3 Tables adds managed migration from Hive to Iceberg News, 4 Nov 2026 **The short answer.** AWS announced on 2026-11-04 that Amazon S3 Tables can manage migration of Apache Hive tables to Apache Iceberg. AWS says S3 Tables validate, convert and copy data while production workloads run, and that you pay only for the data the job copies, with no charge for the conversion. Per the AWS What's New post, S3 Tables validate, convert and copy the data while production workloads keep running. You can review each step, sync destination tables with source changes as often as needed, or roll back to the source data. AWS states the charge is for data the job copies only, with no charge for the Hive-to-Iceberg conversion, and that source objects in S3 Intelligent-Tiering stay in their current access tier. For tables that use Snappy compression, AWS says S3 Tables recompress them to Zstandard and compact them, which it says reduces storage and object count. The post names the console and the S3 Tables migration APIs as entry points. These are AWS's statements; the post gives no benchmark. ## Sources - AWS What's New, 2026-11-04: https://aws.amazon.com/about-aws/whats-new/2026/11/amazon-s3-tables-managed-migration-hive-to-iceberg/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/s3-tables-hive-to-iceberg-migration/ --- # AWS Glue Data Catalog supports vended credentials for federated Iceberg REST catalogs News, 6 Nov 2026 **The short answer.** AWS announced on 2026-11-06 that Glue Data Catalog supports an external catalog's vended credentials in Iceberg REST catalog federation. Per AWS, Athena, EMR, Redshift and Glue can read external Iceberg tables using coarse-grained table grants, with Lake Formation controlling access down to rows and columns. Glue Data Catalog already federated to external Iceberg REST catalogs through an IAM role that you configure with access to your S3 buckets. The new option connects to an external catalog and reads only the tables that catalog authorizes. AWS says it uses temporary security credentials scoped to the S3 storage prefix and vended by the external catalog. Lake Formation manages access control when you query the external table from any AWS analytics engine. Per the post, the feature is available through the Lake Formation console, the AWS CLI, and the Glue and Lake Formation SDKs, in all commercial AWS Regions where Glue and Lake Formation are supported. ## Sources - AWS What's New, 2026-11-06: https://aws.amazon.com/about-aws/whats-new/2026/11/glue-catalog-vended-credentials-iceberg/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/glue-catalog-vended-credentials-iceberg-rest/ --- # Amazon Athena system tables store usage data as Iceberg tables on S3 Tables News, 9 Nov 2026 **The short answer.** AWS announced on 2026-11-09 that Amazon Athena system tables capture usage data within minutes of query completion. AWS says they are service-managed, read-only Iceberg tables on Amazon S3 Tables with configurable retention, and that AWS handles schema, partitioning, compaction and snapshot maintenance. Per the AWS post, usage data was already available through GetQueryExecution, CloudWatch metrics and CloudTrail logs. System tables add the same data as Iceberg tables you can search, filter, aggregate and join in SQL. AWS gives one example: platform teams can query the `query_executions` table to build cost allocation dashboards showing data scanned by workgroup, user or table, without custom ETL. The post says AWS manages the tables, including compaction and snapshot maintenance. It is a case of an AWS service using S3 Tables maintenance for its own data; the post does not state what it costs. ## Sources - AWS What's New, 2026-11-09: https://aws.amazon.com/about-aws/whats-new/2026/11/athena-system-tables/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/athena-system-tables-on-s3-tables/ --- # Snowflake makes in-place Iceberg v2 to v3 upgrade generally available News, 16 Oct 2026 **The short answer.** Snowflake's release notes dated 2026-10-16 say in-place upgrade of Apache Iceberg tables from v2 to v3 is generally available. Per Snowflake, it works for Snowflake-managed tables and for externally managed tables with CATALOG_SOURCE = ICEBERG_REST, and v3 tables gain expression defaults. Snowflake documents the command as `ALTER ICEBERG TABLE ... UPGRADE TO ICEBERG VERSION 3`. The upgraded table keeps its name, privileges and downstream references. Snowflake lists default values, row lineage, deletion vectors and v3 data types as the v3 features it covers. The notes state that upgrading is permanent, though for Snowflake-managed tables the pre-upgrade v2 table can be restored with Time Travel within the retention period. Snowflake-managed v3 tables now support SQL expressions, sequences and identity columns as defaults. Defaults such as `CURRENT_DATE()` are evaluated only for Snowflake writes and need the new `ALLOW_V3_EXPRESSION_DEFAULTS` parameter. Other engines that write the table need to handle that. ## Sources - Snowflake release notes, 2026-10-16: https://docs.snowflake.com/en/release-notes/2026/other/2026-10-16-iceberg-v3-upgrade-expression-defaults-ga (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/snowflake-iceberg-v3-upgrade-ga/ --- # Snowflake opens externally managed Iceberg tables to outside engines via Horizon News, 29 Oct 2026 **The short answer.** Snowflake's release notes dated 2026-10-29 say access to externally managed Iceberg tables through Snowflake Horizon Catalog is generally available. Per Snowflake, Spark, Trino, DuckDB or PyIceberg can read and write them through the Horizon Iceberg REST Catalog API, with Snowflake RBAC applied. The notes say the tables can sit in a catalog-linked database or in a standard Snowflake database, and that Snowflake RBAC applies to both. Two limits are stated. Access to an externally managed Iceberg table that has a Snowflake data protection policy remains in preview. Access through Horizon Catalog is not available on FedRAMP (Moderate) or FedRAMP (High) deployments. A separate Snowflake note dated 2026-10-20 says that when you raise TARGET_FILE_SIZE on a Snowflake-managed Iceberg table, compaction recompacts older files; its example goes from 64 MB to 128 MB. ## Sources - Snowflake release notes, 2026-10-29: https://docs.snowflake.com/en/release-notes/2026/other/2026-10-29-externally-managed-iceberg-tables-horizon-irc-ga (read 10 Nov 2026) - Snowflake release notes, 2026-10-20: https://docs.snowflake.com/en/release-notes/2026/other/2026-10-20-iceberg-target-file-size-recompact (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/snowflake-horizon-external-iceberg-ga/ --- # Databricks: multi-table transactions generally available for managed Delta and Iceberg News, 14 Oct 2026 **The short answer.** Databricks's release notes dated 2026-10-14 say transactions are generally available for Unity Catalog managed Delta Lake and managed Apache Iceberg tables. Per Databricks, BEGIN ATOMIC ... END or BEGIN TRANSACTION ... COMMIT groups SQL statements across one or more tables, and catalog commits must be enabled. Databricks documents two syntaxes: `BEGIN ATOMIC ... END;` and `BEGIN TRANSACTION; ... COMMIT;`. Statements can span one or more managed Delta or managed Iceberg tables in a single atomic transaction. The stated requirement is that catalog commits are enabled on participating tables. This ties the feature to Unity Catalog managed tables, so external tables are outside what the note describes. An earlier Databricks note, dated 2026-09-11, says sharing foreign Iceberg tables with OpenSharing is generally available, including to recipients that use external Iceberg clients. ## Sources - Databricks platform release notes, October 2026: https://docs.databricks.com/aws/en/release-notes/product/2026/october (read 10 Nov 2026) - Databricks platform release notes, September 2026: https://docs.databricks.com/aws/en/release-notes/product/2026/september (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/databricks-transactions-delta-iceberg-ga/ --- # BigQuery continuous queries can write to Iceberg managed tables News, 28 Sep 2026 **The short answer.** Google's BigQuery release notes dated 2026-09-28 say output rows from continuous queries can be written directly into Apache Iceberg managed tables with an INSERT statement. Google describes it as a way to process streaming data and land it in an open-format lakehouse. The 2026-09-28 entry is one line in the BigQuery release notes. It says rows produced by continuous queries can now go straight into Iceberg managed tables using an `INSERT` DML statement. A later entry, dated 2026-10-14, says the Apache Hive catalog endpoint in the Lakehouse runtime catalog is generally available and that metadata migration from Dataproc Metastore to Lakehouse is generally available, to either an Iceberg REST catalog endpoint or a Hive catalog endpoint. The notes are Google's own descriptions and carry no performance or cost figures. ## Sources - BigQuery release notes, 2026-09-28: https://docs.cloud.google.com/bigquery/docs/release-notes#September_28_2026 (read 10 Nov 2026) - BigQuery release notes, 2026-10-14: https://docs.cloud.google.com/bigquery/docs/release-notes#October_14_2026 (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/news/bigquery-continuous-queries-to-iceberg/ --- # Questions about lakehouse platforms, answered from published facts FAQ, read 10 Nov 2026 **The short answer.** Thirty questions a data engineer asks when choosing or re-choosing a lakehouse platform, answered in a few sentences each with a source link: formats, compaction, Spark versions, regions, billing units, published prices and benchmarks. Where nothing was found, the answer says so with a date. ## 30 questions and answers **What is a data lakehouse?** Databricks's documentation defines a data lakehouse as a data management system that combines the benefits of data lakes and data warehouses. In practice that means table data in open files on object storage such as Amazon S3, plus a metadata layer that adds transactions and schema enforcement, so BI and machine learning jobs read the same copy. Databricks's own lakehouse uses Delta Lake for storage and Unity Catalog for governance. **How do Delta Lake, Hudi and Iceberg describe themselves?** Delta Lake's documentation describes an open source storage layer with ACID transactions, scalable metadata handling and time travel over S3, ADLS, GCS and HDFS. Apache Hudi says it pioneered the transactional data lake and is built for high-performance writes on incremental pipelines. Apache Iceberg describes a high-performance format for huge analytic tables that Spark, Trino, Flink, Presto, Hive and Impala can use at the same time. These are each project's own descriptions. **Is a table format the same thing as a file format?** No. Data files are usually Parquet, and the table format is the metadata that says which files make up the table. In Iceberg, table state is kept in metadata files, each change writes a new metadata file and swaps it in atomically, and snapshots point to manifest files that list the data files with their partition data and metrics. Delta Lake keeps a transaction log and Hudi a timeline for the same job. **Can one table be read as Delta, Iceberg and Hudi?** Two published routes exist. Delta UniForm, documented from Databricks Runtime 14.3 LTS, generates Iceberg metadata for a Delta table registered in Unity Catalog without rewriting the Parquet files. Apache XTable, an incubating Apache project, translates metadata between Hudi, Delta Lake and Iceberg in any direction and writes it under the table's base path (_delta_log, metadata or .hoodie). Which engines accept the generated metadata depends on the engine and version. **What does Databricks publish about the Photon engine?** Databricks's documentation describes Photon as a native C++ vectorized engine that keeps Spark's Catalyst planner, works with existing code and falls back to the Spark runtime for unsupported operations. It is on by default for classic all-purpose compute, jobs compute and classic pipelines. Databricks claims up to 5x better price/performance than other cloud data warehouses on that page, and the page gives no method for the figure. **How does Databricks bill compute, and what is the price per DBU?** Databricks bills compute in DBUs at per-second granularity, with pay-as-you-go or committed-use discounts and no up-front cost. Jobs pricing offers serverless (Performance Optimized and Standard modes) and classic clusters, and Photon instances consume DBUs at a different rate than the same instance without Photon. The pricing pages draw their rates by script, but the data file behind the pricing calculator lists Jobs Compute on AWS at $0.15 per DBU on the Premium plan and $0.20 on the Enterprise plan, and 4.08 DBU per node-hour for r6i.4xlarge (read 2026-11-10). On the Compute Bill that is $12.24 of DBUs plus $20.16 of EC2. **How does Snowflake bill compute, and what is the price per credit?** Snowflake bills virtual warehouses in credits per second, with a 60-second minimum each time a warehouse starts. Each size step roughly doubles the rate: Small is 2 credits per hour and Medium is 4. Serverless features, compute pools and cloud services also consume credits. Snowflake's [Service Consumption Table](https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf), effective 2026-11-09 and read for the Compute Bill on 2026-11-10, lists on-demand credits on AWS US East (Northern Virginia) at $2.00 for Standard, $3.00 for Enterprise and $4.00 for Business Critical. Other clouds and regions are priced separately. **Can Snowflake run existing PySpark code?** Snowflake's documentation says Snowpark Connect for Apache Spark runs PySpark DataFrame code against the Snowflake engine over the Spark Connect protocol, with no Spark cluster. It supports Apache Spark 3.5 only; code written for 3.4 and below or 4.0 and above is not supported. Interval types and user-defined types are unsupported, and byte, short and integer columns come back as LongType, so test result schemas before relying on it. **Which Apache Spark versions do recent Databricks Runtime releases ship?** Per Databricks's release notes, Runtime 18 LTS ships Apache Spark 4.1.0 (released June 10, 2026), Runtime 17.3 LTS ships Spark 4.0.0, and Runtimes 16.4 LTS and 15.4 LTS ship Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). Spark version is the first check on a move to or from another platform, since Snowpark Connect supports only 3.5 and Amazon EMR 7.x builds list Spark 3.5. **Which open table formats does Snowflake read and write?** Snowflake's documentation describes Iceberg tables in Parquet, with data in customer-managed S3, Google Cloud Storage or Azure storage through external volumes, an external catalog integration such as AWS Glue, and Delta files read through Delta Direct. The page read did not list Hudi. Snowflake is available on AWS, Azure and Google Cloud, and each account is hosted in a single region. **What table maintenance does Apache Iceberg require?** The Iceberg maintenance page lists expiring snapshots, removing old metadata files and deleting orphan files as recommended. Compacting data files with the Spark rewriteDataFiles action (the sample uses a 500 MB target file size) and rewriting manifests are optional. Each is a job you schedule unless a platform or catalog runs it for you. A retention interval shorter than a write takes can make orphan removal delete live files; the default interval is 3 days. **Who runs Iceberg compaction on AWS?** It depends on the service, listed alphabetically. Amazon Athena compaction is manual: you run OPTIMIZE with REWRITE DATA USING BIN_PACK and VACUUM yourself. AWS Glue Data Catalog compaction is off by default for new tables and is enabled per table with an IAM role. Amazon S3 Tables compaction is on by default for every table, with a 512 MB default target file size you can set between 64 MB and 512 MB. **What does Amazon S3 Tables charge for compaction?** In us-east-1, per the AWS price list read on 2026-11-10, S3 Tables compaction costs $0.005 per GB processed plus $0.002 per 1,000 objects compacted. Sort or Z-order compaction costs $0.01 per GB plus $0.002 per 1,000 objects. Storage is $0.0265 per GB-month for the first 50 TB. Other regions and tiers are priced separately, and these figures are list prices, not a bill for any particular table. **What do Athena and Glue charge in us-east-1?** Per the AWS price lists read on 2026-11-10, Athena SQL costs $5.00 per TB scanned, Athena for Apache Spark costs $0.35 per DPU-hour and provisioned capacity costs $0.30 per DPU-hour. AWS Glue Iceberg compaction costs $0.44 per DPU-hour, Glue 6.0 and later ETL jobs cost $0.308 per DPU-hour, and older Glue ETL costs $0.44. These are list prices for one region. **What does Amazon EMR add to the EC2 price?** EMR on EC2 adds a per-instance charge to the EC2 and EBS price, billed per second with a one-minute minimum. AWS's pricing page example in us-east-1 shows a c4.2xlarge at $0.105 per hour for EMR next to $0.398 for EC2, and one master with two core nodes for a month at $1,101.57. EMR on EKS adds an uplift per vCPU-hour and per GB-hour, plus $0.10 per hour per EKS cluster. **What does EMR Serverless charge?** AWS's EMR pricing page uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour in its EMR Serverless worked example. In that example a job with 100 vCPU and 750 GB for 30 minutes, followed by 200 more vCPU for 15 minutes, costs $9.596275. The page also notes that Spot instances on EMR on EC2 are priced up to 90% below On-Demand. Rates differ by region. **What does Dremio Cloud charge?** Dremio's pricing page lists Dremio Cloud at $0.20 per Dremio Compute Unit, with a free trial of a $400 credit for 30 days. Dremio Enterprise, the self-managed product, is priced through sales. Per Dremio's engine docs, DCUs accrue only while engines run, with a 60-second minimum, at published hourly rates from 16 DCU for a 2XSmall engine up to 2,048 DCU for a 3XLarge engine. **What does Starburst Galaxy charge?** Starburst's pricing page lists Galaxy credits from $0.50 (Pro), $0.75 (Enterprise) and $1.00 (Mission Critical) per credit on AWS in US East (N. Virginia), US East (Ohio) and US West (Oregon). Prices differ by cloud and region; Europe (Frankfurt) is $0.60, $0.90 and $1.20. Cluster compute uses 6 credits per hour for one worker and 48 for eight workers. Annual commitments may be discounted. **What does Microsoft Fabric charge for Spark?** The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour in eastus on pay-as-you-go, read on 2026-11-10. In Fabric each capacity unit maps to two Spark VCores, and jobs are admitted by available VCores with queueing and throttling. Fabric runs on Azure; the region list names no AWS regions, though OneLake shortcuts can read S3 data without copying it. **What does BigQuery publish about its pricing?** Google's pricing page describes on-demand pricing per TiB scanned, with the first 1 TiB per month free, and capacity pricing per slot-hour. BigQuery Omni, which queries S3 data through BigLake tables in six AWS regions, has its own on-demand per-TiB pricing. The page text read on 2026-11-10 lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate; that is the rate the Compute Bill uses. Per-TiB and Spark DCU rates are not used on this site. Omni does not support DML or CREATE TABLE. **What does moving from Databricks to an open stack on AWS involve?** AWS describes the EMR runtime for Spark as fully API compatible with open source Spark and Iceberg, so Spark jobs run as is. Amazon EMR 7.14.0 packages Spark 3.5.8, Delta 3.3.2, Iceberg 1.10.1 and Hudi 1.0.2, whereas Databricks Runtime 18 LTS ships Spark 4.1.0. The team then owns table maintenance, tuning and operations: expiring snapshots, compaction and catalog choice become jobs on its own schedule. **Which Spark features are unavailable on Databricks serverless compute?** Databricks's documentation says serverless compute for notebooks and jobs supports only Spark Connect APIs. Spark RDD APIs, R, Maven coordinates, global temp views, DBFS mounts with instance profiles and Databricks Container Services are not supported. ANSI mode is on by default and the default time zone is UTC, which can change results. The Spark UI and Spark logs are not available; the query profile and client-side logs are used instead. **How do Delta Lake and Hudi handle compaction on the open stack?** Delta Lake's open source documentation lists file management features that the user applies: bin-packing compaction with OPTIMIZE, auto compaction, data skipping, Z-ordering, multi-part checkpointing, log compactions and optimized write. For Hudi, merge-on-read tables need compaction to merge log files into base files and bound read latency; it has two steps, scheduling a plan on the timeline and then executing it. **Which open source catalogs exist for Iceberg tables?** Two open source projects are Project Nessie, a transactional catalog with Git-like semantics and cross-table transactions, and Apache Polaris, which implements Iceberg's REST API and lists Apache Doris, Flink, Spark, Dremio OSS, StarRocks and Trino among supported engines. AWS Glue Data Catalog is a managed option. Catalogs and governance are covered here in one sentence only; check engine support for the catalog protocol you pick. **How does Onehouse bill, and where does it run?** Per Onehouse's billing documentation, usage is metered in Onehouse Compute Units: one OCU is a 4-core instance for 1 hour in your own AWS or Google Cloud account, plus support costs, and you pay your own cloud bill. Standard AWS instance types map to Graviton m8g sizes. The pages read give no dollar rate per OCU, so the price is none found on 2026-11-10. Onehouse claims 2 to 3x better price-performance for ETL. **What benchmark has AWS published for EMR against open source Spark?** AWS ran TPC-DS at 3 TB (104 SparkSQL queries, 3 rounds, 9 r5d.4xlarge nodes, Iceberg) and reports EMR 7.12 averaging 1,349.62 seconds against 6,113.92 seconds for open source Spark 3.5.6 with Iceberg 1.10.0. Its stated cost was $4.81 for EMR 7.12 and $17.65 for open source Spark. AWS ran it, AWS notes the results are not comparable to official TPC-DS, and AWS publishes the setup on GitHub. **What does DualBird claim for Spark and Iceberg compaction?** These are DualBird's own claims. Its home page says 10x-30x faster data processing with 50%-90% lower costs and no changes required. For sort-based Iceberg compaction of 100 GB (synthetic data, about 370 million rows), DualBird reports roughly 12-20x faster single-task execution and about $2 per TB compacted against about $6 rising past $13 per TB on CPU clusters. DualBird ran the test, and the white paper with the full method is behind a form. **Where does DualBird fit in a lakehouse?** As read on 2026-11-10: DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. DualBird says its plugin needs AWS EC2 F2 instances plus a plugin change, with no code changes. **Does DualBird publish a price or a Spark version list?** None found on 2026-11-10. DualBird's site shows percentages (50%-90% lower costs, 55%-85% lower EC2 cost) and the compaction figure of about $2 per TB, but no price list, marketplace listing or pricing model for the plugin. AWS lists F2 instances in eight regions (us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2). DualBird says workloads dominated by Python UDFs fall back to CPU. **Which platforms write Hudi tables?** Documentation differs by platform, listed alphabetically. Dremio's data formats page lists Iceberg and Delta Lake and does not list Hudi. Snowflake's Iceberg page did not list Hudi. Starburst Galaxy reads Hudi tables only: its docs say Galaxy cannot create new Hudi tables or write to them, while Iceberg tables can be created and written. Onehouse says its Quanton jobs read and write Hudi, Iceberg and Delta Lake. ## Sources - What is a data lakehouse? (Databricks on AWS): https://docs.databricks.com/aws/en/lakehouse/ (read 10 Nov 2026) - Apache Iceberg home page: https://iceberg.apache.org/ (read 10 Nov 2026) - Iceberg table spec: https://iceberg.apache.org/spec/ (read 10 Nov 2026) - Apache XTable (Incubating): https://xtable.apache.org/ (read 10 Nov 2026) - Photon (Databricks on AWS): https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Databricks pricing data file (AWS), loaded by its pricing calculator: https://www.databricks.com/en-pricing-assets/data/pricing/AWS.json (read 10 Nov 2026) - Understanding compute cost (Snowflake documentation): https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026) - Snowpark Connect limitations (Snowflake documentation): https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Databricks Runtime release notes: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Apache Iceberg tables (Snowflake documentation): https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Iceberg documentation: maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Maintenance for tables (Amazon S3 User Guide): https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - AWS price list: Amazon S3: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - AWS price list: Amazon Athena: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026) - Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Dremio pricing: https://www.dremio.com/pricing (read 10 Nov 2026) - Starburst Galaxy pricing details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Spark job concurrency and queueing in Fabric: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - BigQuery pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Amazon EMR 7.x release versions: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Serverless compute limitations (Databricks on AWS): https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Delta Lake documentation: optimizations: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026) - Apache Polaris: https://polaris.apache.org/ (read 10 Nov 2026) - Onehouse usage and billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - AWS Big Data Blog: Spark and Iceberg on EMR: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - DualBird blog: Iceberg compaction benchmark: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - DualBird home page: https://dualbird.io/ (read 10 Nov 2026) - Starburst Galaxy: Hudi table format: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/faq/ --- # Five situations, what to check in each Situations, read 10 Nov 2026 **The short answer.** Five buyer situations, each with the checks a team in that situation should run and the published facts that answer them: a Spark bill that doubled on EMR or EKS, nightly pipelines that miss the morning, AI data preparation, leaving Databricks or Snowflake, and daily Iceberg compaction cost. ## Five situations - [Spark bill doubled on EMR or EKS](/situations/bill-doubled/): What to check when an EMR or EKS Spark bill doubled: billing units, cost claims, code changes, settings that move spend, and region, with published facts. - [Nightly pipelines finish after the morning deadline](/situations/late-pipelines/): For nightly pipelines that miss the morning: published runtimes, who ran them, whether a reader can repeat them, failure behavior on large jobs, and Spark versions. - [AI and feature pipelines wait on data preparation](/situations/ai-data-prep/): For AI training and feature pipelines that wait on data prep: table formats, unchanged Spark code, UDF and fallback behavior, and where data has to live. - [Leaving Databricks or Snowflake for an open stack](/situations/leaving-managed/): For a team weighing a move from Databricks or Snowflake to an open stack: what you own afterward, shared table formats, Spark changes and cloud coverage. - [Large Iceberg tables with daily compaction cost](/situations/iceberg-compaction/): For teams paying for daily Iceberg compaction: who runs it, the published unit prices, strategies, delete handling and the benchmarks that exist. Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/situations/ --- # Spark bill doubled on EMR or EKS: what to check on a lakehouse platform Situation, read 10 Nov 2026 **The short answer.** A data platform team on EMR or EKS sees the Spark bill double in a year and asks whether another way of running the same work costs less. The checks below cover the billing unit, the cost claims and who tested them, whether jobs run unchanged, the settings that move the bill, and the AWS region. ## 5 things to check, with the published facts that answer each ### Which unit does each platform bill compute in, and is the rate published? A bill that doubled is a volume times a rate. The unit tells you which of the two you can change, and an unpublished rate cannot be compared before a trial. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | The AWS price list shows an EMR uplift of $0.048 per hour for an m5.xlarge in us-east-1, added to the EC2 price; EMR on EKS adds $0.007084 per vCPU-hour and $0.000777875 per GB-hour. | [AWS price list](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Databricks bills compute in DBUs per second; the pricing pages render their rates by script, and the data file behind the pricing calculator lists Jobs Compute on AWS at $0.15 per DBU on the Premium plan (read 2026-11-10). | [Databricks pricing data file (AWS), loaded by its pricing calculator](https://www.databricks.com/en-pricing-assets/data/pricing/AWS.json), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio Cloud is priced at $0.20 per DCU, billed only while engines run with a 60-second minimum; Dremio Enterprise pricing is "contact sales". | [Dremio page](https://www.dremio.com/pricing), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | A price or pricing model for the DualBird plugin: none found on 2026-11-10. | [DualBird page](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigQuery offers on-demand pricing per TiB scanned or capacity pricing per slot-hour; the page text read on 2026-11-10 lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate. | [Google Cloud documentation](https://cloud.google.com/bigquery/pricing), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus, pay-as-you-go. | [Azure retail price API](https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse bills in OCUs, where 1 OCU is a 4-core instance for 1 hour in your account, plus support costs; you also pay your own AWS or GCP bill. A dollar rate per OCU: none found on 2026-11-10. | [Onehouse documentation](https://docs.onehouse.ai/product/project-settings/usage-and-billing), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | AWS's example in us-east-1 shows a c4.2xlarge EMR charge of $0.105 per hour next to $0.398 per hour for EC2; spot instances are up to a 90% discount versus On-Demand. | [AWS page](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowflake's Service Consumption Table, effective 2026-11-09, lists on-demand credits on AWS US East (Northern Virginia) at $2.00 (Standard), $3.00 (Enterprise) and $4.00 (Business Critical); warehouses bill per second with a 60-second minimum on each start. | [Snowflake Service Consumption Table](https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst Galaxy cluster compute uses 6 credits per hour for 1 worker and 48 for 8 workers; credit prices on AWS start at $0.50 (Pro) in US East (N. Virginia). | [Starburst page](https://www.starburst.io/pricing/pricing-details/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### What does each vendor claim about cost, who ran the test, and is the method published? A cost claim without a workload, a baseline or a method cannot be tied to your jobs. The party that ran the test matters as much as the percentage. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | AWS states the EMR runtime runs up to 4.5x faster than open-source Spark with 2.7x faster Iceberg writes; the page gives no workload, scale or configuration. | [AWS page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Databricks documentation claims up to 5x better price/performance for Photon versus other cloud data warehouses, with no method given on that page. | [Databricks documentation](https://docs.databricks.com/aws/en/compute/photon), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | DualBird says 50% to 90% lower costs with 10x to 30x faster processing; no method is given for that range, and the claim is DualBird's own. | [DualBird page](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | Google states Lightning Engine delivers up to 4.9x faster performance than open-source Spark; no method is given on the pricing page. | [Google Cloud documentation](https://cloud.google.com/dataproc-serverless/pricing), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | Microsoft reports up to six times faster than open-source Spark on TPC-DS scale factor 1000 with Delta, with about 83% compute cost savings on a fixed-size cluster; its method: none found on 2026-11-10. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse states a guaranteed cost reduction of more than 50% for existing Hudi users moving Spark jobs from EMR or open-source Spark; the guarantee is a vendor statement. | [Onehouse page](https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | AWS reports $4.81 for EMR 7.12, $5.47 for EMR 7.5 and $17.65 for open-source Spark 3.5.6 with Iceberg 1.10.0 on TPC-DS 3 TB, On-Demand pricing; AWS ran the test. | [AWS page](https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | A published Snowflake benchmark of Spark batch jobs on Snowpark Connect with its method: none found on 2026-11-10. | [Snowflake documentation](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst reports GPU-accelerated Trino at 3.4x overall on ClickBench in a single-node development environment; that is a SQL query test, not Spark batch work. | [Starburst page](https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Do your existing jobs run unchanged, or does a move to another option need a rewrite? A rewrite is a cost that does not appear on the vendor price page. Check what each platform says about running existing Spark code. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | AWS describes the EMR runtime for Spark as 100% API-compatible with open-source Spark, so existing Spark jobs need no rewrite. | [AWS page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Databricks serverless compute supports only Spark Connect APIs; RDD APIs, R and Maven coordinates are unsupported, and ANSI mode on by default can change results of existing jobs. | [Databricks documentation](https://docs.databricks.com/aws/en/compute/serverless/limitations), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio runs queries on its own Arrow-based engine, not on Spark, so Spark ETL would stay on Spark or be rewritten as Dremio SQL. | [Dremio documentation](https://docs.dremio.com/current/what-is-dremio/architecture), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | DualBird says onboarding is two changes: switch the EC2 instance type to F2 and add the DualBird Spark plugin. | [DualBird page](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigQuery runs PySpark or Scala/Java JAR code from SQL through CREATE PROCEDURE ... OPTIONS(engine = "SPARK"), so code must be wrapped as procedures. | [Google Cloud documentation](https://cloud.google.com/bigquery/docs/spark-procedures), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | Microsoft says the Fabric native execution engine needs no code changes, falls back to JVM Spark for unsupported operators and supports Parquet and Delta. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse documentation lists the EMR migration work: map EMR steps to Onehouse Jobs created through its API or Airflow operators, and configure Glue catalog sync. | [Onehouse documentation](https://docs.onehouse.ai/product/jobs/emr-migration), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | AWS says the EMR runtime provides full API compatibility with open-source Apache Spark and Apache Iceberg; self-run Spark on EKS uses the open-source Spark operator and needs no job rewrite. | [AWS page](https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowpark Connect supports Apache Spark 3.5 only; workloads written for 3.4 and below or 4.0 and above are unsupported and may fail. | [Snowflake documentation](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst Galaxy runs SQL on Trino, so Spark DataFrame code, UDFs and PySpark would be rewritten as Trino SQL or kept on Spark against the same Iceberg tables. | [Starburst documentation](https://docs.starburst.io/starburst-galaxy/index.html), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which settings or limits can raise the bill without a code change? Idle capacity, scaling limits and instance choices change spend. A setting you must tune is a cost you own. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | EMR enables adaptive query execution, dynamic partition pruning, bloom filter join and optimized join reorder; AQE adaptive join conversion is on by default and can be switched off. | [AWS documentation](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Photon instance types consume DBUs at a different rate than the same instance without Photon, and API users must set runtime_engine to PHOTON. | [Databricks documentation](https://docs.databricks.com/aws/en/compute/photon), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio engine settings the admin chooses include size, minimum and maximum replicas, last-replica auto-stop (default 2 hours) and a query runtime limit. | [Dremio documentation](https://docs.dremio.com/dremio-cloud/admin/engines/), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | DualBird claims many Spark settings become irrelevant and cluster sizing reduces to the number of F2 nodes; these are vendor claims, and DualBird says skew still costs time, but less. | [DualBird page](https://dualbird.io/product?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | Fabric Autoscale Billing for Spark is opt-in, runs on dedicated serverless resources at pay-as-you-go CU-hour rates, and a base capacity is still required. | [Azure pricing page](https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse clusters need minimum and maximum OCU limits; the documentation warns that at the maximum a cluster stops scaling up, which can delay processing. | [Onehouse documentation](https://docs.onehouse.ai/product/compute/create-cluster), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | EMR on EKS adds an uplift per vCPU-hour and per GB-hour plus $0.10 per hour for each EKS cluster, and the uplift applies to any pod using an EMR runtime image. | [AWS page](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowflake documents Optimizing and Troubleshooting sections for Snowpark Connect workloads, so some job-level adjustment is expected after migration. | [Snowflake documentation](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst Galaxy offers a fault tolerant execution mode for batch-style work that spools intermediate data; its documentation says it can be slightly slower than normal execution. | [Starburst documentation](https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Does the price or availability change with the AWS region you run in? The reference prices on this site are us-east-1. A region you cannot use, or a platform tied to another cloud, changes the comparison. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | All services in this entry are AWS services and run only on AWS; EMR, Athena, Glue and S3 are priced here in us-east-1. | [AWS page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Databricks lists many AWS regions including us-east-1, us-east-2, eu-west-1 and eu-central-1, and is also hosted on Azure and Google Cloud; some features are only in a subset of regions. | [Databricks documentation](https://docs.databricks.com/aws/en/resources/supported-regions), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio Cloud runs on AWS in us-east-1, us-east-2, us-west-2, eu-west-1 and eu-central-1; Azure is listed as coming soon. | [Dremio documentation](https://docs.dremio.com/dremio-cloud/about/regions), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | AWS lists EC2 F2 instances in eight regions; us-east-2 and eu-west-1 are not among them. DualBird's pages mention no cloud other than AWS. | [AWS documentation](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigQuery Omni reads data in S3 in us-east-1, us-west-2, ap-northeast-2, ap-southeast-2, eu-west-1 and eu-central-1. | [Google Cloud documentation](https://cloud.google.com/bigquery/docs/omni-introduction), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | Fabric runs over OneLake on Azure; the region page lists Azure regions and no AWS regions, so Spark compute does not run inside AWS. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse runs inside your AWS or GCP account; a project is scoped to one region and account, and adding a region needs a support ticket. | [Onehouse documentation](https://docs.onehouse.ai/product/project-settings/usage-and-billing), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | Amazon EMR has regional endpoints in us-east-1, us-east-2, us-west-1, us-west-2 and many other AWS regions; you choose the EC2 instance types. | [AWS documentation](https://docs.aws.amazon.com/general/latest/gr/emr.html), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowflake runs on AWS, Azure and Google Cloud; each account is hosted in a single region. | [Snowflake documentation](https://docs.snowflake.com/en/user-guide/intro-regions.md), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst lists a different credit price by AWS region: Pro credits cost $0.50 in US East (N. Virginia), $0.56 in Ireland, $0.60 in Frankfurt and $0.64 in Tokyo. | [Starburst page](https://www.starburst.io/pricing/pricing-details/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ## Sources - AWS price list: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json (read 10 Nov 2026) - Databricks pricing data file (AWS), loaded by its pricing calculator: https://www.databricks.com/en-pricing-assets/data/pricing/AWS.json (read 10 Nov 2026) - Dremio page: https://www.dremio.com/pricing (read 10 Nov 2026) - DualBird page: https://dualbird.io/ (read 10 Nov 2026) - Google Cloud documentation: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Azure retail price API: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption' (read 10 Nov 2026) - Onehouse documentation: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - AWS page: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Snowflake Service Consumption Table: https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf (read 10 Nov 2026) - Starburst page: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - AWS page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Databricks documentation: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Google Cloud documentation: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) - Microsoft Learn documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Onehouse page: https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton (read 10 Nov 2026) - AWS page: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Snowflake documentation: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Starburst page: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026) - Databricks documentation: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Dremio documentation: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Google Cloud documentation: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - Onehouse documentation: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - Snowflake documentation: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) - Starburst documentation: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - AWS documentation: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - Dremio documentation: https://docs.dremio.com/dremio-cloud/admin/engines/ (read 10 Nov 2026) - DualBird page: https://dualbird.io/product (read 10 Nov 2026) - Azure pricing page: https://azure.microsoft.com/en-us/pricing/details/microsoft-fabric/ (read 10 Nov 2026) - Onehouse documentation: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Snowflake documentation: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-troubleshooting.md (read 10 Nov 2026) - Starburst documentation: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026) - Databricks documentation: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Dremio documentation: https://docs.dremio.com/dremio-cloud/about/regions (read 10 Nov 2026) - AWS documentation: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html (read 10 Nov 2026) - Google Cloud documentation: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Microsoft Learn documentation: https://learn.microsoft.com/en-us/fabric/fundamentals/microsoft-fabric-overview (read 10 Nov 2026) - AWS documentation: https://docs.aws.amazon.com/general/latest/gr/emr.html (read 10 Nov 2026) - Snowflake documentation: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/situations/bill-doubled/ --- # Nightly pipelines no longer finish by morning: what to check on a lakehouse platform Situation, read 10 Nov 2026 **The short answer.** A team whose nightly Spark pipelines run past the morning deadline needs to know what each platform publishes about batch runtime. The checks below cover which benchmarks exist and who ran them, whether they resemble batch ETL, whether a reader can repeat them, behavior on very large jobs, and Spark versions. ## 5 things to check, with the published facts that answer each ### Which runtime results are published, and who ran each one? A result from the vendor is a claim until another party repeats it. Check the workload, the data size and the comparison baseline before reading the multiple. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | AWS states up to 4.5x faster than open-source Spark and 2.7x faster Iceberg writes on its EMR features page, with no workload or scale given. | [AWS page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Databricks announced an audited 100 TB TPC-DS result for Databricks SQL on 2021-11-02; it is a SQL warehouse result, not a Spark batch ETL result. | [Databricks page](https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio's founder reported all 99 TPC-DS queries at 1 TB in 22 seconds on an 8-node m7gd.4xlarge cluster (2025-06-16); a BI query test run by Dremio, not batch ETL. | [Dremio page](https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | DualBird reports about 12x to 20x faster single-task execution for sort-based Iceberg compaction on 100 GB (June 11, 2026); the test was run by DualBird. | [DualBird page](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | Google's pricing page claims up to 4.9x faster performance for its Lightning Engine tier than open-source Spark and gives no method. | [Google Cloud documentation](https://cloud.google.com/dataproc-serverless/pricing), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | Microsoft reports up to six times faster than open-source Spark on TPC-DS scale factor 1000 with Delta; Microsoft ran the test. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse reports TPC-DS up to about 3.1x, TPCx-BB 2x and TPC-DI (1 TB) 2x against open-source Spark on Iceberg, with 4.1x for LakeLoader writes; Onehouse ran the tests. | [Onehouse page](https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | AWS ran TPC-DS 3 TB (104 queries, 3 rounds, 9 x r5d.4xlarge) and reports EMR 7.12 averaging 1,349.62 s against 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0. | [AWS page](https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | A published benchmark of Spark batch jobs on Snowpark Connect with its method: none found on 2026-11-10. | [Snowflake documentation](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst reports TPC-H up to 6x on accelerated queries with GPU Trino in a single-node environment; it is a SQL query test, not Spark batch work. | [Starburst page](https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Is the published workload similar to a nightly batch job? TPC-DS style query tests, compaction and a customer pipeline stress different parts of an engine. A result only transfers to your job if the work is similar. | Option | Published fact | Source, read | | --- | --- | --- | | [Databricks](/platforms/databricks/) | A Databricks benchmark of Spark batch ETL (not SQL warehouse) with a published method: none found on 2026-11-10. | [Databricks documentation](https://docs.databricks.com/aws/en/compute/photon), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Published cost for the same Spark batch work on Dremio: none found on 2026-11-10; the 22-second result is query only. | [Dremio page](https://www.dremio.com/pricing), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | DualBird shows a 110 GB enrichment pipeline (JOIN denormalization) at 6x to 18x higher performance ceiling and 50% to 85% lower EC2 cost, with no hardware, baseline, date or method. | [DualBird page](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | An unnamed B2B SaaS customer ran a UDF-heavy pipeline on EMR and on Onehouse Quanton in its own AWS account; Quanton was faster at p50, p97 and p99, and the write-up is published by Onehouse. | [Onehouse page](https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | A published benchmark of Iceberg, Delta or Hudi compaction time or cost on EMR or EKS: none found on 2026-11-10. | [Apache Iceberg documentation](https://iceberg.apache.org/docs/latest/maintenance/), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | A Starburst benchmark of ETL, joins or compaction on Galaxy CPU clusters with data size and scripts: none found on 2026-11-10. | [Starburst page](https://www.starburst.io/sitemap.xml), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Can a reader repeat the test with published scripts? A repeatable test lets you run your own data through the same setup. A result without scripts can only be read, not checked. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | Published scripts for the AWS EMR marketing claim of 4.5x: none found on 2026-11-10. | [AWS page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Repeatable scripts for the Databricks TPC-DS record: none found on 2026-11-10. | [Databricks page](https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio published a 2020 TPC-DS methodology post with JMeter test plans; the 2025 22-second post gives cluster type but no scripts or configuration repository. | [Dremio page](https://www.dremio.com/blog/dremio-benchmarking-methodology/), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | DualBird's white paper dated July 1, 2026 holds the method behind a download form; no public repository was found (github.com/dualbird returns 404). | [DualBird page](https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | Method and scripts for Microsoft's native execution engine benchmark: none found on 2026-11-10. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse open-sourced LakeLoader, an Apache-2.0 tool that generates incremental loads for write benchmarks; the Quanton engine and the full Iceberg benchmark configurations were not found as a repository. | [GitHub repository data](https://api.github.com/repos/onehouseinc/lake-loader), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | AWS publishes its TPC-DS Iceberg setup in GitHub repositories (aws-samples/emr-on-eks-benchmark and aws-samples/emr-spark-benchmark) with build instructions. | [GitHub page](https://github.com/aws-samples/emr-on-eks-benchmark), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Repeatable benchmark scripts for Snowpark Connect: none found on 2026-11-10. | [Snowflake documentation](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### What happens when a job is large, skewed or fails partway? Missed deadlines usually come from the longest tenant or the retried stage, not the median job. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | EMR enables adaptive query execution and dynamic partition pruning; AQE adaptive join conversion is on by default and can be switched off with a Spark setting. | [AWS documentation](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | A DualBird engineer says the FPGA path accelerates a defined set of operators; workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | [DualBird page](https://dualbird.io/resources/blog/a-simpler-spark?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | In Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | In the customer write-up published by Onehouse, the largest tenant (21 billion rows) filled local disks on 180 EMR nodes and was aborted after 3.7 hours; EMR finished on 300 nodes. | [Onehouse page](https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst Galaxy fault tolerant execution spools intermediate data and retries failed parts of a query; the docs advise against it when most queries run under about a minute. | [Starburst documentation](https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which Spark version do your jobs need? A job pinned to a Spark version that a platform does not run cannot move there without an upgrade project. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | The EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0), and the EMR Spark page says EMR supports Apache Iceberg v3 and Spark 4.0. | [AWS documentation](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Databricks Runtime 18 LTS ships Spark 4.1.0 (June 10, 2026), Runtime 17.3 LTS ships 4.0.0, Runtime 16.4 LTS and 15.4 LTS ship 3.5.x, and Runtime 19 ships 4.2.0 (July 23, 2026). | [Databricks documentation](https://docs.databricks.com/aws/en/release-notes/runtime/), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | A list of Spark versions DualBird supports: none found on 2026-11-10. | [DualBird page](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | Fabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Spark versions supported by Onehouse Quanton clusters: none found on 2026-11-10. | [Onehouse documentation](https://docs.onehouse.ai/product/jobs/), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. | [AWS documentation](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowpark Connect supports Apache Spark 3.5 only. | [Snowflake documentation](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ## Sources - AWS page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Databricks page: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026) - Dremio page: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026) - DualBird page: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - Google Cloud documentation: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026) - Microsoft Learn documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Onehouse page: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026) - AWS page: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026) - Snowflake documentation: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Starburst page: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026) - Databricks documentation: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Dremio page: https://www.dremio.com/pricing (read 10 Nov 2026) - DualBird page: https://dualbird.io/ (read 10 Nov 2026) - Onehouse page: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - Apache Iceberg documentation: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Starburst page: https://www.starburst.io/sitemap.xml (read 10 Nov 2026) - Dremio page: https://www.dremio.com/blog/dremio-benchmarking-methodology/ (read 10 Nov 2026) - DualBird page: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - GitHub repository data: https://api.github.com/repos/onehouseinc/lake-loader (read 10 Nov 2026) - GitHub page: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026) - AWS documentation: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026) - DualBird page: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - Microsoft Learn documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026) - Starburst documentation: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026) - AWS documentation: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Databricks documentation: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026) - Microsoft Learn documentation: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Onehouse documentation: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - Snowflake documentation: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/situations/late-pipelines/ --- # AI training and feature pipelines wait on data prep: what to check on a lakehouse platform Situation, read 10 Nov 2026 **The short answer.** A team feeding AI training and feature pipelines waits on the Spark jobs that prepare the data. The checks below cover which table formats each platform reads and writes, whether Spark code runs unchanged, what happens to Python UDFs and unsupported operators, and where the data has to live. ## 4 things to check, with the published facts that answer each ### Which table formats can the prep jobs read and write? Feature and training data is often shared with other teams and tools. A platform that reads a format but cannot write it changes the pipeline design. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | Athena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only for tables in the AWS Glue Data Catalog; DDL is not supported on tables registered with Lake Formation. | [AWS documentation](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Delta tables can generate Iceberg metadata (UniForm) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files. | [Databricks documentation](https://docs.databricks.com/aws/en/delta/uniform), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio Cloud lists Apache Iceberg and Delta Lake as table formats; Hudi is not listed on that page, and the Open Catalog supports the Iceberg REST Catalog API. | [Dremio documentation](https://docs.dremio.com/dremio-cloud/developer/data-formats/), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | DualBird lists compatibility with Apache Spark, Amazon EMR and Amazon EKS; Iceberg, Delta and Hudi support lists: none found on 2026-11-10 on its home, product, about or resources pages. | [DualBird page](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, and export Iceberg V2 snapshots for open engines. | [Google Cloud documentation](https://cloud.google.com/bigquery/docs/iceberg-tables), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | OneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and the reverse. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake; for Delta the user installs Delta Spark as a dependency. | [Onehouse documentation](https://docs.onehouse.ai/product/jobs/), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | Amazon EMR 7.14.0 packages Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three table formats are available. | [AWS documentation](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowflake supports Iceberg tables with data in customer-managed S3, GCS or Azure storage, and reads Delta files through Delta Direct; Hudi was not listed on the page. | [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst Galaxy lists Iceberg, Delta Lake, Hive and Hudi table formats; Hudi tables are read-only, while Iceberg tables can be created and written. | [Starburst documentation](https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Do the prep jobs run as Spark, or must they be rewritten? Feature code is often PySpark with custom functions. A rewrite delays the pipeline you are trying to speed up. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | Athena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client. | [AWS documentation](https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Photon is a native C++ vectorized engine that keeps the Spark Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime. | [Databricks documentation](https://docs.databricks.com/aws/en/compute/photon), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio documentation describes Spark and Flink only as external engines that can use its Open Catalog; Spark ETL would stay on Spark or be rewritten as Dremio SQL. | [Dremio documentation](https://docs.dremio.com/current/what-is-dremio/architecture), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | DualBird lists feature engineering and AI data prep among its use cases and says no pipeline rewrite is needed; its one worked example with a blog write-up is Iceberg compaction. | [DualBird page](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigQuery stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud, so code must be wrapped as procedures. | [Google Cloud documentation](https://cloud.google.com/bigquery/docs/spark-procedures), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | Microsoft says the native execution engine needs no code changes and supports Parquet and Delta. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | At its May 20, 2025 launch Onehouse said existing Spark and SQL jobs can run on new cluster types, and existing Hudi/Spark jobs can be pointed at Onehouse with no code changes. | [Onehouse page](https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect; there is no Spark cluster and all work runs on a Snowflake warehouse. | [Snowflake documentation](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst Galaxy documentation contains no Spark execution mode; Spark jobs would be rewritten as Trino SQL or kept on Spark against the same Iceberg tables. | [Starburst documentation](https://docs.starburst.io/starburst-galaxy/index.html), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### What happens to Python UDFs and operators the engine does not support? AI prep jobs lean on custom Python and wide rows. Fallback paths decide whether the engine helps or just adds a layer. | Option | Published fact | Source, read | | --- | --- | --- | | [Databricks](/platforms/databricks/) | Databricks serverless compute does not support Spark RDD APIs, R or Maven coordinates; with ANSI mode on by default and UTC as the default time zone, results of existing jobs can differ. | [Databricks documentation](https://docs.databricks.com/aws/en/compute/serverless/limitations), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | A DualBird engineer says workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet; coverage is described as expanding. | [DualBird page](https://dualbird.io/resources/blog/a-simpler-spark?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | Microsoft says the Fabric native execution engine falls back to JVM Spark for unsupported operators. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse published a customer write-up of a UDF-heavy pipeline on EMR and Quanton on m8gd nodes; the customer is unnamed and the write-up is by Onehouse. | [Onehouse page](https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowpark Connect does not support DayTimeIntervalType, YearMonthIntervalType or user-defined types, and returns ByteType, ShortType and IntegerType as LongType. | [Snowflake documentation](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Where does the data have to live, and can the compute reach it? Training data is large and often already sits in S3. Moving it adds cost and delay. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | The EMR, Athena, Glue and S3 services in this entry run only on AWS. | [AWS page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Databricks serverless compute runs without provisioning compute in your cloud account and requires Unity Catalog to reach external data sources. | [Databricks documentation](https://docs.databricks.com/aws/en/compute/serverless/), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | DualBird's product page says deployment runs on the customer's own servers in their AWS environment. | [DualBird page](https://dualbird.io/product?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security. | [Google Cloud documentation](https://cloud.google.com/bigquery/docs/biglake-intro), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; writes to S3 are not possible through the shortcut. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse runs EC2 or Compute Engine instances inside the customer's AWS or GCP account. | [Onehouse documentation](https://docs.onehouse.ai/product/project-settings/usage-and-billing), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowflake Iceberg tables can keep data in customer-managed S3 storage through external volumes. | [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | A Starburst Galaxy cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature. | [Starburst page](https://www.starburst.io/pricing/pricing-details/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ## Sources - AWS documentation: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html (read 10 Nov 2026) - Databricks documentation: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Dremio documentation: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - DualBird page: https://dualbird.io/ (read 10 Nov 2026) - Google Cloud documentation: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Microsoft Learn documentation: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Onehouse documentation: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026) - AWS documentation: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026) - Snowflake documentation: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Starburst documentation: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - AWS documentation: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026) - Databricks documentation: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026) - Dremio documentation: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026) - Google Cloud documentation: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026) - Microsoft Learn documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026) - Onehouse page: https://www.onehouse.ai/blog/announcing-spark-and-sql-on-the-onehouse-compute-runtime-with-quanton (read 10 Nov 2026) - Snowflake documentation: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Starburst documentation: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026) - Databricks documentation: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - DualBird page: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026) - Onehouse page: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026) - Snowflake documentation: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-compatibility.md (read 10 Nov 2026) - AWS page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Databricks documentation: https://docs.databricks.com/aws/en/compute/serverless/ (read 10 Nov 2026) - DualBird page: https://dualbird.io/product (read 10 Nov 2026) - Google Cloud documentation: https://cloud.google.com/bigquery/docs/biglake-intro (read 10 Nov 2026) - Microsoft Learn documentation: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026) - Onehouse documentation: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Starburst page: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - Delta Lake UniForm documentation: https://docs.delta.io/latest/delta-uniform.html (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/situations/ai-data-prep/ --- # Leaving a managed platform for an open stack: what to check on a lakehouse platform Situation, read 10 Nov 2026 **The short answer.** A team on Databricks or Snowflake weighs a move to an open stack of Spark and open table formats on AWS. The checks below cover what the team would own afterward, whether data stays readable in both places during the move, which Spark features change, and which clouds each option supports. ## 5 things to check, with the published facts that answer each ### What table maintenance would the team own after the move? A managed platform runs compaction and cleanup for you. On an open stack, someone schedules it and pays for it. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | Athena compaction is manual: run OPTIMIZE ... REWRITE DATA USING BIN_PACK and VACUUM yourself on Iceberg tables; S3 Tables compaction is on by default. | [AWS documentation](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio runs automatic optimization on Iceberg tables in its Open Catalog: compaction, row-level delete handling, clustering, partition evolution and manifest rewriting. | [Dremio documentation](https://docs.dremio.com/dremio-cloud/manage-govern/optimization), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigQuery Iceberg managed tables run automatic storage optimization (adaptive file sizing, clustering, garbage collection); it is a separate billed item. | [Google Cloud documentation](https://cloud.google.com/bigquery/docs/iceberg-tables), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse claims automatic scaling and automated async table services; its docs pages give no per-job Spark tuning list. | [Onehouse documentation](https://docs.onehouse.ai/product/compute/create-cluster), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | Iceberg documentation says expiring snapshots, removing old metadata files and deleting orphan files are recommended, and compacting with rewriteDataFiles is optional, all as jobs you schedule. | [Apache Iceberg documentation](https://iceberg.apache.org/docs/latest/maintenance/), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst Galaxy runs Iceberg maintenance with ALTER TABLE ... EXECUTE commands: optimize, drop_extended_stats, expire_snapshots and remove_orphan_files. | [Starburst documentation](https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Can the same tables stay readable from both platforms during the move? A staged move needs one copy of the data readable by both engines. Check which format bridges the platforms document. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | Athena supports Iceberg 1.4.2 only for tables in the AWS Glue Data Catalog, with Glue optimistic locking. | [AWS documentation](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Databricks UniForm generates Iceberg metadata for Delta tables from Runtime 14.3 LTS, so Iceberg clients read Delta data without rewriting files. | [Databricks documentation](https://docs.databricks.com/aws/en/delta/uniform), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio Cloud documentation lists Iceberg and Delta Lake and a Polaris-based Open Catalog with the Iceberg REST Catalog API, and lists AWS Glue Data Catalog as connectable. | [Dremio documentation](https://docs.dremio.com/dremio-cloud/developer/data-formats/), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigQuery Iceberg managed tables export Iceberg V2 snapshots for open engines. | [Google Cloud documentation](https://cloud.google.com/bigquery/docs/iceberg-tables), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | Fabric OneLake handles Delta and Iceberg through metadata virtualization. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Apache XTable (formerly OneTable) translates table metadata between Hudi, Delta Lake and Iceberg; it is incubating at the Apache Software Foundation, and Onehouse says it created Hudi and XTable. | [Apache XTable site](https://xtable.apache.org/), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowflake supports Iceberg tables in customer-managed storage with an external catalog integration such as AWS Glue, and reads Delta files through Delta Direct. | [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst Galaxy creates and writes Iceberg tables and reads Delta Lake; Hudi tables are read-only. | [Starburst documentation](https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which Spark behavior changes when the code leaves the managed runtime? Each platform documents gaps against open-source Spark. The list of gaps is the migration work. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | AWS's documentation states 100% API compatibility for the EMR runtime for Spark; that is AWS's claim about its own runtime. | [AWS page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | On Databricks serverless compute, Spark RDD APIs, R, Maven coordinates, global temp views and Databricks Container Services are unsupported. | [Databricks documentation](https://docs.databricks.com/aws/en/compute/serverless/limitations), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse documentation says EMR code can be reused unless it uses AWS-specific features, and lists the EMR step and catalog work. | [Onehouse documentation](https://docs.onehouse.ai/product/jobs/emr-migration), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | EMR on EKS supports the Kubernetes Operator for Apache Spark on your own EKS clusters; the Kubeflow Spark Operator is Apache-2.0 licensed with Helm installation. | [AWS documentation](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowflake documentation says Spark DataFrame and Spark SQL code can be reused with minimal migration, and that Spark Connect defers plan resolution, which changes behavior versus Spark Classic. | [Snowflake documentation](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which rates are published, and which are unreadable? An exit is justified by a bill, so the rates must be readable. Some managed platforms render their rates by script, so the line says which were read. | Option | Published fact | Source, read | | --- | --- | --- | | [Databricks](/platforms/databricks/) | Databricks bills DBUs per second with pay-as-you-go or committed-use discounts; the data file behind the pricing calculator lists Jobs Compute on AWS at $0.15 per DBU on the Premium plan, with 4.08 DBU per node-hour for r6i.4xlarge (read 2026-11-10). | [Databricks pricing data file (AWS), loaded by its pricing calculator](https://www.databricks.com/en-pricing-assets/data/pricing/AWS.json), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigQuery capacity pricing is per slot-hour; the pricing page text read on 2026-11-10 lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate. | [Google Cloud documentation](https://cloud.google.com/bigquery/pricing), 10 Nov 2026 | | [Microsoft Fabric](/platforms/microsoft-fabric/) | The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour for eastus. | [Azure retail price API](https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption'), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | AWS's EMR pricing example in us-east-1 shows an EMR charge of $0.105 per hour next to EC2 at $0.398 per hour for a c4.2xlarge, billed per second with a one-minute minimum. | [AWS page](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowflake's Service Consumption Table, effective 2026-11-09, lists $3.00 per credit for the Enterprise edition on AWS US East (Northern Virginia); warehouses bill credits per second with a 60-second minimum. | [Snowflake Service Consumption Table](https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst Galaxy credit prices start at $0.50 (Pro) and $0.75 (Enterprise) per credit, with a Free tier of up to 3 clusters. | [Starburst page](https://www.starburst.io/pricing/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which clouds and regions does each option document? A move off one platform can tie the team to a single cloud when the target runs in only one. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | The Amazon EMR, Athena, Glue and S3 services run only on AWS. | [AWS page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 | | [Databricks](/platforms/databricks/) | Databricks lists many AWS regions and is also hosted on Azure and Google Cloud. | [Databricks documentation](https://docs.databricks.com/aws/en/resources/supported-regions), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio Cloud runs on AWS; Dremio Enterprise can be deployed on Kubernetes, on premises, or in AWS, Azure or GCP. | [Dremio documentation](https://docs.dremio.com/dremio-cloud/about/regions), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigQuery Omni reads S3 data in six AWS regions, and runs on Google Cloud. | [Google Cloud documentation](https://cloud.google.com/bigquery/docs/omni-introduction), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse runs in the customer's AWS or GCP account; Azure is not listed in its billing documentation. | [Onehouse documentation](https://docs.onehouse.ai/product/project-settings/usage-and-billing), 10 Nov 2026 | | [Snowflake](/platforms/snowflake/) | Snowflake runs on AWS, Azure and Google Cloud, with each account in a single region. | [Snowflake documentation](https://docs.snowflake.com/en/user-guide/intro-regions.md), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst Galaxy prices are listed for AWS, Azure and GCP regions. | [Starburst page](https://www.starburst.io/pricing/pricing-details/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ## Sources - AWS documentation: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Dremio documentation: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Google Cloud documentation: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Onehouse documentation: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026) - Apache Iceberg documentation: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Starburst documentation: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - AWS documentation: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html (read 10 Nov 2026) - Databricks documentation: https://docs.databricks.com/aws/en/delta/uniform (read 10 Nov 2026) - Dremio documentation: https://docs.dremio.com/dremio-cloud/developer/data-formats/ (read 10 Nov 2026) - Microsoft Learn documentation: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026) - Apache XTable site: https://xtable.apache.org/ (read 10 Nov 2026) - Snowflake documentation: https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026) - Starburst documentation: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026) - AWS page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026) - Databricks documentation: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026) - Onehouse documentation: https://docs.onehouse.ai/product/jobs/emr-migration (read 10 Nov 2026) - AWS documentation: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator.html (read 10 Nov 2026) - Snowflake documentation: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026) - Databricks pricing data file (AWS), loaded by its pricing calculator: https://www.databricks.com/en-pricing-assets/data/pricing/AWS.json (read 10 Nov 2026) - Google Cloud documentation: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026) - Azure retail price API: https://prices.azure.com/api/retail/prices?$filter=serviceName%20eq%20'Microsoft%20Fabric'%20and%20armRegionName%20eq%20'eastus'%20and%20priceType%20eq%20'Consumption' (read 10 Nov 2026) - AWS page: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - Snowflake Service Consumption Table: https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf (read 10 Nov 2026) - Starburst page: https://www.starburst.io/pricing/ (read 10 Nov 2026) - Databricks documentation: https://docs.databricks.com/aws/en/resources/supported-regions (read 10 Nov 2026) - Dremio documentation: https://docs.dremio.com/dremio-cloud/about/regions (read 10 Nov 2026) - Google Cloud documentation: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026) - Onehouse documentation: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Snowflake documentation: https://docs.snowflake.com/en/user-guide/intro-regions.md (read 10 Nov 2026) - Starburst page: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/situations/leaving-managed/ --- # Large Iceberg tables and daily compaction cost: what to check on a lakehouse platform Situation, read 10 Nov 2026 **The short answer.** A team that maintains large Iceberg tables pays for compaction every day, whether through a managed service, scheduled Spark jobs or an accelerator. The checks below cover who runs compaction, the published unit price, which strategies are documented, how deletes are handled, and what has been benchmarked. ## 6 things to check, with the published facts that answer each ### Who runs compaction on each platform: the platform or your team? Compaction that runs by default costs money you did not schedule. Compaction you schedule costs engineering time. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | S3 Tables compaction is on by default for all tables with a 512 MB default target file size (configurable 64 to 512 MB); Glue Iceberg compaction is disabled by default for new tables, and Athena compaction is manual. | [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio runs automatic optimization on a dedicated engine it configures, and the same operations can be run by hand with OPTIMIZE TABLE. | [Dremio documentation](https://docs.dremio.com/dremio-cloud/manage-govern/optimization), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigQuery Iceberg managed tables run automatic storage optimization, so compaction needs no job configuration on Google Cloud. | [Google Cloud documentation](https://cloud.google.com/bigquery/docs/iceberg-tables), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse lists a Table Optimizer among its product claims. | [Onehouse page](https://www.onehouse.ai/about-us), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | Iceberg documentation shows compaction as the Spark rewriteDataFiles action, with target-file-size-bytes of 500 MB in its example, run as a job you schedule. | [Apache Iceberg documentation](https://iceberg.apache.org/docs/latest/maintenance/), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst Galaxy compacts with ALTER TABLE ... EXECUTE optimize; files under file_size_threshold (default 100MB) are merged. | [Starburst documentation](https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### What is the published unit price for compaction? A per-unit price lets you multiply by your own daily volume. Unit prices differ in what they count: gigabytes, objects, DPU-hours or credits. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | S3 Tables in us-east-1: compaction is $0.005 per GB processed plus $0.002 per 1,000 objects; sort or Z-order compaction is $0.01 per GB plus $0.002 per 1,000 objects. | [AWS price list](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio bills compute in DCUs only while engines are running; billable usage is DCUs for compute, LLM tokens for AI features and storage. | [Dremio documentation](https://docs.dremio.com/dremio-cloud/admin/subscription/usage), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | DualBird states about $2 per TB compacted for itself against about $6 per TB rising past $13 per TB for CPU clusters; DualBird ran the test and part of the saving is attributed to F2 local NVMe. | [DualBird page](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigQuery storage optimization is a separate billed item; the dollar rate for it: none found on 2026-11-10 in the pricing page text read. | [Google Cloud documentation](https://cloud.google.com/bigquery/docs/iceberg-tables), 10 Nov 2026 | | [Onehouse](/platforms/onehouse/) | Onehouse bills in OCUs (1 OCU is a 4-core instance for 1 hour in your account); no dollar rate per OCU was found on 2026-11-10. | [Onehouse documentation](https://docs.onehouse.ai/product/project-settings/usage-and-billing), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | Starburst Galaxy lists table maintenance and data transforms at 0.30 credits per hour and file ingest at 0.60 credits per hour. | [Starburst page](https://www.starburst.io/pricing/pricing-details/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which Glue and EMR line items apply when your own Spark job compacts? Self-run compaction is billed as ordinary compute plus storage requests. The AWS price list gives the rates. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | Glue Iceberg compaction costs $0.44 per DPU-hour in us-east-1; Glue 6.0+ ETL jobs cost $0.308 per DPU-hour and older Glue ETL $0.44. | [AWS price list](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | EMR Serverless bills vCPU-hours and GB-hours of workers; the us-east-1 rates in AWS's pricing example are $0.052624 and $0.0057785. | [AWS page](https://aws.amazon.com/emr/pricing/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### Which compaction and layout strategies does each platform document? Bin-pack, sort and Z-order rewrite files differently and cost differently. Check which one the price covers. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | Athena documents OPTIMIZE ... REWRITE DATA USING BIN_PACK; S3 Tables prices sort and Z-order compaction separately from standard compaction. | [AWS documentation](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio documents data file compaction, clustering, partition evolution and manifest rewriting for Iceberg tables in its Open Catalog. | [Dremio documentation](https://docs.dremio.com/dremio-cloud/manage-govern/optimization), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | DualBird's published benchmark is sort-based Iceberg compaction on a synthetic table of about 370 million rows and 64 columns with ZSTD level 3. | [DualBird page](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | BigQuery Iceberg managed tables document automatic compaction, clustering and garbage collection. | [Google Cloud documentation](https://cloud.google.com/bigquery/docs/iceberg-tables), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | Iceberg documents rewriteDataFiles and manifest rewriting as optional maintenance; Hudi merge-on-read tables need compaction to bound read latency, and Delta Lake documents OPTIMIZE. | [Apache Iceberg documentation](https://iceberg.apache.org/docs/latest/maintenance/), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### How are row-level deletes handled during compaction? Tables with frequent deletes carry delete files. Compaction either applies them or leaves them for readers. | Option | Published fact | Source, read | | --- | --- | --- | | [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | S3 Tables compaction also applies row-level deletes. | [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html), 10 Nov 2026 | | [Dremio](/platforms/dremio/) | Dremio handles row-level deletes for Iceberg tables with position deletes on v2 and deletion vectors on v3. | [Dremio documentation](https://docs.dremio.com/dremio-cloud/manage-govern/optimization), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ### What has been benchmarked for compaction, and by whom? Most published results are SQL query or ETL tests, and the one compaction benchmark found is run by its vendor. | Option | Published fact | Source, read | | --- | --- | --- | | [Dremio](/platforms/dremio/) | Dremio's published benchmark is a 1 TB TPC-DS query test run by Dremio; it is not a compaction test. | [Dremio page](https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/), 10 Nov 2026 | | [DualBird](/platforms/dualbird/) | DualBird's compaction white paper (July 1, 2026) restates the results and holds methodology, instance counts and Spark version behind a download form; the blog gives dataset shape and baselines. | [DualBird page](https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 | | [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | Iceberg, Delta or Hudi compaction benchmarks for EMR or EKS, with time or cost: none found on 2026-11-10. | [Apache Iceberg documentation](https://iceberg.apache.org/docs/latest/maintenance/), 10 Nov 2026 | | [Starburst](/platforms/starburst/) | A Starburst benchmark of compaction on Galaxy CPU clusters with data size and scripts: none found on 2026-11-10. | [Starburst page](https://www.starburst.io/sitemap.xml), 10 Nov 2026 | Options in alphabetical order. Vendor statements are the vendor's own. ## Sources - AWS documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026) - Dremio documentation: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026) - Google Cloud documentation: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026) - Onehouse page: https://www.onehouse.ai/about-us (read 10 Nov 2026) - Apache Iceberg documentation: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026) - Starburst documentation: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026) - AWS price list: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026) - Dremio documentation: https://docs.dremio.com/dremio-cloud/admin/subscription/usage (read 10 Nov 2026) - DualBird page: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026) - Onehouse documentation: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026) - Starburst page: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026) - AWS price list: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026) - AWS page: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026) - AWS documentation: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026) - Dremio page: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026) - DualBird page: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026) - Starburst page: https://www.starburst.io/sitemap.xml (read 10 Nov 2026) Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/situations/iceberg-compaction/ --- # The Compute Bill, the platform facts and the format facts as downloads Datasets, read 10 Nov 2026 **The short answer.** Three downloadable data files sit behind the pages: the Lakehouse Compute Bill, the platform facts and the table-format facts. Each row carries its source URL and read date, so a reader can check it, reuse it or load it into a spreadsheet. The bill file holds each line's formula, unit prices and cost note. ## Downloads | Dataset | Rows | CSV | JSON | | --- | --- | --- | --- | | Lakehouse Compute Bill | 12 | [lakehouse-compute-bill.csv](/datasets/lakehouse-compute-bill.csv) | [lakehouse-compute-bill.json](/datasets/lakehouse-compute-bill.json) | | Platform facts | 122 | [platform-facts.csv](/datasets/platform-facts.csv) | [platform-facts.json](/datasets/platform-facts.json) | | Table format facts | 24 | [table-format-facts.csv](/datasets/table-format-facts.csv) | [table-format-facts.json](/datasets/table-format-facts.json) | Edition 10 Nov 2026. License CC BY 4.0. ### Lakehouse Compute Bill Nightly Spark batch job, 10 nodes for 2 hours (us-east-1): the stated cost on each of 12 lines from published unit prices, with the formula and assumptions. ### Platform facts Every published fact behind the platform profiles, with its topic, source URL and read date. Alphabetical by platform. ### Table format facts Apache Iceberg, Delta Lake and Apache Hudi in the same eight rows, each with its source. ## Cite this Lakehouse Compare, "Lakehouse Compute Bill", edition 10 Nov 2026, https://datalakehousecompare.com/datasets/ (CC BY 4.0). Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/datasets/