# Questions about lakehouse platforms, answered from published facts

FAQ, read 10 Nov 2026

**The short answer.** Thirty questions a data engineer asks when choosing or re-choosing a lakehouse platform, answered in a few sentences each with a source link: formats, compaction, Spark versions, regions, billing units, published prices and benchmarks. Where nothing was found, the answer says so with a date.

## 30 questions and answers

**What is a data lakehouse?**

Databricks's documentation defines a data lakehouse as a data management system that combines the benefits of data lakes and data warehouses. In practice that means table data in open files on object storage such as Amazon S3, plus a metadata layer that adds transactions and schema enforcement, so BI and machine learning jobs read the same copy. Databricks's own lakehouse uses Delta Lake for storage and Unity Catalog for governance.

**How do Delta Lake, Hudi and Iceberg describe themselves?**

Delta Lake's documentation describes an open source storage layer with ACID transactions, scalable metadata handling and time travel over S3, ADLS, GCS and HDFS. Apache Hudi says it pioneered the transactional data lake and is built for high-performance writes on incremental pipelines. Apache Iceberg describes a high-performance format for huge analytic tables that Spark, Trino, Flink, Presto, Hive and Impala can use at the same time. These are each project's own descriptions.

**Is a table format the same thing as a file format?**

No. Data files are usually Parquet, and the table format is the metadata that says which files make up the table. In Iceberg, table state is kept in metadata files, each change writes a new metadata file and swaps it in atomically, and snapshots point to manifest files that list the data files with their partition data and metrics. Delta Lake keeps a transaction log and Hudi a timeline for the same job.

**Can one table be read as Delta, Iceberg and Hudi?**

Two published routes exist. Delta UniForm, documented from Databricks Runtime 14.3 LTS, generates Iceberg metadata for a Delta table registered in Unity Catalog without rewriting the Parquet files. Apache XTable, an incubating Apache project, translates metadata between Hudi, Delta Lake and Iceberg in any direction and writes it under the table's base path (_delta_log, metadata or .hoodie). Which engines accept the generated metadata depends on the engine and version.

**What does Databricks publish about the Photon engine?**

Databricks's documentation describes Photon as a native C++ vectorized engine that keeps Spark's Catalyst planner, works with existing code and falls back to the Spark runtime for unsupported operations. It is on by default for classic all-purpose compute, jobs compute and classic pipelines. Databricks claims up to 5x better price/performance than other cloud data warehouses on that page, and the page gives no method for the figure.

**How does Databricks bill compute, and what is the price per DBU?**

Databricks bills compute in DBUs at per-second granularity, with pay-as-you-go or committed-use discounts and no up-front cost. Jobs pricing offers serverless (Performance Optimized and Standard modes) and classic clusters, and Photon instances consume DBUs at a different rate than the same instance without Photon. The pricing pages draw their rates by script, but the data file behind the pricing calculator lists Jobs Compute on AWS at $0.15 per DBU on the Premium plan and $0.20 on the Enterprise plan, and 4.08 DBU per node-hour for r6i.4xlarge (read 2026-11-10). On the Compute Bill that is $12.24 of DBUs plus $20.16 of EC2.

**How does Snowflake bill compute, and what is the price per credit?**

Snowflake bills virtual warehouses in credits per second, with a 60-second minimum each time a warehouse starts. Each size step roughly doubles the rate: Small is 2 credits per hour and Medium is 4. Serverless features, compute pools and cloud services also consume credits. Snowflake's [Service Consumption Table](https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf), effective 2026-11-09 and read for the Compute Bill on 2026-11-10, lists on-demand credits on AWS US East (Northern Virginia) at $2.00 for Standard, $3.00 for Enterprise and $4.00 for Business Critical. Other clouds and regions are priced separately.

**Can Snowflake run existing PySpark code?**

Snowflake's documentation says Snowpark Connect for Apache Spark runs PySpark DataFrame code against the Snowflake engine over the Spark Connect protocol, with no Spark cluster. It supports Apache Spark 3.5 only; code written for 3.4 and below or 4.0 and above is not supported. Interval types and user-defined types are unsupported, and byte, short and integer columns come back as LongType, so test result schemas before relying on it.

**Which Apache Spark versions do recent Databricks Runtime releases ship?**

Per Databricks's release notes, Runtime 18 LTS ships Apache Spark 4.1.0 (released June 10, 2026), Runtime 17.3 LTS ships Spark 4.0.0, and Runtimes 16.4 LTS and 15.4 LTS ship Spark 3.5.x. Runtime 19 ships Spark 4.2.0 (July 23, 2026). Spark version is the first check on a move to or from another platform, since Snowpark Connect supports only 3.5 and Amazon EMR 7.x builds list Spark 3.5.

**Which open table formats does Snowflake read and write?**

Snowflake's documentation describes Iceberg tables in Parquet, with data in customer-managed S3, Google Cloud Storage or Azure storage through external volumes, an external catalog integration such as AWS Glue, and Delta files read through Delta Direct. The page read did not list Hudi. Snowflake is available on AWS, Azure and Google Cloud, and each account is hosted in a single region.

**What table maintenance does Apache Iceberg require?**

The Iceberg maintenance page lists expiring snapshots, removing old metadata files and deleting orphan files as recommended. Compacting data files with the Spark rewriteDataFiles action (the sample uses a 500 MB target file size) and rewriting manifests are optional. Each is a job you schedule unless a platform or catalog runs it for you. A retention interval shorter than a write takes can make orphan removal delete live files; the default interval is 3 days.

**Who runs Iceberg compaction on AWS?**

It depends on the service, listed alphabetically. Amazon Athena compaction is manual: you run OPTIMIZE with REWRITE DATA USING BIN_PACK and VACUUM yourself. AWS Glue Data Catalog compaction is off by default for new tables and is enabled per table with an IAM role. Amazon S3 Tables compaction is on by default for every table, with a 512 MB default target file size you can set between 64 MB and 512 MB.

**What does Amazon S3 Tables charge for compaction?**

In us-east-1, per the AWS price list read on 2026-11-10, S3 Tables compaction costs $0.005 per GB processed plus $0.002 per 1,000 objects compacted. Sort or Z-order compaction costs $0.01 per GB plus $0.002 per 1,000 objects. Storage is $0.0265 per GB-month for the first 50 TB. Other regions and tiers are priced separately, and these figures are list prices, not a bill for any particular table.

**What do Athena and Glue charge in us-east-1?**

Per the AWS price lists read on 2026-11-10, Athena SQL costs $5.00 per TB scanned, Athena for Apache Spark costs $0.35 per DPU-hour and provisioned capacity costs $0.30 per DPU-hour. AWS Glue Iceberg compaction costs $0.44 per DPU-hour, Glue 6.0 and later ETL jobs cost $0.308 per DPU-hour, and older Glue ETL costs $0.44. These are list prices for one region.

**What does Amazon EMR add to the EC2 price?**

EMR on EC2 adds a per-instance charge to the EC2 and EBS price, billed per second with a one-minute minimum. AWS's pricing page example in us-east-1 shows a c4.2xlarge at $0.105 per hour for EMR next to $0.398 for EC2, and one master with two core nodes for a month at $1,101.57. EMR on EKS adds an uplift per vCPU-hour and per GB-hour, plus $0.10 per hour per EKS cluster.

**What does EMR Serverless charge?**

AWS's EMR pricing page uses $0.052624 per vCPU-hour and $0.0057785 per GB-hour in its EMR Serverless worked example. In that example a job with 100 vCPU and 750 GB for 30 minutes, followed by 200 more vCPU for 15 minutes, costs $9.596275. The page also notes that Spot instances on EMR on EC2 are priced up to 90% below On-Demand. Rates differ by region.

**What does Dremio Cloud charge?**

Dremio's pricing page lists Dremio Cloud at $0.20 per Dremio Compute Unit, with a free trial of a $400 credit for 30 days. Dremio Enterprise, the self-managed product, is priced through sales. Per Dremio's engine docs, DCUs accrue only while engines run, with a 60-second minimum, at published hourly rates from 16 DCU for a 2XSmall engine up to 2,048 DCU for a 3XLarge engine.

**What does Starburst Galaxy charge?**

Starburst's pricing page lists Galaxy credits from $0.50 (Pro), $0.75 (Enterprise) and $1.00 (Mission Critical) per credit on AWS in US East (N. Virginia), US East (Ohio) and US West (Oregon). Prices differ by cloud and region; Europe (Frankfurt) is $0.60, $0.90 and $1.20. Cluster compute uses 6 credits per hour for one worker and 48 for eight workers. Annual commitments may be discounted.

**What does Microsoft Fabric charge for Spark?**

The Azure retail price API lists Fabric Spark capacity usage at $0.18 per CU-hour in eastus on pay-as-you-go, read on 2026-11-10. In Fabric each capacity unit maps to two Spark VCores, and jobs are admitted by available VCores with queueing and throttling. Fabric runs on Azure; the region list names no AWS regions, though OneLake shortcuts can read S3 data without copying it.

**What does BigQuery publish about its pricing?**

Google's pricing page describes on-demand pricing per TiB scanned, with the first 1 TiB per month free, and capacity pricing per slot-hour. BigQuery Omni, which queries S3 data through BigLake tables in six AWS regions, has its own on-demand per-TiB pricing. The page text read on 2026-11-10 lists Enterprise edition compute at $0.06 per slot-hour, with no region shown beside the rate; that is the rate the Compute Bill uses. Per-TiB and Spark DCU rates are not used on this site. Omni does not support DML or CREATE TABLE.

**What does moving from Databricks to an open stack on AWS involve?**

AWS describes the EMR runtime for Spark as fully API compatible with open source Spark and Iceberg, so Spark jobs run as is. Amazon EMR 7.14.0 packages Spark 3.5.8, Delta 3.3.2, Iceberg 1.10.1 and Hudi 1.0.2, whereas Databricks Runtime 18 LTS ships Spark 4.1.0. The team then owns table maintenance, tuning and operations: expiring snapshots, compaction and catalog choice become jobs on its own schedule.

**Which Spark features are unavailable on Databricks serverless compute?**

Databricks's documentation says serverless compute for notebooks and jobs supports only Spark Connect APIs. Spark RDD APIs, R, Maven coordinates, global temp views, DBFS mounts with instance profiles and Databricks Container Services are not supported. ANSI mode is on by default and the default time zone is UTC, which can change results. The Spark UI and Spark logs are not available; the query profile and client-side logs are used instead.

**How do Delta Lake and Hudi handle compaction on the open stack?**

Delta Lake's open source documentation lists file management features that the user applies: bin-packing compaction with OPTIMIZE, auto compaction, data skipping, Z-ordering, multi-part checkpointing, log compactions and optimized write. For Hudi, merge-on-read tables need compaction to merge log files into base files and bound read latency; it has two steps, scheduling a plan on the timeline and then executing it.

**Which open source catalogs exist for Iceberg tables?**

Two open source projects are Project Nessie, a transactional catalog with Git-like semantics and cross-table transactions, and Apache Polaris, which implements Iceberg's REST API and lists Apache Doris, Flink, Spark, Dremio OSS, StarRocks and Trino among supported engines. AWS Glue Data Catalog is a managed option. Catalogs and governance are covered here in one sentence only; check engine support for the catalog protocol you pick.

**How does Onehouse bill, and where does it run?**

Per Onehouse's billing documentation, usage is metered in Onehouse Compute Units: one OCU is a 4-core instance for 1 hour in your own AWS or Google Cloud account, plus support costs, and you pay your own cloud bill. Standard AWS instance types map to Graviton m8g sizes. The pages read give no dollar rate per OCU, so the price is none found on 2026-11-10. Onehouse claims 2 to 3x better price-performance for ETL.

**What benchmark has AWS published for EMR against open source Spark?**

AWS ran TPC-DS at 3 TB (104 SparkSQL queries, 3 rounds, 9 r5d.4xlarge nodes, Iceberg) and reports EMR 7.12 averaging 1,349.62 seconds against 6,113.92 seconds for open source Spark 3.5.6 with Iceberg 1.10.0. Its stated cost was $4.81 for EMR 7.12 and $17.65 for open source Spark. AWS ran it, AWS notes the results are not comparable to official TPC-DS, and AWS publishes the setup on GitHub.

**What does DualBird claim for Spark and Iceberg compaction?**

These are DualBird's own claims. Its home page says 10x-30x faster data processing with 50%-90% lower costs and no changes required. For sort-based Iceberg compaction of 100 GB (synthetic data, about 370 million rows), DualBird reports roughly 12-20x faster single-task execution and about $2 per TB compacted against about $6 rising past $13 per TB on CPU clusters. DualBird ran the test, and the white paper with the full method is behind a form.

**Where does DualBird fit in a lakehouse?**

As read on 2026-11-10: DualBird is not a lakehouse. It speeds up Spark on EMR and EKS; a team on a fully managed platform cannot add it. It documents no Spark version list and no table-format support list; Iceberg appears only in its own compaction benchmark. DualBird says its plugin needs AWS EC2 F2 instances plus a plugin change, with no code changes.

**Does DualBird publish a price or a Spark version list?**

None found on 2026-11-10. DualBird's site shows percentages (50%-90% lower costs, 55%-85% lower EC2 cost) and the compaction figure of about $2 per TB, but no price list, marketplace listing or pricing model for the plugin. AWS lists F2 instances in eight regions (us-east-1, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ca-central-1, eu-central-1 and eu-west-2). DualBird says workloads dominated by Python UDFs fall back to CPU.

**Which platforms write Hudi tables?**

Documentation differs by platform, listed alphabetically. Dremio's data formats page lists Iceberg and Delta Lake and does not list Hudi. Snowflake's Iceberg page did not list Hudi. Starburst Galaxy reads Hudi tables only: its docs say Galaxy cannot create new Hudi tables or write to them, while Iceberg tables can be created and written. Onehouse says its Quanton jobs read and write Hudi, Iceberg and Delta Lake.

## Sources

- What is a data lakehouse? (Databricks on AWS): https://docs.databricks.com/aws/en/lakehouse/ (read 10 Nov 2026)
- Apache Iceberg home page: https://iceberg.apache.org/ (read 10 Nov 2026)
- Iceberg table spec: https://iceberg.apache.org/spec/ (read 10 Nov 2026)
- Apache XTable (Incubating): https://xtable.apache.org/ (read 10 Nov 2026)
- Photon (Databricks on AWS): https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026)
- Databricks pricing data file (AWS), loaded by its pricing calculator: https://www.databricks.com/en-pricing-assets/data/pricing/AWS.json (read 10 Nov 2026)
- Understanding compute cost (Snowflake documentation): https://docs.snowflake.com/en/user-guide/cost-understanding-compute (read 10 Nov 2026)
- Snowpark Connect limitations (Snowflake documentation): https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026)
- Databricks Runtime release notes: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026)
- Apache Iceberg tables (Snowflake documentation): https://docs.snowflake.com/en/user-guide/tables-iceberg (read 10 Nov 2026)
- Iceberg documentation: maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026)
- Maintenance for tables (Amazon S3 User Guide): https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026)
- AWS price list: Amazon S3: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026)
- AWS price list: Amazon Athena: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAthena/current/index.json (read 10 Nov 2026)
- Amazon EMR pricing: https://aws.amazon.com/emr/pricing/ (read 10 Nov 2026)
- Dremio pricing: https://www.dremio.com/pricing (read 10 Nov 2026)
- Starburst Galaxy pricing details: https://www.starburst.io/pricing/pricing-details/ (read 10 Nov 2026)
- Spark job concurrency and queueing in Fabric: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026)
- BigQuery pricing: https://cloud.google.com/bigquery/pricing (read 10 Nov 2026)
- Amazon EMR 7.x release versions: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026)
- Serverless compute limitations (Databricks on AWS): https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026)
- Delta Lake documentation: optimizations: https://docs.delta.io/latest/optimizations-oss.html (read 10 Nov 2026)
- Apache Polaris: https://polaris.apache.org/ (read 10 Nov 2026)
- Onehouse usage and billing: https://docs.onehouse.ai/product/project-settings/usage-and-billing (read 10 Nov 2026)
- AWS Big Data Blog: Spark and Iceberg on EMR: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026)
- DualBird blog: Iceberg compaction benchmark: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026)
- DualBird home page: https://dualbird.io/ (read 10 Nov 2026)
- Starburst Galaxy: Hudi table format: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-hudi.html (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/faq/
