Skip to content
Lakehouse Compare

Situation, read 10 Nov 2026

Nightly pipelines no longer finish by morning: what to check on a lakehouse platform

All situationsOpen the Compute Bill

5 things to check, with the published facts that answer each

Which runtime results are published, and who ran each one?

A result from the vendor is a claim until another party repeats it. Check the workload, the data size and the comparison baseline before reading the multiple.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR plus Athena on IcebergAWS states up to 4.5x faster than open-source Spark and 2.7x faster Iceberg writes on its EMR features page, with no workload or scale given.AWS page, 10 Nov 2026
DatabricksDatabricks announced an audited 100 TB TPC-DS result for Databricks SQL on 2021-11-02; it is a SQL warehouse result, not a Spark batch ETL result.Databricks page, 10 Nov 2026
DremioDremio's founder reported all 99 TPC-DS queries at 1 TB in 22 seconds on an 8-node m7gd.4xlarge cluster (2025-06-16); a BI query test run by Dremio, not batch ETL.Dremio page, 10 Nov 2026
DualBirdDualBird reports about 12x to 20x faster single-task execution for sort-based Iceberg compaction on 100 GB (June 11, 2026); the test was run by DualBird.DualBird page, 10 Nov 2026
Google BigQuery and BigLakeGoogle's pricing page claims up to 4.9x faster performance for its Lightning Engine tier than open-source Spark and gives no method.Google Cloud documentation, 10 Nov 2026
Microsoft FabricMicrosoft reports up to six times faster than open-source Spark on TPC-DS scale factor 1000 with Delta; Microsoft ran the test.Microsoft Learn documentation, 10 Nov 2026
OnehouseOnehouse reports TPC-DS up to about 3.1x, TPCx-BB 2x and TPC-DI (1 TB) 2x against open-source Spark on Iceberg, with 4.1x for LakeLoader writes; Onehouse ran the tests.Onehouse page, 10 Nov 2026
Open stack on EMR and EKSAWS ran TPC-DS 3 TB (104 queries, 3 rounds, 9 x r5d.4xlarge) and reports EMR 7.12 averaging 1,349.62 s against 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0.AWS page, 10 Nov 2026
SnowflakeA published benchmark of Spark batch jobs on Snowpark Connect with its method: none found on 2026-11-10.Snowflake documentation, 10 Nov 2026
StarburstStarburst reports TPC-H up to 6x on accelerated queries with GPU Trino in a single-node environment; it is a SQL query test, not Spark batch work.Starburst page, 10 Nov 2026

Is the published workload similar to a nightly batch job?

TPC-DS style query tests, compaction and a customer pipeline stress different parts of an engine. A result only transfers to your job if the work is similar.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
DatabricksA Databricks benchmark of Spark batch ETL (not SQL warehouse) with a published method: none found on 2026-11-10.Databricks documentation, 10 Nov 2026
DremioPublished cost for the same Spark batch work on Dremio: none found on 2026-11-10; the 22-second result is query only.Dremio page, 10 Nov 2026
DualBirdDualBird shows a 110 GB enrichment pipeline (JOIN denormalization) at 6x to 18x higher performance ceiling and 50% to 85% lower EC2 cost, with no hardware, baseline, date or method.DualBird page, 10 Nov 2026
OnehouseAn unnamed B2B SaaS customer ran a UDF-heavy pipeline on EMR and on Onehouse Quanton in its own AWS account; Quanton was faster at p50, p97 and p99, and the write-up is published by Onehouse.Onehouse page, 10 Nov 2026
Open stack on EMR and EKSA published benchmark of Iceberg, Delta or Hudi compaction time or cost on EMR or EKS: none found on 2026-11-10.Apache Iceberg documentation, 10 Nov 2026
StarburstA Starburst benchmark of ETL, joins or compaction on Galaxy CPU clusters with data size and scripts: none found on 2026-11-10.Starburst page, 10 Nov 2026

Can a reader repeat the test with published scripts?

A repeatable test lets you run your own data through the same setup. A result without scripts can only be read, not checked.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR plus Athena on IcebergPublished scripts for the AWS EMR marketing claim of 4.5x: none found on 2026-11-10.AWS page, 10 Nov 2026
DatabricksRepeatable scripts for the Databricks TPC-DS record: none found on 2026-11-10.Databricks page, 10 Nov 2026
DremioDremio published a 2020 TPC-DS methodology post with JMeter test plans; the 2025 22-second post gives cluster type but no scripts or configuration repository.Dremio page, 10 Nov 2026
DualBirdDualBird's white paper dated July 1, 2026 holds the method behind a download form; no public repository was found (github.com/dualbird returns 404).DualBird page, 10 Nov 2026
Microsoft FabricMethod and scripts for Microsoft's native execution engine benchmark: none found on 2026-11-10.Microsoft Learn documentation, 10 Nov 2026
OnehouseOnehouse open-sourced LakeLoader, an Apache-2.0 tool that generates incremental loads for write benchmarks; the Quanton engine and the full Iceberg benchmark configurations were not found as a repository.GitHub repository data, 10 Nov 2026
Open stack on EMR and EKSAWS publishes its TPC-DS Iceberg setup in GitHub repositories (aws-samples/emr-on-eks-benchmark and aws-samples/emr-spark-benchmark) with build instructions.GitHub page, 10 Nov 2026
SnowflakeRepeatable benchmark scripts for Snowpark Connect: none found on 2026-11-10.Snowflake documentation, 10 Nov 2026

What happens when a job is large, skewed or fails partway?

Missed deadlines usually come from the longest tenant or the retried stage, not the median job.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR plus Athena on IcebergEMR enables adaptive query execution and dynamic partition pruning; AQE adaptive join conversion is on by default and can be switched off with a Spark setting.AWS documentation, 10 Nov 2026
DualBirdA DualBird engineer says the FPGA path accelerates a defined set of operators; workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet.DualBird page, 10 Nov 2026
Microsoft FabricIn Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling.Microsoft Learn documentation, 10 Nov 2026
OnehouseIn the customer write-up published by Onehouse, the largest tenant (21 billion rows) filled local disks on 180 EMR nodes and was aborted after 3.7 hours; EMR finished on 300 nodes.Onehouse page, 10 Nov 2026
StarburstStarburst Galaxy fault tolerant execution spools intermediate data and retries failed parts of a query; the docs advise against it when most queries run under about a minute.Starburst documentation, 10 Nov 2026

Which Spark version do your jobs need?

A job pinned to a Spark version that a platform does not run cannot move there without an upgrade project.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR plus Athena on IcebergThe EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0), and the EMR Spark page says EMR supports Apache Iceberg v3 and Spark 4.0.AWS documentation, 10 Nov 2026
DatabricksDatabricks Runtime 18 LTS ships Spark 4.1.0 (June 10, 2026), Runtime 17.3 LTS ships 4.0.0, Runtime 16.4 LTS and 15.4 LTS ship 3.5.x, and Runtime 19 ships 4.2.0 (July 23, 2026).Databricks documentation, 10 Nov 2026
DualBirdA list of Spark versions DualBird supports: none found on 2026-11-10.DualBird page, 10 Nov 2026
Microsoft FabricFabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1.Microsoft Learn documentation, 10 Nov 2026
OnehouseSpark versions supported by Onehouse Quanton clusters: none found on 2026-11-10.Onehouse documentation, 10 Nov 2026
Open stack on EMR and EKSAmazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3.AWS documentation, 10 Nov 2026
SnowflakeSnowpark Connect supports Apache Spark 3.5 only.Snowflake documentation, 10 Nov 2026

Sources

  1. AWS page, read 10 Nov 2026
  2. Databricks page, read 10 Nov 2026
  3. Dremio page, read 10 Nov 2026
  4. DualBird page, read 10 Nov 2026
  5. Google Cloud documentation, read 10 Nov 2026
  6. Microsoft Learn documentation, read 10 Nov 2026
  7. Onehouse page, read 10 Nov 2026
  8. AWS page, read 10 Nov 2026
  9. Snowflake documentation, read 10 Nov 2026
  10. Starburst page, read 10 Nov 2026
  11. Databricks documentation, read 10 Nov 2026
  12. Dremio page, read 10 Nov 2026
  13. DualBird page, read 10 Nov 2026
  14. Onehouse page, read 10 Nov 2026
  15. Apache Iceberg documentation, read 10 Nov 2026
  16. Starburst page, read 10 Nov 2026
  17. Dremio page, read 10 Nov 2026
  18. DualBird page, read 10 Nov 2026
  19. GitHub repository data, read 10 Nov 2026
  20. GitHub page, read 10 Nov 2026
  21. AWS documentation, read 10 Nov 2026
  22. DualBird page, read 10 Nov 2026
  23. Microsoft Learn documentation, read 10 Nov 2026
  24. Starburst documentation, read 10 Nov 2026
  25. AWS documentation, read 10 Nov 2026
  26. Databricks documentation, read 10 Nov 2026
  27. Microsoft Learn documentation, read 10 Nov 2026
  28. Onehouse documentation, read 10 Nov 2026
  29. Snowflake documentation, read 10 Nov 2026

Last reviewed 10 Nov 2026.