# Nightly pipelines no longer finish by morning: what to check on a lakehouse platform

Situation, read 10 Nov 2026

**The short answer.** A team whose nightly Spark pipelines run past the morning deadline needs to know what each platform publishes about batch runtime. The checks below cover which benchmarks exist and who ran them, whether they resemble batch ETL, whether a reader can repeat them, behavior on very large jobs, and Spark versions.

## 5 things to check, with the published facts that answer each

### Which runtime results are published, and who ran each one?

A result from the vendor is a claim until another party repeats it. Check the workload, the data size and the comparison baseline before reading the multiple.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | AWS states up to 4.5x faster than open-source Spark and 2.7x faster Iceberg writes on its EMR features page, with no workload or scale given. | [AWS page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 |
| [Databricks](/platforms/databricks/) | Databricks announced an audited 100 TB TPC-DS result for Databricks SQL on 2021-11-02; it is a SQL warehouse result, not a Spark batch ETL result. | [Databricks page](https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html), 10 Nov 2026 |
| [Dremio](/platforms/dremio/) | Dremio's founder reported all 99 TPC-DS queries at 1 TB in 22 seconds on an 8-node m7gd.4xlarge cluster (2025-06-16); a BI query test run by Dremio, not batch ETL. | [Dremio page](https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/), 10 Nov 2026 |
| [DualBird](/platforms/dualbird/) | DualBird reports about 12x to 20x faster single-task execution for sort-based Iceberg compaction on 100 GB (June 11, 2026); the test was run by DualBird. | [DualBird page](https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 |
| [Google BigQuery and BigLake](/platforms/bigquery-biglake/) | Google's pricing page claims up to 4.9x faster performance for its Lightning Engine tier than open-source Spark and gives no method. | [Google Cloud documentation](https://cloud.google.com/dataproc-serverless/pricing), 10 Nov 2026 |
| [Microsoft Fabric](/platforms/microsoft-fabric/) | Microsoft reports up to six times faster than open-source Spark on TPC-DS scale factor 1000 with Delta; Microsoft ran the test. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime), 10 Nov 2026 |
| [Onehouse](/platforms/onehouse/) | Onehouse reports TPC-DS up to about 3.1x, TPCx-BB 2x and TPC-DI (1 TB) 2x against open-source Spark on Iceberg, with 4.1x for LakeLoader writes; Onehouse ran the tests. | [Onehouse page](https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads), 10 Nov 2026 |
| [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | AWS ran TPC-DS 3 TB (104 queries, 3 rounds, 9 x r5d.4xlarge) and reports EMR 7.12 averaging 1,349.62 s against 6,113.92 s for open-source Spark 3.5.6 with Iceberg 1.10.0. | [AWS page](https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/), 10 Nov 2026 |
| [Snowflake](/platforms/snowflake/) | A published benchmark of Spark batch jobs on Snowpark Connect with its method: none found on 2026-11-10. | [Snowflake documentation](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark), 10 Nov 2026 |
| [Starburst](/platforms/starburst/) | Starburst reports TPC-H up to 6x on accelerated queries with GPU Trino in a single-node environment; it is a SQL query test, not Spark batch work. | [Starburst page](https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

### Is the published workload similar to a nightly batch job?

TPC-DS style query tests, compaction and a customer pipeline stress different parts of an engine. A result only transfers to your job if the work is similar.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Databricks](/platforms/databricks/) | A Databricks benchmark of Spark batch ETL (not SQL warehouse) with a published method: none found on 2026-11-10. | [Databricks documentation](https://docs.databricks.com/aws/en/compute/photon), 10 Nov 2026 |
| [Dremio](/platforms/dremio/) | Published cost for the same Spark batch work on Dremio: none found on 2026-11-10; the 22-second result is query only. | [Dremio page](https://www.dremio.com/pricing), 10 Nov 2026 |
| [DualBird](/platforms/dualbird/) | DualBird shows a 110 GB enrichment pipeline (JOIN denormalization) at 6x to 18x higher performance ceiling and 50% to 85% lower EC2 cost, with no hardware, baseline, date or method. | [DualBird page](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 |
| [Onehouse](/platforms/onehouse/) | An unnamed B2B SaaS customer ran a UDF-heavy pipeline on EMR and on Onehouse Quanton in its own AWS account; Quanton was faster at p50, p97 and p99, and the write-up is published by Onehouse. | [Onehouse page](https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs), 10 Nov 2026 |
| [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | A published benchmark of Iceberg, Delta or Hudi compaction time or cost on EMR or EKS: none found on 2026-11-10. | [Apache Iceberg documentation](https://iceberg.apache.org/docs/latest/maintenance/), 10 Nov 2026 |
| [Starburst](/platforms/starburst/) | A Starburst benchmark of ETL, joins or compaction on Galaxy CPU clusters with data size and scripts: none found on 2026-11-10. | [Starburst page](https://www.starburst.io/sitemap.xml), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

### Can a reader repeat the test with published scripts?

A repeatable test lets you run your own data through the same setup. A result without scripts can only be read, not checked.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | Published scripts for the AWS EMR marketing claim of 4.5x: none found on 2026-11-10. | [AWS page](https://aws.amazon.com/emr/features/spark/), 10 Nov 2026 |
| [Databricks](/platforms/databricks/) | Repeatable scripts for the Databricks TPC-DS record: none found on 2026-11-10. | [Databricks page](https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html), 10 Nov 2026 |
| [Dremio](/platforms/dremio/) | Dremio published a 2020 TPC-DS methodology post with JMeter test plans; the 2025 22-second post gives cluster type but no scripts or configuration repository. | [Dremio page](https://www.dremio.com/blog/dremio-benchmarking-methodology/), 10 Nov 2026 |
| [DualBird](/platforms/dualbird/) | DualBird's white paper dated July 1, 2026 holds the method behind a download form; no public repository was found (github.com/dualbird returns 404). | [DualBird page](https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 |
| [Microsoft Fabric](/platforms/microsoft-fabric/) | Method and scripts for Microsoft's native execution engine benchmark: none found on 2026-11-10. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime), 10 Nov 2026 |
| [Onehouse](/platforms/onehouse/) | Onehouse open-sourced LakeLoader, an Apache-2.0 tool that generates incremental loads for write benchmarks; the Quanton engine and the full Iceberg benchmark configurations were not found as a repository. | [GitHub repository data](https://api.github.com/repos/onehouseinc/lake-loader), 10 Nov 2026 |
| [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | AWS publishes its TPC-DS Iceberg setup in GitHub repositories (aws-samples/emr-on-eks-benchmark and aws-samples/emr-spark-benchmark) with build instructions. | [GitHub page](https://github.com/aws-samples/emr-on-eks-benchmark), 10 Nov 2026 |
| [Snowflake](/platforms/snowflake/) | Repeatable benchmark scripts for Snowpark Connect: none found on 2026-11-10. | [Snowflake documentation](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

### What happens when a job is large, skewed or fails partway?

Missed deadlines usually come from the longest tenant or the retried stage, not the median job.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | EMR enables adaptive query execution and dynamic partition pruning; AQE adaptive join conversion is on by default and can be switched off with a Spark setting. | [AWS documentation](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html), 10 Nov 2026 |
| [DualBird](/platforms/dualbird/) | A DualBird engineer says the FPGA path accelerates a defined set of operators; workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet. | [DualBird page](https://dualbird.io/resources/blog/a-simpler-spark?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 |
| [Microsoft Fabric](/platforms/microsoft-fabric/) | In Fabric each capacity unit maps to two Spark VCores, and Spark jobs are admitted by available VCores with FIFO queueing and throttling. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing), 10 Nov 2026 |
| [Onehouse](/platforms/onehouse/) | In the customer write-up published by Onehouse, the largest tenant (21 billion rows) filled local disks on 180 EMR nodes and was aborted after 3.7 hours; EMR finished on 300 nodes. | [Onehouse page](https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs), 10 Nov 2026 |
| [Starburst](/platforms/starburst/) | Starburst Galaxy fault tolerant execution spools intermediate data and retries failed parts of a query; the docs advise against it when most queries run under about a minute. | [Starburst documentation](https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

### Which Spark version do your jobs need?

A job pinned to a Spark version that a platform does not run cannot move there without an upgrade project.

| Option | Published fact | Source, read |
| --- | --- | --- |
| [Amazon EMR plus Athena on Iceberg](/platforms/aws-emr-athena-iceberg/) | The EMR 7.x release table lists Spark 3.5.x builds (up to 3.5.8-amzn-0), and the EMR Spark page says EMR supports Apache Iceberg v3 and Spark 4.0. | [AWS documentation](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html), 10 Nov 2026 |
| [Databricks](/platforms/databricks/) | Databricks Runtime 18 LTS ships Spark 4.1.0 (June 10, 2026), Runtime 17.3 LTS ships 4.0.0, Runtime 16.4 LTS and 15.4 LTS ship 3.5.x, and Runtime 19 ships 4.2.0 (July 23, 2026). | [Databricks documentation](https://docs.databricks.com/aws/en/release-notes/runtime/), 10 Nov 2026 |
| [DualBird](/platforms/dualbird/) | A list of Spark versions DualBird supports: none found on 2026-11-10. | [DualBird page](https://dualbird.io/?utm_source=datalakehousecompare.com&utm_medium=referral&utm_campaign=situation), 10 Nov 2026 |
| [Microsoft Fabric](/platforms/microsoft-fabric/) | Fabric Runtime 1.3 uses Apache Spark 3.5 and Runtime 2.0 uses Spark 4.1. | [Microsoft Learn documentation](https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables), 10 Nov 2026 |
| [Onehouse](/platforms/onehouse/) | Spark versions supported by Onehouse Quanton clusters: none found on 2026-11-10. | [Onehouse documentation](https://docs.onehouse.ai/product/jobs/), 10 Nov 2026 |
| [Open stack on EMR and EKS](/platforms/open-stack-emr-eks/) | Amazon EMR 7.14.0 packages Spark 3.5.8-amzn-0, Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3. | [AWS documentation](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html), 10 Nov 2026 |
| [Snowflake](/platforms/snowflake/) | Snowpark Connect supports Apache Spark 3.5 only. | [Snowflake documentation](https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md), 10 Nov 2026 |

Options in alphabetical order. Vendor statements are the vendor's own.

## Sources

- AWS page: https://aws.amazon.com/emr/features/spark/ (read 10 Nov 2026)
- Databricks page: https://www.databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html (read 10 Nov 2026)
- Dremio page: https://www.dremio.com/blog/breakthrough-announcement-dremio-is-the-fastest-lakehouse-20x-faster-on-tpc-ds/ (read 10 Nov 2026)
- DualBird page: https://dualbird.io/resources/blog/iceberg-compaction-benchmark-faster-spark-lower-cost (read 10 Nov 2026)
- Google Cloud documentation: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026)
- Microsoft Learn documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026)
- Onehouse page: https://www.onehouse.ai/blog/apache-iceberg-on-quanton-3x-faster-apache-spark-workloads (read 10 Nov 2026)
- AWS page: https://aws.amazon.com/blogs/big-data/run-apache-spark-and-iceberg-4-5x-faster-than-open-source-spark-with-amazon-emr/ (read 10 Nov 2026)
- Snowflake documentation: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026)
- Starburst page: https://www.starburst.io/blog/gpu-accelerated-sql-analytics-how-starburst-and-nvidia-deliver-industry-benchmark-speedups-on-gpu-infrastructure/ (read 10 Nov 2026)
- Databricks documentation: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026)
- Dremio page: https://www.dremio.com/pricing (read 10 Nov 2026)
- DualBird page: https://dualbird.io/ (read 10 Nov 2026)
- Onehouse page: https://www.onehouse.ai/blog/emr-migration-to-quanton-spark-udf-jobs (read 10 Nov 2026)
- Apache Iceberg documentation: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026)
- Starburst page: https://www.starburst.io/sitemap.xml (read 10 Nov 2026)
- Dremio page: https://www.dremio.com/blog/dremio-benchmarking-methodology/ (read 10 Nov 2026)
- DualBird page: https://dualbird.io/resources/blog/spark-iceberg-compaction-benchmark-whitepaper (read 10 Nov 2026)
- GitHub repository data: https://api.github.com/repos/onehouseinc/lake-loader (read 10 Nov 2026)
- GitHub page: https://github.com/aws-samples/emr-on-eks-benchmark (read 10 Nov 2026)
- AWS documentation: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-performance.html (read 10 Nov 2026)
- DualBird page: https://dualbird.io/resources/blog/a-simpler-spark (read 10 Nov 2026)
- Microsoft Learn documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing (read 10 Nov 2026)
- Starburst documentation: https://docs.starburst.io/starburst-galaxy/data-engineering/optimization-performance-and-quality/workload-optimization/clusters-with-fte.html (read 10 Nov 2026)
- AWS documentation: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026)
- Databricks documentation: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026)
- Microsoft Learn documentation: https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables (read 10 Nov 2026)
- Onehouse documentation: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026)
- Snowflake documentation: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/situations/late-pipelines/
