Skip to content
Lakehouse Compare

Guide, read 10 Nov 2026

Where Spark batch work runs on each platform

All guidesOpen the Compute Bill

The paths side by side

Vendors alphabetical after the AWS row. Versions and notes from each vendor's documentation, read 2026-11-10.
PlatformWhere Spark code runsSpark version documentedCode change to expect
Amazon EMR, AWS Glue and AthenaEMR on EC2, EKS or Serverless; Glue ETL; Athena for Apache SparkEMR 7.14.0: 3.5.8-amzn-0; Athena: 3.5None stated for EMR; AWS describes 100% API compatibility
BigQuery and Google CloudStored procedures for Apache Spark, run on Google Cloudruntime_version 2.2 in the exampleWrap code as a procedure; none found for full job migration
DatabricksClassic or serverless jobs computeRuntime 18 LTS: 4.1.0; 17.3 LTS: 4.0.0; 16.4 and 15.4 LTS: 3.5.xServerless supports only Spark Connect APIs
DremioNot Spark; Dremio's Arrow-based engineNot applicableRewrite as Dremio SQL or keep Spark on the same Iceberg tables
Microsoft FabricSpark pools on AzureRuntime 1.3: 3.5; Runtime 2.0: 4.1Compute does not run inside AWS
OnehouseQuanton clusters in your AWS accountSpark versions: none found on 2026-11-10Jobs created through its API or Airflow operators
SnowflakeSnowflake warehouse via Snowpark Connect3.5 onlySemantic differences listed in the compatibility notes
Starburst GalaxyNot Spark; managed TrinoNot applicableRewrite as Trino SQL or keep Spark

AWS: three places to run the same job

Athena for Apache Spark runs Spark 3.5 and accepts code from SageMaker Unified Studio notebooks or any Spark Connect compatible client. It costs $0.35 per DPU-hour in us-east-1. AWS Glue ETL bills DPUs at $0.308 per hour for Glue 6.0 and later, with a minimum of 2 DPUs per Spark job. EMR on EC2 bills instance-hours plus an uplift, and EMR Serverless bills vCPU-hours and GB-hours. All three accept ordinary Spark code; what differs is the unit of billing and who manages the cluster.

Databricks

Databricks jobs run on classic clusters or serverless. Photon is on by default for classic all-purpose and jobs compute, and falls back to the Spark runtime for unsupported operations. Serverless compute does not give you a Spark UI or Spark logs; query profile and client-side logs replace them. Jobs pricing offers serverless Performance Optimized and Standard modes next to classic clusters.

Snowflake, BigQuery and Fabric

Snowflake runs PySpark DataFrame code on a warehouse through the Spark Connect protocol, with no Spark cluster. Batch jobs are submitted with the Snowpark Submit CLI, and Snowflake lists a separate optimization and troubleshooting section for workloads after migration.

BigQuery creates a Spark procedure with CREATE PROCEDURE and engine = "SPARK", using PySpark or Scala and Java code from Cloud Storage or inline. It runs on Google Cloud. BigQuery Omni can run analytics on S3 data through BigLake tables, but DML and CREATE TABLE are not supported there, so it does not run write-heavy Spark batch ETL on S3. Google's Spark service has a Standard tier for batch ETL and a Premium tier on Lightning Engine.

In Fabric, starter pools start in 5 to 10 seconds with no setup, and custom pools tune node size and scaling. Fabric Spark runs on Azure capacity, and OneLake S3 shortcuts are read-only.

Onehouse, Dremio and Starburst

Onehouse documents that Quanton runs Spark and SQL jobs reading and writing Hudi, Iceberg and Delta Lake in your AWS account. Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog such as AWS Glue; for Delta Lake you add Delta Spark as a dependency. Each cluster needs minimum and maximum OCU limits, and at the maximum the cluster stops scaling up, which can delay processing.

Dremio and Starburst Galaxy are query engines for SQL. Dremio's docs describe Spark and Flink only as external engines that use its Open Catalog. Galaxy offers a fault tolerant execution mode that spools intermediate data for batch-style SQL; the docs say it can be slightly slower than normal execution.

What to check for a batch job

  • The Spark major version the code needs, against the table above.
  • Whether the job uses RDD APIs, UDFs, custom containers or mounts that a serverless path rejects.
  • The billing unit and minimum, since nightly jobs with many short stages meet minimums often.
  • Which table formats and catalogs the engine can write to.
  • Whether a published benchmark names its method. AWS's EMR 7.12 run publishes its setup on GitHub, Onehouse describes its setup in blog posts with results as charts, and each was run by the vendor.

Sources

  1. Amazon EMR 7.x release versions, read 10 Nov 2026
  2. Athena for Apache Spark, read 10 Nov 2026
  3. AWS Glue pricing, read 10 Nov 2026
  4. AWS price list, AWS Glue, us-east-1, read 10 Nov 2026
  5. Amazon EMR Serverless, read 10 Nov 2026
  6. Databricks Runtime release notes, read 10 Nov 2026
  7. Databricks Photon, read 10 Nov 2026
  8. Databricks serverless compute limitations, read 10 Nov 2026
  9. Databricks jobs pricing, read 10 Nov 2026
  10. Snowpark Connect for Apache Spark, read 10 Nov 2026
  11. Snowpark Connect limitations, read 10 Nov 2026
  12. BigQuery stored procedures for Apache Spark, read 10 Nov 2026
  13. BigQuery Omni, read 10 Nov 2026
  14. Google serverless Spark pricing, read 10 Nov 2026
  15. Fabric runtime, read 10 Nov 2026
  16. Fabric OneLake S3 shortcuts, read 10 Nov 2026
  17. Onehouse jobs, read 10 Nov 2026
  18. Onehouse create cluster, read 10 Nov 2026
  19. Dremio architecture, read 10 Nov 2026
  20. Starburst Galaxy documentation, read 10 Nov 2026

Last reviewed 10 Nov 2026.