# Where Spark batch work runs on each platform

Guide, read 10 Nov 2026

**The short answer.** Spark batch jobs run natively on Amazon EMR, AWS Glue, Athena and Databricks, through a Spark Connect layer on Snowflake, as stored procedures in BigQuery and on Azure in Fabric. Dremio and Starburst Galaxy run SQL engines instead. This page lists each documented path.

## The paths side by side

| Platform | Where Spark code runs | Spark version documented | Code change to expect |
| --- | --- | --- | --- |
| Amazon EMR, AWS Glue and Athena | EMR on EC2, EKS or Serverless; Glue ETL; Athena for Apache Spark | EMR 7.14.0: 3.5.8-amzn-0; Athena: 3.5 | None stated for EMR; AWS describes 100% API compatibility |
| BigQuery and Google Cloud | Stored procedures for Apache Spark, run on Google Cloud | runtime_version 2.2 in the example | Wrap code as a procedure; none found for full job migration |
| Databricks | Classic or serverless jobs compute | Runtime 18 LTS: 4.1.0; 17.3 LTS: 4.0.0; 16.4 and 15.4 LTS: 3.5.x | Serverless supports only Spark Connect APIs |
| Dremio | Not Spark; Dremio's Arrow-based engine | Not applicable | Rewrite as Dremio SQL or keep Spark on the same Iceberg tables |
| Microsoft Fabric | Spark pools on Azure | Runtime 1.3: 3.5; Runtime 2.0: 4.1 | Compute does not run inside AWS |
| Onehouse | Quanton clusters in your AWS account | Spark versions: none found on 2026-11-10 | Jobs created through its API or Airflow operators |
| Snowflake | Snowflake warehouse via Snowpark Connect | 3.5 only | Semantic differences listed in the compatibility notes |
| Starburst Galaxy | Not Spark; managed Trino | Not applicable | Rewrite as Trino SQL or keep Spark |

Vendors alphabetical after the AWS row. Versions and notes from each vendor's documentation, read 2026-11-10.

## AWS: three places to run the same job

Athena for Apache Spark runs Spark 3.5 and accepts code from SageMaker Unified Studio notebooks or any Spark Connect compatible client. It costs $0.35 per DPU-hour in us-east-1. AWS Glue ETL bills DPUs at $0.308 per hour for Glue 6.0 and later, with a minimum of 2 DPUs per Spark job. EMR on EC2 bills instance-hours plus an uplift, and EMR Serverless bills vCPU-hours and GB-hours. All three accept ordinary Spark code; what differs is the unit of billing and who manages the cluster.

## Databricks

Databricks jobs run on classic clusters or serverless. Photon is on by default for classic all-purpose and jobs compute, and falls back to the Spark runtime for unsupported operations. Serverless compute does not give you a Spark UI or Spark logs; query profile and client-side logs replace them. Jobs pricing offers serverless Performance Optimized and Standard modes next to classic clusters.

## Snowflake, BigQuery and Fabric

Snowflake runs PySpark DataFrame code on a warehouse through the Spark Connect protocol, with no Spark cluster. Batch jobs are submitted with the Snowpark Submit CLI, and Snowflake lists a separate optimization and troubleshooting section for workloads after migration.

BigQuery creates a Spark procedure with CREATE PROCEDURE and engine = "SPARK", using PySpark or Scala and Java code from Cloud Storage or inline. It runs on Google Cloud. BigQuery Omni can run analytics on S3 data through BigLake tables, but DML and CREATE TABLE are not supported there, so it does not run write-heavy Spark batch ETL on S3. Google's Spark service has a Standard tier for batch ETL and a Premium tier on Lightning Engine.

In Fabric, starter pools start in 5 to 10 seconds with no setup, and custom pools tune node size and scaling. Fabric Spark runs on Azure capacity, and OneLake S3 shortcuts are read-only.

## Onehouse, Dremio and Starburst

Onehouse documents that Quanton runs Spark and SQL jobs reading and writing Hudi, Iceberg and Delta Lake in your AWS account. Iceberg 1.10.0 is installed automatically when a job uses an Iceberg REST Catalog such as AWS Glue; for Delta Lake you add Delta Spark as a dependency. Each cluster needs minimum and maximum OCU limits, and at the maximum the cluster stops scaling up, which can delay processing.

Dremio and Starburst Galaxy are query engines for SQL. Dremio's docs describe Spark and Flink only as external engines that use its Open Catalog. Galaxy offers a fault tolerant execution mode that spools intermediate data for batch-style SQL; the docs say it can be slightly slower than normal execution.

## What to check for a batch job

- The Spark major version the code needs, against the table above.
- Whether the job uses RDD APIs, UDFs, custom containers or mounts that a serverless path rejects.
- The billing unit and minimum, since nightly jobs with many short stages meet minimums often.
- Which table formats and catalogs the engine can write to.
- Whether a published benchmark names its method. AWS's EMR 7.12 run publishes its setup on GitHub, Onehouse describes its setup in blog posts with results as charts, and each was run by the vendor.

## Sources

- Amazon EMR 7.x release versions: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-7.x.html (read 10 Nov 2026)
- Athena for Apache Spark: https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark.html (read 10 Nov 2026)
- AWS Glue pricing: https://aws.amazon.com/glue/pricing/ (read 10 Nov 2026)
- AWS price list, AWS Glue, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026)
- Amazon EMR Serverless: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html (read 10 Nov 2026)
- Databricks Runtime release notes: https://docs.databricks.com/aws/en/release-notes/runtime/ (read 10 Nov 2026)
- Databricks Photon: https://docs.databricks.com/aws/en/compute/photon (read 10 Nov 2026)
- Databricks serverless compute limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (read 10 Nov 2026)
- Databricks jobs pricing: https://www.databricks.com/product/pricing/jobs (read 10 Nov 2026)
- Snowpark Connect for Apache Spark: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-apache-spark (read 10 Nov 2026)
- Snowpark Connect limitations: https://docs.snowflake.com/en/developer-guide/snowpark-connect/snowpark-connect-limitations.md (read 10 Nov 2026)
- BigQuery stored procedures for Apache Spark: https://cloud.google.com/bigquery/docs/spark-procedures (read 10 Nov 2026)
- BigQuery Omni: https://cloud.google.com/bigquery/docs/omni-introduction (read 10 Nov 2026)
- Google serverless Spark pricing: https://cloud.google.com/dataproc-serverless/pricing (read 10 Nov 2026)
- Fabric runtime: https://learn.microsoft.com/en-us/fabric/data-engineering/runtime (read 10 Nov 2026)
- Fabric OneLake S3 shortcuts: https://learn.microsoft.com/en-us/fabric/onelake/create-s3-shortcut (read 10 Nov 2026)
- Onehouse jobs: https://docs.onehouse.ai/product/jobs/ (read 10 Nov 2026)
- Onehouse create cluster: https://docs.onehouse.ai/product/compute/create-cluster (read 10 Nov 2026)
- Dremio architecture: https://docs.dremio.com/current/what-is-dremio/architecture (read 10 Nov 2026)
- Starburst Galaxy documentation: https://docs.starburst.io/starburst-galaxy/index.html (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/guides/batch-spark-on-a-lakehouse/
