Situation, read 10 Nov 2026
AI training and feature pipelines wait on data prep: what to check on a lakehouse platform
All situationsOpen the Compute Bill
4 things to check, with the published facts that answer each
Which table formats can the prep jobs read and write?
Feature and training data is often shared with other teams and tools. A platform that reads a format but cannot write it changes the pipeline design.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR plus Athena on Iceberg | Athena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only for tables in the AWS Glue Data Catalog; DDL is not supported on tables registered with Lake Formation. | AWS documentation, 10 Nov 2026 |
| Databricks | Delta tables can generate Iceberg metadata (UniForm) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files. | Databricks documentation, 10 Nov 2026 |
| Dremio | Dremio Cloud lists Apache Iceberg and Delta Lake as table formats; Hudi is not listed on that page, and the Open Catalog supports the Iceberg REST Catalog API. | Dremio documentation, 10 Nov 2026 |
| DualBird | DualBird lists compatibility with Apache Spark, Amazon EMR and Amazon EKS; Iceberg, Delta and Hudi support lists: none found on 2026-11-10 on its home, product, about or resources pages. | DualBird page, 10 Nov 2026 |
| Google BigQuery and BigLake | BigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, and export Iceberg V2 snapshots for open engines. | Google Cloud documentation, 10 Nov 2026 |
| Microsoft Fabric | OneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and the reverse. | Microsoft Learn documentation, 10 Nov 2026 |
| Onehouse | Onehouse Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake; for Delta the user installs Delta Spark as a dependency. | Onehouse documentation, 10 Nov 2026 |
| Open stack on EMR and EKS | Amazon EMR 7.14.0 packages Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three table formats are available. | AWS documentation, 10 Nov 2026 |
| Snowflake | Snowflake supports Iceberg tables with data in customer-managed S3, GCS or Azure storage, and reads Delta files through Delta Direct; Hudi was not listed on the page. | Snowflake documentation, 10 Nov 2026 |
| Starburst | Starburst Galaxy lists Iceberg, Delta Lake, Hive and Hudi table formats; Hudi tables are read-only, while Iceberg tables can be created and written. | Starburst documentation, 10 Nov 2026 |
Do the prep jobs run as Spark, or must they be rewritten?
Feature code is often PySpark with custom functions. A rewrite delays the pipeline you are trying to speed up.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR plus Athena on Iceberg | Athena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client. | AWS documentation, 10 Nov 2026 |
| Databricks | Photon is a native C++ vectorized engine that keeps the Spark Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime. | Databricks documentation, 10 Nov 2026 |
| Dremio | Dremio documentation describes Spark and Flink only as external engines that can use its Open Catalog; Spark ETL would stay on Spark or be rewritten as Dremio SQL. | Dremio documentation, 10 Nov 2026 |
| DualBird | DualBird lists feature engineering and AI data prep among its use cases and says no pipeline rewrite is needed; its one worked example with a blog write-up is Iceberg compaction. | DualBird page, 10 Nov 2026 |
| Google BigQuery and BigLake | BigQuery stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud, so code must be wrapped as procedures. | Google Cloud documentation, 10 Nov 2026 |
| Microsoft Fabric | Microsoft says the native execution engine needs no code changes and supports Parquet and Delta. | Microsoft Learn documentation, 10 Nov 2026 |
| Onehouse | At its May 20, 2025 launch Onehouse said existing Spark and SQL jobs can run on new cluster types, and existing Hudi/Spark jobs can be pointed at Onehouse with no code changes. | Onehouse page, 10 Nov 2026 |
| Snowflake | Snowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect; there is no Spark cluster and all work runs on a Snowflake warehouse. | Snowflake documentation, 10 Nov 2026 |
| Starburst | Starburst Galaxy documentation contains no Spark execution mode; Spark jobs would be rewritten as Trino SQL or kept on Spark against the same Iceberg tables. | Starburst documentation, 10 Nov 2026 |
What happens to Python UDFs and operators the engine does not support?
AI prep jobs lean on custom Python and wide rows. Fallback paths decide whether the engine helps or just adds a layer.
| Option | Published fact | Source, read |
|---|---|---|
| Databricks | Databricks serverless compute does not support Spark RDD APIs, R or Maven coordinates; with ANSI mode on by default and UTC as the default time zone, results of existing jobs can differ. | Databricks documentation, 10 Nov 2026 |
| DualBird | A DualBird engineer says workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet; coverage is described as expanding. | DualBird page, 10 Nov 2026 |
| Microsoft Fabric | Microsoft says the Fabric native execution engine falls back to JVM Spark for unsupported operators. | Microsoft Learn documentation, 10 Nov 2026 |
| Onehouse | Onehouse published a customer write-up of a UDF-heavy pipeline on EMR and Quanton on m8gd nodes; the customer is unnamed and the write-up is by Onehouse. | Onehouse page, 10 Nov 2026 |
| Snowflake | Snowpark Connect does not support DayTimeIntervalType, YearMonthIntervalType or user-defined types, and returns ByteType, ShortType and IntegerType as LongType. | Snowflake documentation, 10 Nov 2026 |
Where does the data have to live, and can the compute reach it?
Training data is large and often already sits in S3. Moving it adds cost and delay.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR plus Athena on Iceberg | The EMR, Athena, Glue and S3 services in this entry run only on AWS. | AWS page, 10 Nov 2026 |
| Databricks | Databricks serverless compute runs without provisioning compute in your cloud account and requires Unity Catalog to reach external data sources. | Databricks documentation, 10 Nov 2026 |
| DualBird | DualBird's product page says deployment runs on the customer's own servers in their AWS environment. | DualBird page, 10 Nov 2026 |
| Google BigQuery and BigLake | BigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security. | Google Cloud documentation, 10 Nov 2026 |
| Microsoft Fabric | OneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; writes to S3 are not possible through the shortcut. | Microsoft Learn documentation, 10 Nov 2026 |
| Onehouse | Onehouse runs EC2 or Compute Engine instances inside the customer's AWS or GCP account. | Onehouse documentation, 10 Nov 2026 |
| Snowflake | Snowflake Iceberg tables can keep data in customer-managed S3 storage through external volumes. | Snowflake documentation, 10 Nov 2026 |
| Starburst | A Starburst Galaxy cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature. | Starburst page, 10 Nov 2026 |
Sources
- AWS documentation, read 10 Nov 2026
- Databricks documentation, read 10 Nov 2026
- Dremio documentation, read 10 Nov 2026
- DualBird page, read 10 Nov 2026
- Google Cloud documentation, read 10 Nov 2026
- Microsoft Learn documentation, read 10 Nov 2026
- Onehouse documentation, read 10 Nov 2026
- AWS documentation, read 10 Nov 2026
- Snowflake documentation, read 10 Nov 2026
- Starburst documentation, read 10 Nov 2026
- AWS documentation, read 10 Nov 2026
- Databricks documentation, read 10 Nov 2026
- Dremio documentation, read 10 Nov 2026
- Google Cloud documentation, read 10 Nov 2026
- Microsoft Learn documentation, read 10 Nov 2026
- Onehouse page, read 10 Nov 2026
- Snowflake documentation, read 10 Nov 2026
- Starburst documentation, read 10 Nov 2026
- Databricks documentation, read 10 Nov 2026
- DualBird page, read 10 Nov 2026
- Onehouse page, read 10 Nov 2026
- Snowflake documentation, read 10 Nov 2026
- AWS page, read 10 Nov 2026
- Databricks documentation, read 10 Nov 2026
- DualBird page, read 10 Nov 2026
- Google Cloud documentation, read 10 Nov 2026
- Microsoft Learn documentation, read 10 Nov 2026
- Onehouse documentation, read 10 Nov 2026
- Starburst page, read 10 Nov 2026
- Delta Lake UniForm documentation, read 10 Nov 2026
Last reviewed 10 Nov 2026.