Skip to content
Lakehouse Compare

Situation, read 10 Nov 2026

AI training and feature pipelines wait on data prep: what to check on a lakehouse platform

All situationsOpen the Compute Bill

4 things to check, with the published facts that answer each

Which table formats can the prep jobs read and write?

Feature and training data is often shared with other teams and tools. A platform that reads a format but cannot write it changes the pipeline design.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR plus Athena on IcebergAthena supports Apache Iceberg 1.4.2 for read, time travel, write and DDL, only for tables in the AWS Glue Data Catalog; DDL is not supported on tables registered with Lake Formation.AWS documentation, 10 Nov 2026
DatabricksDelta tables can generate Iceberg metadata (UniForm) from Databricks Runtime 14.3 LTS so Iceberg clients read Delta data without rewriting files.Databricks documentation, 10 Nov 2026
DremioDremio Cloud lists Apache Iceberg and Delta Lake as table formats; Hudi is not listed on that page, and the Open Catalog supports the Iceberg REST Catalog API.Dremio documentation, 10 Nov 2026
DualBirdDualBird lists compatibility with Apache Spark, Amazon EMR and Amazon EKS; Iceberg, Delta and Hudi support lists: none found on 2026-11-10 on its home, product, about or resources pages.DualBird page, 10 Nov 2026
Google BigQuery and BigLakeBigQuery Iceberg managed tables store data in buckets you own, support DML, streaming and Spark connectors, and export Iceberg V2 snapshots for open engines.Google Cloud documentation, 10 Nov 2026
Microsoft FabricOneLake handles Delta and Iceberg through metadata virtualization so Iceberg tables read as Delta and the reverse.Microsoft Learn documentation, 10 Nov 2026
OnehouseOnehouse Quanton runs Spark and SQL jobs that read and write Hudi, Iceberg and Delta Lake; for Delta the user installs Delta Spark as a dependency.Onehouse documentation, 10 Nov 2026
Open stack on EMR and EKSAmazon EMR 7.14.0 packages Delta 3.3.2-amzn-3, Iceberg 1.10.1-amzn-1 and Hudi 1.0.2-amzn-3, so all three table formats are available.AWS documentation, 10 Nov 2026
SnowflakeSnowflake supports Iceberg tables with data in customer-managed S3, GCS or Azure storage, and reads Delta files through Delta Direct; Hudi was not listed on the page.Snowflake documentation, 10 Nov 2026
StarburstStarburst Galaxy lists Iceberg, Delta Lake, Hive and Hudi table formats; Hudi tables are read-only, while Iceberg tables can be created and written.Starburst documentation, 10 Nov 2026

Do the prep jobs run as Spark, or must they be rewritten?

Feature code is often PySpark with custom functions. A rewrite delays the pipeline you are trying to speed up.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR plus Athena on IcebergAthena for Apache Spark (version 3.5) accepts Spark code from SageMaker Unified Studio notebooks or any Spark Connect compatible client.AWS documentation, 10 Nov 2026
DatabricksPhoton is a native C++ vectorized engine that keeps the Spark Catalyst planner and works with existing Spark API code; unsupported operations fall back to the Spark runtime.Databricks documentation, 10 Nov 2026
DremioDremio documentation describes Spark and Flink only as external engines that can use its Open Catalog; Spark ETL would stay on Spark or be rewritten as Dremio SQL.Dremio documentation, 10 Nov 2026
DualBirdDualBird lists feature engineering and AI data prep among its use cases and says no pipeline rewrite is needed; its one worked example with a blog write-up is Iceberg compaction.DualBird page, 10 Nov 2026
Google BigQuery and BigLakeBigQuery stored procedures for Apache Spark run PySpark or Scala/Java JAR code from SQL on Google Cloud, so code must be wrapped as procedures.Google Cloud documentation, 10 Nov 2026
Microsoft FabricMicrosoft says the native execution engine needs no code changes and supports Parquet and Delta.Microsoft Learn documentation, 10 Nov 2026
OnehouseAt its May 20, 2025 launch Onehouse said existing Spark and SQL jobs can run on new cluster types, and existing Hudi/Spark jobs can be pointed at Onehouse with no code changes.Onehouse page, 10 Nov 2026
SnowflakeSnowpark Connect runs existing PySpark DataFrame code against the Snowflake engine over Spark Connect; there is no Spark cluster and all work runs on a Snowflake warehouse.Snowflake documentation, 10 Nov 2026
StarburstStarburst Galaxy documentation contains no Spark execution mode; Spark jobs would be rewritten as Trino SQL or kept on Spark against the same Iceberg tables.Starburst documentation, 10 Nov 2026

What happens to Python UDFs and operators the engine does not support?

AI prep jobs lean on custom Python and wide rows. Fallback paths decide whether the engine helps or just adds a layer.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
DatabricksDatabricks serverless compute does not support Spark RDD APIs, R or Maven coordinates; with ANSI mode on by default and UTC as the default time zone, results of existing jobs can differ.Databricks documentation, 10 Nov 2026
DualBirdA DualBird engineer says workloads dominated by Python UDFs or unsupported operators fall back to the CPU and do not see the same benefit yet; coverage is described as expanding.DualBird page, 10 Nov 2026
Microsoft FabricMicrosoft says the Fabric native execution engine falls back to JVM Spark for unsupported operators.Microsoft Learn documentation, 10 Nov 2026
OnehouseOnehouse published a customer write-up of a UDF-heavy pipeline on EMR and Quanton on m8gd nodes; the customer is unnamed and the write-up is by Onehouse.Onehouse page, 10 Nov 2026
SnowflakeSnowpark Connect does not support DayTimeIntervalType, YearMonthIntervalType or user-defined types, and returns ByteType, ShortType and IntegerType as LongType.Snowflake documentation, 10 Nov 2026

Where does the data have to live, and can the compute reach it?

Training data is large and often already sits in S3. Moving it adds cost and delay.

Options in alphabetical order. Vendor statements are the vendor's own.
OptionPublished factSource, read
Amazon EMR plus Athena on IcebergThe EMR, Athena, Glue and S3 services in this entry run only on AWS.AWS page, 10 Nov 2026
DatabricksDatabricks serverless compute runs without provisioning compute in your cloud account and requires Unity Catalog to reach external data sources.Databricks documentation, 10 Nov 2026
DualBirdDualBird's product page says deployment runs on the customer's own servers in their AWS environment.DualBird page, 10 Nov 2026
Google BigQuery and BigLakeBigLake tables let you query structured data in Cloud Storage, Amazon S3 and Blob Storage with access delegation and row and column security.Google Cloud documentation, 10 Nov 2026
Microsoft FabricOneLake S3 shortcuts give read-only access to data in an S3 bucket in any file format without copying; writes to S3 are not possible through the shortcut.Microsoft Learn documentation, 10 Nov 2026
OnehouseOnehouse runs EC2 or Compute Engine instances inside the customer's AWS or GCP account.Onehouse documentation, 10 Nov 2026
SnowflakeSnowflake Iceberg tables can keep data in customer-managed S3 storage through external volumes.Snowflake documentation, 10 Nov 2026
StarburstA Starburst Galaxy cluster and its catalogs are normally in the same cloud and region; cross-region querying is a private preview feature.Starburst page, 10 Nov 2026

Sources

  1. AWS documentation, read 10 Nov 2026
  2. Databricks documentation, read 10 Nov 2026
  3. Dremio documentation, read 10 Nov 2026
  4. DualBird page, read 10 Nov 2026
  5. Google Cloud documentation, read 10 Nov 2026
  6. Microsoft Learn documentation, read 10 Nov 2026
  7. Onehouse documentation, read 10 Nov 2026
  8. AWS documentation, read 10 Nov 2026
  9. Snowflake documentation, read 10 Nov 2026
  10. Starburst documentation, read 10 Nov 2026
  11. AWS documentation, read 10 Nov 2026
  12. Databricks documentation, read 10 Nov 2026
  13. Dremio documentation, read 10 Nov 2026
  14. Google Cloud documentation, read 10 Nov 2026
  15. Microsoft Learn documentation, read 10 Nov 2026
  16. Onehouse page, read 10 Nov 2026
  17. Snowflake documentation, read 10 Nov 2026
  18. Starburst documentation, read 10 Nov 2026
  19. Databricks documentation, read 10 Nov 2026
  20. DualBird page, read 10 Nov 2026
  21. Onehouse page, read 10 Nov 2026
  22. Snowflake documentation, read 10 Nov 2026
  23. AWS page, read 10 Nov 2026
  24. Databricks documentation, read 10 Nov 2026
  25. DualBird page, read 10 Nov 2026
  26. Google Cloud documentation, read 10 Nov 2026
  27. Microsoft Learn documentation, read 10 Nov 2026
  28. Onehouse documentation, read 10 Nov 2026
  29. Starburst page, read 10 Nov 2026
  30. Delta Lake UniForm documentation, read 10 Nov 2026

Last reviewed 10 Nov 2026.