Situation, read 10 Nov 2026
Large Iceberg tables and daily compaction cost: what to check on a lakehouse platform
All situationsOpen the Compute Bill
6 things to check, with the published facts that answer each
Who runs compaction on each platform: the platform or your team?
Compaction that runs by default costs money you did not schedule. Compaction you schedule costs engineering time.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR plus Athena on Iceberg | S3 Tables compaction is on by default for all tables with a 512 MB default target file size (configurable 64 to 512 MB); Glue Iceberg compaction is disabled by default for new tables, and Athena compaction is manual. | AWS documentation, 10 Nov 2026 |
| Dremio | Dremio runs automatic optimization on a dedicated engine it configures, and the same operations can be run by hand with OPTIMIZE TABLE. | Dremio documentation, 10 Nov 2026 |
| Google BigQuery and BigLake | BigQuery Iceberg managed tables run automatic storage optimization, so compaction needs no job configuration on Google Cloud. | Google Cloud documentation, 10 Nov 2026 |
| Onehouse | Onehouse lists a Table Optimizer among its product claims. | Onehouse page, 10 Nov 2026 |
| Open stack on EMR and EKS | Iceberg documentation shows compaction as the Spark rewriteDataFiles action, with target-file-size-bytes of 500 MB in its example, run as a job you schedule. | Apache Iceberg documentation, 10 Nov 2026 |
| Starburst | Starburst Galaxy compacts with ALTER TABLE ... EXECUTE optimize; files under file_size_threshold (default 100MB) are merged. | Starburst documentation, 10 Nov 2026 |
What is the published unit price for compaction?
A per-unit price lets you multiply by your own daily volume. Unit prices differ in what they count: gigabytes, objects, DPU-hours or credits.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR plus Athena on Iceberg | S3 Tables in us-east-1: compaction is $0.005 per GB processed plus $0.002 per 1,000 objects; sort or Z-order compaction is $0.01 per GB plus $0.002 per 1,000 objects. | AWS price list, 10 Nov 2026 |
| Dremio | Dremio bills compute in DCUs only while engines are running; billable usage is DCUs for compute, LLM tokens for AI features and storage. | Dremio documentation, 10 Nov 2026 |
| DualBird | DualBird states about $2 per TB compacted for itself against about $6 per TB rising past $13 per TB for CPU clusters; DualBird ran the test and part of the saving is attributed to F2 local NVMe. | DualBird page, 10 Nov 2026 |
| Google BigQuery and BigLake | BigQuery storage optimization is a separate billed item; the dollar rate for it: none found on 2026-11-10 in the pricing page text read. | Google Cloud documentation, 10 Nov 2026 |
| Onehouse | Onehouse bills in OCUs (1 OCU is a 4-core instance for 1 hour in your account); no dollar rate per OCU was found on 2026-11-10. | Onehouse documentation, 10 Nov 2026 |
| Starburst | Starburst Galaxy lists table maintenance and data transforms at 0.30 credits per hour and file ingest at 0.60 credits per hour. | Starburst page, 10 Nov 2026 |
Which Glue and EMR line items apply when your own Spark job compacts?
Self-run compaction is billed as ordinary compute plus storage requests. The AWS price list gives the rates.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR plus Athena on Iceberg | Glue Iceberg compaction costs $0.44 per DPU-hour in us-east-1; Glue 6.0+ ETL jobs cost $0.308 per DPU-hour and older Glue ETL $0.44. | AWS price list, 10 Nov 2026 |
| Open stack on EMR and EKS | EMR Serverless bills vCPU-hours and GB-hours of workers; the us-east-1 rates in AWS's pricing example are $0.052624 and $0.0057785. | AWS page, 10 Nov 2026 |
Which compaction and layout strategies does each platform document?
Bin-pack, sort and Z-order rewrite files differently and cost differently. Check which one the price covers.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR plus Athena on Iceberg | Athena documents OPTIMIZE ... REWRITE DATA USING BIN_PACK; S3 Tables prices sort and Z-order compaction separately from standard compaction. | AWS documentation, 10 Nov 2026 |
| Dremio | Dremio documents data file compaction, clustering, partition evolution and manifest rewriting for Iceberg tables in its Open Catalog. | Dremio documentation, 10 Nov 2026 |
| DualBird | DualBird's published benchmark is sort-based Iceberg compaction on a synthetic table of about 370 million rows and 64 columns with ZSTD level 3. | DualBird page, 10 Nov 2026 |
| Google BigQuery and BigLake | BigQuery Iceberg managed tables document automatic compaction, clustering and garbage collection. | Google Cloud documentation, 10 Nov 2026 |
| Open stack on EMR and EKS | Iceberg documents rewriteDataFiles and manifest rewriting as optional maintenance; Hudi merge-on-read tables need compaction to bound read latency, and Delta Lake documents OPTIMIZE. | Apache Iceberg documentation, 10 Nov 2026 |
How are row-level deletes handled during compaction?
Tables with frequent deletes carry delete files. Compaction either applies them or leaves them for readers.
| Option | Published fact | Source, read |
|---|---|---|
| Amazon EMR plus Athena on Iceberg | S3 Tables compaction also applies row-level deletes. | AWS documentation, 10 Nov 2026 |
| Dremio | Dremio handles row-level deletes for Iceberg tables with position deletes on v2 and deletion vectors on v3. | Dremio documentation, 10 Nov 2026 |
What has been benchmarked for compaction, and by whom?
Most published results are SQL query or ETL tests, and the one compaction benchmark found is run by its vendor.
| Option | Published fact | Source, read |
|---|---|---|
| Dremio | Dremio's published benchmark is a 1 TB TPC-DS query test run by Dremio; it is not a compaction test. | Dremio page, 10 Nov 2026 |
| DualBird | DualBird's compaction white paper (July 1, 2026) restates the results and holds methodology, instance counts and Spark version behind a download form; the blog gives dataset shape and baselines. | DualBird page, 10 Nov 2026 |
| Open stack on EMR and EKS | Iceberg, Delta or Hudi compaction benchmarks for EMR or EKS, with time or cost: none found on 2026-11-10. | Apache Iceberg documentation, 10 Nov 2026 |
| Starburst | A Starburst benchmark of compaction on Galaxy CPU clusters with data size and scripts: none found on 2026-11-10. | Starburst page, 10 Nov 2026 |
Sources
- AWS documentation, read 10 Nov 2026
- Dremio documentation, read 10 Nov 2026
- Google Cloud documentation, read 10 Nov 2026
- Onehouse page, read 10 Nov 2026
- Apache Iceberg documentation, read 10 Nov 2026
- Starburst documentation, read 10 Nov 2026
- AWS price list, read 10 Nov 2026
- Dremio documentation, read 10 Nov 2026
- DualBird page, read 10 Nov 2026
- Onehouse documentation, read 10 Nov 2026
- Starburst page, read 10 Nov 2026
- AWS price list, read 10 Nov 2026
- AWS page, read 10 Nov 2026
- AWS documentation, read 10 Nov 2026
- Dremio page, read 10 Nov 2026
- DualBird page, read 10 Nov 2026
- Starburst page, read 10 Nov 2026
Last reviewed 10 Nov 2026.