# Iceberg compaction: rewrite_data_files, strategies and what platforms automate

Table maintenance, read 10 Nov 2026

**The short answer.** Iceberg compaction rewrites many small data files into fewer files near a 512 MB target, using the rewrite_data_files procedure with the binpack, sort or z-order strategy. Open-source Iceberg leaves scheduling to you; AWS, Databricks, Dremio, Snowflake and BigQuery document managed versions.

## The procedure and the action

Iceberg tracks each data file in manifest files, so more small files means more metadata and more file-open cost. The Spark procedure rewrite_data_files combines small files into larger ones in parallel. In SQL: CALL catalog_name.system.rewrite_data_files(table => 'db.sample'). The Java and Scala equivalent is SparkActions.rewriteDataFiles, which accepts a filter and options such as target-file-size-bytes.

The where argument limits the rewrite to files that may contain matching data. Iceberg's maintenance page lists compaction under optional maintenance, next to rewriting manifests, and notes that streaming queries may produce small data files. The related procedures are rewrite_manifests and rewrite_position_delete_files.

## Strategies

| Strategy | How it is set | What it does |
| --- | --- | --- |
| binpack | strategy => 'binpack' (the default) | Combines small files into files near the target size without reordering rows |
| sort | strategy => 'sort', sort_order => 'id DESC NULLS LAST' | Rewrites files with rows sorted by the listed columns; defaults to the table's sort order |
| z-order | strategy => 'sort', sort_order => 'zorder(c1,c2)' | Interleaves several columns into one sort key |
| hilbert | strategy => 'sort', sort_order => 'hilbert(c1,c2)' | Hilbert curve ordering; every column contributes 8 bytes; cannot be combined with zorder() or plain columns |

Source: Apache Iceberg Spark procedures page, read 2026-11-10.

## Defaults that decide when files are rewritten

The target file size is 512 MB, the default of write.target-file-size-bytes. A file under 75% of the target (384 MB) or over 180% of it (about 922 MB) is selected for rewriting regardless of other criteria. Any file group with 5 or more files (min-input-files) is rewritten, and a single group is capped at 100 GB of input (max-file-group-size-bytes). Up to 5 file groups are rewritten at once (max-concurrent-file-group-rewrites).

Deletes count too. A data file with a deletion ratio of 0.3 or more (delete-ratio-threshold) is considered for rewriting, and remove-dangling-deletes, off by default, adds a commit that drops delete files that no longer apply. Set partial-progress.enabled to commit groups as they finish, with 10 commits allowed by default, so a long rewrite on a big table survives a failure part way.

Rewrite-all forces every file to be rewritten, which is what a sort or z-order run over the whole table needs. AWS states that sort and z-order compaction may cost more than binpack in S3 Tables, and prices them higher, as the last section shows.

## What each platform automates for Iceberg

Vendors alphabetical, with the AWS services grouped first. Details and sources are on [compaction on each platform](/maintenance/compaction-on-each-platform/).

AWS Glue Data Catalog compaction is off for new tables and enabled per table; it starts when a table or partition has more than 100 files, each smaller than 75% of the target. S3 Tables compaction is on by default and offers auto, binpack, sort and z-order, with a target of 64 to 512 MB. Athena runs compaction only when you issue OPTIMIZE ... REWRITE DATA USING BIN_PACK. BigQuery Iceberg managed tables compact automatically. Databricks predictive optimization covers Unity Catalog managed Iceberg tables. Dremio optimizes Iceberg tables in its Open Catalog automatically. Snowflake compacts Snowflake-managed Iceberg tables by default and does nothing for externally managed ones. Starburst Galaxy runs optimize when you issue ALTER TABLE ... EXECUTE.

## Cost of running it yourself

On EMR the rewrite is a Spark job billed as EC2 hours plus the EMR uplift; see [EMR cost optimization](/guides/emr-cost-optimization-lakehouse/). The managed AWS options bill separately: Glue at $0.44 per DPU-hour, S3 Tables at $0.005 per GB processed plus $0.002 per 1,000 objects, or $0.01 per GB for sort and z-order. No published benchmark comparing these on one table was found on 2026-11-10; the one compaction benchmark with a stated cost is DualBird's own and is described on the platforms page.

## Other maintenance pages

- [Iceberg snapshot expiry and orphan file removal](/maintenance/snapshot-expiry-and-orphan-files/)
- [Delta Lake OPTIMIZE and VACUUM: defaults and platform automation](/maintenance/delta-optimize-and-vacuum/)
- [Apache Hudi table services: compaction, clustering, cleaning](/maintenance/hudi-table-services/)
- [Which platforms automate Iceberg compaction](/maintenance/compaction-on-each-platform/)

## Sources

- Apache Iceberg Spark procedures: https://iceberg.apache.org/docs/latest/spark-procedures/ (read 10 Nov 2026)
- Apache Iceberg maintenance: https://iceberg.apache.org/docs/latest/maintenance/ (read 10 Nov 2026)
- Apache Iceberg configuration: https://iceberg.apache.org/docs/latest/configuration/ (read 10 Nov 2026)
- Amazon S3 Tables maintenance: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html (read 10 Nov 2026)
- AWS Glue compaction optimization: https://docs.aws.amazon.com/glue/latest/dg/compaction-management.html (read 10 Nov 2026)
- Athena Iceberg optimization: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-data-optimization.html (read 10 Nov 2026)
- BigQuery Iceberg tables: https://cloud.google.com/bigquery/docs/iceberg-tables (read 10 Nov 2026)
- Databricks predictive optimization: https://docs.databricks.com/aws/en/optimizations/predictive-optimization (read 10 Nov 2026)
- Dremio Cloud table optimization: https://docs.dremio.com/dremio-cloud/manage-govern/optimization (read 10 Nov 2026)
- Snowflake Iceberg table management: https://docs.snowflake.com/en/user-guide/tables-iceberg-manage (read 10 Nov 2026)
- Starburst Galaxy Iceberg: https://docs.starburst.io/starburst-galaxy/data-engineering/working-with-data-lakes/table-formats/gl-iceberg.html (read 10 Nov 2026)
- AWS price list, AWS Glue, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AWSGlue/current/index.json (read 10 Nov 2026)
- AWS price list, Amazon S3, us-east-1: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonS3/current/index.json (read 10 Nov 2026)

Last reviewed 10 Nov 2026. Canonical: https://datalakehousecompare.com/maintenance/iceberg-compaction/
