- Query and restore prior versions
- VACUUM and physical deletion
- OPTIMIZE and file compaction
- Liquid Clustering
Delta Lake keeps a transaction log (_delta_log) alongside the Parquet files in cloud storage. Every write — insert, update, delete, schema change — creates a new log entry. That log is what makes time travel, ACID guarantees, and safe file cleanup possible.
Query and restore prior versions
Every Delta table version is queryable without any setup.
-- by version number
SELECT * FROM catalog.silver.orders VERSION AS OF 5;
-- by timestamp
SELECT * FROM catalog.silver.orders TIMESTAMP AS OF '2025-01-01 00:00:00';
DESCRIBE HISTORY shows which version maps to which operation and timestamp:
DESCRIBE HISTORY catalog.silver.orders;
To roll a table back to a prior state, RESTORE creates a new version that matches the old one. It does not delete the intervening log entries — the table just continues from a new version number that mirrors the old state:
RESTORE TABLE catalog.silver.orders TO VERSION AS OF 3;
This is useful when a bad merge or schema migration needs to be undone quickly without rebuilding the table from source.
Time travel stops working once VACUUM removes the underlying Parquet files. If a version’s files have been vacuumed, querying that version returns an error — the data is gone from storage, not just from the log.
VACUUM and physical deletion
Delta DELETE and UPDATE operations write new Parquet files and mark old ones for removal — they do not immediately purge the old files from cloud storage. Time travel still works on those files until VACUUM clears them.
VACUUM removes the physical files outside the retention window:
VACUUM catalog.silver.orders; -- removes files older than 7 days (default)
VACUUM catalog.silver.orders RETAIN 168 HOURS; -- same, explicit
Before running VACUUM on production tables, a dry run confirms what will be deleted without actually deleting anything:
VACUUM catalog.silver.orders RETAIN 168 HOURS DRY RUN;
The retention safety check
Running VACUUM with a retention window below 7 days fails by default:
VACUUM catalog.silver.orders RETAIN 0 HOURS;
-- Error: Are you sure you would like to vacuum files with such a short retention period?
Delta Lake blocks sub-7-day retention to protect concurrent readers and active time travel windows. Bypassing the guard requires a config change:
SET spark.databricks.delta.retentionDurationCheck.enabled = false;
VACUUM catalog.silver.orders RETAIN 0 HOURS;
SET spark.databricks.delta.retentionDurationCheck.enabled = true;
Re-enable the check immediately after. Leaving it disabled is a risk because it silently allows future VACUUM calls to delete files that concurrent readers may still need.
Physical deletion for GDPR compliance
A DELETE statement removes rows from the table but leaves the old Parquet files on cloud storage. Time travel can still reach the deleted data. For a true physical deletion the sequence is:
-- soft delete
DELETE FROM catalog.silver.orders WHERE customer_id = 99;
-- still on S3 — time travel still works
SELECT COUNT(*) FROM catalog.silver.orders VERSION AS OF 1 WHERE customer_id = 99;
-- preview what VACUUM would remove
VACUUM catalog.silver.orders RETAIN 0 HOURS DRY RUN;
-- disable the safety check
SET spark.databricks.delta.retentionDurationCheck.enabled = false;
-- physically purge
VACUUM catalog.silver.orders RETAIN 0 HOURS;
-- re-enable
SET spark.databricks.delta.retentionDurationCheck.enabled = true;
-- verify data is gone from storage
SELECT COUNT(*) FROM catalog.silver.orders VERSION AS OF 1 WHERE customer_id = 99;
-- Error: the referenced files have been vacuumed.
Never delete files from S3 or ADLS directly. The _delta_log references every file by path. Removing files outside of VACUUM corrupts the table — queries start failing with file-not-found errors that are difficult to recover from.
OPTIMIZE and file compaction
Small files accumulate in any write-heavy pipeline. Many small Parquet files slow read queries because each file requires a separate read request. OPTIMIZE compacts them:
OPTIMIZE catalog.silver.orders;
For tables registered under Unity Catalog, Predictive Optimization runs OPTIMIZE and VACUUM automatically based on usage patterns — no scheduled job needed:
ALTER CATALOG my_catalog ENABLE PREDICTIVE OPTIMIZATION;
ALTER SCHEMA my_catalog.silver ENABLE PREDICTIVE OPTIMIZATION;
Liquid Clustering
Traditional performance tuning in Delta Lake relied on PARTITION BY for coarse-grained data layout and ZORDER BY for fine-grained co-location. Liquid Clustering replaces both.
Liquid Clustering incrementally re-clusters data based on which columns appear most often in filters and joins, without requiring a full table rewrite. The clustering columns are defined at table creation:
CREATE TABLE catalog.silver.orders
CLUSTER BY (customer_id, order_ts)
USING DELTA;
Or added to an existing table:
ALTER TABLE catalog.silver.orders CLUSTER BY (customer_id, order_ts);
Good clustering columns are high-cardinality columns that appear frequently in WHERE and JOIN conditions — typically 1 to 4. After altering the clustering columns, OPTIMIZE triggers the first re-clustering pass:
OPTIMIZE catalog.silver.orders;
What Liquid Clustering replaces
The older approach combined static partitioning with Z-Order:
-- old pattern
CREATE TABLE orders PARTITIONED BY (order_date) USING DELTA;
OPTIMIZE orders ZORDER BY (customer_id);
This worked but created ongoing maintenance problems. Partitioning by date on write-heavy tables produces large numbers of small partition directories. Z-Order had to be re-applied manually and reset on each OPTIMIZE run. And picking the wrong partition column was hard to undo without rebuilding the table.
Liquid Clustering handles layout incrementally and can be changed without a full rebuild.
Migrating a partitioned table to Liquid Clustering
Liquid Clustering and PARTITION BY cannot coexist on the same table. There is no in-place conversion — the table has to be recreated:
CREATE TABLE catalog.silver.orders_clustered
CLUSTER BY (customer_id, order_ts)
USING DELTA
AS SELECT * FROM catalog.silver.orders;
DROP TABLE catalog.silver.orders;
ALTER TABLE catalog.silver.orders_clustered RENAME TO catalog.silver.orders;
For large tables this rebuild has a real cost, so it is worth getting the clustering columns right early.
Delta Lake table operations — time travel for debugging and recovery, VACUUM for physical purging, OPTIMIZE for read performance, and Liquid Clustering for query-aware layout — cover most of what a data platform team needs to keep tables healthy as they grow.