When a Databricks job runs slow or fails, there are a few specific places to look. The Lakeflow Jobs UI shows what happened at the job level; the Spark UI shows what happened inside each task.
Job run history
The Lakeflow Jobs UI keeps a run history for each job. From the job page, the Runs tab shows:
- Duration per run — useful for spotting trends: a job that slowly grew from 5 to 45 minutes has a data growth or skew problem
- Start time and trigger source (scheduled, manual, file arrival)
- Task-level status for each run (which task failed, which were SKIPPED)
- Links to logs and Spark UI for each task
Looking at duration trends over multiple runs is more useful than investigating a single slow run in isolation. If duration jumped after a specific date, check what changed: new data volume, a schema change, a code deployment.
Spark UI
Each notebook or job cluster exposes a Spark UI at the cluster detail page. The most useful tabs:
Jobs: shows each Spark action (count, write, collect) as a job entry. Links to the stages that make up the job.
Stages: shows each stage in the query plan. Each stage corresponds to a shuffle boundary. The task summary bar shows min/median/max task duration — if max is much higher than median, there is skew.
SQL: shows the query plan for Spark SQL operations. The physical plan shows which join strategy was chosen (BroadcastHashJoin vs SortMergeJoin) and where data is moving between stages.
Data skew
Data skew happens when one or a few partition keys contain a disproportionate share of the data. In a join or groupBy, one executor gets most of the work while others sit idle.
How to spot it: on the Stages tab in the Spark UI, click into a slow stage. In the task summary, if max duration is 10× the median, you have skew. The task with the max duration is processing a skewed partition.
The fix is salting — adding a random suffix to the skewed key to spread it across more partitions:
from pyspark.sql import functions as F
orders_salted = orders.withColumn(
"customer_id_salted",
F.concat(F.col("customer_id"), F.lit("_"), (F.rand() * 10).cast("int").cast("string"))
)
customers_replicated = (
customers
.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(10)])))
.withColumn(
"customer_id_salted",
F.concat(F.col("customer_id"), F.lit("_"), F.col("salt").cast("string"))
)
)
result = orders_salted.join(customers_replicated, on="customer_id_salted", how="inner")
Spill to disk
Spill happens when a shuffle operation needs more memory than the executor has and starts writing intermediate data to disk. Disk I/O is orders of magnitude slower than memory, so spill dramatically slows down shuffles.
How to spot it: in the Stages tab, the task summary columns include “Spill (disk)”. If that column shows non-zero values for many tasks, spill is a factor.
Common fixes:
- Increase executor memory (
spark.executor.memory) - Reduce the number of shuffle partitions if data volume does not warrant many partitions
- Use broadcast joins to eliminate the shuffle entirely for small-table joins
Shuffle partition tuning
spark.sql.shuffle.partitions controls how many partitions are created after a shuffle (a groupBy, join, or distinct). The default is 200.
200 is often too high for small datasets (many near-empty partitions) and too low for very large ones (each partition too large to fit in memory). Adaptive Query Execution handles this automatically when enabled:
SET spark.sql.adaptive.enabled = true;
SET spark.sql.adaptive.coalescePartitions.enabled = true;
With AQE on, Databricks measures the actual shuffle output size and merges small partitions post-shuffle. On Databricks, AQE is enabled by default and manual shuffle partition tuning is rarely needed.
Cluster startup failures
When a cluster fails to start, the error appears in the cluster event log (Compute > cluster name > Event Log tab). Common causes:
Instance type unavailable: the requested VM type has no capacity in the chosen availability zone. Fix: use a different instance type or enable instance fallback in the cluster configuration.
Init script failure: a shell script attached to the cluster config failed during startup. The init script logs in the event log show which command failed. Common issues: missing packages, broken download URLs, permission errors.
Library installation failure: a Python or Maven library in the cluster config could not be installed (version conflict, missing dependency). The event log shows which library failed and why.
Credential or network issues: the cluster cannot reach a secret scope, external database, or storage location. Usually a VPC routing, firewall rule, or IAM permissions problem.
For Job Clusters specifically: if the cluster cannot start within the job’s configured timeout, the task fails with a cluster startup error. The run logs show “Cluster terminated before becoming ready.”