Orchestrate Work with Lakeflow Jobs

• Task types
• Task dependencies
• Retries and timeouts
• Branching
• Trigger types
• SKIPPED vs FAILED

Lakeflow Jobs is the built-in orchestration layer in Databricks. A job is a sequence — or graph — of tasks, each of which can be a notebook, Python script, SQL query, dbt project, Delta Live Tables pipeline, or JAR. Jobs can run on a schedule, in response to an event, or triggered manually.

Task types

Each task in a job runs a specific type of work:

Python script tasks are generally preferred over notebook tasks for production jobs because they are easier to test, version, and import as modules.

Task dependencies

Tasks are wired together using depends_on. A task runs only after all its upstream tasks have completed successfully.

tasks:
  - task_key: ingest_raw
    notebook_task:
      notebook_path: /notebooks/01_ingest

  - task_key: clean_silver
    depends_on:
      - task_key: ingest_raw
    notebook_task:
      notebook_path: /notebooks/02_clean

  - task_key: build_gold
    depends_on:
      - task_key: clean_silver
    notebook_task:
      notebook_path: /notebooks/03_gold

Multiple tasks can depend on the same upstream task, creating fan-out patterns. A single task can depend on multiple upstream tasks (fan-in) and will wait for all of them before starting.

Retries and timeouts

Transient failures (network blips, cluster startup issues) are common in distributed jobs. Retry settings handle them without manual intervention:

tasks:
  - task_key: load_external_api
    max_retries: 3
    min_retry_interval_millis: 30000
    retry_on_timeout: true
    timeout_seconds: 300
    python_wheel_task:
      entry_point: fetch_data

retry_on_timeout: true means a task that times out counts as a retryable failure rather than a hard stop. min_retry_interval_millis adds a wait between attempts to avoid hammering a struggling dependency.

Branching

Branching tasks let a job take different paths based on whether an upstream task succeeded, failed, or was skipped. This is useful for sending alerts on failure or running cleanup logic only when needed.

tasks:
  - task_key: run_transform
    notebook_task:
      notebook_path: /notebooks/transform

  - task_key: on_success
    depends_on:
      - task_key: run_transform
        outcome: "succeeded"
    notebook_task:
      notebook_path: /notebooks/notify_success

  - task_key: on_failure
    depends_on:
      - task_key: run_transform
        outcome: "failed"
    notebook_task:
      notebook_path: /notebooks/alert_and_rollback

An outcome of "succeeded" or "failed" on a depends_on entry turns the dependency into a conditional edge. Without an outcome, the dependency requires success.

Trigger types

Lakeflow Jobs supports four trigger mechanisms:

Scheduled: runs on a cron schedule (e.g., every day at 02:00 UTC)

schedule:
  quartz_cron_expression: "0 0 2 * * ?"
  timezone_id: "UTC"

File arrival: triggers when new files appear in a cloud storage path. Useful for event-driven pipelines where ingestion timing is unpredictable.

Table update: triggers when a Unity Catalog table is written to. Downstream jobs can wait for an upstream job to publish data before running.

Continuous: the job runs in a loop, with a configurable pause between iterations. Suitable for near-real-time workloads that cannot use streaming but need frequent updates.

SKIPPED vs FAILED

FAILED: the task ran and errored. The task code threw an exception, a notebook command failed, or the cluster could not start.

SKIPPED: the task did not run because an upstream task did not meet the dependency condition. If a task depends on an upstream task succeeding and that upstream task failed, the downstream task is skipped — not failed. SKIPPED tasks do not consume compute.

This distinction matters when reading job run histories. A long chain of SKIPPED tasks usually points to one root-cause FAILED task upstream. Fix the upstream failure; the skipped tasks will re-run normally on the next run.