Declarative Automation Bundles (DABs) let you define Databricks resources — jobs, pipelines, clusters — as code in a databricks.yml file. A bundle lives in Git, deploys consistently across environments, and runs through the Databricks CLI.
Project structure
A DABs project has a predictable layout:
my-project/
databricks.yml
resources/
jobs/
customer_refresh.yml
pipelines/
bronze_ingest.yml
src/
notebooks/
01_ingest.py
02_transform.py
modules/
quality_checks.py
conf/
dev.yaml
prod.yaml
The databricks.yml file declares the bundle name, workspace URLs per environment, and references to resource files:
bundle:
name: customer-pipeline
workspace:
host: https://${var.workspace_host}
include:
- resources/jobs/*.yml
- resources/pipelines/*.yml
targets:
dev:
variables:
workspace_host: adb-dev-workspace.azuredatabricks.net
mode: development
prod:
variables:
workspace_host: adb-prod-workspace.azuredatabricks.net
mode: production
Environment targets
Targets define how the bundle behaves in each environment. mode: development prefixes resource names with the current user’s name, which prevents dev deployments from colliding with production resources.
Variable overrides per target let you point to different storage paths, catalog names, or cluster configurations:
targets:
dev:
variables:
catalog: dev_catalog
cluster_size: Small
prod:
variables:
catalog: prod_catalog
cluster_size: Large
Resources reference variables using ${var.variable_name}:
resources:
jobs:
customer_refresh:
name: customer_refresh_${var.catalog}
tasks:
- task_key: transform
notebook_task:
notebook_path: src/notebooks/02_transform.py
Bundle commands
The main CLI commands for working with a bundle:
# validate the bundle configuration without deploying
databricks bundle validate --target dev
# deploy resources to the workspace
databricks bundle deploy --target dev
# trigger a specific job
databricks bundle run --target dev customer_refresh
# destroy all deployed resources
databricks bundle destroy --target dev
bundle validate checks syntax, variable resolution, and resource references. It does not contact the workspace — it is safe to run in CI before deploying. A common mistake is assuming validate confirms that referenced workspace resources exist; it only confirms the configuration is well-formed.
bundle deploy pushes the job and pipeline definitions to the target workspace. It does not run anything.
Git Folders
Git Folders (formerly Databricks Repos) connects a workspace folder to a remote Git repository. You can pull from a branch, switch branches, and commit changes through the Databricks UI or API.
Git Folders are useful for interactive development — you can edit a notebook in the workspace and push changes back to Git without leaving Databricks. For production deployments, DABs with a CI/CD pipeline is the more reliable path.
To connect a Git Folder: go to Workspace > Repos > Add Repo, enter the Git repository URL, and select the branch. From a notebook, %run and Python imports work on files in the same repo path.
GitHub Actions integration
A typical setup has two pipelines:
CI (on pull request): validate the bundle configuration
name: CI
on:
pull_request:
branches: [main]
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install Databricks CLI
run: pip install databricks-cli
- name: Validate bundle
env:
DATABRICKS_HOST: $
DATABRICKS_TOKEN: $
run: databricks bundle validate --target dev
CD (on merge to main): deploy to production
name: CD
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install Databricks CLI
run: pip install databricks-cli
- name: Deploy to production
env:
DATABRICKS_HOST: $
DATABRICKS_TOKEN: $
run: databricks bundle deploy --target prod
Store workspace URLs and tokens as GitHub Secrets, not in the repository. The token should belong to a service principal, not a personal account, so it is not tied to an individual’s employment.