- Control Plane and Data Plane
- Unity Catalog and the 3-level namespace
- Compute types
- Delta Lake as the storage format
Databricks splits its architecture into two zones: the control plane, which Databricks manages, and the data plane, which runs in your cloud account. Most of what you interact with day-to-day passes through the control plane, but your actual data never leaves the data plane.
Control Plane and Data Plane
The control plane handles the management layer: the web UI, job scheduler, cluster manager, Unity Catalog metadata, and API endpoints. Databricks hosts and operates it.
The data plane is where compute runs and data lives. Spark clusters, SQL Warehouses, and the Parquet files in your storage bucket all exist in your cloud account (AWS, Azure, or GCP). Network traffic between the two planes goes through a secure channel — customer data does not flow through the control plane.
This split is why Databricks can manage cluster lifecycle without needing access to your actual data. The cluster manager in the control plane tells your cloud account to spin up VMs; those VMs run in your account and read from your storage.
Unity Catalog and the 3-level namespace
Unity Catalog is the governance layer that spans all Databricks workspaces in an account. It organizes data objects into a 3-level hierarchy:
catalog.schema.table
- Catalog: top-level container, typically mapped to an environment or domain (
prod,raw,finance) - Schema: a namespace within a catalog, equivalent to a database (
bronze,silver,gold) - Table or view: the actual data object
USE CATALOG prod;
USE SCHEMA silver;
SELECT * FROM orders LIMIT 10;
-- or fully qualified:
SELECT * FROM prod.silver.orders LIMIT 10;
Unity Catalog also manages External Locations — named references to cloud storage paths with access credentials attached. External Locations let you reference storage paths using logical names rather than embedding credentials in notebooks.
CREATE EXTERNAL LOCATION my_data_lake
URL 's3://my-bucket/data/'
WITH (STORAGE CREDENTIAL my_credential);
Compute types
Databricks offers four main compute types, each suited to different workloads:
All-Purpose Cluster: Long-running clusters for interactive development and exploration. Multiple users can share one. They persist until manually terminated or after a configurable idle timeout. Higher per-hour cost due to uptime.
Job Cluster: Spun up for a single job run and terminated when the job finishes. No idle time, no manual cleanup. Job clusters are standard for scheduled production jobs — isolated and cost-efficient.
SQL Warehouse: Purpose-built for SQL Analytics with two modes:
- Serverless: Databricks manages the compute infrastructure; faster startup, no cluster configuration
- Classic (Pro): runs in your VPC, more configuration options
Serverless Compute (for notebooks and jobs): No cluster configuration needed. Databricks handles instance selection and scaling. Good for teams that want to skip cluster management entirely.
The practical difference between All-Purpose and Job clusters shows up in cost and isolation. A nightly job that runs on a shared All-Purpose cluster shares resources with other users and keeps the cluster alive indefinitely. The same job on a Job cluster gets its own compute, runs to completion, and costs nothing while idle.
Delta Lake as the storage format
Every table created in Databricks defaults to Delta format. Delta Lake stores data as Parquet files in cloud storage alongside a _delta_log transaction log.
The transaction log is what provides ACID guarantees on object storage:
- Every write — insert, update, delete, schema change — creates a new log entry
- Concurrent readers and writers are coordinated through the log without file locking
- Schema enforcement is on by default: writes that do not match the existing schema fail
Additional capabilities built on the transaction log:
- Time travel: query any prior version of a table
- MERGE: upsert operations without rebuilding the whole table
- Change Data Feed: downstream systems can consume only the changed rows
- Deletion Vectors: row-level deletes without rewriting entire Parquet files
Delta tables also support OPTIMIZE (compact small files), VACUUM (remove old file versions), and Liquid Clustering (incremental data layout optimization by column value).