Where to put Databricks job standards in a DAB

The setup

To show how presets, Python mutators and cluster policies work, I use a small orders bundle in medallion style. I built it for this article, and you can find it in this repository. It has three jobs:

• bronze loads raw order files from a Unity Catalog volume with Auto Loader, once a day.
• silver cleans up types, column names and duplicates. It starts when the bronze table changes.
• gold builds a SQL mart on top of silver, on a SQL warehouse. It starts when the silver table changes.

The jobs are simple on purpose: they give the standards something to work on. I deploy them to three targets: dev for development, acc for acceptance and prd for production. The team behind the bundle agreed on a few standards. Every job has an owner, layer and project tag and alerts the team when it fails. Serverless jobs pin one environment version so a new runtime doesn’t sneak in unannounced. At the start, each job file holds those standards itself. The bundle looks like this:

orders/
├── databricks.yml
├── pyproject.toml
├── resources/
│   ├── orders_bronze.job.yml
│   ├── orders_silver.job.yml
│   └── orders_gold.job.yml
├── src/orders/
└── tests/

databricks.yml is the center of the bundle, the place where everything comes together. It names the bundle, pulls in the job files from resources/, and defines the targets. For now, that is about all it does:

# Omitted: artifacts, variables, all but mode per target
bundle:
  name: orders
  databricks_cli_version: "1.23.0"

include:
  - resources/*.yml

targets:
  dev:
    default: true
    mode: development
  acc:
    mode: production
  prd:
    mode: production

Each job has its own file under resources/. The silver job, with the team’s standards still in it, looks like this:

# Omitted: trigger, named_parameters
resources:
  jobs:
    orders_silver:
      name: orders_silver
      tags:
        owner: orders-team      # required on every job
        layer: silver
        project: orders         # same on every job
      email_notifications:      # same on every job
        on_failure:
          - [email protected]
      tasks:
        - task_key: standardize
          environment_key: default
          python_wheel_task:
            package_name: orders
            entry_point: silver
      environments:
        - environment_key: default
          spec:
            environment_version: "6"   # same in every environment
            dependencies:
              - ../dist/*.whl

The commented lines aren’t really about this job; they are there because the team agreed on them. The project tag, the failure alert and the environment version repeat, word for word, across the job files. The owner tag is a bit different. Its value changes from job to job, but the team wants every job to have one. In the rest of this article I take those lines out of the job files, one at a time, and put each in the place that fits it. I start with the simplest: presets.

Presets

A preset is a setting in databricks.yml that the CLI applies to the resources in the bundle that support it, so you write it once and all jobs get it. The CLI offers a limited set of presets, listed in the reference: a name prefix, tags, whether schedules and triggers are paused, the maximum number of concurrent runs per job, and a few options for pipelines and artifacts. Presets live at two levels:

• Top level, directly in databricks.yml. These apply to all targets.
• Target level, under one target in targets. These apply only when you deploy to that target.

The example below shows both levels: a project tag at the top level, and an environment tag per target:

# Omitted: everything but the presets, the acc target
presets:
  tags:
    project: orders

targets:
  dev:
    presets:
      tags:
        environment: dev
  prd:
    presets:
      tags:
        environment: prd

When you deploy to dev, each job gets both tags: project: orders and environment: dev. If the two levels set the same key, the target level wins. A tag the job sets itself always wins, so a job with its own environment tag keeps it.

A target’s mode already sets some presets for you. In development mode the CLI puts [dev <user>] in front of the resource names and pauses schedules and triggers.

In the example, all three jobs get the project tag from the top-level preset, so I can remove it from each job file:

# Omitted: all but the tags
resources:
  jobs:
    orders_silver:
      tags:
        owner: orders-team
        layer: silver
        project: orders

Presets don’t cover everything the team wants to standardize. There is no preset for the failure alert or the environment version, so those stay in the job files for now. And a preset puts the same value on every resource: it can’t add a failure alert only where one is missing, and it can’t refuse a job without an owner. Those cases need code, which is what Python mutators are for.

Python mutators

A Python mutator is a function the CLI runs on every job when it loads the bundle. The function receives the job and can do one of three things: return it unchanged, return a changed copy, or raise an error that stops the command. Whatever it returns is the job that gets deployed. The name comes from programming in general, where a mutator is a method that changes an object, like list.append, but a bundle mutator doesn’t change the job in place. A job is a dataclass, so dataclasses.replace makes the changed copy, a new object with only the fields you name changed.

Mutators are more flexible than presets, and they work differently. A preset is a fixed value: the CLI puts the same value on every resource, unless the resource sets its own. A mutator is code: it looks at each resource and decides what to do with it. That lets a mutator do three things presets can’t:

1. Change any field. A mutator can change any field of a resource, including the many fields that have no preset, such as notifications, timeouts or retries on a job.
2. Add conditions. A mutator sees what a resource already has, so it can fill a field only when it is empty, or follow any other rule you can write in Python.
3. Enforce rules. A mutator can stop the command when a resource breaks a rule.

A mutator runs every time the CLI loads the bundle, so on validate and plan as well as on deploy. It runs after the presets, so it sees the job as it will be deployed, with the [dev <user>] name and the preset tags. Below, I build a mutator in three steps: change a field, add a condition and enforce a rule.

1. Change any field

The smallest mutator there is adds the failure alert to every job:

from dataclasses import replace

from databricks.bundles.core import Bundle, job_mutator
from databricks.bundles.jobs import Job, JobEmailNotifications

ALERT_EMAIL = "[email protected]"


@job_mutator
def add_failure_email(bundle: Bundle, job: Job) -> Job:
    alert = JobEmailNotifications(on_failure=[ALERT_EMAIL])
    return replace(job, email_notifications=alert)

The function returns a copy of the job with the alert set, so the email_notifications block can go from the job files. The @job_mutator decorator makes the CLI call it for every job, also for jobs defined in YAML. Pipelines, schemas and volumes have their own decorator, but here I stick to jobs.

The function lives in mutators.py, next to databricks.yml. To register it, list it under mutators in databricks.yml:

python:
  venv_path: .venv
  mutators:
    - "mutators:add_failure_email"

2. Add conditions

The function above replaces the notifications a job already has, so I add a condition: a job that already has notifications keeps them.

# Omitted: imports, ALERT_EMAIL
@job_mutator
def add_failure_email(bundle: Bundle, job: Job) -> Job:
    if job.email_notifications:
        return job
    alert = JobEmailNotifications(on_failure=[ALERT_EMAIL])
    return replace(job, email_notifications=alert)

The environment version works the same way. A job can have several environments, so the mutator goes through them and only fills in the version where it is missing:

# Omitted: imports
@job_mutator
def add_environment_version(bundle: Bundle, job: Job) -> Job:
    envs = []
    for env in job.environments:
        if env.spec and not env.spec.environment_version:
            spec = replace(env.spec, environment_version="6")
            env = replace(env, spec=spec)
        envs.append(env)
    return replace(job, environments=envs)

The version sits inside the environment’s spec, so it takes two replace calls. Register this mutator next to add_failure_email in databricks.yml. The example repository combines both in one function, add_defaults.

3. Enforce rules

The last standard is an ownership rule: every job needs an owner tag. Its value belongs to the job, so a mutator can’t fill it in, but it can check that the tag is there and refuse the job if it isn’t:

# Omitted: imports, add_defaults and its helpers
@job_mutator
def require_owner(bundle: Bundle, job: Job) -> Job:
    if "owner" not in job.tags:
        raise ValueError(
            f'Job "{job.name}" has no "owner" tag. '
            'Add the tag "owner" to the job in its resource file.'
        )
    return job

If a mutator raises an exception, the command stops. For example, databricks bundle validate fails like this when a job has no owner tag:

# (traceback omitted, user replaced)
Error: Failed to apply 'require_owner' mutator
  at resources.jobs.orders_gold
  in mutators.py:23:1

ValueError: Job "[dev <user>] orders_gold" has no "owner" tag. Add the tag "owner" to the job in its resource file.

Back to the silver job from the presets section: with the mutators in place, I can also remove the alert block and the environment version from the job files:

# Omitted: trigger, tasks
resources:
  jobs:
    orders_silver:
      name: orders_silver
      tags:
        owner: orders-team
        layer: silver
      email_notifications:
        on_failure:
          - [email protected]
      environments:
        - environment_key: default
          spec:
            environment_version: "6"
            dependencies:
              - ../dist/*.whl

The job file now only holds what makes this job different. The presets add the project and environment tags, and the mutators add the alert and the version. That is also the price: the job file no longer contains all of the job’s configuration.

Cluster policies

A cluster policy is a set of rules for cluster settings, stored in the workspace. A cluster that uses it gets the policy’s defaults and can’t break its limits. Presets and mutators run in the CLI and only cover one bundle, but a policy lives in the workspace, so it applies to every cluster that uses it, whichever bundle or user creates that cluster. Use a policy for compute rules that every team has to follow, not just yours.

Not every job has a cluster. A job gets its compute in one of two ways:

• Classic compute. The job defines its own cluster, a job cluster: the runtime version, the VM type, the number of workers.
• Serverless compute. The job defines no cluster, because Databricks manages compute settings for serverless compute. At most, it picks an environment, as in the example above.

If all your jobs run on serverless, you don’t need a cluster policy, because policies only apply to classic compute. Serverless has serverless usage policies instead, set up by an admin. They only tag serverless usage, so you can track its cost. They don’t set defaults or limits.

The orders jobs run on serverless compute, so the repository has a second example: a weather bundle with one job that runs on classic compute. That job picks its own runtime, VM type and number of workers, like any classic job. A policy sets the rules instead. Here it is, simplified to three rules:

resources:
  cluster_policies:
    job_compute:
      name: job-compute-${bundle.target}
      definition:
        spark_version:
          type: unlimited
          defaultValue: auto:latest-lts
        node_type_id:
          type: allowlist
          values:
            - Standard_D4ds_v5
            - Standard_D8ds_v5
          defaultValue: Standard_D4ds_v5
        custom_tags.policy:
          type: fixed
          value: job-compute

Each rule is a different type:

• A default. spark_version accepts any version, but a job that leaves it out gets auto:latest-lts, the latest long-term support runtime.
• An allowlist. A job can only pick a VM type from the list. The default sits in that list, so a job that doesn’t specify a VM type gets the standard size.
• A fixed value. Every cluster that uses the policy carries the tag policy: job-compute, and a job can’t change it. It is a cluster tag (custom_tags), not a job tag like the ones the presets set.

A job can override a default, but it can’t break an allowlist or a fixed value: Databricks then refuses the cluster. Unlike presets and mutators, this isn’t checked in your DAB. The workspace enforces it when the job’s cluster is created. Defaults have one more catch: the API only applies them when the cluster asks for it, with apply_policy_default_values: true, which the job cluster sets.

The job finds the policy through a lookup variable, job_policy, which uses the policy’s name because the ID differs per workspace. The job cluster then points to the policy instead of choosing a runtime and a VM type itself:

# Omitted: name, tags, schedule, tasks, custom_tags
resources:
  jobs:
    weather_ingest:
      job_clusters:
        - job_cluster_key: default
          new_cluster:
            spark_version: 18.2.x-scala2.13
            node_type_id: Standard_D4ds_v5
            policy_id: ${var.job_policy}
            apply_policy_default_values: true
            autoscale:
              min_workers: 1
              max_workers: 4

The runtime and the VM type are gone from the job, because the policy fills them in and adds the policy tag. The job still sets its own worker count.

Choosing the right place

Presets, mutators and cluster policies each do a different job. A preset puts the same value on every resource. A mutator looks at each job, and can fill a gap or refuse it. A cluster policy sets rules for compute, in the workspace, for every cluster that uses it. Four questions help you pick one:

1. Where does it live? Presets and mutators live in your DAB project and only cover what it deploys. A policy lives in the workspace, even if you deploy it from a bundle, and covers every cluster that uses it.
2. When does it apply? Presets and mutators run in the CLI, every time it loads the bundle. A policy applies in the workspace.
3. Can it refuse, or only set a default? Presets only set values, and a job can override most of them. A mutator can also refuse a job. A policy does both, and no cluster can break its limits.
4. Can I see the effect before deploying? Presets and mutators run during databricks bundle validate, so you can check them before deploying. A policy doesn’t, so you only see its effect on the cluster.

Pick the simplest option that works. A preset is one line of YAML, while a mutator is code you have to maintain, so only use a mutator for what a preset can’t do. Mutators and policies also work well together: a policy only applies to clusters that name it, so a mutator can set policy_id on every job cluster in the bundle, or refuse a job cluster without one.

There is one more way to share standards: a bundle template. Every bundle you create from it starts with these standards in place. But a template only acts once, when the bundle is created. If a standard changes later, existing bundles don’t follow. Presets and mutators apply again on every deploy, and a policy on every cluster that uses it.

Closing

So where does a standard belong? Start with presets: they are a few lines of YAML, and validate runs them. Move a standard to a mutator when no preset covers it, or when it needs a condition or a check. Use a cluster policy when the rule is about classic compute and every team has to follow it.