PySpark habits that make Databricks jobs cost more

These sixteen Spark code habits make Databricks jobs run longer than they need to. The costliest eight come with bad and better PySpark, and the guide ends with how to catch them before they merge.

Spark code and the Databricks bill

Databricks bills DBUs for every hour a cluster or warehouse runs (serverless jobs bill only while work runs), and on classic compute your cloud provider bills the virtual machines underneath for the same time. So a job that runs twice as long costs about twice as much, and code that needs more memory than it should pushes teams onto bigger, more expensive machines.

Spark code is where a lot of that extra time comes from. The patterns below make Spark move more data, run more jobs or do work in Python that the engine could do faster. The eight that usually cost the most each get a bad and a better version and the rule that flags them. Eight more are in a short table after them.

The rule codes are from cylint, a free linter by Cluster Yield, and I ran every snippet through cylint 0.1.5 to check them. I use cylint's rule codes here because it is the more mature tool. My own linter, sparklint, is at the end.

Code is only part of the bill. The rest is how compute is set up and used, which is not in your code at all. More on that further down.

Start with the jobs that cost the most

A slow notebook that runs once a month matters less than a mediocre one that runs every hour. Rank your jobs by cost first and review the code behind the top rows. The system.billing.usage table has every DBU, and system.billing.list_prices has the price of each SKU:

SQL Top 25 jobs by list cost, last 30 days

-- Which jobs cost the most? Last 30 days, DBUs at list price.
-- (List price, before your discount. Classic compute also has a cloud VM bill
-- that is not in this table.)
SELECT
  u.usage_metadata.job_id                                         AS job_id,
  round(sum(u.usage_quantity * p.pricing.effective_list.default), 2) AS list_cost
FROM system.billing.usage u
JOIN system.billing.list_prices p
  ON  u.sku_name   = p.sku_name
  AND u.cloud      = p.cloud
  AND u.usage_unit = p.usage_unit
  AND u.usage_start_time >= p.price_start_time
  AND (p.price_end_time IS NULL OR u.usage_start_time < p.price_end_time)
WHERE u.usage_date >= current_date() - INTERVAL 30 DAYS
  AND u.usage_metadata.job_id IS NOT NULL
GROUP BY ALL
ORDER BY list_cost DESC
LIMIT 25;

1–3: work on the driver

The driver is one machine. Anything that pulls rows onto it, or loops over rows on it, turns a parallel cluster into a single-threaded program while every worker waits and bills.

1. collect() or toPandas() on a big DataFrame

cylint: CY001, CY006 (and CY014 when both run on the same DataFrame)

Every row has to travel to the driver and fit in its memory. At best the job is slow. At worst the driver runs out of memory and the whole run is lost, and retried if retries are set.

PySpark Aggregate or limit before you collect

from pyspark.sql import functions as F

# Bad: every row of the table lands in the driver's memory
orders = spark.read.table("sales.orders")
rows = orders.collect()
pdf = orders.toPandas()

# Better: let Spark do the work, bring back only the answer
orders = spark.read.table("sales.orders")
totals = (
    orders.groupBy("country_code")
    .agg(F.sum("amount").alias("amount"))
)
pdf = totals.toPandas()                 # a few hundred rows
sample = orders.limit(1000).toPandas()  # or a small sample

2. Looping over collect() results

cylint: CY031 (and CY001)

Each pass of the loop starts its own Spark job, one after another, and often writes a tiny batch each time. A join does the same work in one job, in parallel.

PySpark Replace the driver loop with a join

# Bad: a loop on the driver, one Spark job
# (and one small write) per row
risky = spark.read.table("crm.customers").where("risk_score > 0.9")
for row in risky.collect():
    orders = (
        spark.read.table("sales.orders")
        .where(f"customer_id = {row.customer_id}")
    )
    orders.write.mode("append").saveAsTable("review.risky_orders")

# Better: one join, one write
risky = (
    spark.read.table("crm.customers")
    .where("risk_score > 0.9")
    .select("customer_id")
)
orders = spark.read.table("sales.orders")
(
    orders.join(risky, "customer_id", "left_semi")
    .write.mode("append")
    .saveAsTable("review.risky_orders")
)

3. show(), display() and count() left in production

cylint: CY012, CY020, CY014

Each of these is an action. A count() runs the whole query again, and a show() or display() runs it until it has the rows it shows, which can still mean most of the work. A show() and two count() calls before a write can mean four passes over the data instead of one. To test for an empty result, use isEmpty(), which stops at the first row, not count() == 0.

PySpark One action per job, not four

# Bad: show(), count() and the emptiness check
# each run the query again
events = (
    spark.read.table("raw.events")
    .where("event_date = current_date()")
)
events.show()
print(events.count())
if events.count() == 0:
    raise ValueError("no events today")
events.write.mode("overwrite").saveAsTable("silver.events_today")

# Better: no debug output in the job, and an
# emptiness check that stops at one row
events = (
    spark.read.table("raw.events")
    .where("event_date = current_date()")
)
if events.isEmpty():  # Spark 3.3+; or len(events.take(1)) == 0
    raise ValueError("no events today")
events.write.mode("overwrite").saveAsTable("silver.events_today")

4–5: work the optimizer can't touch

4. Python UDFs where a built-in function exists

cylint: CY002, CY009 (UDF inside a filter)

A plain Python UDF sends every row from the JVM to a Python worker and back, and Spark can't optimize through it. Used in a filter, it also stops the filter from being pushed down to the scan, so Spark reads everything first. Built-in functions in pyspark.sql.functions avoid all of that. When there is no built-in, a pandas UDF is the middle ground: still Python, but data moves in Arrow batches instead of row by row.

PySpark Built-ins first, pandas UDFs when you must

import pandas as pd
from pyspark.sql import functions as F

# Bad: every row goes to a Python process and back,
# and the filter can't be pushed down
is_gmail = F.udf(lambda s: s.lower().endswith("@gmail.com"), "boolean")
users = spark.read.table("crm.users")
gmail = users.where(is_gmail(F.col("email")))

# Better: the built-in function runs inside the JVM
users = spark.read.table("crm.users")
gmail = users.where(F.lower("email").endswith("@gmail.com"))

# No built-in for it? A pandas UDF moves whole Arrow
# batches, not single rows.
@F.pandas_udf("string")
def browser_family(ua: pd.Series) -> pd.Series:
    # parse_browser: your own (or a library's) Python function
    return ua.map(parse_browser)

5. Filtering late, or on a wrapped column

cylint: no rule for this one (it does flag the cache without unpersist, CY025)

Spark's optimizer moves many filters earlier on its own, but not past a cache() or collect(), and not a filter that itself calls a Python UDF. So filter before you cache or collect. A filter on a function of a column, like year(order_date) = 2026, often can't use file statistics to skip data; the same filter as a range on the raw column can.

PySpark Filter first, on the raw column

from pyspark.sql import functions as F

# Bad: caches the whole table, then filters it,
# on a column wrapped in a function
orders = spark.read.table("sales.orders").cache()
this_year = orders.where(F.year("order_date") == 2026)

# Better: filter first, as a plain range on the raw column
this_year = spark.read.table("sales.orders").where(
    (F.col("order_date") >= "2026-01-01")
    & (F.col("order_date") < "2027-01-01")
)

6–8: joins

Adaptive Query Execution (AQE, on by default in current Spark and Databricks runtimes) fixes some join problems while the query runs: it can switch to a broadcast join and split skewed partitions in sort-merge joins. These three it fixes only partly, or not at all.

6. Cross joins and joins with only a range condition

cylint: CY007, CY015 (and CY010 for a join without how=)

Without an equality key, Spark has to compare every row on one side with every row on the other: a million events against a hundred thousand sessions is a hundred billion comparisons. Add an equality key when the data has one. On Databricks, the range_join hint helps when a pure range join can't be avoided. Pass how= explicitly too, for clarity.

PySpark Give the join a key

events = spark.read.table("raw.events")
sessions = spark.read.table("silver.sessions")

# Bad: no join key, so every event is compared with every session
hits = events.join(
    sessions,
    (events.event_ts >= sessions.start_ts)
    & (events.event_ts < sessions.end_ts),
)

# Better: add an equality key, so Spark can hash or
# sort-merge, then apply the range
hits = events.join(
    sessions,
    (events.user_id == sessions.user_id)
    & (events.event_ts >= sessions.start_ts)
    & (events.event_ts < sessions.end_ts),
    "inner",
)

# A pure range join you can't avoid: Databricks'
# range join hint (bin size in seconds)
hits = events.hint("range_join", 300).join(
    sessions,
    (events.event_ts >= sessions.start_ts)
    & (events.event_ts < sessions.end_ts),
    "inner",
)

7. Exploding joins (duplicate keys on both sides)

cylint: no rule; check key uniqueness

When both sides have many rows per key, the result has their product: 50 orders and 20 tickets for one customer become 1,000 rows. The job shuffles and writes far more than anyone meant, and a later distinct() often hides it. Bring one side down to one row per key before the join.

PySpark One row per key before you join

from pyspark.sql import functions as F

orders = spark.read.table("sales.orders")      # many rows per customer
tickets = spark.read.table("support.tickets")  # many rows per customer

# Bad: orders x tickets rows for every customer
joined = orders.join(tickets, "customer_id", "inner")

# Better: bring one side down to one row per key first
ticket_counts = (
    tickets.groupBy("customer_id")
    .agg(F.count("*").alias("tickets"))
)
joined = orders.join(ticket_counts, "customer_id", "left")

8. Skewed joins and aggregations

cylint: no rule; look at the key counts

When a few keys hold most of the rows (guest users, a default ID, NULL), one task gets them all and the rest of the cluster waits for it, billing the whole time. AQE splits skewed partitions in sort-merge joins, but not every case. Take a single hot key out and handle it apart, or salt the key to spread it. The salting example uses a deliberate 16-row cross join, so it carries a # cy:ignore CY007 comment: that is what the suppression is for.

PySpark Split out the hot key, or salt it

from pyspark.sql import functions as F

orders = spark.read.table("sales.orders")
customers = spark.read.table("crm.customers")

# Look first: a few keys holding most of the rows is skew
(
    orders.groupBy("customer_id").count()
    .orderBy(F.desc("count"))
    .limit(10)
    .collect()
)

# Fix 1: hot keys (guests: 0, -1 or NULL)? Take them
# out of the join and handle them apart
is_guest = (
    F.col("customer_id").isNull()
    | F.col("customer_id").isin(0, -1)
)
guests = orders.where(is_guest)
known = orders.where(~is_guest).join(customers, "customer_id", "left")
result = known.unionByName(guests, allowMissingColumns=True)

# Fix 2: salting. Spread each key over N buckets;
# copy the small side N times
N = 16
salted = orders.withColumn("salt", (F.rand() * N).cast("int"))
salts = spark.range(N).withColumnRenamed("id", "salt")
customers_n = customers.crossJoin(salts)  # cy:ignore CY007
joined = (
    salted.join(customers_n, ["customer_id", "salt"], "inner")
    .drop("salt")
)

Eight more, in short

These cost less on a typical job, or are easy to catch once you know them.

Habitcylint ruleFix
withColumn() in a loopCY003 (CY011 for renames and drops)Add the columns in one withColumns() (Spark 3.3+) or one select(), so the plan is analyzed once.
SELECT * into a tableCY004Name the columns you need. A written copy keeps every column, and every later reader pays for it.
Schema inference on big CSV or JSONCY018State the schema, so Spark reads the files once instead of twice.
Reading the same table once per country, customer or dayNone; catch it in reviewRead it once, groupBy() the key and write once.
Single-use cache, or a cache never releasedCY005, CY025, CY014Cache only what two or more actions read, then unpersist() it.
Broadcasting the big side of a joinNone; check table sizesHint a broadcast only on the side you know is small. Spark broadcasts small tables on its own.
A window with orderBy() and no partitionBy()CY017Partition the window by the key the logic is about, so one task does not sort every row.
repartition() or coalesce(1) before a write, or small appends in a loopCY008, CY013 (CY014 for the loop)Read everything at once and write once, and let Delta size the files.

Catch them with cylint

cylint is a free, open-source (Apache 2.0) PySpark linter. I use it and recommend it. It reads your .py files with Python's ast module, follows DataFrames from spark.read, spark.sql() and spark.table() through your code, and flags patterns like the ones above. It needs no Spark, no cluster and no account, so it runs on a laptop and in CI in seconds.

Shell Install it, list the rules, lint a folder

pip install cylint

cy rules                              # every rule and its code
cy lint src/                          # a file or a folder of .py files
cy lint --min-severity warning src/   # skip info-level findings
cy lint --format json src/ > cy.json  # machine-readable output

Exit codes make it easy to gate on: 0 means no findings, 1 warnings or info, 2 critical findings. Databricks notebooks committed in source format are plain .py files (they start with # Databricks notebook source), and cylint lints them like any other file; I checked. Turn rules off or skip folders with a config file.

YAML .cylint.yml

# .cylint.yml in the repo root (or [tool.cylint] in pyproject.toml)
min-severity: warning

rules:
  CY004: off        # e.g. you use SELECT * on purpose in dynamic SQL

exclude:
  - tests/
  - notebooks/scratch/

Silence a single finding you have checked with a comment on the line, for example # cy:ignore CY001. In GitHub Actions, --format github turns findings into notes on the pull request diff.

YAML Lint every pull request

# .github/workflows/pyspark-lint.yml
name: PySpark Lint
on: pull_request

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install cylint
      # Fails on critical findings only (exit code 2). Once the
      # warnings are cleared, delete "|| [ $? -eq 1 ]" to fail on them too.
      - run: cy lint --min-severity warning --format github src/ || [ $? -eq 1 ]

Know its limits. cylint looks at code, not data, so it can't tell a collect() on ten rows from one on ten billion, and it has no rule for reading the same table over and over, exploding joins, skew or broadcasting the wrong side. Some findings will be fine. Check them with the author.

How to fix these in a team

A linter catches the obvious cases. A short review checklist catches the rest.

  • Does anything pull a large DataFrame to the driver, or loop over rows there?
  • Are there show(), display() or count() calls that only served debugging?
  • Is every UDF needed, or is there a built-in? If not, is it a pandas UDF?
  • Does every join have an equality key, an explicit how=, and a side that is unique per key?
  • Is anything cached that is used once, or never unpersisted?
  • Are filters applied early, on raw columns, and are only the needed columns read?
  • Do CSV and JSON reads state a schema?
  • Does every window have a partitionBy()?
  • Does the job write once, without a repartition() or coalesce(1) in front?

Then add a CI gate. Add it in steps.

  1. Run cy lint on the whole repo once and fix or suppress what is there. Exclude scratch folders in .cylint.yml.
  2. Lint every pull request with --min-severity warning and fail the build on exit code 2 (critical findings: withColumn loops, cross and non-equi joins, UDFs in filters, repeated actions without cache).
  3. Once the warnings are cleared, fail on exit code 1 too: delete || [ $? -eq 1 ] from the workflow above.
  4. For the big jobs, note the cost per run before a fix and a week or two after, so you know which fixes paid off.

Code fixes don't reach the rest of the bill

Clean Spark code makes each job cheaper to run. It does nothing for compute that runs when no job does, for clusters bigger than the work, for pipelines and tables nobody uses any more, or for paying more per hour than you need to. In five years of working with Azure Databricks at several companies, I often found the bigger savings on that infrastructure side. It lives in how compute is set up and used, not in code, so no linter sees it. Broadly it falls into six areas.

  • Clusters with more machine than the work needs.
  • Compute that keeps billing while nothing runs.
  • Runs that fail or retry, which you pay for and throw away.
  • Start-up time on work that starts compute often.
  • Jobs and tables whose output no one reads.
  • Compute types that cost more than they look.

You can dig into these by hand in the system tables, and it pays to.

By hand or with a tool

Use a linter for code in every case. For the rest, it depends on who has the time.

  • Do it yourself if you have one or two workspaces and someone who owns cost will go through the system tables every month and check what each change saved. It costs only their time.
  • Databricks' own tools (cost dashboards, budgets, tags) show where money goes and stop some new waste. Use them either way.
  • ClusterBite fits when you want the infrastructure changes found, priced in dollars and measured for you, and you run Azure Databricks. Your team still makes the changes.
By handcylintNative Databricks toolsClusterBite
Looks atWhatever you have time forYour PySpark codeUsage and spendInfrastructure and running costs, from the system tables
Finds and prices specific changesYes, if someone has the timeFinds code patterns; no pricesShows where money goes, not which change to makeYes: a ranked list in dollars per month
Measures what each change savedRarelyNoTrends and budgets, not per changeYes, each change (how)
CostEngineer timeFreeIncluded30% of the measured saving for three months per change made
LimitsNeeds an owner, or it lapsesCan't see data sizes or anything outside codeLeave the analysis to youAzure Databricks today; doesn't read your repos or notebooks

ClusterBite is the service I built for this. It has read-only access to the Databricks system tables, ranks the changes by dollars a month and measures what each one saved (how it measures). Your team decides what to change. It works with Azure Databricks today, and the pricing page has the fee details.

Get started

Check your own code

I also wrote sparklint, a free, open-source (Apache 2.0) PySpark linter. It knows which objects are Spark DataFrames, so a count() on a Python list or a pandas DataFrame is not reported, and it reads Databricks notebooks as well as plain .py files. It accepts cylint's # cy:ignore comments, so the two can run side by side. It is early (0.x): if it flags code that is fine, please open an issue.

Shell Install it and lint your code

pip install sparklint
# .py files, Databricks notebooks and .ipynb
sparklint check src/ notebooks/

Sources

Checked on 3 November 2026.

About the author

Marthinus Bosman

Marthinus Bosman is a data engineer. He has worked with Azure Databricks for five years, at several companies, in data engineering and MLOps. He built ClusterBite.