TL;DR
Photon is the native C++ vectorized engine built into every Databricks SQL warehouse and serverless cluster. It replaces JVM-based Spark execution at the operator level, processing data in columnar batches using SIMD instructions for 3x to 8x average speedups on SQL and ETL workloads. The catch: Photon clusters cost more per DBU, and Python UDFs, stateful streaming, and some operators fall back to the JVM, meaning you pay the Photon rate for partial acceleration. This blog covers how Photon works, what it accelerates, what it skips, how to enable it correctly across compute types, and how to measure whether it is paying off on your specific workloads before committing production clusters to it.
Photon is on by default across every Databricks SQL warehouse and serverless cluster in 2026. Most data teams enabling it cannot tell you what percentage of their last job ran on Photon versus the JVM fallback path. That gap has a direct cost: Photon clusters cost more per DBU, and the speedup only appears on the operators Photon covers natively. Teams enabling Photon without auditing their workloads routinely pay a premium for partial acceleration.
Databricks built Photon to solve a ceiling that JVM-based Spark execution cannot break through. Garbage collection, object overhead, and row-by-row processing leave CPU SIMD registers and cache-aware memory layouts mostly unused. Photon is written in C++ specifically to use those capabilities. In this article, we cover how Photon works, what it accelerates, what it skips, how to enable it correctly, and how to measure whether it is paying off on your workloads.
Key Takeaways Photon is a native C++ vectorized engine that replaces JVM execution at the operator level while leaving Apache Spark’s Catalyst query planner untouched It processes data in columnar batches using SIMD instructions, delivering 3x to 8x speedups on average across SQL and ETL workloads Python UDFs, stateful streaming, and some ML operators fall back to the JVM, which means you pay the Photon DBU rate for partial acceleration Photon is always-on for SQL warehouses and has been the default serverless runtime since 2025; classic clusters require an explicit checkbox or API flag Sub-two-second queries, Python UDF-heavy pipelines, and stateful streaming workloads see minimal benefit and may cost more on Photon than on standard runtime Reading the SQL warehouse query profile or Spark UI physical plan before committing to Photon on a production workload takes fifteen minutes and prevents weeks of overpaying
Model Your DBU Costs Before Enabling Photon at Scale Kanerika runs cost modeling on representative production workloads so cluster configuration decisions are grounded in real numbers.
See Our Databricks in Practice
What is Databricks Photon? Photon is the native vectorized query engine built into the Databricks Runtime. It replaces JVM-based Spark SQL execution at the operator level, processing data in columnar batches rather than row by row. Apache Spark’s Catalyst optimizer still generates the query plan. Photon takes over at the execution layer for every operator it supports natively.
1. How Photon Fits Inside the Databricks Runtime Catalyst generates the logical and physical query plan as it always has. Photon handles execution for supported operators using SIMD instructions that process thousands of rows per CPU cycle. When Photon hits an unsupported operator, it falls back to the JVM transparently for the duration of that operator, then picks up again on the next supported one. The fallback is invisible in query results but shows clearly in the Spark UI execution details.
2. Why Databricks Wrote a New Engine From Scratch JVM-based execution creates a ceiling on what modern hardware can deliver. Garbage collection, object overhead, and row-by-row processing leave SIMD registers, cache-aware memory layouts, and CPU branch prediction mostly unused. Photon was written in C++ from scratch to use all of these rather than retrofit Spark. The result is an engine that can outperform existing warehouses on SQL workloads while remaining compatible with existing Apache Spark APIs.
3. Photon in 2026: From Preview to Platform Default Photon first appeared as a preview in 2020, reached general availability in 2022, and became the default serverless runtime in 2025. By 2026 it powers every SQL warehouse, every serverless workload, and is enabled by default on classic all-purpose and jobs compute. The question for data teams in 2026 is where Photon accelerates their specific workload and where it falls short, which is where most documentation ends and where actual cost problems begin.
Source: Databricks How Photon Executes Queries Differently 1. Columnar Batch Execution vs Row-by-Row Processing Classic Spark processes one row at a time through a JVM pipeline. Photon loads entire column segments into CPU registers and operates on batches of thousands of rows simultaneously. For large scans, joins, and aggregations, this removes a bottleneck that cluster resizing cannot solve. Databricks customers have observed 3x to 8x speedups on average on production workloads, with some workloads exceeding 12x on the TPC-DS 100TB benchmark.
2. SIMD Instructions and What They Change for SQL Workloads SIMD (Single Instruction, Multiple Data) lets one CPU instruction process multiple values in parallel. Photon exposes this to Spark workloads without requiring any code changes. A query computing an aggregate sum over a large dataset loads entire column segments into CPU registers and applies SIMD-optimized aggregation functions in parallel rather than iterating over individual rows. Queries that previously needed large clusters to hit latency targets can often reach the same threshold on smaller Photon-enabled configurations.
3. The Catalyst Plus Photon Division of Work Catalyst remains the optimizer. It generates both the logical and physical query plan. Photon replaces the JVM engine at the execution stage for every operator it supports. The split is invisible in query results but shows up clearly in the SQL warehouse query profile, where Photon execution time appears as a percentage of total query time, and in the Spark UI physical plan, where Photon operators are explicitly marked.
Photon Coverage: Supported Operators and JVM Fallbacks This is where most Photon documentation stays vague and where teams make expensive configuration mistakes.
1. Workloads With Strong Native Coverage Photon delivers full native coverage across several workload types:
SQL analytics and BI: Default engine for all SQL warehouses, powering dashboards, ad hoc queries, and scheduled reportsETL and data engineering: Batch jobs built with SQL or the DataFrame API, covering scans, joins, aggregations, and writesDelta Lake and Parquet writes:Stateless streaming: Writing to Delta or Parquet sinks from sources including Delta, Parquet, CSV, JSON, Kafka, and Kinesis
For data engineering workloads running large-scale ETL on Databricks , Photon is the right default across all of these patterns.
2. Operators That Fall Back to the JVM Python UDFs are the most impactful fallback case. The moment a Python UDF enters the execution plan, Photon hands that operation back to the JVM for the duration of that operator. Stateful streaming is not supported at all . Some ML operators fall through depending on the specific operation, though GraphFrames and standard feature engineering operations are supported. The Spark UI physical plan shows exactly which operators ran natively and which fell back.
3. The Partial Coverage Problem Photon clusters consume DBUs at a higher rate than standard runtime. A job that is 80% Photon-accelerated and 20% JVM pays the full Photon DBU rate for the entire run and gets vectorized execution on 80% of the work. For Python-UDF-heavy jobs, this math works against you. Enabling Photon on a pipeline with dense UDF usage can produce a higher bill for the same or slower wall-clock performance compared to standard runtime.
Photon Runtime vs Standard Spark Runtime: Dimension Photon Runtime Standard Spark Runtime Execution engine Native C++ vectorized JVM row-based SIMD support Yes No Python UDF support Falls back to JVM Native Stateful streaming Not supported Supported DBU rate Higher Standard Wide table writes Accelerated Standard Default on SQL warehouses Always on No Best for Large SQL, ETL scans, Delta writes UDF-heavy, stateful streaming
Enabling Photon Across Compute Types 1. SQL Warehouses: Always On Photon is the default engine for all SQL warehouses and cannot be disabled. Pro and Serverless tiers are Photon-only. Any query that ran on a DBSQL warehouse had Photon execute whatever portion of the plan it covers natively. There is no configuration decision to make here.
2. Classic and Jobs Compute: Default On, Toggleable Classic all-purpose and jobs compute enable Photon by default in 2026. The UI toggle is the “Use Photon Acceleration” checkbox under Performance when creating or editing a cluster. Via the Clusters API or Jobs API, set runtime_engine: "PHOTON" to enable it explicitly. Photon is supported on most modern instance families including AWS m6g, c6g, r6g, m5, c5, and r5 series; Azure Eav5 and Dav5; and GCP n2 and c3. It is not supported on smaller general-purpose instances such as AWS t3 or Azure D-series.
3. Serverless Compute: Automatic Since 2025 Photon has been the default serverless runtime since 2025. There is no checkbox to configure. Databricks selects a Photon-enabled runtime automatically and the serverless pricing already incorporates the Photon DBU rate. The same partial coverage rules apply: a job with unsupported operators runs those portions on the JVM regardless of the compute type.
4. Lakeflow Pipelines: Explicit Configuration Required For Lakeflow Spark Declarative Pipelines , set photon: true in the pipeline configuration. Classic Lakeflow Spark Declarative Pipelines enable Photon by default. Enabling Photon in the pipeline configuration accelerates execution across all supported operators in the pipeline DAG, which is particularly effective for ETL pipelines doing heavy Delta writes and scans.
# Enable Photon on a cluster via the Clusters API
{
"cluster_name": "photon-etl-cluster",
"spark_version": "15.4.x-scala2.12",
"runtime_engine": "PHOTON",
"node_type_id": "m5.4xlarge",
"num_workers": 4
}Databricks Lakehouse Architecture: What’s Changed in 2026 Explore Databricks Lakehouse Architecture and discover how it simplifies data engineering, analytics, governance, and AI workloads.
Learn More
The DBU Math: Where Photon Earns Its Premium 1. How the DBU Rate Works Photon-enabled instance types consume DBUs at a higher rate than the same instance type running standard runtime. The exact multiplier varies by instance type and cloud provider, published on the Databricks pricing page . The economic premise is that fewer compute hours offset the higher DBU rate per hour. That works when Photon coverage is high and the workload duration is long enough for the speedup to compound.
2. Workloads Where the Premium Pays Off Longer-running queries processing large datasets consistently justify the Photon premium:
Queries that normally take over two seconds Heavy scan and join workloads over large Delta tables Large Delta writes including MERGE INTO operations at scale Concurrent BI dashboard queries on SQL warehouses ETL batch jobs with full SQL and DataFrame API coverage
For data analytics and data integration workloads running these patterns, Photon almost always reduces total cost compared to standard runtime despite the higher DBU rate.
3. Workloads Where the Premium Works Against You Sub-two-second queries see minimal improvement because planning and scheduling overhead dominate execution time at that scale. Python UDF-heavy jobs pay the Photon rate while running large portions on the JVM. Databricks documentation confirms that Photon provides the greatest benefit for longer-running queries processing large datasets, and explicitly notes that queries completing in under two seconds see no meaningful improvement from vectorized execution.
4. How to Measure Before Committing Run identical workloads on Photon and standard clusters using the same instance type. Compare wall-clock time, DBU consumption, and total dollar cost. The SQL warehouse query profile shows Photon execution percentage for DBSQL queries. The Spark UI physical plan shows which operators ran natively for cluster workloads. Running this measurement on a representative sample before configuring production clusters is standard practice in any serious Databricks deployment.
Reading the Query Profile for Real Photon Coverage 1. The DBSQL Query Profile The SQL warehouse query profile shows Photon execution time as a percentage of total query time. A query at 95% Photon coverage is getting near-full vectorized acceleration. A query at 40% means the JVM is handling most of the work and the Photon DBU premium is largely wasted on that workload. The profile also shows per-operator timing, which identifies exactly which operations are pulling coverage down.
2. The Spark UI Physical Plan The Spark UI physical plan marks which operators ran as Photon operators and which ran on the JVM. Operators that fell back show the fallback reason in the plan detail. Reviewing this before enabling Photon on a production cluster takes fifteen minutes and prevents weeks of paying the wrong DBU rate for workloads where standard runtime would deliver lower total cost.
3. Diagnosing and Fixing Low Coverage Low Photon coverage almost always traces back to Python UDFs, unsupported operators, or execution patterns that do not suit columnar batch processing. The most common fix is replacing Python UDFs with Spark SQL equivalents or Databricks built-in functions. Databricks documents a full list of supported and unsupported operations. For workloads where Python UDFs are unavoidable in production, standard runtime is the correct cluster choice.
Choosing Between Photon and Standard Runtime Workload decision framework:
Workload Type Recommendation Reason SQL analytics and BI Always on Default on SQL warehouses, no configuration needed Large ETL with SQL or DataFrame API On Full operator coverage, strong speedup Delta Lake writes at scale On Native Parquet writer, major write improvement Stateless streaming to Delta or Parquet On Supported and accelerated Python UDF-heavy pipelines Evaluate first UDFs fall to JVM, pay Photon rate for partial coverage Stateful streaming Off Not supported Sub-two-second interactive queries No impact Planning overhead dominates execution time ML feature engineering with Spark SQL On Supported for Spark SQL and DataFrame operations Lakeflow pipelines with heavy Delta writes On Explicit photon: true in pipeline config
Four Mistakes Teams Make With Photon 1. Enabling Photon on Python UDF-Heavy Jobs Without Checking Coverage A pipeline built around Python UDFs will have low Photon coverage and high Photon DBU costs. Check the operator mix in the Spark UI physical plan before enabling Photon on any job with heavy UDF usage. Where UDFs can be rewritten as built-in Spark SQL functions, that rewrite recovers Photon coverage and pays back the engineering time within weeks on high-volume workloads.
2. Assuming Serverless Equals Full Photon Acceleration Photon-enabled serverless means the runtime is Photon-capable, not that every operator in every job runs natively. The same partial coverage rules apply: a job with unsupported operators runs those portions on the JVM regardless of compute type. Teams moving workloads to serverless without auditing operator coverage first sometimes see higher bills without understanding why.
3. Using Unsupported Instance Types Photon supports most modern instance families but not smaller general-purpose instances such as AWS t3 and Azure D-series. Enabling Photon on an unsupported instance type results in silent fallback to standard runtime at standard DBU rates, meaning the cluster is misconfigured without a visible error. Check instance type compatibility against the Databricks supported instance list before deploying Photon-enabled clusters.
4. Comparing Photon and Standard Runs on Different Instance Types Photon runs on specific instance families. A Photon versus standard comparison that uses different instance types benchmarks hardware as much as execution engines, producing a misleading result. Use identical instance families across comparison runs to isolate the Photon execution variable from the hardware variable.
How Kanerika Configures Photon in Enterprise Databricks Environments Kanerika is a Databricks Consulting Partner with production implementations across financial services, manufacturing, healthcare, and logistics. Photon configuration is part of every engagement, scoped to the actual workload mix rather than applied as a default.
Our Photon work across enterprise deployments covers three areas where most teams leave performance on the table:
Operator coverage audit: Before recommending Photon on any cluster, we profile the actual operator mix using Spark UI plan analysis. For jobs with high UDF density, we rewrite UDF logic into Spark SQL or built-in functions where viable before enabling Photon. Where UDFs are irreplaceable, we route those workloads to standard runtime clusters.DBU cost modeling: We model projected DBU consumption on Photon versus standard runtime using representative production workloads before any cluster configuration change. This prevents teams from paying the Photon premium on jobs that see minimal acceleration, which is a common and expensive mistake on first Databricks deployments.Workload routing: SQL warehouses and large ETL jobs run on Photon. Stateful streaming and UDF-heavy pipelines route to standard runtime. FLIP , Kanerika’s migration accelerator, configures workload routing as part of the migration output when moving pipelines from legacy platforms like Informatica into Databricks.
Kanerika holds ISO 27001/27701, SOC II Type II, and CMMI Level 3 certifications across 100+ enterprise clients with a 98% retention rate.
One of the largest retail corporations in the United States ran multiple business-critical applications on distributed on-premise databases, PostgreSQL and Cassandra, spread across systems with no centralized governance or lineage. As data volumes grew and analytics demands increased, the architecture created mounting operational risk: fragmented data, high infrastructure overhead, and limited visibility across the enterprise.
Challenge The organization needed to migrate off on-premise infrastructure without taking any application offline. Production application dependencies made a standard cutover approach too risky, and distributed databases created data silos with no consistent lineage or governance across business units.
Solution Kanerika designed and executed a three-phase migration to Databricks using PySpark notebooks and Spark connectors to migrate full historical data from PostgreSQL and Cassandra into Delta Lake tables under Unity Catalog-managed schemas. Continuous incremental synchronization using timestamp-based CDC logic and Delta MERGE operations kept source databases live throughout the transition. Each application was redirected to Databricks workspaces individually after validation, followed by full decommissioning of on-premise infrastructure.
Results Zero production downtime across the full migration 100% of legacy infrastructure decommissioned 100% centralized governance, lineage, and data access through Unity Catalog
Wrapping Up Photon is the execution engine underneath every Databricks SQL warehouse and serverless workload in 2026. The teams getting the most out of it are the ones who audit operator coverage before enabling it, measure the DBU math on representative production workloads, and route Python UDF-heavy jobs to standard runtime where Photon coverage is too low to justify the premium. The configuration is simple. The judgment about where it belongs takes more care, but the fifteen minutes spent reading a query profile or Spark UI plan before a production deployment consistently pays back in lower bills and better performance.
Reduce Databricks Costs With the Right Architecture From cluster design to workload routing, Kanerika handles the architecture decisions so your team focuses on the data work.
Book a Meeting
FAQs 1. What is Databricks Photon? Databricks Photon is the native vectorized query engine built into the Databricks Runtime. Written in C++, it replaces JVM-based Spark SQL execution at the operator level, processing data in columnar batches using SIMD instructions. Apache Spark’s Catalyst optimizer still plans the query. Photon handles execution for supported operators, delivering 3x to 8x average speedups on SQL and ETL workloads.
2. How does Photon differ from Apache Spark? Apache Spark processes data row by row through a JVM-based execution engine. Photon processes data in columnar batches using SIMD CPU instructions, which allows multiple values to be evaluated per CPU cycle. The Spark APIs are unchanged and code requires no modification. Photon replaces only the execution layer beneath those APIs for supported operators.
3. Does Photon require code changes to existing workloads? Photon is designed to be fully compatible with existing Apache Spark DataFrame and SQL APIs. No code changes are required. Enabling Photon on a cluster or enabling it in a pipeline configuration is sufficient for supported workloads to benefit from vectorized execution. Unsupported operations fall back to the JVM automatically without any code-level handling required.
4. Why does Photon sometimes show no performance improvement? Photon provides the greatest benefit for longer-running queries processing large datasets. Queries completing in under two seconds see minimal improvement because planning and scheduling overhead dominates execution time at that scale. Python UDF-heavy jobs pay the Photon DBU rate while running substantial portions on the JVM fallback path. Checking the SQL warehouse query profile or Spark UI physical plan identifies whether Photon coverage is high enough to justify the DBU premium on any given workload.
5. How do I enable Photon on a Databricks cluster? For SQL warehouses, Photon is always on and cannot be disabled. For classic all-purpose and jobs compute, use the “Use Photon Acceleration” checkbox in the cluster UI under Performance, or set runtime_engine: “PHOTON” via the Clusters API or Jobs API. For Lakeflow Spark Declarative Pipelines, set photon: true in the pipeline configuration. Serverless compute has used Photon by default since 2025 with no configuration required.
6. What is the DBU multiplier for Photon-enabled compute? Photon-enabled instance types consume DBUs at a higher rate than the same instance type running standard runtime. The exact multiplier varies by instance type and cloud provider and is published on the Databricks pricing page. The premise is that Photon’s speedup reduces compute hours enough to offset the higher per-DBU rate. That calculation holds for long-running workloads with high Photon operator coverage, and works against you on short queries or UDF-heavy jobs.
7. Do Python UDFs work with Photon? Python UDFs are supported in the sense that they execute without error on Photon-enabled clusters. When a Python UDF appears in the execution plan, Photon hands that operation back to the JVM for the duration of that operator, then resumes native execution on the next supported operator. Pipelines with high UDF density will have low Photon coverage. Replacing Python UDFs with Spark SQL equivalents or Databricks built-in functions restores full Photon acceleration for those operations.
8. How do I check what percentage of a query ran on Photon? For SQL warehouse queries, open the query profile in the Databricks SQL UI. It shows Photon execution time as a percentage of total query time and per-operator timing details. For cluster workloads, open the Spark UI and inspect the physical execution plan. Photon operators are explicitly marked in the plan, and operators that fell back to the JVM show the fallback reason in the plan detail.
