TL;DR
The Databricks API is a set of REST surfaces that give developers programmatic control over clusters, jobs, SQL warehouses, pipelines, model serving, Unity Catalog, and the Supervisor Agent and Lakebase layers added in 2026. OAuth 2.0 is now the recommended authentication method over personal access tokens, and the Python SDK handles auth, retries, and pagination automatically. The Supervisor API, in beta since April 2026, lets teams build multi-tool AI agents in a single call with Unity Catalog governing what those agents can access. Rate limits return HTTP 429 with a Retry-After header and should be handled with exponential backoff. Models available through the Foundation Model API as of mid-2026 include Claude Opus 4.8, GPT-5.5, and Llama 4 Maverick. Getting the most from the API requires clean Unity Catalog governance first.
The gap between knowing Databricks and building on it is wider than most teams expect. The platform ships feature releases every few weeks, authentication defaults have shifted from personal access tokens to OAuth 2.0, and the agent-layer APIs introduced in 2026 have changed how production AI systems connect to data. Engineers working from patterns learned a year ago may be operating on methods that are now deprecated or restructured.
This guide covers the Databricks API as it stands in mid-2026: the architecture, authentication options, core REST endpoints, SDK choices across Python, Java, and Go, and the newer surfaces including the Supervisor API, Foundation Model APIs, and Unity Catalog REST operations. What follows covers the endpoints that matter, the patterns that hold up at enterprise scale, and the mistakes that cause production failures.
This article breaks down how the Databricks API works, what has changed in 2026, and how enterprise teams can build reliable automations and AI agent integrations on top of it.
Key Takeaways The Databricks REST API covers clusters, jobs, SQL warehouses, pipelines, model serving, Unity Catalog, and the Supervisor Agent and Lakebase surfaces added in 2026. OAuth 2.0 is the recommended authentication method for both user and service principal access; personal access tokens remain supported but Databricks flags them as higher security risk. The Python SDK handles authentication, retries, and pagination automatically, making it the right default for most automation work over raw HTTP calls. Rate limits return HTTP 429 with a Retry-After header; exponential backoff with jitter is the standard pattern and is built into the SDK. The Supervisor API (Beta from April 2026) lets teams build multi-tool AI agents in a single API call, with Unity Catalog governing agent access to tools, data, and external MCP servers.
What the Databricks API Actually Covers The Databricks API is not a single endpoint. It is a collection of REST surfaces organized at two levels: account-level and workspace-level. Account-level APIs manage users, workspaces, billing, and account-wide settings, hosted at accounts.cloud.databricks.com and requiring account admin credentials.
Workspace-level APIs manage everything inside a single workspace: clusters, jobs, notebooks, SQL warehouses, Delta pipelines, model serving endpoints, and Unity Catalog objects . Each workspace has its own URL, and API calls must target the correct host.
As of mid-2026, the API surface has expanded significantly. The Supervisor Agent surface, Lakebase database provisioning endpoints, and Unity Catalog write access from external Delta clients are all new additions. Teams should reference the Databricks REST API reference directly rather than relying on documentation from more than a few months ago, since the 2026 release cadence has been rapid.
The Data + AI Summit 2026 announcements added several new API surfaces that changed how agent and database integrations work, and many existing documentation examples now reference deprecated authentication patterns.
The API version in the path matters. The /2.1/jobs endpoint introduced async job runs and better pagination over the older /2.0/jobs surface. When multiple API versions appear in documentation, the current Databricks reference shows the recommended version per endpoint. Teams running complex Databricks workflows should verify which version their SDK is targeting before a major deployment.
API Group Key Endpoints Use Cases Clusters /api/2.0/clusters/create, list, start, deleteProgrammatic cluster lifecycle, CI/CD pipelines Jobs / Lakeflow /api/2.1/jobs/create, run-now, runs/listScheduled job automation, workflow orchestration SQL Warehouses /api/2.0/sql/warehousesBI workload provisioning, cost control Delta Pipelines /api/2.0/pipelinesLakeflow Spark Declarative Pipelines lifecycle Model Serving /api/2.0/serving-endpointsReal-time ML inference, Foundation Model API access Supervisor Agent /api/2.0/agents/supervisors/createMulti-tool AI agent builds in one call Unity Catalog /api/2.1/unity-catalog/tables, schemas, catalogsGovernance, lineage, access control Lakebase /api/2.0/database-instancesServerless Postgres database provisioning SCIM / Identity /api/2.0/accounts/{id}/scim/v2/UsersUser lifecycle management Permissions /api/2.0/permissions/{type}/{id}Fine-grained ACL management
How to Authenticate with the Databricks API 1. OAuth 2.0 as the Default Databricks recommends OAuth 2.0 for all new integrations , both user-facing and automated. Personal access tokens (PATs) remain supported but carry higher security risk: they are long-lived by default, cannot be automatically rotated, and Databricks documentation now flags them as legacy. The recommended PAT configuration limits token lifetime to fewer than 90 days, and Databricks automatically revokes any PAT unused for 90 or more days.
OAuth 2.0 works through two flows in Databricks. User-to-machine (U2M) is for interactive CLI commands where a human authenticates and the token is generated on their behalf. Machine-to-machine (M2M) uses a service principal with a client ID and secret, which is the correct pattern for automated pipelines, CI/CD systems, and applications that run without human interaction.
Databricks security practices increasingly center on service principals with M2M OAuth rather than per-user tokens for production workloads, which limits the blast radius of a compromised credential to the service principal’s scope rather than a human user’s full account.
2. Getting an M2M Token The token request goes to the workspace or account OIDC endpoint. Each token is valid for one hour and the SDK and CLI handle refresh automatically. For environments needing a federated identity approach, such as GitHub Actions or Azure Entra ID, Databricks supports OAuth 2.0 Token Exchange: a federated JWT from the identity provider is exchanged for a Databricks OAuth token at runtime.
This pattern is particularly useful for Databricks deployment pipelines that run from CI/CD infrastructure, since no credentials need to be stored in the application itself or in the CI/CD environment’s secrets store.
3. Unified Authentication Unified authentication is a configuration convention that works across the CLI, Python SDK, Terraform provider, and Java SDK. Setting DATABRICKS_HOST, DATABRICKS_CLIENT_ID, and DATABRICKS_CLIENT_SECRET as environment variables is enough for most tools. These can also be stored in a .databrickscfg profile, making it easy to switch between workspaces without code changes.
One common setup mistake: SCIM account-level operations require an account admin token generated separately from workspace PATs, a distinction that trips up teams when first building programmatic user provisioning.
Databricks Consulting and Implementation Kanerika’s certified Databricks engineers design and implement lakehouse architectures, data pipelines, and AI agent integrations tailored to enterprise requirements and existing infrastructure.
Explore Databricks Services
Which Databricks SDK Should Your Team Use 1. Python SDK The Python SDK (databricks-sdk) is the standard choice for data engineering , ML workflows, and general automation. It wraps all public REST API operations, handles pagination automatically, and implements retry logic for transient errors including 429 rate-limit responses. Authentication is handled through unified auth without additional configuration when environment variables are set. For teams working on Databricks MLflow implementation , the SDK simplifies experiment tracking API calls significantly over raw HTTP.
2. Java and Go SDKs The Java SDK targets enterprise applications and JVM-based systems. The Go SDK is designed for high-performance, cloud-native integrations where low memory overhead matters. Both are maintained by Databricks and cover the same public API surface. The Python SDK documentation is more complete for newer API surfaces like the Supervisor API, where Go and Java SDK support typically follows a few weeks behind the REST and Python releases.
3. When to Use Raw HTTP Raw HTTP calls with curl or an HTTP client make sense for quick testing, shell scripts, or environments where installing a dependency is not practical. They are not the right default for production because they require manual pagination, token refresh, and retry logic. The SDK handles all three and reduces the surface area for bugs in automation code. Teams exploring Databricks serverless options will find the SDK particularly useful since the serverless compute APIs have more configuration options than the UI surfaces.
Core Endpoint Groups and How They Work 1. Clusters API The Clusters API manages the full lifecycle of all-purpose and job clusters. Runtime versions are string identifiers tied to Databricks Runtime releases. As of July 2026, Databricks Runtime 18.2 is in Beta and 18.1 is generally available. Hard-coding a specific runtime version into pipeline code causes failures when that version is retired; parameterized configuration or dynamic version resolution is the correct pattern for long-running Databricks performance optimization work.
Cluster termination is automatic for job clusters when a run completes. All-purpose clusters must be terminated explicitly. The API returns HTTP 200 immediately on start commands, but the cluster state moves through PENDING and RUNNING asynchronously. Production code should poll the get endpoint until state is RUNNING before submitting work, rather than assuming the cluster is available after receiving the 200 response.
2. Jobs and Lakeflow API The Jobs API at /api/2.1/jobs is the operational surface for all scheduled and triggered work. The run-now endpoint triggers an immediate run and returns a run_id; the runs/get endpoint polls for completion. In March 2026, Databricks renamed Databricks Asset Bundles to Declarative Automation Bundles. For teams who previously relied on Databricks Asset Bundles for deployment, the rename carries no functional breaking change, but the CLI commands and documentation paths have been updated to reflect the new naming.
April 2026 added the ability to disable individual tasks in a job without removing them. The disabled flag keeps run history intact and allows re-enabling without rebuilding the job definition. This is useful for temporarily skipping a failing downstream task while a fix is in progress, without losing the task’s full run history. Teams managing Databricks Lakeflow pipelines at scale will find this granular task control reduces the need for full job recreation during incident recovery.
3. SQL Warehouses API SQL warehouses are the compute layer for Databricks SQL. The API at /api/2.0/sql/warehouses manages warehouse creation, resizing, start, and stop. As of April 2026, 5X-Large warehouses (512 workers) are available in public preview for serverless and pro tiers. The Statement Execution API at /api/2.0/sql/statements is the programmatic interface for running SQL against a warehouse and supports both synchronous and asynchronous execution with result polling. Teams comparing Databricks vs Snowflake for SQL-heavy analytical workloads will find the warehouse API differences particularly relevant to their compute cost modeling.
4. Pipelines API The Pipelines API manages Lakeflow Spark Declarative Pipelines. April 2026 extended update history retention in the pipeline API response from 30 to 60 days. Real-time mode for Lakeflow Spark Declarative Pipelines, delivering sub-100ms latency for streaming tables, entered public preview in May 2026 alongside the update_flow API for runtime flow configuration. Teams building near-real-time analytics should evaluate whether the Databricks real-time analytics streaming tables path or the real-time pipeline mode better fits their latency requirements.
Foundation Model APIs and AI Serving 1. Pay-Per-Token Endpoints The Foundation Model API gives developers access to hosted LLMs through standard REST endpoints. The request format follows the OpenAI API convention, which means libraries built for the OpenAI interface work with Databricks-hosted models without modification. Models available on Databricks-hosted infrastructure as of July 2026 include Claude Opus 4.8 (added May 2026), GPT-5.5 and GPT-5.5 Pro (added April 2026), Llama 4 Maverick, Gemini 3.1 Pro, and Qwen3-Next Instruct. Each model carries separate input token per minute (ITPM) and output token per minute (OTPM) rate limits by workspace tier.
The platform implements a credit-back mechanism for OTPM limits: when a request specifies max_tokens, the system reserves that count upfront against the OTPM allowance, then credits back the unused portion after the response is generated. This means actual throughput is typically higher than the raw OTPM limit suggests when requests complete below their max token ceiling. For teams building Databricks generative AI applications, understanding the credit-back mechanism is important for capacity planning under load.
2. Model Serving Endpoints API The Serving Endpoints API at /api/2.0/serving-endpoints deploys custom models, fine-tuned models, and external model proxies for real-time inference. Customer-managed keys for model artifact encryption became generally available in April 2026 for new endpoints. Enterprise AI teams working in regulated environments should configure customer-managed keys before deploying fine-tuned models containing proprietary training data. The Databricks Mosaic AI platform manages the underlying compute and scaling for serving endpoints without requiring infrastructure configuration from the application team.
3. Unity AI Gateway The Unity AI Gateway , generally available in 2026, is the governance layer for all LLM endpoint traffic. It enforces access control, tracks usage, and audits activity across Foundation Model API calls and MCP server interactions. Configuration is through workspace admin settings or the Gateway REST API. Teams that skip gateway configuration during initial deployment often discover they have no audit trail for AI spending or data access patterns when a compliance review arrives.
Limit Type What It Controls Response on Breach Input Tokens Per Minute (ITPM) Prompt tokens processed in 60s window HTTP 429 Output Tokens Per Minute (OTPM) Response tokens generated in 60s window HTTP 429, with credit-back on completion Queries Per Hour (QPH) Total request count in 60-minute window HTTP 429
Building AI Agents with the Databricks Supervisor API 1. What the Supervisor API Does The Supervisor API , introduced in beta in April 2026, lets developers define an AI agent in a single API call. The request specifies the model, the set of tools available, and an instruction set. Databricks handles tool selection, execution, and response synthesis. The agent has access to Unity Catalog-governed tools, custom MCP servers added to the workspace, and managed agent memory backed by Lakebase, so context and session history persist across sessions. Over 100,000 agents have been built on Agent Bricks since launch, processing more than one quadrillion tokens per year as of the Data + AI Summit 2026 announcement.
2. Connecting External MCP Servers April 2026 added support for custom MCP servers as Supervisor Agent sub-agents. Databricks provides managed OAuth flows for select external providers including Google Drive, GitHub, Glean, and SharePoint, eliminating the need to register a separate OAuth application for each integration. The Unity AI Gateway governs and audits all MCP traffic through a single control plane. The Supervisor API can also call Databricks Genie as a sub-agent for natural language data queries, which allows agents to answer business questions by routing through the Genie interface rather than requiring custom SQL generation logic.
3. Contextual Policies for Agent Security A security capability released at the Data + AI Summit 2026 allows contextual policies for agent tools, written in SQL with Python support coming. Policies are stateful and react based on runtime data attributes rather than fixed role-based rules. An agent that accesses a column tagged with PII can be blocked from writing that data to a public-facing table while still being allowed to email it internally. Salesforce updates can be configured to require human approval before the agent proceeds. Teams building agentic workflows on Databricks should review the Databricks security model for agents before moving to production, since the default agent trust boundaries differ from traditional application permission models.
Using the Unity Catalog API for Enterprise Governance 1. Core Catalog Endpoints The Unity Catalog REST API at /api/2.1/unity-catalog covers catalogs, schemas, tables, volumes, and governed tags. Unity Catalog is enabled by default for all Databricks workspaces created after November 2023. The API enforces access control automatically: a token with insufficient grants returns 403 regardless of what the endpoint accepts at the API level, which means governance enforcement is the platform’s responsibility rather than the application layer’s. For teams evaluating Unity Catalog vs Microsoft Purview vs Collibra , the REST API coverage of the Unity Catalog governance layer is a significant differentiator for programmatic governance automation.
2. External Delta Client Access April 2026 added write and create access to Unity Catalog managed and external Delta tables from external Delta clients such as Apache Spark. External Delta tables support full read, write, and create operations. Creating and writing to managed Delta tables is in Beta and requires catalog-managed commits to be enabled. This change is significant for teams using Databricks Delta Sharing and for organizations that run mixed Spark environments and need to write to Databricks-governed tables from outside the platform without moving data through an intermediate storage layer.
3. Data Classification API Databricks Data Classification became generally available in April 2026. The API allows triggering classification jobs that scan Unity Catalog tables, detect sensitive data, and apply system-governed tags in the class.* namespace. Teams in financial services, healthcare, and insurance benefit most from automated classification because it eliminates the manual column-by-column tagging process that becomes a bottleneck as data volumes grow. Data governance programs that previously required manual lineage documentation can significantly reduce that effort when the classification and lineage APIs run on a scheduled basis.
Error Handling and Retry Patterns for the Databricks API 1. Rate Limit Structure Databricks enforces per-endpoint, per-workspace rate limits. The API returns HTTP 429 with a Retry-After header when a limit is hit. The Python SDK raises TooManyRequests for 429 responses and TemporarilyUnavailable for 503s. Both are distinct error types that should be handled separately rather than caught in a generic exception block. Rate limits are not publicly documented for all endpoints; HTTP 429 is the runtime signal, and exponential backoff is the correct response regardless of the specific limit in place.
2. Exponential Backoff Implementation The recommended retry pattern uses exponential backoff with jitter. Starting from a one-second base delay, the wait doubles on each retry and caps at 60 seconds. Jitter (random noise added to the delay) prevents synchronized retry storms when multiple concurrent callers hit the same rate limit simultaneously. The Python SDK’s TooManyRequests exception exposes a retry_after_secs attribute that should be used when present, since it reflects the server’s specific guidance rather than a client-side estimate. Teams running bulk operations against Databricks should consider a data integration pattern that throttles outbound API calls through a token bucket rather than relying purely on reactive backoff.
3. Idempotency and Bulk Operations For bulk job submissions, idempotency keys prevent duplicate side effects if a request times out and is retried. When writing to SQL destinations from Databricks at scale, the repartition() parameter controls connection concurrency and prevents downstream databases from becoming bottlenecks. A common mistake is treating Spark’s default parallelism as appropriate for external API and database targets, which can trigger 429s or connection exhaustion on the receiving end. Databricks troubleshooting work often traces production failures back to this exact pattern.
HTTP Code Error Typical Cause Resolution 400 Bad Request Invalid payload, missing required field Validate JSON against API reference 401 Unauthorized Expired or invalid token Refresh OAuth token or verify PAT 403 Forbidden Insufficient Unity Catalog privileges Grant required privileges for the principal 404 Not Found Resource does not exist or wrong workspace URL Verify IDs and workspace hostname 429 Too Many Requests Rate limit exceeded Implement exponential backoff with Retry-After header 500 Internal Server Error Platform-side issue Retry with backoff; escalate to support if persistent
How Enterprise Teams Build on the Databricks API 1. CI/CD with Declarative Automation Bundles Declarative Automation Bundles define jobs, clusters, and permissions as YAML configuration files stored in Git. The CLI deploys bundles to target environments. This pattern separates configuration from code and enables code review for pipeline changes. For teams managing Databricks dbt integrations, bundle-based deployment provides a unified way to manage both dbt project files and the Databricks jobs that run them through a single version-controlled pipeline. Teams using bundles for deployment eliminate the class of production failures that come from manually configuring jobs through the UI, where there is no audit trail or rollback mechanism.
2. Infrastructure as Code with Terraform The Databricks Terraform Provider exposes the full workspace API surface as Terraform resources. Teams managing multiple workspaces use Terraform to enforce consistent cluster policies, permission boundaries, and job configurations across environments. The provider also handles service principal provisioning and Unity Catalog privilege grants, which makes it possible to manage the full governance lifecycle through code rather than the workspace UI. For organizations evaluating a migration from legacy systems to Databricks , Terraform provides a way to codify the target architecture before any data movement begins.
3. Event-Driven Architectures External orchestrators like Apache Airflow use the Jobs API to trigger Databricks runs based on upstream events. The run-now endpoint returns a run_id immediately; the orchestrator polls runs/get for completion status and propagates failures upstream. This decoupling keeps scheduling logic out of the Databricks workspace. Teams using Databricks Workflows for internal orchestration can combine both approaches: use Databricks Workflows for intra-platform dependencies and the Jobs API via Airflow for cross-system coordination.
Databricks API at Scale: How Kanerika Builds on It Kanerika is a Microsoft Solutions Partner for Data and AI with Analytics Specialization, a Databricks Consulting Partner, a Snowflake Select Tier Partner, and holds ISO 27001, ISO 27701, SOC II Type II, and CMMI Level 3 certifications. With 300-plus professionals serving 100-plus enterprise clients, the firm delivers data engineering , AI implementation , and migration services on Databricks and adjacent platforms.
Kanerika’s engineering teams work with the Databricks API across the full range of enterprise use cases: cluster lifecycle automation in CI/CD pipelines, programmatic Unity Catalog governance at multi-workspace scale, Foundation Model API integration for AI agent builds, and Lakeflow job orchestration for production ETL. FLIP, Kanerika’s proprietary DataOps platform, automates pipeline migration into Databricks environments and covers 12 source-to-target paths, cutting migration effort by 50 to 60% and reducing annual licensing costs by 75% compared to legacy tools. Informatica to Databricks migration and Hadoop to Databricks migration are two paths where FLIP’s automation substantially reduces the manual work of rewriting pipeline logic for the new platform.
Client engagements at Kanerika consistently demonstrate that teams adopting programmatic API-driven governance through Unity Catalog reduce access control errors that reach production and cut compliance audit preparation time compared to manual workspace configuration. Kanerika’s Databricks practice covers full-lifecycle implementation, migration acceleration with FLIP, and AI agent development on the platform.
Building or Modernizing a Databricks Environment? Kanerika’s certified Databricks engineers help enterprise teams design API-driven automation, Unity Catalog governance, and AI agent architectures that scale. Talk to the team to scope your engagement.
Schedule a Meeting →
Case Study: Modernizing Retail Analytics Infrastructure With Databricks The client is one of the largest retail corporations in the United States, operating supermarkets and multi-department stores at national scale. With thousands of store locations and a complex web of business applications dependent on operational data, the organization required a highly reliable, reliable data platform to support enterprise-wide analytics, reporting, and day-to-day decision-making.
Client’s Challenges Distributed on-premise databases created data silos with no centralized lineage, governance, or consistent visibility across business units and downstream applications High infrastructure and maintenance overhead consumed IT capacity in hardware upkeep, backup management, and scaling work that limited focus on analytics and business growth Production application dependencies made a standard cutover approach too risky. So, migration had to run with zero downtime and full parallel availability until each system was verified
Our Solutions Designed a three-phase migration framework using PySpark notebooks and Spark connectors to migrate full historical data from PostgreSQL and Cassandra into Delta Lake tables under Unity Catalog-managed schemas Implemented continuous incremental synchronization using timestamp-based CDC-style logic and Delta MERGE operations, keeping source databases live and current throughout the transition Executed a controlled application cutover phase with parallel system availability, redirecting each application to Databricks workspaces individually after validation, followed by full decommissioning of on-premise infrastructure
Results Zero Downtime, no production interruption 100% Legacy Infrastructure Decommissioned 100% Centralized governance, lineage, and data access
Wrapping Up The Databricks API in 2026 covers far more than cluster management and job scheduling. OAuth 2.0 unified authentication, the Supervisor API for agent builds, Foundation Model API access to hosted LLMs, and Unity Catalog REST operations for programmatic governance have made it a full enterprise integration surface. Teams that treat the API as infrastructure code rather than operational tooling get more reliable pipelines, faster deployments, and auditable governance at scale. Whether a team is migrating legacy ETL into Databricks or building AI agents on top of a governed lakehouse, the patterns here provide a foundation for production-grade work.
FAQs What Is the Databricks API and What Does It Cover? The Databricks API is a set of REST endpoints that provide programmatic control over the Databricks platform. It covers cluster lifecycle, job scheduling, SQL warehouse management, Delta pipelines, model serving, Unity Catalog governance, the Supervisor Agent for AI agent builds, and Lakebase database provisioning. Operations span both account-level administration and workspace-level execution, with all surfaces accessible through the CLI, SDKs, or direct HTTP calls. The 2.1 Jobs endpoint and the Supervisor API are two of the most significant surfaces added in recent releases and represent the direction the platform is moving toward agent-first automation.
What Is the Difference Between the Databricks REST API and the SDK? The REST API is the underlying HTTP interface for all Databricks operations. The SDK wraps those calls in typed language-level functions and adds automatic authentication, pagination, retry logic, and error handling. For most production use cases the Python SDK reduces code complexity significantly over raw HTTP. The SDK is maintained by Databricks and covers the complete public API surface, though newer surfaces like the Supervisor API sometimes appear in the REST reference a few weeks before full SDK parity arrives. Raw REST access is appropriate for testing, shell scripts, and environments where installing a Python dependency is not practical.
How Does Authentication Work with the Databricks API in 2026? Databricks recommends OAuth 2.0 for all API access. For human-interactive tools, user-to-machine OAuth generates tokens after browser-based login. For automated pipelines, machine-to-machine OAuth uses a service principal with a client ID and secret to request short-lived tokens from the OIDC endpoint. Personal access tokens remain supported but are considered legacy due to higher security risk and lack of automatic rotation. The unified authentication convention in the SDK handles token refresh across all supported tools without requiring manual credential management in application code. Federated token exchange via OAuth 2.0 Token Exchange is the preferred pattern for GitHub Actions and cloud-native CI/CD pipelines.
What Is the Supervisor API and When Should Teams Use It? The Supervisor API, in beta since April 2026, lets teams build multi-tool AI agents in a single API call. A developer specifies the model, the tools available (Unity Catalog-governed data tools, custom MCP servers, web search), and an instruction set, and Databricks handles tool selection, execution, and response assembly. It is the right choice for production AI agents that need governed access to enterprise data without custom orchestration code. Teams managing compliance-sensitive data benefit most from the contextual policy controls that restrict agent behavior based on runtime data attributes rather than static role assignments. Agents built on the Supervisor API can persist memory across sessions through Lakebase, which removes the need for custom session state management.
How Should Teams Handle Rate Limiting in the Databricks API? Databricks rate limits return HTTP 429 with a Retry-After header. The recommended pattern is exponential backoff with jitter: start with a one-second base delay, double it on each retry, cap at 60 seconds, and add random noise to prevent synchronized retry storms. The Python SDK raises TooManyRequests for 429 responses with a retry_after_secs attribute that reflects the server’s specific guidance. For bulk operations, a token-bucket rate limiter on the client side prevents hitting the limit in the first place and produces more predictable throughput than reactive backoff alone, since it spaces requests out proactively rather than waiting for failures to occur.
What Is Unity Catalog and How Does the API Interact with It? Unity Catalog is the account-wide governance layer built into Databricks, managing data access control, lineage, audit logging, and data classification across all workspaces. The Unity Catalog REST API allows programmatic management of catalogs, schemas, tables, volumes, and governed tags. All workspace APIs enforce Unity Catalog permissions automatically: a token with insufficient grants returns 403 regardless of the endpoint. This makes Unity Catalog the single governance enforcement point, so application code does not need to implement its own access checks on top of API calls. The Data Classification API, generally available since April 2026, extends programmatic governance to sensitive data identification.
Can the Databricks API Connect to External Systems? The Supervisor API supports custom MCP servers as agent tools, with Databricks providing managed OAuth flows for external providers including Google Drive, GitHub, Glean, and SharePoint. External orchestrators like Apache Airflow connect to the Jobs API to trigger runs and poll for completion. The Lakebase API exposes a serverless Postgres-compatible interface for applications that need transactional read/write alongside lakehouse analytics. JDBC and ODBC connectors let BI tools and legacy applications query Databricks SQL warehouses without requiring API code changes on the client side. The Unity REST API added external Delta client write access in April 2026, enabling direct writes to Unity Catalog tables from external Spark environments.
What Databricks API Changes Are Most Important for 2026? Several 2026 changes affect production integrations directly. OAuth 2.0 is now the recommended default over personal access tokens for all new integrations. The Supervisor API and Foundation Model APIs with support for Claude Opus 4.8, GPT-5.5, and Llama 4 Maverick are stable options for AI agent builds. Databricks Asset Bundles was renamed to Declarative Automation Bundles in March 2026. Unity Catalog write access from external Delta clients entered beta in April 2026. Real-time mode for Lakeflow Spark Declarative Pipelines, delivering sub-100ms latency, entered public preview in May 2026. Teams running older API patterns should review the Databricks migration guides for each of these changes before their next major deployment cycle to avoid unexpected breaking changes.
