TL;DR
Databricks AutoML automates data preparation, algorithm selection, and hyperparameter tuning to produce baseline classification, regression, and forecasting models with fully editable notebooks. It is no longer included as a built-in library starting Databricks Runtime 18.0 ML, so teams need to check their runtime version and compute access mode before building new workloads on it.
Key Takeaways Databricks AutoML automates data preparation, feature engineering, algorithm selection, and hyperparameter tuning for classification, regression, and forecasting problems. Every AutoML trial generates a fully editable Python notebook, the glass box design that lets a data scientist inspect and modify the exact code behind each model. AutoML is no longer bundled as a built-in library starting Databricks Runtime 18.0 ML, and it requires Dedicated or No Isolation Shared compute access mode, not Standard. The right evaluation metric matters more than the leaderboard score, since the top-ranked trial can still be the wrong choice for an imbalanced dataset or a cost-sensitive decision. Turning an AutoML baseline into a production model requires MLflow tracking, Unity Catalog registration, and monitoring that AutoML itself does not provide. Kanerika, a Databricks Consulting Partner, helps enterprise teams turn AutoML baselines into governed, production-ready ML pipelines instead of one-off notebooks. Watch on YouTube
What Do Databricks 2026 Updates Mean for Enterprise AI?
Kanerika breaks down what changed across Databricks’ 2026 platform updates, including LTAP, Genie ZeroOps, and AI governance, and what it means for enterprise machine learning work.
When an Old AutoML Notebook Suddenly Stops Working A data scientist reruns a notebook that has worked for two years without incident. This time, import databricks.automl throws an error instead of returning a summary object. The team just upgraded to Databricks Runtime 18.0 ML. Nothing in the error message explains why a familiar library suddenly disappeared.
That scenario is playing out across Databricks workspaces in 2026, because AutoML quietly moved from a built-in library to an optional package. This guide covers what Databricks AutoML actually automates, how classification, regression, and forecasting experiments work under the hood, and exactly what changed with Runtime 18.0 ML so a team is not caught mid-migration.
What Is Databricks AutoML? Databricks AutoML is a tool inside the Databricks Machine Learning workspace that automates the repetitive parts of building a model. Point it at a dataset and a target column. It profiles the data, trains multiple algorithms, and tunes their hyperparameters, then ranks the results on a leaderboard, as described on Databricks’ own AutoML product page .
It covers three problem types: classification, regression, and forecasting. Rather than treating model training as a black box, Databricks calls this a glass box approach, because every trial produces a complete, editable Python notebook. A data scientist can open, reproduce, and modify that notebook with domain knowledge the algorithm cannot see on its own, a design explained further in Databricks’ AutoML documentation .
AutoML sits inside a larger Databricks machine learning stack. That stack also includes Databricks Mosaic AI for generative and agentic workloads, Databricks Model Serving for real-time inference, and Unity Catalog for governance across every asset a model touches. AutoML is the fastest on-ramp to a working model. It is not a replacement for problem framing, data quality work, or the deployment controls covered later in this guide.
Case Study
64% Less Wastage With AI and ML Implementation in Healthcare
A healthcare workforce platform used AI and ML with Kanerika for document verification, cutting wastage by 64% and its operations team from 500 to 320 staff.
Read the Case Study → Is Databricks AutoML Still Available in Runtime 18.0 ML? This is the question that current documentation answers inconsistently, and it deserves a direct answer. Databricks’ own release notes state plainly that Databricks Runtime 18.0 ML has removed AutoML from the default machine learning runtime image . A workspace running that version or later does not have databricks.automl preinstalled the way it was on every ML runtime before it.
Removed as a built-in library is not the same as removed as a capability. The databricks-automl-runtime package that AutoML depends on is still published and installable on PyPI . Existing clusters pinned to earlier runtimes can generally continue running AutoML experiments. So can clusters where a team installs the package manually.
That gap between what changed and what current pages still show is exactly what confused users on the Databricks community forum . A thread there on AutoML deprecation drew replies noting that the documentation had added warning notes but that the removal itself lacked clear migration guidance. Older tutorials, cached blog posts, and even some current product pages still show AutoML screenshots without a version qualifier.
Four Facts That Settle the AutoML Confusion A few facts settle the confusion:
AutoML as a built-in library is gone from Databricks Runtime 18.0 ML and above, confirmed directly in Databricks’ release notes. The underlying databricks-automl-runtime package still exists on PyPI and can be installed manually on supported runtimes. AutoML has never supported the Standard (shared) compute access mode. It requires Dedicated (formerly single user) or No Isolation Shared clusters, a constraint that predates the Runtime 18.0 ML change and still applies. Databricks’ broader machine learning investment in 2026 has shifted toward Agent Bricks for building governed AI agents, a different product built for a different job than tabular classification, regression, and forecasting. Before starting any new AutoML work, confirm the workspace’s Databricks Runtime version, cloud provider, and region. Also confirm the compute access mode assigned to the target cluster. A five-minute check avoids the exact failure mode that opened this article.
Compatibility note last checked against Databricks’ official release notes and Microsoft Learn’s Azure Databricks documentation, both current as of 2026.
How Databricks AutoML Works From Dataset to Best Trial Every AutoML experiment follows the same sequence, whether it runs through the low-code UI or the Python API.
Infer the problem and validate the target. AutoML checks the target column and confirms the problem type matches what was requested.Profile the data. It detects semantic types such as numeric, categorical, text, and datetime columns, beyond what raw data types alone would suggest.Prepare features. Missing values get imputed, categories get encoded, and features get scaled using built-in best practices.Split the data. AutoML creates training, validation, and test sets, and it supports a custom or time-based split when random splitting would leak information.Train and tune. Multiple algorithms run in parallel across the cluster, each with its own hyperparameter search.Apply early stopping. On Databricks Runtime 10.4 LTS ML and above, unpromising trials stop early instead of consuming the full time budget.Rank the trials. Results are ranked by the primary metric chosen for the experiment, whether that is F1 score, R-squared, or SMAPE.Generate notebooks and log to MLflow. Every trial produces a source-code notebook, and every run’s parameters, metrics, and artifacts are tracked in MLflow automatically.The distributed part matters at enterprise scale. Hyperparameter search runs across worker nodes rather than a single machine. That is why AutoML experiments on large Databricks clusters finish faster than an equivalent grid search run locally.
Choosing the Right Problem Type for Your Machine Learning Task AutoML only helps if the problem type matches the business question. Forecasting demand is not a regression problem in disguise. Treating it as one throws away the time-ordering information that makes a forecast useful.
Problem Type Business Question Example Algorithms Used Default Primary Metric Classification Will this customer churn this quarter? Decision trees, random forest, logistic regression, XGBoost, LightGBM F1 score Regression What will this shipment cost? Decision trees, random forest, linear regression with SGD, XGBoost, LightGBM R-squared Forecasting What will demand look like next month? Prophet, Auto-ARIMA, DeepAR on serverless compute SMAPE
Table 1: Databricks AutoML Problem Types and Algorithms. Each problem type draws from open source libraries including scikit-learn, XGBoost, LightGBM, Prophet, and Auto-ARIMA, evaluated against a metric appropriate to that problem type by default.
A workload that does not fit any of these three categories, such as image classification, ranking, or reinforcement learning , needs custom training code. Databricks deliberately scopes AutoML to tabular and time series problems, not a general-purpose model builder.
Forecasting has its own nuances worth flagging before treating it as a shortcut version of regression. For multiple related time series, such as sales by store or demand by region, the identity_col parameter tells AutoML which columns identify each individual series. It then trains a separate model per series rather than blending them into one. A team can add external factors like price changes or promotions through exogenous_cols for ARIMA-based models. Or it can join a covariate feature table via Feature Store for broader model coverage, both documented in Databricks’ forecasting covariates guide .
How Databricks AutoML Compares to Other AutoML Platforms Teams evaluating Databricks AutoML often compare it against Azure Machine Learning AutoML, DataRobot, and H2O.ai. The right fit usually depends on where the data and compute already sit, not on a feature checklist alone.
Platform Best Fit Governance Model Notable Constraint Databricks AutoML Teams whose data and Spark pipelines already live in Databricks MLflow plus Unity Catalog Removed as a built-in library on Runtime 18.0 ML and above Azure Machine Learning AutoML Azure-native teams that need Purview and Active Directory governance built in Azure ML plus Microsoft Purview A separate Azure service from Azure Databricks, with its own setup DataRobot Enterprise teams that want a polished interface and vendor support DataRobot’s own governance and explainability layer Licensing cost scales with usage H2O.ai Teams that want open source transparency and cost control Self-managed, open source Needs more in-house MLOps maturity to operate at scale
Table 2: Databricks AutoML vs Other AutoML Platforms. A platform choice driven by where governed data already lives tends to hold up better over time than one driven by a feature comparison alone.
Databricks AutoML Requirements and Pre-Run Checks A few requirements catch teams off guard, mostly because they are easy to miss until an experiment fails to start.
The cluster needs a supported compute access mode. Dedicated (formerly single user) and No Isolation Shared clusters support AutoML. Standard (formerly shared) clusters do not, regardless of runtime version. AutoML also needs network ports 1017 and 1021 open to access workspace files, a detail worth confirming with a cloud administrator before an experiment silently stalls.
AutoML accepts a Spark DataFrame, a pandas DataFrame, a pandas-on-Spark DataFrame, or a Unity Catalog table name as input. Whichever format a team uses, a short list of data quality checks before starting saves far more time than they cost:
Remove target leakage columns, meaning any column that would not be known at prediction time. Check cardinality on categorical columns, since extremely high-cardinality fields can distort feature encoding. Review missingness, outliers, and duplicate rows before assuming the leaderboard score reflects real performance. Confirm labels are valid and consistently formatted, particularly for classification targets. AutoML automatically samples very large datasets before training. That keeps experiments fast, but it also means the trained model reflects a sample, not the full table, which matters when a business decision hinges on a rare event or a small subgroup.
Joining Feature Store Tables Into an AutoML Run A detail many introductions to AutoML skip entirely: classification and regression experiments on Databricks Runtime 11.3 LTS ML and above accept a feature_store_lookups parameter. That parameter joins governed feature tables into the training set automatically. Each entry names a table_name and a lookup_key, and AutoML handles the join before training starts, documented in Databricks’ Feature Store integration guide .
That matters for any team with an existing data governance practice around Unity Catalog. It means an AutoML baseline can train against the same governed, versioned features a production model will eventually use. That beats a one-off flat file extract that drifts out of sync the moment the source data changes.
Using Databricks AutoML Through the UI and Python API Databricks AutoML supports two paths to the same result. A low-code UI handles a quick first pass, while a Python API covers anything that needs to run as part of a repeatable pipeline.
The UI Walkthrough From the Machine Learning workspace, creating an experiment means selecting a Dedicated or No Isolation Shared cluster, choosing the source dataset, and picking a prediction target and problem type. AutoML then asks for a primary evaluation metric and an experiment timeout, and it lets a team exclude specific frameworks that do not fit the use case.
A team can leave training and validation splits to AutoML’s defaults or set them manually, including a time-based split for problems where order matters. Once the experiment starts, a trial leaderboard fills in live. A data scientist can open the best trial’s notebook directly from the results page to inspect the exact code behind the top model.
A Python API Example The Python API exposes three entry points that map directly to the three problem types. They are databricks.automl.classify(), regress(), and forecast(). A minimal classification call looks like this:
import databricks.automl as automl
summary = automl.classify(
dataset=train_df,
target_col="churned",
primary_metric="f1",
timeout_minutes=30,
)
best_trial = summary.best_trial
print(best_trial.metrics)
print(best_trial.notebook_id)The timeout_minutes parameter controls how long the search runs. Databricks deprecated the older max_trials parameter that capped the number of trials directly, so time-based control is now the supported approach.
The returned AutoMLSummary object exposes best_trial, along with every trial’s metrics, parameters, and generated notebook. All of that is also logged to MLflow automatically. From there, a data scientist recalculates the metrics that matter on a held-out test set. Next, they pick a decision threshold based on the real cost of a false positive versus a false negative. Finally, they register the approved model in Unity Catalog with a description, an owner, and validation evidence attached.
Kanerika Service
Hire Databricks Developers
Bring in Kanerika’s Databricks-certified engineers to run AutoML experiments, build the Python API into a pipeline, and take a baseline model to production.
Hire Databricks Developers How to Improve Databricks AutoML Model Accuracy The trial at the top of the leaderboard is not automatically the right model to ship. Reading the score correctly matters as much as the search itself.
Start with the metric, not the score. Accuracy alone hides failure on an imbalanced dataset. A model that always predicts the majority class can still post a high accuracy number while missing every case that actually matters.
Problem Type Metrics to Check Classification F1 score, precision, recall, ROC-AUC, PR-AUC, log loss Regression R-squared, RMSE, MAE, MSE Forecasting SMAPE, MAPE, MDAPE, MAE, RMSE
Table 3: Metrics to Check by Problem Type. Precision and recall matter more than raw accuracy whenever the classes are imbalanced or a specific error type carries more business cost than another.
A few checks catch most accuracy problems before a model reaches production:
Compare validation and test performance. A large gap between the two signals overfitting to the validation set. Use time-based splits whenever the data has a natural order, so future information cannot leak into training. Review what AutoML sampled on large datasets before trusting a result that looks unusually strong. Tune the classification threshold and check probability calibration rather than accepting the default 0.5 cutoff. Compare the winning trial against a simple baseline, such as predicting the historical average, and against whatever model is already running in production. Evaluate performance across segments such as region, product line, or customer group, since an aggregate score can hide weak performance on a segment that matters most to the business. Improving features, labels, and the observation window usually moves accuracy further than extending the search timeout does. AutoML can only tune what the data supports.
From Generated Notebook to Production-Ready Model Every AutoML trial notebook contains the same building blocks. These include data preparation code, the algorithm and hyperparameter configuration for that trial, training and evaluation logic, MLflow logging calls, and optional Shapley value calculations for feature importance using the SHAP package.
AutoML disables SHAP calculations by default because they are memory intensive, and it skips them automatically on datasets with a datetime column on older runtime versions. Enabling them means setting shap_enabled = True in the feature importance section of the generated notebook and rerunning it.
Turning a trial notebook into something a production system depends on takes real engineering work, not a copy-paste. Reasonable checks before a generated notebook enters CI/CD include reviewing pinned library versions and removing exploratory code that has no business running unattended. Another key check: confirming the notebook’s logic still matches the approved model rather than a discarded earlier trial.
From there, the workflow runs through two systems every Databricks ML team should already know. MLflow tracks parameters, metrics, models, and environment files for every trial, which makes comparing candidates by performance, latency, size, and stability straightforward. A team registers the approved model in Unity Catalog under a three-level name, with ownership, permissions, tags, and a description attached, replacing the legacy workspace model registry entirely.
Promoting a model from there uses aliases and a controlled deployment job, whether the destination is batch scoring or real-time Databricks Model Serving . Capturing the input schema, model signature, and dependency list at this stage prevents the most common production incident. That incident is a model that behaves differently in serving than it did in the notebook that trained it.
What Databricks AutoML Does Not Do AutoML automates model search. It does not automate the governance and monitoring work that keeps a model safe to run in production. Treating a leaderboard win as production-ready is where most AutoML-related incidents start.
Several controls sit entirely outside what AutoML provides:
Data validation before training and before every inference call, catching schema drift before it reaches a model. Bias, fairness, and explainability review for any model whose output affects a person’s access to credit, employment, insurance, or similar outcomes. Feature and prediction drift monitoring once a model is live, not just at training time. Business KPI monitoring alongside statistical model metrics, since a model can stay statistically accurate while the business outcome it drives quietly degrades. Retraining triggers, approval rules, and a tested rollback path for when a new model version underperforms the one it replaced. Audit logs, lineage records, and access reviews tied to the model, not just the underlying data. None of this is a knock against AutoML. Databricks built it to accelerate the search for a working model, and it does that well. The governance layer around it is a separate, deliberate build, which is exactly where AI governance practices and platform ownership decisions belong.
The gap shows up most often on forecasting and regression models feeding a business process rather than a dashboard. That includes predictive analytics outputs that drive an inventory order, a staffing decision, or a pricing change. A model that quietly drifts on one of those workloads costs real money before anyone notices the leaderboard score from six months ago no longer reflects reality.
Databricks AutoML Cost and Performance Controls AutoML cost scales with compute type, worker count, data size, the algorithms included, and how long the search runs, and nothing caps any of that automatically.
A few habits keep AutoML experiments from becoming a silent line item on the Databricks bill:
Run a short baseline experiment before committing to a long search, to see whether extra time actually moves the metric. Use timeout_minutes to bound the search, since the older max_trials parameter is deprecated and no longer the supported control. Exclude frameworks that clearly do not fit the use case rather than letting AutoML evaluate every available algorithm by default. Track the metric gained per additional compute hour, and stop the search once that curve flattens out. Factor inference latency and serving cost into model selection, not only training-time accuracy. Cluster sizing has an outsized effect that is easy to overlook. A larger worker pool speeds up the parallel search, but AutoML bills by cluster uptime regardless of whether every trial is contributing useful information. An oversized cluster left running past the point of diminishing returns is the single most common source of AutoML cost overruns.
Recording experiment cost next to accuracy and business impact turns AutoML spend from a surprise into a managed line item. That mirrors the way a mature MLOps practice already tracks deployment metrics.
Datasheet
Databricks Funding Support Through Kanerika
See how Kanerika helps qualifying enterprise teams access Databricks funding support to offset the cost of a governed AutoML and machine learning rollout.
View the Datasheet → Common Databricks AutoML Errors and How to Fix Them Most AutoML failures fall into a short list of recurring patterns.
Symptom Likely Cause Fix databricks.automl import failsRunning on Databricks Runtime 18.0 ML or later without the package installed Install databricks-automl-runtime from PyPI, or confirm whether an earlier runtime is required AutoML option missing from the UI Cluster is on Standard (shared) compute access mode Switch to a Dedicated or No Isolation Shared cluster Experiment stops with no useful trial Timeout set too low, or data volume too small for a stable split Increase timeout_minutes and confirm the dataset has enough rows for training, validation, and test splits Driver memory errors on a large dataset AutoML’s automatic sampling did not reduce the working set enough for the driver’s memory Use a larger driver node or pre-aggregate the dataset before starting the experiment Generated notebook fails on rerun Underlying library versions changed since the notebook was generated Pin library versions in the notebook, or regenerate the trial on the current runtime Strong leaderboard score, weak production results Data leakage, an unrepresentative sample, or metric mismatch with the real business question Re-check target leakage, review what was sampled, and compare against a baseline and the current production model
Table 4: Common Databricks AutoML Failures and Fixes. Most of these trace back to a runtime or compute access mode mismatch, which is worth ruling out first before investigating the data itself.
Checklist
Enterprise AI Checklist: Readiness, Governance, Adoption
Use Kanerika’s Enterprise AI Checklist alongside the list below to plan the readiness and governance work around any AutoML-built model.
Get the Checklist → Keep, Replace, or Rebuild After the Runtime 18.0 ML Change Teams with existing AutoML workloads have three real options once a workspace approaches Databricks Runtime 18.0 ML. The right one depends on risk tolerance and how central the workload is.
Keep temporarily. If the current runtime is still supported and an exit plan exists, staying put buys time without immediate risk. This is not a permanent answer, since every unsupported runtime eventually loses security patching.
Replace the search layer. Teams that want a supported AutoML product with similar controls can install databricks-automl-runtime manually on a compatible runtime. Or they can evaluate Azure Machine Learning AutoML where Azure-native governance is a firm requirement.
Rebuild from the generated notebooks. Where transparency, stability, and long-term customization matter most, the generated trial notebooks are a legitimate starting point for hand-built training code. That code typically uses MLflow alongside scikit-learn, XGBoost, LightGBM, Optuna, or Ray Tune for hyperparameter search, the same toolset Kanerika’s applied machine learning engagements use once a workload graduates past a baseline.
Before any of these paths, inventory every notebook, workflow, job, cluster policy, and registered model that imports databricks.automl. Test the highest-risk workloads on the target runtime in an isolated environment first, and set a rollback point before changing the runtime on anything running in production. Runtime pinning without a tested exit plan is how a two-year-old notebook becomes an emergency instead of a planned migration.
Databricks AutoML: How Kanerika Builds Governed ML Pipelines Kanerika is a Databricks Consulting Partner , and AutoML shows up constantly in early-stage machine learning engagements because it gets a client’s team to a working baseline fast. The harder, more valuable work starts after that baseline exists.
Kanerika’s approach to AutoML engagements follows a consistent path. First, the team assesses the client’s Databricks runtime, workspace configuration, and compute access mode before touching a single notebook. Next, it establishes an AutoML baseline against a governed Unity Catalog feature table rather than an ad hoc extract. From there, it wraps the winning trial in MLflow tracking and a controlled promotion path. Finally, it validates the model against business metrics, not just the leaderboard score, before the model goes anywhere near production traffic.
In one representative engagement (anonymized example), a manufacturing client’s data science team had built several AutoML classification baselines directly in personal notebooks. The team had no shared experiment tracking and no consistent way to promote a winning trial into production. Kanerika rebuilt the workflow around a governed feature table, standardized the AutoML baseline step, and added MLflow-tracked evaluation and Unity Catalog registration before any model reached a serving endpoint. The generated trial code looked ready to ship on the leaderboard, but it needed real refactoring first. The main issues were pinned dependencies and a leakage column the original extract had left in by accident.
Results From a Fashion Retail Forecasting Engagement That kind of gap between a fast AutoML baseline and a production-safe model shows up across Kanerika’s broader forecasting and applied ML work. In one fashion retail engagement, Kanerika built AI-driven demand forecasting for seasonal and capsule collections , which cut inventory carrying costs by 37 percent. The forecast gave planners something they could actually act on, instead of a model that looked accurate in a notebook and fell apart against real seasonal demand.
Enterprise teams considering Databricks for machine learning can use Kanerika’s AI Maturity Assessment to see where their current setup stands against a governed, production-ready target state. This applies whether they are starting with AutoML or already have workloads running on it. Or they can talk with Kanerika’s team directly about a specific Databricks ML environment.
Case Study
Zero-Downtime Databricks Migration and Analytics Modernization
A national retail corporation moved off on-premise Postgres and Cassandra to Databricks with Kanerika, achieving zero downtime and retiring all legacy infrastructure.
Read the Case Study → A Production-Ready Databricks AutoML Checklist Confirm the current Databricks Runtime version and AutoML’s support status on it. Validate dataset quality, target definition, and leakage controls before starting the experiment. Choose a primary metric that matches the real business question, not the default. Preserve a true held-out test set the model never saw during training or validation. Review sampling, class balance, and time ordering in the underlying data. Inspect the generated notebook’s code and dependency versions before trusting it. Compare the winning trial against a simple baseline and the current production model. Test accuracy across the data segments that matter most to the business, not just in aggregate. Register only approved models in Unity Catalog, with ownership and documentation attached. Set monitoring, retraining, and rollback rules, and document the Runtime 18.0 ML response plan for this workload specifically. Getting Databricks AutoML Right Before the Next Runtime Upgrade Databricks AutoML remains one of the fastest ways to get a working classification, regression, or forecasting model out of raw data. The glass box notebooks it generates are a genuine advantage over opaque AutoML tools. What changed with Runtime 18.0 ML is not AutoML’s usefulness, but its default availability. That distinction is easy to miss until an old notebook stops importing without warning.
Check the runtime, the compute access mode, and the governance path before an AutoML experiment starts. That habit saves far more time than the search itself. Some teams need help auditing existing AutoML workloads, and others need help designing a governed Databricks ML pipeline from the ground up. Either way, they can reach out to Kanerika’s team directly.
Frequently Asked Questions
What Is Databricks AutoML? Databricks AutoML is a low-code and code-based tool built into the Databricks Machine Learning workspace that automates data preparation, algorithm selection, and hyperparameter tuning for classification, regression, and forecasting problems. Each experiment produces a leaderboard of trials plus an editable Python notebook for every run, so a data scientist gets a working baseline model along with the exact code behind it rather than a black box.
Has Databricks AutoML Been Deprecated? Databricks AutoML has not been fully discontinued, but it is no longer included as a built-in library starting Databricks Runtime 18.0 ML and above. Databricks’ own release notes for that runtime state plainly that AutoML has been removed from the default install, which is why community forum users have reported confusion about its future. Existing workloads on earlier runtimes are not immediately affected.
Can Databricks AutoML Still Be Used on Runtime 18.0 ML or Later? Databricks AutoML still works with newer runtimes through the separate databricks-automl-runtime package available on PyPI, rather than as a preinstalled library. It also requires Dedicated or No Isolation Shared compute access mode, since Standard (shared) access mode does not support it. Teams should confirm both the package installation and compute mode before assuming an old AutoML notebook will run unchanged.
How Do You Use Databricks AutoML Through the Python API? The Python API exposes three entry points, databricks.automl.classify(), regress(), and forecast(), which accept a target column, a primary metric, and a timeout in minutes. Each call returns an AutoMLSummary object with a best_trial attribute pointing to the top-performing run, its metrics, and its generated notebook. Results and parameters are logged to MLflow automatically for every trial in the experiment.
Why Is My AutoML Experiment Missing From the Databricks UI? The AutoML option can disappear from a workspace after a runtime upgrade, most commonly when a cluster moves to Databricks Runtime 18.0 ML or later without the databricks-automl-runtime package installed. It can also be hidden by an unsupported compute access mode, since Standard (shared) clusters cannot run AutoML experiments. Checking the cluster’s runtime version and access mode resolves most cases.
How Can Databricks AutoML Model Accuracy Be Improved? Accuracy improves less from longer search time and more from picking the right primary metric, checking for class imbalance, and using a time-aware split when data has a temporal order. Reviewing the generated notebook for data leakage and comparing the winning trial against a simple baseline catches problems a leaderboard score alone will not show. Feature quality typically matters more than search duration.
What Is the Difference Between Databricks AutoML and Azure Machine Learning AutoML? Databricks AutoML runs inside the Databricks workspace and integrates directly with MLflow and Unity Catalog, which suits teams whose data and Spark pipelines already live on the platform. Azure Machine Learning AutoML is a separate Azure service with its own governance model built around Microsoft Purview and Active Directory. The right choice usually depends on where the underlying data and compute already sit.
What Should Teams Use Instead of Databricks AutoML for New Production Workloads? For new production workloads, many teams combine MLflow with open source libraries such as scikit-learn, XGBoost, or Optuna to keep the same experiment tracking without relying on a built-in AutoML library. Azure-native teams sometimes evaluate Azure Machine Learning AutoML instead. The right replacement depends on how much control, governance, and platform integration the workload actually needs.
