TL;DR: Databricks Autoloader is an incremental ingestion source (the cloudFiles format) that watches cloud object storage, processes only new files since the last run, and handles schema inference and evolution so new columns do not silently break the pipeline. Use it inside a Structured Streaming job with checkpoints to land raw files into Delta tables, choosing directory listing for small volumes and file notification mode for high-throughput streams.
Watch on YouTube
Why Databricks’ Platform Wins With 2026 Data Insights
Why so many enterprise data teams standardize on Databricks for engineering, analytics, and ML, and why ingestion is usually where that decision starts.
Every data team eventually hits the same wall. Files keep landing in cloud storage from upstream data engineering systems, the same hand-rolled ingestion script runs again, and one day a new column appears and the pipeline silently drops it, or noisily breaks at 2 AM.
Databricks Autoloader exists to make that whole class of work disappear. It treats cloud object storage as a streaming source, tracks which files it has already seen, evolves the schema as the data changes, and recovers cleanly from failure, all without you maintaining the bookkeeping.
This guide walks through what Databricks Autoloader actually is, how the cloudFiles source works under the hood, how schema inference and evolution behave, when to use directory listing versus file notification mode, how it differs from COPY INTO, and the production patterns and pitfalls Kanerika sees most often.
It assumes a basic familiarity with the Databricks lakehouse architecture and how raw files turn into Bronze, Silver, and Gold tables. If you are new to that pattern, the linked guide is the right place to start.
It is scoped to incremental file ingestion from cloud storage, the part of the pipeline where new files turn into Bronze tables. Downstream transformations into Silver and Gold are a separate problem, covered briefly only where Autoloader’s output shapes them.
Key Takeaways Databricks Autoloader is a Structured Streaming source called cloudFiles that incrementally ingests files from S3, ADLS, GCS, or Unity Catalog volumes into Delta tables, tracking processed files in a RocksDB-backed checkpoint so it gives exactly-once processing across restarts. Schema inference samples the first 50 GB or 1,000 files, writes the schema to a configured cloudFiles.schemaLocation, and adds a _rescued_data column that captures fields that did not fit, which is the production safety net for any source you do not fully control. Schema evolution has four modes set in cloudFiles.schemaEvolutionMode, addNewColumns is the default with no schema, rescue captures drift without failing the stream, failOnNewColumns enforces strict contracts, and addNewColumnsWithTypeWidening also widens compatible numeric types automatically. Directory listing mode works for any source with minimal setup, file notification mode plugs into S3 events plus SQS, Event Grid plus Azure Queue Storage, or GCS plus Pub/Sub, and is the right switch the day file discovery cost shows up in the storage bill. Autoloader is the streaming choice for continuous ingestion at scale, COPY INTO is the SQL command for idempotent batch loads, and many production stacks run both against the same target table when reprocessing a slice of files. Kanerika, a Databricks consulting partner, builds Autoloader pipelines that land Bronze tables within minutes of file arrival, expose schema drift through the rescued data column, and run on Lakeflow Jobs with the schema-restart pattern. What Is Databricks Autoloader? Databricks Autoloader is the recommended way to incrementally ingest files from cloud storage into a Databricks lakehouse . It is implemented as a Structured Streaming source called cloudFiles, and it processes only the new files that have arrived since the last run.
Listen on Spotify
Data Intelligence Is Your Competitive Edge
For teams choosing between platforms, this is also one of the practical reasons Databricks fits file-heavy ingestion patterns well. Snowflake handles this with Snowpipe; Databricks handles it with Autoloader, and the engine is built into the same notebook and job surface you already use.
The mental model is simple. You point it at a folder in S3, ADLS Gen2, GCS, or a Unity Catalog volume. You tell it the file format. It does the rest, including discovering new files, remembering what it has already processed, inferring and evolving the schema, and writing into a Delta table.
Autoloader sits inside the broader Databricks Data Intelligence Platform and is the standard ingestion source for the Bronze layer of a medallion architecture on Databricks , where files land in Bronze, are shaped into Silver, and are aggregated for analytics in Gold. See the official medallion architecture reference for the canonical Databricks framing. Anything that lands as a file, JSON events from a SaaS export, CSV drops from a partner that you would otherwise wire through an ETL pipeline , Parquet snapshots from a CDC or data-transformation tool, can flow through it.
The point of the abstraction is that ingestion stops being your problem. You stop hand-rolling file lists, watermark tables, and “what did we process last night” queries. The platform tracks all of it, and your data pipeline automation can focus on the transformations that actually matter.
Databricks now markets the same engine inside its Lakeflow Declarative Pipelines as the recommended file ingestion path. The terminology has shifted, but the underlying cloudFiles source is the same one teams have used for years, and most of the configuration carries over.
How cloudFiles Works Under the Hood Autoloader’s job is to answer one question on every micro-batch: which files in the source path are new, and which have I already processed? Getting that right at scale is harder than it looks, and the design choices behind cloudFiles are what make it different from a hand-written file-watcher script.
The source is a Structured Streaming reader. Every trigger fires a micro-batch. The reader looks at the source path, figures out the new files using whichever detection mode you chose, hands them to Spark for parsing, and commits a record of which files it processed into the streaming checkpoint.
That record lives in a RocksDB-backed key-value store inside the checkpoint location, the same one Databricks uses across its Delta-based features . RocksDB is the same embedded store Spark uses for stateful streaming, and it lets Autoloader track billions of file identifiers without blowing up driver memory. Restart the stream after a crash, and the checkpoint tells it exactly where to resume, which is how Autoloader gives you the exactly-once processing guarantee that real data pipelines depend on.
Schema information lives alongside the checkpoint, in a separate _schemas directory inside the cloudFiles.schemaLocation path you configure. Every schema version is written there, so the stream can detect, recover from, and audit changes over time without losing earlier versions.
What this means in practice is that Autoloader is closer to a managed CDC engine for object storage than to a file-list helper. The bookkeeping that used to fail your previous ingestion jobs, deduping new files, surviving restarts, evolving the schema, is now something the source owns.
Supported Sources, Formats, and Cloud Storage Autoloader is deliberately broad on the input side, because real enterprise data lakes never speak only one language. Knowing what is and is not in scope saves a planning conversation later.
On cloud storage, Autoloader reads from Amazon S3, Azure Data Lake Storage Gen2, Azure Blob Storage, Google Cloud Storage, and Unity Catalog volumes. Anything that has a supported cloud URI scheme works, which means almost every landing zone built during a modern cloud transformation qualifies.
On file format, eight options are supported: JSON, CSV, XML, Parquet, Avro, ORC, plain text, and binaryFile. The format is set with the required cloudFiles.format option, and the same source can read partitioned Hive-style layouts on any of them.
Kanerika Service
Databricks Consulting and Implementation
Kanerika is a Databricks partner that designs and operates production lakehouses on Databricks, from Bronze ingestion with Autoloader to governed, observable pipelines across Silver and Gold.
Explore Databricks Services The choice of format affects schema behavior. JSON, CSV, XML, Avro, and Parquet support schema inference and evolution. ORC, text, and binaryFile have fixed schemas, so the schema location still helps with partition inference but evolution is not in play. Databricks Runtime version matters here: XML, for example, needs DBR 14.3 LTS or above, and Parquet schema evolution needs DBR 11.3 LTS or above.
One thing Autoloader does not do is read streaming sources directly. It is not a Kafka consumer, it does not read Delta change feeds, and it does not pull from a database directly. Those belong to Structured Streaming and Lakeflow ingestion connectors. Autoloader’s contract is specifically files in cloud storage, and confusing that boundary is the first design mistake to avoid.
Schema Inference and the Rescued Data Column Schema is where ingestion projects most often go off the rails. Autoloader’s defaults are conservative on purpose, because the alternative, types drifting silently between runs, is a debugging nightmare.
When you point Autoloader at a folder for the first time and pass cloudFiles.schemaLocation, it samples the first 50 GB or 1,000 files of data, whichever limit it hits first, and writes the inferred schema into the _schemas directory. For JSON, CSV, and XML, it infers every column as a string by default, even nested fields inside JSON.
That default looks aggressive. It is deliberate. String typing means a stray integer that suddenly arrives as "42 USD" does not silently corrupt a column you thought was numeric. If you want stricter inference, you turn it on with cloudFiles.inferColumnTypes = true, and Autoloader uses Spark’s DataFrameReader logic to pick concrete types from samples.
For Parquet and Avro, which encode their own schema, Autoloader merges the schemas of the sampled files into a global schema. When files disagree, the widest type wins, and you can override that with schema hints.
Mode Behavior on a new column Best for addNewColumns (default)Stream fails with UnknownFieldException, the new column is appended to the stored schema, then a restart resumes with the updated schema. Most pipelines on structured sources where you want every new column captured. addNewColumnsWithTypeWideningSame as above, but Autoloader also widens supported types automatically, like int to long or float to double. Unsupported widenings go to _rescued_data. Pipelines that see numeric type drift on upstream changes. rescueSchema never evolves, stream never fails on a new column, and the new field is captured in the _rescued_data column. Noisy JSON sources where you want resilience first and structured fields later, in Silver. failOnNewColumnsStream fails and stays failed until you update the supplied schema or remove the offending file. Strict contracts with upstream producers, often for regulated data. noneSchema does not evolve, new columns are silently dropped, and nothing is rescued unless rescuedDataColumn is set explicitly. Locked schemas where new fields are not a business signal.
The most important thing Autoloader adds to inference is the _rescued_data column. It is automatically added to your schema, and it captures any field that did not fit the current schema, because the column is missing, the type does not match, or the case does not match. The rescued column holds a JSON blob with the bad fields and the source file path of the record, which makes it trivial to debug later.
This is your production safety net. A stricter mode like FAILFAST on its own will kill your stream the moment a malformed record shows up. _rescued_data instead keeps the stream alive, parks the offending fields, and gives you a queryable record of every mismatch. Always keep it on for any pipeline reading from a source you do not fully control.
You can also nudge the inference with cloudFiles.schemaHints. Hints are a comma-separated list of column name and type pairs in SQL syntax, like tags MAP<STRING,STRING>, version INT, documented in the official Auto Loader schema reference . They are useful when you know a JSON field is really a date or a numeric, and you do not want Autoloader to guess.
Schema Evolution Modes: addNewColumns, rescue, failOnNewColumns, none Inference handles what your data looks like today. Evolution handles what it will look like tomorrow. Autoloader gives you four modes to choose from, set with cloudFiles.schemaEvolutionMode, and the default depends on whether you supplied a schema.
The default is addNewColumns when no schema is provided and none when you do provide one. The four behaviors are worth knowing precisely.
Two patterns matter most in production. The first is the addNewColumns plus restart pattern, where you let the stream fail when a new column appears, the schema gets updated in the schema location, and an orchestrator like Databricks Workflows or Lakeflow Jobs restarts the stream automatically. This gives you a clean append of every new field with full auditability.
Case Study
40% Faster Reporting: Retail Analytics Modernized on Databricks
A national retail corporation eliminated data silos and modernized its analytics on Databricks, with Autoloader-style incremental ingestion at the core, delivering 40% faster reporting, 30% more data accessibility, and a 25% reduction in processing time.
Read the Case Study → The second is the Bronze JSON pattern. For raw JSON sources where the producer drifts, many teams use rescue mode for the Bronze table and parse the structured fields out into Silver. The Bronze stream then never fails on a new field, the rescued data stays queryable, and the schema work moves downstream where it belongs.
Directory Listing vs File Notification Mode Autoloader has two ways to discover new files in cloud storage. The choice between them is the single biggest operational lever on cost and latency at scale, and it is one that teams often get backwards.
The default is directory listing mode. Every trigger, Autoloader lists the source directory through the cloud’s storage API, compares the listing to its checkpoint, and processes whatever is new. Setup is zero, because the only thing it needs is read access to the bucket. On directories with up to a few thousand files, it is fast and free of side effects, similar to the simpler ingestion paths in a Snowflake data warehouse .
File notification mode is different. You set cloudFiles.useNotifications = true and Autoloader plugs into the cloud’s native event service. New files emit an event, the event lands in a queue, and Autoloader reads the queue instead of re-listing the directory. The mechanics differ slightly by cloud, but the pattern is the same. The Azure variant is described in the Microsoft Learn Auto Loader file detection modes reference.
The economic logic is straightforward. Listing a directory with 10 million objects on every five-minute trigger means hammering the cloud’s list API and paying for it. Reading new-file events from a queue does not scale with the size of the directory, only with the arrival rate. On any directory that grows beyond a few hundred thousand files, notification mode wins on both cost and latency.
The trade-off is setup. File notification mode needs Autoloader to be able to create or read the notification resources, which means IAM or RBAC work in the cloud. Databricks can provision them automatically if file events are enabled at the storage account, and Databricks recommends that path for any large workload. For quick-start pipelines and small directories, directory listing is the right starting point. The day file discovery shows up in your storage bill, switch.
Autoloader vs COPY INTO: When to Use Which If you have spent time in a Databricks SQL warehouse, you have probably already met COPY INTOSnowflake’s batch ingestion and end up running both. They should not.
COPY INTO is the right tool for batch loads. It runs as a SQL statement, has exactly-once idempotency baked in, and is what you want when a partner drops a file twice a day and you want to ingest each new drop into a managed table. It supports mergeSchema for schema evolution and works inside any SQL workflow.
Autoloader is the right tool for continuous ingestion at scale. It is a streaming source. It tracks millions of files efficiently. It supports the four-mode schema evolution above, with type widening, which is what real data integration at scale needs. And it is what you build when files arrive throughout the day and you want them in Bronze within minutes.
Cloud Event service Queue AWS S3 S3 event notifications, fanned out via SNS SQS Azure ADLS Gen2 or Blob Event Grid system topic Azure Queue Storage Google Cloud Storage GCS Object change notifications Pub/Sub
Many production stacks use both, the same way data warehouse automation stacks mix scheduled and event-driven loads. Autoloader runs the continuous Bronze ingestion, and COPY INTO handles the occasional partner backfill or the one-off historical reload of a slice of files. Running them in parallel against the same target table is supported and is a clean pattern when you need to reprocess a subset without disrupting the live stream.
Configuration Options You Will Actually Touch The cloudFiles source, fully documented in the Auto Loader options reference , exposes more options than any one pipeline needs. The eight below are the ones that show up in most production runs.
cloudFiles.formatavro, binaryFile, csv, json, orc, parquet, text, xml.cloudFiles.schemaLocationcloudFiles.schemaEvolutionModecloudFiles.schemaHintscloudFiles.useNotificationstrue to switch from directory listing to file notification mode.cloudFiles.includeExistingFilestrue by default, set to false when you want the stream to start fresh from the moment of deployment and ignore the backlog.cloudFiles.maxFilesPerTriggercloudFiles.maxBytesPerTriggercloudFiles.backfillIntervalTwo more options are worth knowing even if you do not set them often. cloudFiles.allowOverwrites controls whether an overwritten file is re-ingested, off by default. cloudFiles.partitionColumns tells Autoloader to read Hive-style partition keys out of the directory layout, which matters when your source uses date partitions you want to keep as columns.
Dimension Autoloader (cloudFiles) COPY INTO Style Structured Streaming source SQL command File scale Billions, millions/hour Tens of thousands per run Idempotency Streaming checkpoint, exactly-once Per-file dedup table, exactly-once Schema evolution Four modes including type widening and rescue mergeSchema onlyReprocessing Requires checkpoint reset or backfill Trivial, re-run with FORCE = TRUE or a different file list Best fit Continuous, high-volume, schema-drifting sources Batch loads, partner drops, one-time bulk ingest
A Production Pattern: Autoloader Into a Bronze Delta Table Words only get you so far. The minimum viable production Autoloader job has a shape that is worth keeping in your head.
The Python skeleton looks like this:
spark.readStream.format("cloudFiles") \
.option("cloudFiles.format", "json") \
.option("cloudFiles.schemaLocation", "/Volumes/main/bronze/_schemas/orders") \
.option("cloudFiles.schemaEvolutionMode", "addNewColumns") \
.option("cloudFiles.useNotifications", "true") \
.option("cloudFiles.maxFilesPerTrigger", 200) \
.load("/Volumes/main/landing/orders/") \
.writeStream \
.option("checkpointLocation", "/Volumes/main/bronze/_checkpoints/orders") \
.trigger(availableNow=True) \
.toTable("main.bronze.orders")Five decisions make that block production-ready. The schema location lives outside the source path, so re-creating the source folder does not lose schema history. Notifications are on, because the orders directory will outgrow listing in a quarter. The evolution mode is explicit, so the team knows new columns will be appended and the stream restart is expected. The rate limit caps each batch, so a spike of new files cannot blow up cluster sizing. And the trigger is availableNow, which processes every new file in batches and exits, the most common Lakeflow Jobs pattern.
The matching orchestration on top is a Databricks Workflow with a file-arrival trigger or a cron schedule, with retry-on-failure configured. When schema evolution fires the UnknownFieldException, the job retries, the schema location now has the new column, and the next attempt picks up cleanly. That is the pattern Databricks recommends, and it removes most of the operational pain that earlier ingestion patterns left engineers carrying.
Common Pitfalls and How to Avoid Them The failure modes Kanerika sees most often are not in the engine. They are in how it is configured and what surrounds it. Five are worth flagging.
Forgetting the rescued data column. Pipelines that turn _rescued_data off lose the audit trail the moment data drifts. Keep it on unless the source is a strict contract you control end to end.
Skipping the schema location. Without cloudFiles.schemaLocation, Autoloader cannot evolve a schema. The stream still runs, but every restart re-infers from scratch, and every change of behavior shows up as confusing inconsistency.
Running directory listing on a directory that has outgrown it. Listing cost scales with directory size. The first time the storage bill spikes, the cause is usually here. Switch to notifications, or partition the source into smaller directories, and wire the storage and notification queues into your Azure monitoring tools so listing-cost spikes show up as an alert, not an end-of-month surprise.
Not planning for backfills. Autoloader’s checkpoint is sticky on purpose. If you need to reprocess a window of files, you either reset the checkpoint, set cloudFiles.backfillInterval, or run a COPY INTO job in parallel against the same target. None of those should be a surprise the day you need them.
Treating Bronze like Silver. Autoloader’s job is to land raw files in a queryable table. The temptation to parse, clean, and enrich inside the same stream multiplies failure surface. Keep Bronze as a faithful copy. Move shaping and validation into a separate Silver job that reads from the Bronze table, where techniques like liquid clustering can keep query patterns fast and modelling choices such as Data Vault modeling shape how Silver and Gold layers are built downstream.
Watch on YouTube
How to Move Your Enterprise Data Stack to Databricks
A walkthrough of how Kanerika moves enterprise data stacks onto Databricks, including the ingestion layer where Autoloader takes over from hand-rolled file-watchers.
How Kanerika Builds Autoloader Pipelines Kanerika is a Databricks consulting and implementation partner, and ingestion is usually where engagements start. The pattern below is the one we apply on most Autoloader builds, refined over dozens of client lakehouses.
Assess. We look at the existing landing zones and how many directories will need a stream, the file formats and their volumes, the upstream producers and how often their schemas actually drift, and the latency the business needs from raw file to Bronze table. That assessment is what determines whether one Autoloader job per source or one per directory makes sense, and whether notifications need to be set up from day one.
Design. We choose the schema evolution mode per source, not per pipeline. Strict partner contracts get failOnNewColumns, which fits the disciplined transformation patterns teams build with dbt on Databricks . Drifting JSON sources get rescue. Most structured drops get addNewColumns with an automatic restart in the orchestrating job. Schema locations live under Unity Catalog volumes, separate from the source path and the checkpoint, so the three concerns can move independently.
Build. We standardize on the Lakeflow Jobs pattern with Trigger.availableNow, with retries configured for the expected UnknownFieldException on schema evolution. Notifications are enabled at the storage account or bucket, so subsequent streams pick them up without per-source IAM work. Every Bronze table writes _rescued_data through, even if downstream Silver jobs ignore it. The pipeline is checked into a Databricks Asset Bundle so the same job can deploy across dev, staging, and production with parameter overrides.
Govern. Bronze tables land in a Unity Catalog schema with column-level lineage on, the foundation of any serious governance framework . The rescued data column has a downstream Silver query that surfaces drift to the data team in a dashboard, so a schema change is visible before it becomes a stakeholder phone call. Cost dashboards split job-cluster spend by Autoloader pipeline, the same cost-discipline pattern we use on Snowflake estates, so the team can see when a directory has grown enough to justify switching to notifications.
Enable. We hand over the runbook for the two operational events teams actually see: schema evolution restarts and storage-event configuration changes. Engineers leave the engagement able to add a new Autoloader source without our help, including switching to serverless compute where job cadence justifies it, which is the only outcome that matters. Whether the rest of the platform is vector search , ML, or BI is a downstream choice.
The result on a recent retail engagement was a Bronze ingestion layer that handles JSON, Parquet, and CSV drops from a dozen producers, ingests them into Unity Catalog volumes within minutes of arrival, the kind of ingestion bar enterprise data engineering teams measure us against, and surfaces every schema drift to the data team automatically. The team that used to spend a day a week on ingestion bugs spends it on the analytics those tables feed.
Wrapping Up Databricks Autoloader is not a clever feature you bolt onto a pipeline. It is the right way to do file ingestion on a lakehouse, because it solves the four bookkeeping problems that used to silently break every other approach: knowing which files are new, tracking the schema as it changes, recovering cleanly from failure, and scaling to millions of files without doing list-API gymnastics.
The decisions that matter are about your specific data, not about the engine. Choose your evolution mode by how much your upstream drifts. Choose your detection mode by directory size. Keep _rescued_data on. Push transformation into Silver. Run it with Lakeflow Jobs and let the schema-restart pattern do its job.
Do those five things and the rest of the platform stops feeling like glue.
Frequently Asked Questions What is Databricks Autoloader and what problem does it solve? Databricks Autoloader is a Structured Streaming source named cloudFiles that incrementally ingests new files from cloud object storage into Delta tables. It solves the bookkeeping problem of file ingestion at scale, namely tracking which files have already been processed, evolving the schema as new columns appear, recovering cleanly from failure, and handling millions of files without writing custom file-list code. You point it at a folder in S3, ADLS Gen2, GCS, or a Unity Catalog volume, tell it the format, and it does the rest, including writing into a Bronze table you can query the moment the data lands.
How does Autoloader's schema evolution actually work? When Autoloader detects a new column in incoming data, it performs schema inference on the latest micro-batch, appends the new column to the stored schema in cloudFiles.schemaLocation, and stops the stream with an UnknownFieldException. Restarting the stream resumes processing with the updated schema. Behavior is controlled by cloudFiles.schemaEvolutionMode and has four modes: addNewColumns is the default when no schema is provided, rescue captures drift in the _rescued_data column without failing, failOnNewColumns enforces a strict contract, and addNewColumnsWithTypeWidening also widens compatible numeric types like int to long automatically. Databricks recommends running Autoloader inside Lakeflow Jobs so the stream restarts automatically after each schema change.
What is the difference between Databricks Autoloader and COPY INTO? Autoloader is a Structured Streaming source built for continuous, high-volume ingestion. It scales to billions of files and millions of files per hour, supports the full four-mode schema evolution, and tracks processed files in a streaming checkpoint. COPY INTO is a SQL command for idempotent batch loads. It is the right tool when files arrive on a schedule, when you need to reprocess a slice with FORCE = TRUE, or when your workflow is fundamentally SQL-driven. Many production stacks use both, Autoloader for the continuous Bronze ingestion and COPY INTO for one-off backfills against the same target table.
When should I use directory listing versus file notification mode? Use directory listing mode when setup matters more than scale, typically for sources with up to a few thousand files in the directory. It needs only read access to the bucket and works on every supported cloud. Switch to file notification mode the moment listing cost starts to show up in the storage bill, or the directory grows past a few hundred thousand files. Notification mode plugs Autoloader into S3 plus SQS on AWS, Event Grid plus Azure Queue Storage on Azure, or GCS plus Pub/Sub on Google Cloud, so new-file detection scales with arrival rate rather than directory size.
What is the _rescued_data column and should I keep it on? The _rescued_data column is added to your schema automatically when schema inference is enabled. It captures any field that did not fit the current schema because the column is missing, the type does not match, or the case does not match. The rescued column holds a JSON blob with the bad fields and the source file path of the record. Keep it on for every pipeline that reads from a source you do not fully control. It is your production safety net for schema drift, and the alternative, dropping or failing on each mismatch, is much harder to debug after the fact.
Does Autoloader work with Lakeflow Declarative Pipelines? Yes. Lakeflow Declarative Pipelines, formerly Delta Live Tables, uses Autoloader as the recommended file ingestion source for its streaming tables. The cloudFiles options are the same. The difference is that Lakeflow manages the checkpoint location and the schema location for you automatically, which removes a class of configuration mistakes. If you are starting fresh, Lakeflow is the cleanest place to run Autoloader. If you are migrating from a notebook-based job, the underlying engine and options carry over.
How do I reprocess a window of files when Autoloader's checkpoint is sticky? You have three options. First, reset the streaming checkpoint, which forces Autoloader to discover every file in the source again, including the ones already in the target. This is the heaviest option. Second, set cloudFiles.backfillInterval, which schedules an asynchronous backfill that catches files the notification service might have missed without restarting the main stream. Third, run a COPY INTO job against the same Delta table with FORCE = TRUE or a curated file list, which is the cleanest way to reprocess a subset while the live Autoloader stream keeps running.
What are the most common Autoloader configuration mistakes? Five recur. Turning off the _rescued_data column, which loses the audit trail the moment data drifts. Skipping cloudFiles.schemaLocation, which prevents schema evolution and re-infers from scratch on every restart. Running directory listing mode on a directory that has outgrown it, which spikes storage list-API cost. Not planning for backfills, so the first reprocess request becomes a stressful improvisation. And treating Bronze like Silver, parsing and cleaning inside the ingestion stream rather than landing raw files and shaping them in a separate Silver job. Avoiding those five covers most of what Kanerika sees go wrong on real engagements.
