TL;DR
You can read from and write to a Microsoft Fabric warehouse inside a PySpark notebook. The connector is built into Fabric, so there is nothing to install. Both directions use one method, called synapsesql. Reading works on any current runtime. Writing needs Runtime 2.0, which new workspaces do not use by default. The connector cannot update or delete rows, so those jobs finish in T-SQL.
Key Takeaways The method is synapsesql, used as spark.read.synapsesql() and df.write.synapsesql(), and it takes a three-part warehouse.schema.table name. Warehouse writes are available only on Fabric Runtime 2.0, which is GA but not yet the default for new workspaces. The connector supports four save modes: ErrorIfExists, Ignore, Overwrite and Append. There is no update, delete or upsert. You can write to a warehouse, but never to a lakehouse SQL analytics endpoint, which is read-only. Cross-workspace access works through Constants.WorkspaceId plus the item id, not through a connection string. The connector honours the object, row and column-level security already defined in the SQL engine, and it only authenticates an interactive Microsoft Entra user. Watch on YouTube
Why Most Fabric Deployments Fail at Scale | Amit Chandak, Microsoft MVP
Amit Chandak, Kanerika’s Chief Analytics Officer and a Microsoft MVP, walks through the architecture and governance decisions that quietly break Fabric implementations once real workloads arrive, including where Spark belongs and where it does not.
The One Line in Microsoft’s Docs That Decides Whether Your Write Works Microsoft’s connector documentation carries a single sentence that settles most failed attempts at this task. Writing data to a warehouse table, it says, “is available only with the latest GA runtime, Runtime 2.0”. A new Fabric workspace still starts on Runtime 1.3, which has now moved to end-of-support-announced status.
That gap explains a specific and confusing experience. The read half of your notebook works immediately, the write half throws, and the error text says nothing about runtimes. So before you debug a single line of code in a PySpark notebook in Microsoft Fabric Warehouse, check the runtime dropdown. The setup section below shows where it lives.
What a PySpark Notebook in Microsoft Fabric Warehouse Actually Does A Fabric warehouse is a T-SQL engine. It stores tables, enforces permissions and answers SQL queries. It does not run Python, and it does not give you DataFrames.
A PySpark notebook is the other half of that picture. It runs distributed Python over Spark, which is what you want for transformations that are awkward in SQL: machine learning feature work, text parsing, calling an API per batch, or joining warehouse data to files sitting in a Fabric lakehouse .
The Spark connector for Fabric Data Warehouse is the bridge between the two. Microsoft describes it as a connector with a simplified Spark API that “operates with just one line of code” and that “comes preinstalled within the Fabric runtime”. You do not add a JAR, you do not manage a JDBC string, and you do not store a password anywhere.
One detail is worth stating plainly because it decides your architecture. The connector writes to a warehouse . It cannot write to the SQL analytics endpoint of a lakehouse, because that endpoint is read-only. If your target is a lakehouse table, you skip this connector entirely and write Delta directly.
What Changed Since Runtime 1.3 This is the part most published guidance has not caught up with, so it is worth being precise. The figures below come from Microsoft’s Fabric runtime comparison .
Runtime 1.3 Runtime 2.0 Release stage End of support announced Generally available Apache Spark 3.5.5 4.1 Delta Lake 3.2 4.2 Python 3.11 3.13 Scala / Java 2.12.17 / 11 2.13.16 / 21 Warehouse read Supported Supported Warehouse write Not supported Supported
Two practical consequences follow. First, a notebook that reads fine on a default workspace will fail the moment you add a write, and the error will not say “wrong runtime” in plain words. Second, moving to Runtime 2.0 changes more than a version number. Scala goes from 2.12 to 2.13 and Java from 11 to 21, so custom JARs built for 1.3 will need rebuilding. Python and R libraries generally migrate cleanly.
Runtime 2.0 also carries the Native Execution Engine , which Microsoft benchmarks at up to six times faster than open-source Spark on TPC-DS workloads, with no code changes. You turn it on per environment, or per application with spark.native.enabled.
Prerequisites, Permissions and Runtime Setup Before any code runs, four things need to be true.
A warehouse you can reach. You need at least Read permission on the warehouse item, which behaves like CONNECT in SQL Server, plus object-level rights on the specific tables or views.An interactive Microsoft Entra sign-in. The connector supports interactive Entra user authentication only. Service principal authentication is not supported, which matters the moment you try to productionise.Runtime 2.0, if you intend to write. Set it at workspace level under Workspace settings, Data Engineering/Science, Spark settings, Environment, then pick 2.0 (Spark 4.1, Delta 4.2) . Or set it on a single Environment item and attach that to the notebook, which overrides the workspace default and is the safer way to test.A lakehouse attached to the notebook for default workspace context, so the connector can discover the warehouse endpoint without you naming ids.That third point about service principals deserves a warning. Because the connector passes the signed-in user’s credentials to the SQL engine, a notebook scheduled inside a pipeline runs as the identity that owns the schedule. Permissions that work when you click Run can fail at 2 a.m. under a different owner. Run the test under the schedule itself before you trust it.
Checklist
Microsoft Fabric Readiness Checklist
Work through the workspace, capacity, security and governance decisions that need to be settled before Fabric carries production workloads, so runtime and permission surprises do not land late.
Get the Checklist →
Importing the Connector in a PySpark Notebook Run these two lines first, in their own cell, before anything else touches the connector.
import com.microsoft.spark.fabric
from com.microsoft.spark.fabric.Constants import ConstantsTwo mistakes are common here. The class is Constants, plural, not Constant. And its members are CamelCase, so it is Constants.WorkspaceId, never Constants.WORKSPACE_ID. Both raise attribute errors that look like connector faults and are not.
How to Read a Fabric Warehouse Table With PySpark Reading is the first thing most people try with a PySpark notebook in Microsoft Fabric Warehouse, and it is one line of code. The argument is a three-part name, meaning warehouse, then schema, then table or view.
df = spark.read.synapsesql("warehouse01.dbo.geography")
df.show()Standard DataFrame operations chain onto it, so you can keep test reads small while you develop.
# first 10 rows only
df = spark.read.synapsesql("warehouse01.dbo.geography").limit(10)
# filter on a column value
df = spark.read.synapsesql("warehouse01.dbo.geography").filter("country == 'India'")
# select specific columns
df = spark.read.synapsesql("warehouse01.dbo.geography").select("city", "country")When the shaping is better expressed in SQL, push the whole query to the warehouse engine and let Spark receive only the result. This keeps the heavy filtering and aggregation where the indexes are.
df = (spark.read
.option(Constants.DatabaseName, "warehouse01")
.synapsesql("SELECT country, COUNT(*) AS cities FROM dbo.geography GROUP BY country"))For large tables, the connector supports parallel reads in the same shape as spark.read.jdbc. You give it a numeric partition column and a range, and Spark splits the read across partitions instead of pulling everything through one connection.
df = (spark.read
.option("partitionColumn", "customer_id")
.option("lowerBound", 1)
.option("upperBound", 5000000)
.option("numPartitions", 16)
.synapsesql("warehouse01.dbo.customers"))How to Write a PySpark DataFrame Back to the Warehouse Writing uses the same method on the writer side, and it requires Runtime 2.0.
filtered = df.filter("country == 'India'")
filtered.write.mode("overwrite").synapsesql("warehouse01.dbo.geography_india")Four save modes are supported, and choosing the wrong one is the most common cause of a pipeline that silently does nothing.
Save mode If the table already exists If it does not errorifexists (default)Write aborts and raises an exception Table is created with the data ignoreWrite is skipped, no error raised Table is created with the data overwriteExisting rows are replaced Table is created with the data appendNew rows are added Table is created with the data
Note what ignore does. It returns success and writes nothing. A scheduled job using ignore against an existing table will look healthy in every run history and never update a row.
Under the covers the default write is a two-phase operation. The connector stages the DataFrame into intermediate storage, then issues COPY INTO to load it into the warehouse table. That is why throughput holds up as volumes grow, and why a write is not a row-by-row insert.
The default write path: Spark stages the DataFrame, then COPY INTO loads the warehouse table. Tuning Writes Under Private Link If Private Link is enabled at tenant or workspace level, the write path changes. Data goes through JDBC batch inserts with no external staging, and the connector picks a strategy for you.
writeStrategy Behaviour When to pick it MULTI_THREAD_NO_DUPLICATES_BATCH_INSERTMulti-threaded writes through staging tables Default under Private Link, best throughput NO_DUPLICATES_BATCH_INSERTSingle-threaded, streams rows without buffering whole partitions When executors hit OutOfMemoryError
(df.write
.mode("overwrite")
.option("writeStrategy", "MULTI_THREAD_NO_DUPLICATES_BATCH_INSERT")
.option("writerThreadsPerTask", "5")
.synapsesql("warehouse01.dbo.customers"))Both strategies use a two-phase commit. Each partition writes into its own temporary staging table, staging tables are truncated before each attempt, and the driver merges them into the target in a single atomic transaction. That is what makes the write safe when Spark retries a failed task: you get all the data or none of it, never duplicates.
writerThreadsPerTask defaults to 3. Raising it opens more concurrent JDBC connections to the warehouse and increases executor memory use, so treat 5 as an experiment rather than a setting to copy.
Reading and Writing Across Fabric Workspaces By default the connector looks for the warehouse in the workspace of the lakehouse attached to your notebook. To reach another workspace you supply ids explicitly. The workspace id is the GUID in the Fabric URL after /groups/.
# by workspace id alone
df = (spark.read
.option(Constants.WorkspaceId, "")
.synapsesql("warehouse01.dbo.geography"))
# pinned to a specific warehouse item
df = (spark.read
.option(Constants.WorkspaceId, "")
.option(Constants.DatawarehouseId, "")
.synapsesql("warehouse01.dbo.geography")) Use Constants.LakehouseId in place of Constants.DatawarehouseId when the source is a lakehouse SQL analytics endpoint. Pinning the item id is worth the extra line in production, because warehouse names are not unique across workspaces and a rename will not break an id.
What the Connector Cannot Do, and What to Use Instead The gaps matter more than the features, because they decide whether a PySpark notebook is the right tool at all. Microsoft’s own Private Link documentation is explicit that the connector “doesn’t support update, delete, or upsert operations”.
What you need Works with this connector? What to use instead Insert or fully replace a table Yes, append or overwrite Nothing else needed UPDATE, DELETE or MERGE / upsert No Write to a staging table, then run MERGE in a T-SQL notebook or stored procedure Write to a lakehouse table No, the SQL endpoint is read-only Write Delta directly with df.write.format("delta") Query warehouse history No, time travel syntax is not supported Query time travel from T-SQL instead Run unattended as a service principal No, interactive Entra users only Pipeline activities, or a Data Factory copy Simple no-code ingestion Not its purpose Dataflows Gen2 or a pipeline copy activity
The upsert workaround is the one to internalise. You land the Spark output in a staging table with overwrite, then a T-SQL step does the MERGE against the target. Spark handles the transformation, the warehouse handles the transaction. Trying to emulate a merge in Spark against a warehouse ends badly.
When to Use a Notebook Against a Warehouse, and When to Use T-SQL Most of the time the warehouse can answer the question by itself, and reaching for Spark adds a session, a cost and a failure mode for nothing. The split we apply on delivery work is narrow and worth stating.
Use a PySpark notebook when Stay in T-SQL when The transformation needs Python libraries, machine learning or text parsing the SQL engine cannot express The work is joins, filters, aggregation or windowing You are joining warehouse tables to files, or to data from outside Fabric Everything involved already lives in the warehouse You need to process volumes in parallel that would tie up the SQL engine The result set is small and the query is routine The job is a scheduled batch build, where session startup is a small share of the runtime The job is interactive or sub-minute, where Spark startup dominates You are landing external data that has to be reshaped before it is trustworthy You need UPDATE, DELETE or MERGE, which the connector cannot do at all
Which engine takes the job: Spark for Python, files and scale; T-SQL for joins, merges and small result sets. A useful test: if you can write the logic as a single T-SQL statement that finishes quickly, the notebook is overhead. Spark earns its place when the transformation genuinely cannot be said in SQL, or when the data is not all in the warehouse yet.
Kanerika Service
SQL Server to Microsoft Fabric Migration
We move SQL-based reporting estates onto Fabric and decide, table by table, what belongs in Spark and what stays in T-SQL. One US retail chain came out of it with 74% faster reporting cycles.
Plan Your Migration
Security: What Spark Actually Sees A frequent worry is that reading a warehouse through Spark quietly bypasses the controls set in SQL. It does not. Microsoft states that while accessing a table or view, the connector “upholds security models defined at the SQL engine level”, naming object-level, row-level and column-level security.
In practice that means a user restricted to one region’s rows by an RLS predicate sees exactly those rows in a DataFrame. A masked or denied column does not arrive in Spark at all. You do not re-implement those rules in notebook code, and you should not try.
The exposure is elsewhere. Once data is in a DataFrame, anything the notebook then writes, to a lakehouse file or another table, carries no security policy with it. A read that respects RLS followed by a write to an open lakehouse table is a leak with a clean audit trail. Decide who can attach notebooks to a workspace with the same care you use for warehouse grants.
Troubleshooting the Errors You Will Actually Hit Symptom Likely cause Fix AttributeError on snapsql or synapsqlMethod misspelt, a very common typo The method is synapsesql NameError: Constants is not definedImport cell never ran, or kernel restarted Re-run the two import lines AttributeError: WORKSPACE_IDWrong member style Use Constants.WorkspaceId Read works, write fails Workspace is on Runtime 1.3 Switch the workspace or environment to Runtime 2.0 Write reports success, table unchanged Save mode is ignore and the table exists Use append or overwrite Table not found across workspaces Connector defaulted to the attached lakehouse’s workspace Add Constants.WorkspaceId and the item id Permission denied on some columns only Column-level security is doing its job Request object-level grants, do not work around it Executor OutOfMemoryError during a Private Link write Multi-threaded strategy buffering partitions Switch to NO_DUPLICATES_BATCH_INSERT Scheduled run fails, manual run passes Schedule owner has different permissions Grant the owning identity the same access
One behaviour to expect: when a cell fails, the connector cancels the remaining cells in the notebook run. Detailed messages live in the Spark application logs, not in the cell output, so check there before assuming the connector is at fault.
A Production Pattern That Holds Up Once a warehouse is the destination, the same shape keeps working for every PySpark notebook in the workspace. The detail that matters here is where the connector sits and what it is allowed to do on each hop.
Spark prepares, the warehouse serves. Reshaping and enrichment happen in notebooks. The warehouse holds the conformed tables analysts and Power BI read, with row and column rules enforced by the SQL engine.The connector is the seam, and it is one-directional per hop. Spark reads warehouse dimensions to enrich a job, and writes finished output into staging tables with overwrite.T-SQL closes every transaction that changes existing rows, because the connector cannot merge. The staging table is the handoff point.Nothing schedules under a service principal. Because auth is interactive-user only, the owning identity of each scheduled notebook has to be decided up front.How the upstream layers are built, and how Delta behaves underneath them, is a separate topic. We cover the ingestion and transformation side in our guide to the Microsoft Fabric lakehouse .
Listen on Spotify
Why Fabric Is Becoming the Go-To Data Platform | SSIS to Fabric Migration
This is also the cost-sensible split. Spark capacity is billed while a session is alive, so pushing filters and aggregations into the warehouse with a passthrough query, rather than reading whole tables into a DataFrame, usually costs less and finishes sooner.
How Kanerika Builds Fabric Data Platforms We do this work in production, and the numbers below come from named engagements rather than capability claims.
FoodPharma, a contract food manufacturer, ran six core operational systems with no shared reporting layer. Working with Kanerika, they consolidated more than 50 tables and roughly 1 TB of history into Microsoft Fabric with automated daily loads. Cross-functional reporting that took two business days now takes about 90 minutes, and the BI team recovered roughly 15 hours a week previously lost to manual extraction and reconciliation. The build ran seven weeks. Microsoft published the full customer story .
For a leading US retail chain, we migrated SQL-based reporting onto Fabric and measured 74% faster reporting cycles, a 65% increase in reporting stability and 72% faster access to current metrics.
On migrations of that shape we lead with FLIP, our data integration accelerator, to move the existing SQL and ETL logic across, then decide case by case which transformations belong in a PySpark notebook and which stay in the warehouse engine.
Case Study
74% Faster Retail Reporting with Microsoft Fabric
How a leading US retail chain moved SQL-based reporting to Microsoft Fabric and cut reporting cycles by 74% while making refreshes materially more stable.
Read the Case Study →
If you are weighing a move onto Fabric, or a notebook layer is already fighting your warehouse, our Microsoft Fabric practice and data engineering team run these migrations end to end, including the SQL Server to Fabric path .
Talk to Kanerika
Planning a Microsoft Fabric Build or Migration?
We design the Spark and warehouse split, the runtime and capacity plan, and the governance model before code gets written. A short call is usually enough to tell you which parts of your pipeline belong in notebooks and which do not.
Schedule a Demo →
Frequently Asked Questions
Can I write data from PySpark directly into Microsoft Fabric Warehouse? Yes. Use df.write.synapsesql("warehouse.schema.table") from a PySpark notebook. Warehouse writes require Fabric Runtime 2.0, the current GA runtime. New workspaces still default to Runtime 1.3, which supports reads but not writes, so switch the runtime at workspace or environment level first. The connector handles staging and loading for you. Reads, by contrast, work on either runtime without any change.
What is the correct method name for the Fabric warehouse Spark connector? The method is synapsesql. You call it as spark.read.synapsesql() to read and df.write.synapsesql() to write. Variants such as snapsql and synapsql circulate widely in copied sample code and raise an attribute error. The options class is Constants, plural, with CamelCase members such as Constants.WorkspaceId. Upper-case names like WORKSPACE_ID also fail.
What is PySpark used for? Large-scale data processing. That covers ingesting and cleaning raw data, joining datasets too large for a single machine, feature engineering, and batch transformations inside a pipeline. In Fabric it is also the way to combine warehouse tables with lakehouse files and sources outside the platform in a single job. In Fabric it also joins warehouse tables to lakehouse files.
Is Databricks in Microsoft Fabric? No. They are separate products from different vendors, and neither contains the other. Fabric has its own managed Spark runtime, while Azure Databricks is a distinct service with its own workspaces and clusters. The two can interoperate through shared storage and shortcuts, but a Fabric notebook is not a Databricks notebook and the runtimes differ.
What is a PySpark notebook? An interactive document that runs Python code against a Spark session in cells, showing output inline as you go. In Microsoft Fabric a notebook belongs to a workspace and usually attaches to a lakehouse for context. It runs on a Spark pool, and can read from or write to warehouses, lakehouses and external sources.
How do I read from a warehouse in another workspace? Add the workspace id as a connector option before the call, like spark.read.option(Constants.WorkspaceId, "<workspace-id>").synapsesql("warehouse.schema.table"). The id is the GUID in the Fabric URL after /groups/. In production, also pin Constants.DatawarehouseId, because warehouse names are not unique across workspaces and a rename will not break an id. The same option pattern works for writes across workspaces as well.
Is PySpark or SQL faster? For set-based work over data already in the warehouse, SQL usually wins. The engine is optimised for exactly that, and there is no Spark session to start. PySpark wins when the job is large enough to benefit from parallelism, or when the logic simply cannot be expressed in SQL at all.
Is PySpark a tool or framework? PySpark is the Python interface to the Apache Spark framework. Spark is the distributed processing framework that does the work; PySpark is how you drive it from Python rather than Scala or Java. In everyday conversation people call it both a tool and a framework, and very little turns on the distinction.
Can I overwrite an existing warehouse table using PySpark? Yes. Overwrite is one of four supported save modes, alongside ErrorIfExists, Ignore and Append. ErrorIfExists is the default. Use df.write.mode("overwrite").synapsesql(...) to replace the contents of an existing table. Take care with Ignore, which silently skips the write when the target table already exists and still reports the run as successful.
What runtime version is required for Fabric Warehouse read and write? Reads work on current runtimes. Writes to a warehouse table require Runtime 2.0, the latest GA runtime, which ships Apache Spark 4.1, Delta Lake 4.2 and Python 3.13. Runtime 1.3 carries Spark 3.5.5 and Delta 3.2, has end of support announced, and remains the default selection for new workspaces. Check yours before writing.
Do I need to install any external packages to use the Fabric Spark connector? No. The connector comes preinstalled in the Fabric runtime. You only need two import lines before using it, which are import com.microsoft.spark.fabric and from com.microsoft.spark.fabric.Constants import Constants. There is no JAR to upload, no library to manage, and no connection string or password to store anywhere in the notebook. Run those two lines in their own cell first.
Can I filter or limit data when reading from a warehouse in PySpark? Yes. Chain standard DataFrame methods onto the read, for example spark.read.synapsesql("wh.dbo.table").filter("country == 'India'").limit(10). For heavy filtering or aggregation, push a T-SQL query to the engine using the Constants.DatabaseName option. Then only the result set travels into Spark. That keeps memory use down and makes development iterations noticeably quicker. Chaining filters and limits works as it does in plain Spark.
Is it necessary to connect a notebook to a Lakehouse even if I only work with a Warehouse? Attaching a lakehouse sets the default workspace context the connector uses to discover your warehouse, and it gives you relative file paths. It is a convenience rather than a documented hard requirement for warehouse work. If you reach across workspaces, you must pass Constants.WorkspaceId and the item id explicitly regardless of what is attached.
Can we use PySpark in Fabric? Yes. PySpark is the default language in Fabric notebooks, running on the managed Spark runtime with no infrastructure for you to set up. You can read and write lakehouse Delta tables directly. Fabric warehouses are reached through the Spark connector, which is already installed in the runtime. No cluster sizing or driver configuration is required before you start.
How do I create a notebook in Microsoft Fabric? From your workspace, select New and then Notebook, or create one from the Data Engineering experience. Attach a lakehouse for default context, confirm the runtime version in the environment settings, and choose your language. Set the runtime to 2.0 before you start if the notebook will write to a warehouse table.
Does the Fabric Spark connector support UPDATE, DELETE or MERGE? No. Microsoft documents plainly that the connector does not support update, delete or upsert operations. The standard workaround is to write Spark output into a staging table using overwrite. A MERGE in a T-SQL notebook or stored procedure then updates the target. Spark does the transformation, the warehouse owns the transaction.
Can I write to a lakehouse SQL analytics endpoint with this connector? No. The SQL analytics endpoint of a lakehouse is read-only, so the connector reads from it and writes only to a warehouse. To land data in a lakehouse table, write Delta directly with df.write.format("delta").saveAsTable(...) rather than routing it through the warehouse connector. Choosing the wrong target causes confusing permission errors.
Does the connector work with a service principal for scheduled jobs? No. The connector supports interactive Microsoft Entra user authentication only. Service principal authentication is unsupported. A scheduled notebook therefore runs under the identity that owns the schedule, so permissions that work when you press Run can fail unattended at night. Test the scheduled path with the owning identity before you rely on it.
Does reading a warehouse through Spark bypass row-level security? No. The connector upholds the security models defined at the SQL engine level, including object-level, row-level and column-level security. A restricted user sees only the rows they are permitted to see in the DataFrame. The real exposure is downstream, because anything you then write elsewhere carries no security policy with it.
How do I speed up a large warehouse read in PySpark? Use parallel reads. Supply a numeric partition column with partitionColumn, lowerBound, upperBound and numPartitions, and Spark splits the read across partitions instead of pulling everything through one connection. Better still, push filters and aggregation into a T-SQL passthrough query so that far less data ever moves into Spark. Both can be combined when volumes justify it.
What are writeStrategy and writerThreadsPerTask? They tune writes in Private Link environments, where the connector uses JDBC batch inserts rather than external staging. writeStrategy chooses between multi-threaded and single-threaded staging writes, and the multi-threaded option is selected automatically. writerThreadsPerTask defaults to three; raising it adds JDBC connections and executor memory pressure, so change it carefully.
Why does my write succeed but the table never changes? Almost always the save mode. The ignore mode skips the write without raising an error when the target table exists. Every run looks healthy in the history while nothing actually updates. Switch to append or overwrite, depending on whether you are adding rows or replacing the table contents. Append adds rows, while overwrite replaces the contents of the table.
Can I query warehouse history or time travel through the Spark connector? No. Microsoft notes that although Fabric Warehouse itself supports time travel, this connector does not work with time travel query syntax. Run those queries in T-SQL against the warehouse instead. Delta time travel in a lakehouse is a separate mechanism with its own syntax and its own retention rules. Spark returns an error rather than an older snapshot.
Can Spark notebooks write data to a warehouse in Fabric? Yes, through the Spark connector for Fabric Data Warehouse running on Runtime 2.0. The connector stages the DataFrame into intermediate storage and then issues COPY INTO to load the warehouse table, which is why it scales with volume. It can create, append to and overwrite tables, but never update or merge existing rows.
Does Fabric support Spark? Yes. Microsoft Fabric runs a managed Apache Spark service behind notebooks and Spark job definitions. It provides pools, Delta Lake and support for Python, Scala, R and SQL. Runtime 2.0 ships Apache Spark 4.1 and Delta Lake 4.2. It also includes the Native Execution Engine, which Microsoft measures at up to six times the speed of open-source Spark.
Which IDE is best for PySpark? For Fabric work the built-in notebook experience is usually the fastest path, because the connector, authentication and lakehouse context are already wired up for you. If you prefer a local editor, the VS Code extension for Fabric Spark supports Runtime 2.0 for notebooks and Spark job definitions. It also keeps source control simpler.
Is Fabric better than Databricks? They solve overlapping problems differently. Fabric suits organisations already standardised on Microsoft, where Power BI, governance and SQL warehousing matter as much as Spark itself. Databricks tends to win on heavy, code-first data engineering and machine learning at scale. The honest answer depends on your existing estate, your team’s skills and what you already pay for.
Is PySpark good for ETL? It is a strong fit when transformations need Python libraries, or when data volumes justify distributed processing. It also helps when you are combining sources that do not all live in one engine. For work that is joins, filters and aggregation inside a warehouse, T-SQL is usually simpler to write, cheaper to run and quicker to finish.
Is PySpark harder than Pandas? It has more moving parts. Pandas runs in one process on one machine. PySpark distributes work across executors, which introduces partitioning, lazy evaluation and shuffles you have to reason about. The DataFrame API will feel familiar if you know Pandas, but debugging and performance tuning are genuinely different skills that take time to build.
Is PySpark an API or a library? PySpark is the Python API for Apache Spark. You can install it as a Python package, but what it actually gives you is a binding onto the Spark engine running on the JVM. In Microsoft Fabric you do not install anything, because PySpark is part of the managed runtime that backs every notebook.
