Real-time intelligence has moved from a competitive edge to an operational necessity. A logistics team getting a delay alert six hours late has already missed the rerouting window. A fraud system running hourly batch jobs has already processed thousands of transactions through the gap.
Microsoft Fabric’s Real-Time Intelligence stack is built to close that gap. Eventstream connects event sources, KQL Database stores and queries them with millisecond access, Activator fires automated responses, and native dashboards refresh against live data.
In this article, we’ll cover how the Fabric RTI stack works, how KQL handles streaming data differently from SQL, building a pipeline step by step, use cases across industries, implementation challenges, and how Kanerika deploys these systems for enterprise clients.
Key Takeaways Batch analytics creates a fixed visibility lag; by the time a cycle completes, the window for time-sensitive action has already closed. The Fabric RTI stack is a connected system: Eventstream routes events, KQL Database queries them, Activator fires automated responses, and dashboards display live data. KQL is purpose-built for time-series workloads with native operators for time bucketing, rolling windows, anomaly detection , and forecasting that SQL cannot replicate without workarounds. Real-time and batch analytics are complementary; batch remains the better choice for historical reporting, ML training, and cost-sensitive workloads. Pipeline architecture decisions made at setup, including raw tables, update policies, and materialized views, directly determine production performance and query latency. Governance and capacity planning belong in the pipeline design, not after go-live; streaming ingestion draws more compute and retention decisions affect compliance from day one.
Microsoft Fabric Real-Time Intelligence Services Kanerika builds production-ready Fabric RTI solutions, including Eventstream, KQL Databases, Activator, and governance for enterprise-scale operations.
See Fabric in Practice
Why Batch Analytics Creates a Visibility Gap Your Business Pays For Batch processing works by collecting data over a period, processing it all at once, and storing the results for analysis. That model was built for a world where insights took days to act on and data volumes were manageable. The business environment has changed around it.
1. Blind Spots In Every Batch Cycle Batch pipelines process data in fixed windows. Anything that happens between two batch runs is invisible to the systems depending on that data. A fraud pattern that starts at 2:17pm stays undetected until the 3:00pm job runs.
A machine running outside safe parameters at 8:14am keeps running until the 9:00am batch surfaces it. The gap between when something happens and when the business sees it is structural, and it compounds across every system running on the same schedule.
2. The Business Cost of Delayed Insight According to Grand View Research , the global streaming analytics market was valued at USD 23.4 billion in 2023 and is projected to reach USD 128.4 billion by 2030, at a CAGR of 28.3% from 2024 to 2030.
The cost of that lag varies by context. In fraud detection, seconds determine financial exposure. In supply chain, a one-hour delay on a logistics alert can cascade into missed delivery windows and penalties.
3. Operational Areas Most Exposed to Batch Latency The batch visibility gap is most costly in three categories: time-bound decisions (fraud alerts, safety thresholds, inventory depletion alerts), high-frequency event sources that outpace refresh cycles (IoT telemetry, transaction feeds, application logs), and operational workflows where teams need live data to act rather than scheduled reports to review. These are the workloads where event streaming with KQL replaces batch as the right architecture.
How Microsoft Fabric Handles Event Streaming: The RTI Stack Microsoft Fabric’s Real-Time Intelligence is a set of components that work together as a connected pipeline. Understanding what each component does prevents the common mistake of treating the whole stack as one monolithic tool.
Here is how the five components fit together:
Component Role Example Event Sources Feed live data into the pipeline IoT Hub telemetry, transaction events, database CDC feeds Eventstream Collect, route, and filter events Drops PII fields; fans out to KQL Database and Lakehouse KQL Database Store and query events at low latency Time-bucketed aggregations, anomaly detection, threshold alerts Activator Trigger automated actions from conditions Fires Teams alert when temperature exceeds a set threshold Real-Time Dashboards Display live operational metrics Operations floor dashboard refreshing every 5 seconds
1. Event Sources: Supported Connectors and Schema Impact Eventstream connects to over 25 sources as of early 2026. These include the standard event streaming sources most teams start with:
Azure-native: Event Hubs, IoT Hub, Event Grid, Service BusOpen-source / multi-cloud: Apache Kafka, Confluent Cloud Kafka, Amazon Kinesis, Amazon MSK, Google Cloud Pub/SubIoT protocols: MQTT (v3.1/v3.1.1), Azure IoT Operations, custom HTTPS endpointsDatabase CDC: Azure SQL DB, PostgreSQL, MySQL, MongoDB, SQL Server on VM, Azure SQL Managed Instance
The CDC sources are significant for enterprise teams. Change Data Capture streams database changes as events in real time, letting teams treat relational database updates as streaming events without rewriting application code. Schema is handled automatically via DeltaFlow, which autodiscovers source table schemas and adjusts destination tables when the source changes.
2. Eventstream: Routing and Light Transformation Eventstream is the ingestion and routing layer. It collects events from sources, optionally applies lightweight processing (filtering, projection, aggregation, deduplication, windowed joins), and routes output to one or more destinations simultaneously. A single Eventstream can feed a KQL Database, a Fabric Lakehouse , and Activator at the same time.
Eventstream is also where PII and sensitive fields should be filtered. Data removed at this layer never touches storage. A new direct ingestion mode, released as GA in January 2026, allows Eventstream to land events straight into Eventhouse without a processing step. This is useful for high-throughput scenarios where raw event storage takes priority over transformation.
3. KQL Database: The Storage and Analytics Layer Each Eventhouse is the container within Fabric that holds one or more KQL Databases. The Eventhouse handles workspace-level infrastructure; the KQL Database is where events are actually stored and queried. This distinction matters when configuring capacity and retention, which are set at the KQL Database level, not the Eventhouse level.
The storage model uses both an in-memory row store for recent events (accessible in milliseconds) and compressed columnar extents for longer-term data. KQL Databases also support T-SQL queries, so teams more comfortable with SQL can query without learning KQL from scratch. The advanced time-series functions are KQL-native and have no T-SQL equivalents.
4. Activator: From Insight to Automated Action Activator monitors query outputs and fires actions when conditions are met. Those actions include Teams notifications, email alerts, and Power Automate flows. The result is a pipeline that surfaces insights and acts on them automatically, without a human in the loop. A Power BI dashboard requires someone to be watching. Activator fires regardless.
5. Real-Time Dashboards: Live Business Visibility Fabric’s native real-time dashboards auto-refresh against KQL queries, down to a one-second interval. Note that Microsoft is retiring legacy Power BI streaming datasets; new streaming datasets cannot be created after October 2027, making Fabric Real-Time Intelligence the forward path for all live data scenarios. Real-time dashboards in Fabric are the replacement.
Event Streaming With KQL vs. Batch Processing: When to Use Which Real-time and batch are complementary, not competing. Knowing which to use is a more useful framework than treating real-time as universally better.
Dimension Batch Processing Event Streaming With KQL Data freshness Fixed refresh schedule (hourly, daily) Continuous; sub-second to seconds Latency Minutes to hours Milliseconds to seconds Query model Runs against snapshots Runs against live and historical data Compute cost Lower; scheduled and predictable Higher for streaming ingestion mode Best fit Historical reporting, trend analysis, ML training Alerts, operational monitoring, fraud detection Architecture complexity Lower Higher; streaming governance and schema evolution
1. When Batch Processing Is The Right Choice Batch handles workloads where latency is acceptable and cost efficiency is the priority. Monthly financial reporting, historical trend analysis, ML training datasets, and complex joins across slowly-changing dimension tables all fit batch naturally. Replacing a working batch pipeline with event streaming purely for the sake of modernization is a weak justification on its own.
2. When Event Streaming With KQL Is Worth The Complexity Real-time is worth the additional architecture overhead in four scenarios: time-sensitive decisions like fraud detection and safety thresholds, high-frequency event sources like IoT telemetry and transaction feeds, operational workflows where teams act on live data such as logistics coordination and clinical monitoring, and automation scenarios where Fabric Activator removes human latency from the loop entirely.
3. Three Questions To Guide Your Fabric RTI Decision Before committing to a streaming architecture, three questions help clarify whether it fits. Does the decision window actually require sub-minute data, or is hourly refresh sufficient? Can the source systems produce a continuous event stream at the required volume? Does the team have the capacity to manage streaming ingestion, schema evolution, and governance requirements over time? If the answer to any of these is unclear, batch remains the lower-risk starting point.
How KQL Handles Streaming Data Differently From SQL SQL was designed for relational tables and set-based operations. It handles structured queries against static or slowly-changing datasets well. Streaming data is the opposite: it arrives continuously, carries timestamps, and the most valuable queries are about time windows, rates of change, and patterns across sequences.
1. Streaming Data and the Time-Series Engine Advantage KQL’s data model centers on time. Every table has an ingestion timestamp, queries reference time natively, and the query planner optimizes for time-range scans. bin() bucketing, ago() rolling windows, and time-series operators run efficiently against high-volume event tables without the workaround syntax SQL requires.
The underlying storage reinforces this. Fresh events hit the row store first for millisecond access, then compact into columnar extents for efficient historical queries. SQL databases apply uniform storage regardless of data freshness, which creates a performance ceiling on streaming workloads.
2. The KQL Operators That Matter for Event Workloads A focused set of operators covers most streaming analytics patterns. Aggregation and windowing with summarize and bin() form the foundation of every streaming dashboard:
// Time-binned aggregation
SensorEvents
| where EventTime > ago(1h)
| summarize AvgTemp = avg(Temperature), EventCount = count()
by bin(EventTime, 5m), SensorId
| order by EventTime desc
For trend and anomaly analysis, make-series is the correct operator. It generates regular time buckets from raw events and handles missing values automatically, which summarize with bin() alone cannot achieve:
// Regular time-series for anomaly detection
SensorEvents
| make-series AvgTemp = avg(Temperature)
on EventTime from ago(24h) to now() step 5m
by SensorId
| extend (anomalies, score, baseline) = series_decompose_anomalies(AvgTemp)
| mv-expand EventTime, AvgTemp, anomalies, score
| where anomalies == 1
| project EventTime, SensorId, AvgTemp, score
Beyond anomaly detection, series_fit_line() catches linear drift for predictive maintenance, series_decompose_forecast() projects future values, and series_fir()/series_iir() apply moving averages for signal filtering.
Update Policies parse raw JSON at ingestion time rather than on every query, which keeps query performance consistent at scale:
// Parse once at ingestion, not on every query
.create function ParseDeviceEvents() {
RawDeviceEvents
| extend payload = parse_json(Body)
| project
EventTime = todatetime(payload.timestamp),
DeviceId = tostring(payload.device_id),
Temp = todouble(payload.readings.temperature),
Pressure = todouble(payload.readings.pressure)
}
.alter table DeviceReadings policy update
@'[{"IsEnabled": true, "Source": "RawDeviceEvents",
"Query": "ParseDeviceEvents()", "IsTransactional": false}]'
Materialized Views pre-aggregate data so dashboard queries hit a summary table instead of scanning millions of raw events:
// Dashboard queries hit this, not the raw table
.create materialized-view HourlyDeviceSummary on table DeviceReadings {
DeviceReadings
| summarize AvgTemp = avg(Temp), MaxTemp = max(Temp), ReadingCount = count()
by bin(EventTime, 1h), DeviceId
}3. Query Performance at Enterprise Scale Materialized views and hot cache sizing are the two levers that determine whether dashboards are fast or slow. Hot cache should cover the time range dashboards actually query, typically the last 3–7 days for operational workloads, not the full retention window.
For production deployments, Microsoft recommends an F64 Fabric SKU as the practical minimum. Below that, Eventstream and Eventhouse workloads share capacity with other Fabric items, which creates throttling under streaming load. Set CU throttling alerts in the Fabric Admin portal before production traffic arrives.
7 Benefits of Real-Time Intelligence in Microsoft Fabric Explore how Real-Time Intelligence in Microsoft Fabric turns streaming data into actionable insights with Eventstream, KQL, and automation.
Learn More
Building an Event Streaming Pipeline in Microsoft Fabric The setup for a Fabric RTI pipeline follows a consistent sequence regardless of use case.
1. Connect Event Sources to Eventstream From the Fabric workspace, navigate to the Real-Time Intelligence experience and create an Eventstream. Add a source from the available connectors. Fabric’s built-in sample data streams let you test the full pipeline without an external source dependency.
The source type shapes schema design. Kafka sources produce byte payloads that need explicit parsing. Event Hubs and IoT Hub typically produce JSON, which Eventstream can partially structure before routing. CDC sources from relational databases arrive in a structured tabular format, analytics-ready without additional parsing.
2. Route, Filter, and Transform Before Storage Use Eventstream processing to filter high-noise or low-value events, drop PII fields, and fan out to multiple destinations simultaneously. The design principle: reduce data volume at the Eventstream layer; reshape and type-cast structure inside KQL via update policies.
Doing heavy parsing at the Eventstream layer limits reuse if multiple downstream destinations need the raw form of the event.
3. Write KQL Queries for Dashboards and Alerts Define update policies to parse raw events into typed columns at ingestion time. Build materialized views for any aggregation queried more than a few times per minute. Threshold queries that feed Activator should run against the materialized view layer, not the raw event table.
For a full update policy and materialized view example, see the code blocks in the KQL section above.
4. Automate Responses With Activator In Activator, define conditions against KQL query outputs and specify the action: Teams message, email, or Power Automate trigger. Activator evaluates conditions continuously, so responses fire in real time without scheduled checks.
5. Connect Power BI for Business-Level Reporting Real-time dashboards work well for operations teams who need sub-minute data. For business users in Power BI, Direct Query mode against the KQL Database provides low-latency refreshes. Point Power BI queries at materialized views rather than base tables to keep performance consistent under load.
For the data integration context, Data Integration with Microsoft Fabric covers how Eventstream fits alongside other ingestion patterns.
Real-World Event Streaming Use Cases Across Industries 1. IoT Monitoring and Predictive Maintenance Manufacturing and energy clients stream sensor telemetry (temperature, vibration, pressure, flow rate) into KQL Database via Eventstream. KQL’s make-series and series_decompose_anomalies() operators detect equipment degradation patterns before failure occurs. Activator fires maintenance work orders automatically when anomaly scores cross defined thresholds, shifting from reactive to predictive maintenance.
2. Fraud Detection and Transaction Monitoring Financial services clients stream transaction events from payment processors and banking systems. KQL queries detect velocity patterns, geographic inconsistencies, and known fraud signatures. The pipeline from event ingestion to alert delivery operates in seconds. Microsoft Fabric in Finance covers the compliance and data residency architecture for regulated environments.
3. Supply Chain and Logistics Visibility GPS, status, and condition events from fleets and warehouses stream into KQL. Time-binned aggregations surface ETA deviations, dwell-time violations, and temperature excursions on condition-sensitive shipments. Activator fires when a shipment falls outside acceptable parameters, giving teams time to reroute rather than explain.
4. Application Performance Monitoring Development teams stream application logs, error events, and performance metrics into KQL. Queries detect error rate spikes, latency degradation, and dependency failures across distributed services in near real time, replacing or augmenting dedicated APM tools within the same governed data platform.
5. Customer Behavior Analytics E-commerce and SaaS companies stream clickstream and usage events into KQL. Queries track funnel drop-offs, feature adoption rates, and session anomalies in near real time. Marketing and product teams get live data rather than next-morning reports, enabling faster responses to campaign performance and product issues.
Implementation Challenges and How to Address Them 1. Managing High Event Volumes Without Overrunning Capacity Units Microsoft Fabric bills in Capacity Units. Streaming ingestion mode (for sub-second latency) draws more compute than batched ingestion, which uses a default 5-minute batching cycle. Teams that enable streaming ingestion across all tables frequently hit unexpected CU consumption. Use streaming ingestion only where sub-30-second latency genuinely matters, such as fraud detection and safety alerting. Default to batched ingestion for everything else.
2. KQL Query Performance Under Sustained Load Materialized views and hot cache sizing are the primary performance levers. Hot cache should cover only the time range dashboards actually query, typically the last 3–7 days for most operational workloads. A dashboard that scans 90 days of hot-cached data on every refresh is slower and more expensive than one that queries a well-built materialized view.
3. Data Retention and Governance in Streaming Pipelines Streaming data accumulates fast. Retention policies on KQL tables need deliberate configuration before production traffic starts. Governance decisions about who can query what, how long PII can be stored, and what audit trails are required all belong in the pipeline design, not as post-launch additions. Data Governance with Microsoft Purview covers the governance integration with Fabric RTI.
4. Capacity Planning Before Go-Live Fabric RTI performance scales with Capacity Unit allocation. Teams that size capacity for expected average load frequently hit throttling during event volume spikes. An F64 SKU is the practical minimum for production streaming workloads. Set capacity throttling alerts in the Fabric Admin portal before go-live. Understanding Microsoft Fabric Pricing covers the CU billing model across all Fabric workloads.
How Kanerika Builds Real-Time Analytics Solutions on Microsoft Fabric Kanerika has deployed Fabric RTI pipelines across manufacturing, logistics, financial services, and healthcare clients. As a Microsoft Solutions Partner for Data & AI with Analytics Specialization and Microsoft Fabric Featured Partner , Kanerika brings production delivery experience across the full RTI stack.
With 300+ professionals, 100+ enterprise clients, and 98% client retention, Kanerika handles architecture design, pipeline implementation, governance setup, and ongoing optimization. The KARL AI agent runs on top of Fabric RTI stacks for clients who want natural language queries against streaming data, removing the KQL requirement entirely for operations teams. Kanerika’s KANGuard and KANComply frameworks extend Microsoft Purview governance into KQL Database environments from day one.
Real-World Impact: 98% Efficiency Gain With Microsoft Fabric Data Management Challenges An enterprise client was managing data across fragmented systems with poor visibility, manual processes, and no unified governance layer. Reporting was slow, data quality was inconsistent, and teams had limited ability to act on timely insights.
Solution Kanerika implemented Microsoft Fabric to consolidate the client’s data management environment into a single platform. The engagement covered data engineering, lakehouse architecture, pipeline automation, and governance setup, replacing disconnected tools with a governed, unified data layer.
Results 55% increase in operational efficiency 98% improvement in data system scalability 31% decrease in data processing costs
Wrapping Up Event streaming with KQL in Microsoft Fabric closes the visibility gap that batch analytics leaves open. Eventstream routes events from any source into a KQL Database built for time-series workloads, Activator fires automated responses without human latency, and real-time dashboards give operations teams continuous visibility.
The architecture suits time-sensitive operational decisions best; batch remains the right choice for historical analysis and cost-sensitive workloads. For organizations ready to assess whether Fabric RTI fits their current environment and workload requirements, schedule a consultation with Kanerika’s Microsoft Fabric team .
Ready to Harness Real-Time Data at Scale? Kanerika Delivers End-to-End Event Streaming and Analytics Solutions on Microsoft Fabric.
Book a Meeting
FAQs 1. What is Event Streaming With KQL in Microsoft Fabric? Event Streaming With KQL in Microsoft Fabric enables organizations to collect, process, analyze, and act on streaming data as it is generated. By combining Eventstream with KQL databases, businesses can continuously monitor operational events from applications, IoT devices, websites, and enterprise systems. This reduces the delay between data generation and insight, helping teams make faster and more informed decisions.
2. What is KQL and why is it important for real-time analytics? KQL, or Kusto Query Language, is Microsoft’s query language designed for high-speed analysis of large volumes of data. It allows users to search, filter, aggregate, and visualize streaming information with minimal latency. In Microsoft Fabric, KQL plays a central role in identifying trends, detecting anomalies, and generating operational insights from continuously arriving event data.
3. How does Eventstream work in Microsoft Fabric? Eventstream serves as the real-time data ingestion layer within Microsoft Fabric. It can capture events from multiple sources, including Azure Event Hubs, Kafka, IoT platforms, applications, and databases. Once ingested, events can be filtered, transformed, and routed to KQL databases or other Fabric destinations for analysis. This creates a seamless pipeline for processing and analyzing live data.
4. What are the benefits of using KQL for event streaming analytics? KQL is optimized for fast, interactive analysis of large-scale streaming datasets. It enables organizations to investigate events, identify patterns, and uncover anomalies in near real time without building complex processing pipelines. Its scalability, performance, and ease of querying make it particularly valuable for operational monitoring, security analytics, IoT workloads, and business intelligence initiatives.
5. Which business use cases benefit from Event Streaming With KQL in Microsoft Fabric? Organizations use Event Streaming With KQL for a wide range of scenarios, including predictive maintenance, fraud detection, application monitoring, supply chain visibility, customer behavior tracking, and operational intelligence. Any use case that requires immediate visibility into changing conditions can benefit from real-time event processing and analytics capabilities.
6. How does Event Streaming With KQL differ from traditional batch analytics? Traditional analytics processes data at scheduled intervals, which can introduce delays between an event occurring and insights becoming available. Event Streaming With KQL analyzes events as they happen, allowing organizations to detect issues, identify opportunities, and trigger actions in near real time. This makes it particularly useful for operational environments where timely decisions are critical.
7. Can Event Streaming With KQL support enterprise-scale workloads? Yes. Microsoft Fabric is built to handle high-volume streaming data from multiple sources across the enterprise. KQL databases are designed for fast ingestion and querying of millions of events, making them suitable for large-scale monitoring, analytics, and observability workloads. Organizations can scale their real-time analytics capabilities without managing separate infrastructure components.
8. How can organizations get started with Event Streaming With KQL in Microsoft Fabric? The best starting point is to identify a business process that would benefit from real-time visibility, such as monitoring operations, tracking customer activity, or analyzing IoT data. Organizations can then connect event sources to Eventstream, route data into a KQL database, and create dashboards or alerts for analysis. Beginning with a focused use case allows teams to demonstrate value before expanding real-time analytics across the enterprise.
