Microsoft Fabric Real-Time Intelligence: Building Live Enterprise Dashboards Without a Data Team

For most Malaysian enterprises, real-time analytics has been the domain of companies with dedicated data engineering teams, Kafka clusters, and Spark expertise. Microsoft Fabric's Real-Time Intelligence capabilities change this equation for many Microsoft-centric analytics teams. Business analysts, application developers, and operations teams can build streaming dashboards and live monitoring solutions without running Kafka or Spark infrastructure themselves — although production designs still need data ownership, governance, and operational accountability.

This article walks through what Real-Time Intelligence in Fabric actually is, when it makes sense, how to use it with real KQL examples, and what Malaysian businesses should consider before adopting it.

What Is Real-Time Intelligence in Fabric?

Microsoft Fabric's Real-Time Intelligence hub is a set of integrated services that ingest, process, analyze, and visualize streaming data — all within the same SaaS platform as your data lake, data warehouse, and Power BI reports.

The core components are:

Eventhouse

Eventhouse is a managed Kusto Query Language (KQL) database optimized for time-series, streaming, semi-structured, and log-style data. Think of it as a purpose-built database for "data that is happening right now" — IoT sensor readings, API logs, application traces, and security events. Microsoft positions Eventhouse for storing and analyzing streaming data, querying billions of events in seconds, and organizing data for fast search based on arrival time.

Key capabilities: - Data is organized for fast searching based on when it arrived, which fits telemetry and log analytics patterns. - Materialized views for pre-computed aggregates. Microsoft documents materialized views as an optimization pattern; validate performance improvement against your own workload rather than relying on a universal percentage. - Governance through Fabric workspace/item permissions, auditability, lineage, and security patterns such as governed views/functions or downstream semantic-model controls where sensitive fields are involved. - Integration with Fabric and Power BI reporting experiences without building a separate streaming database stack.

Real-Time Dashboards

These are KQL-powered dashboards that connect to KQL databases and can be configured with auto-refresh for operational monitoring. Unlike scheduled BI reports, Real-Time Dashboards are designed for low-latency exploration and live operations views; set refresh frequency based on query cost, CU consumption, and the latency the business actually needs.

Eventstreams

Eventstreams is a visual, no-code editor for building streaming data pipelines. You connect a source (Azure Event Hubs, IoT Hub, Kafka, or an HTTPS endpoint), optionally apply transformations (filter, project, aggregate), and send the result to one or more destinations (Eventhouse, Lakehouse, or Real-Time Dashboard).

Mirrored Databases

Mirroring in Fabric lets you create low-latency replicas of supported operational databases and external sources into Fabric OneLake. Microsoft documents support for sources such as Azure SQL Database, Azure SQL Managed Instance, Azure Cosmos DB, Azure Database for PostgreSQL, Azure Database for MySQL (preview), SQL Server, Snowflake, SAP, Oracle, and others. Treat replication as near real-time rather than guaranteed seconds: latency depends on source/destination region, change volume, network latency, and compute resources. This is useful for operational reporting because analytical queries can target the mirrored data rather than the production system.

Why This Matters for Malaysian Enterprises

Malaysia has a strong manufacturing sector (IoT sensor monitoring), a growing fintech scene (real-time transaction monitoring), and increasing adoption of cloud-based ERP systems (Operational reporting). All of these generate streaming data, but most organizations lack the specialized skills to build and maintain traditional streaming infrastructure.

Step-by-Step: Building a Real-Time IoT Monitoring Dashboard

Let us walk through a concrete scenario: a Malaysian manufacturing company wants to monitor equipment temperature, humidity, and vibration from 200 IoT sensors on the factory floor.

Step 1: Create an Eventhouse

In the Fabric portal, navigate to Real-Time Intelligence, select "New" → "Eventhouse". Name it FactoryTelemetry and place the workspace/capacity in the approved data residency region for the organization. If Malaysia West is available for your tenant and Fabric workload, use it; otherwise validate Fabric regional availability and compliance requirements before deployment.

// After creation, verify the Eventhouse is online
.show databases

// Output should show your Eventhouse database

Step 2: Create an Eventstream

Eventstreams use a visual interface for connecting event sources, applying supported transformations, and routing events to destinations. The implementation flow is:

1. Add source — choose Azure IoT Hub, Event Hubs, or the custom HTTPS endpoint. For the factory scenario, IoT Hub is the natural choice because it supports device authentication and twin management.
2. Configure transformation — add a filter step to discard malformed messages, add a project step to select only needed fields.
3. Add destination — select your Eventhouse and specify the target table.

// Conceptual Eventstream configuration (illustrative only; validate against Fabric export/API support before treating as deployable IaC)
{
"sources": [
{
"type": "iotHub",
"name": "factory-sensors",
"iotHubName": "iothub-factory-malaysia",
"consumerGroup": "fabric-ingest"
}
],
"transformations": [
{
"type": "filter",
"name": "filter-in-range",
"expression": "temperature >= -20 and temperature <= 150"
}
],
"destinations": [
{
"type": "eventhouse",
"name": "factory-telemetry",
"eventhouseName": "FactoryTelemetry",
"databaseName": "FactoryTelemetry",
"tableName": "IoTReadings"
}
]
}

Step 3: Create the Target Table with KQL

Before data can flow into Eventhouse, you must define the table schema:

// Define the table schema for IoT sensor readings
.create table IoTReadings (
timestamp: datetime,
deviceId: string,
temperature: real,
humidity: real,
vibration: real,
status: string,
factoryZone: string
)

// Define JSON mapping for incoming data
.create table IoTReadings ingestion json mapping 'IoTMapping'
'['
' {"column":"timestamp", "path":"$.timestamp"},'
' {"column":"deviceId", "path":"$.device.id"},'
' {"column":"temperature", "path":"$.telemetry.temperature"},'
' {"column":"humidity", "path":"$.telemetry.humidity"},'
' {"column":"vibration", "path":"$.telemetry.vibration"},'
' {"column":"status", "path":"$.device.status"},'
' {"column":"factoryZone", "path":"$.metadata.zone"}'
']'

// Create a materialized view for hourly aggregates
.create materialized-view HourlyAggregates on table IoTReadings
{
IoTReadings
| summarize
avg_temp = avg(temperature),
max_temp = max(temperature),
avg_humidity = avg(humidity),
device_count = dcount(deviceId)
by bin(timestamp, 1h), factoryZone
}

Step 4: Create a Real-Time Dashboard

In the Fabric portal, select "New" → "Real-Time Dashboard". Add tiles using KQL queries:

// Tile 1: Live temperature chart (last 15 minutes)
IoTReadings
| where timestamp > ago(15m)
| where status == "online"
| summarize avg(temperature) by bin(timestamp, 10s), deviceId
| render timechart

// Tile 2: Anomaly detection — devices exceeding temperature threshold
IoTReadings
| where timestamp > ago(5m)
| where temperature > 60
| project timestamp, deviceId, temperature, factoryZone
| sort by timestamp desc

// Tile 3: Device count by zone (live count)
IoTReadings
| where timestamp > ago(5m)
| summarize distinct_devices = dcount(deviceId) by factoryZone
| render columnchart

// Tile 4: 5-minute moving average with anomaly bands
let timeWindow = 5m;
IoTReadings
| where timestamp > ago(1h)
| summarize
avg_temp = avg(temperature),
percentile_95_temp = percentile(temperature, 95),
percentile_05_temp = percentile(temperature, 5)
by bin(timestamp, 1m), factoryZone
| render timechart with(ysplit=panels)

The dashboard auto-refreshes without manual refresh. You can pin individual tiles to a Power BI report for embedding in executive dashboards.

Step 5: Set Up Alerts

For alerting, use Fabric Activator or dashboard alerting patterns instead of assuming alert rules are authored directly inside Eventhouse. Activator can monitor streaming data or dashboard/report conditions and trigger actions such as email, Teams notifications, or Power Automate flows:

// Query to identify overheating devices
IoTReadings
| where timestamp > ago(2m) and temperature > 70
| project
AlertTime = now(),
DeviceId = deviceId,
Temperature = temperature,
Zone = factoryZone,
Message = strcat("Device ", deviceId, " in zone ", factoryZone,
" is overheating at ", strcat(tostring(temperature), "°C"))

Use this query as the condition behind the dashboard or Activator rule, then route notifications through Teams, email, Power Automate, or another approved workflow.

Mirrored Databases for Operational Reporting

The Mirroring capability deserves special attention. It creates a low-latency replica of supported operational data in Fabric without building a separate ETL pipeline, but source prerequisites and latency must be validated.

Setting Up a Mirrored Database for PostgreSQL

For Azure Database for PostgreSQL, use the Fabric mirroring workflow documented by Microsoft and confirm source prerequisites before production use. A safe implementation outline is:

1. In Fabric, create a mirrored database for Azure Database for PostgreSQL.
2. Create or select a secure connection to the source database.
3. Select the databases/tables supported by the mirroring workflow.
4. Validate replication health and latency under expected write volume.
5. Query the mirrored data from Fabric analytics experiences such as SQL analytics endpoints, Power BI, notebooks, or other supported Fabric workloads.

Example analytical query pattern after data is available in Fabric:

SELECT
DATEADD(hour, DATEDIFF(hour, 0, LastModified), 0) AS hour_bucket,
OrderStatus,
SUM(Amount) AS TotalRevenue
FROM dbo.orders
WHERE LastModified > DATEADD(day, -1, SYSUTCDATETIME())
GROUP BY DATEADD(hour, DATEDIFF(hour, 0, LastModified), 0), OrderStatus;

Use cases for mirrored databases in Malaysia: - ERP reporting with reduced production impact — mirror a supported operational source into Fabric and run analytical queries against the mirrored data instead of the production database. Validate source support, licensing, and replication overhead for systems such as Dynamics, SAP, SQL Server, or PostgreSQL. - Cross-database joins — join mirrored CRM data with mirrored inventory data to build a real-time order fulfillment dashboard. - Compliance queries — auditors need access to historical data without touching production. Point them at the mirror.

Cost Model and Capacity Planning

Fabric uses Capacity Units (CU). Real-Time Intelligence consumes CU for:

• Ingestion — each MB of data ingested consumes CU based on the source and transformations.
• Querying — each KQL query consumes CU proportional to the data scanned.
• Storage — Eventhouse data is stored in Fabric OneLake, billed separately from CU.

Capacity Sizing Guide

Note: The estimates below are based on Fabric pay-as-you-go capacity pricing. Actual costs depend on CU consumption patterns, data ingestion volume, and query complexity. Monitor usage with the Fabric Capacity Metrics app.

Cost Optimization Strategies

Use materialized views. Instead of scanning raw data for every dashboard query, pre-compute hourly or daily aggregates. Treat materialized views as a workload-specific optimization: measure ingestion cost, query latency, and refresh behaviour before declaring the improvement.

// Compare: raw query (expensive)
IoTReadings
| where timestamp > ago(7d)
| summarize avg(temperature) by bin(timestamp, 1h)
// This scans 7 days of raw data per refresh

// vs. materialized view (cheap)
HourlyAggregates
| where timestamp > ago(7d)
// This scans 168 rows instead of millions

Set data retention limits. Kusto databases support retention and caching policies. For the factory use case, consider keeping raw hot-path data for 7 days and longer-term aggregates for 1 year, then validate against regulatory and operational requirements:

In the Fabric Eventhouse UI, you can configure retention and caching policies per table: - IoTReadings: Set retention to 7 days (raw telemetry, short-lived) - HourlyAggregates: Set retention to 365 days with a 30-day hot cache

These settings are configured through the Eventhouse table management interface rather than KQL commands.

Use caching policies. Configure hot-cache policy deliberately for frequently queried data. Do not assume a universal default is right for every workload; tune cache duration against query latency and capacity consumption.

Comparison with Traditional Streaming Solutions

Before Fabric Real-Time Intelligence, Malaysian organizations building real-time dashboards typically used one of these approaches:

Apache Kafka + Spark Streaming

• Pros: Maximum flexibility, battle-tested, works with any data source.
• Cons: Requires Kafka operations skills, broker/controller management, stream-processing skills, and a separate monitoring stack unless you use a managed service. VM-based cost varies widely by sizing, storage, replication factor, support model, and region before any stream-processing compute.
• Best for: Organizations with existing Kafka investment or complex stream processing needs (joins across multiple streams, stateful aggregations, ML inference in-stream).

Azure Stream Analytics

• Pros: SQL-based, managed service, native connections to Azure services.
• Cons: Job-oriented (not dashboard-oriented), you still need a separate visualization layer. Output must go to a database or Power BI.
• Best for: Simple ETL patterns where you need to filter and route streaming data to multiple destinations.

Fabric Real-Time Intelligence

• Pros: SaaS analytics platform, no self-managed streaming cluster for common scenarios, KQL-based, built-in dashboards, unified with the rest of Fabric.
• Cons: Tied to Microsoft ecosystem, limited to Fabric's CU model for costing, some advanced processing patterns (exactly-once semantics, stateful joins) require careful design.
• Best for: Organizations that want a unified analytics platform and need to get a real-time dashboard running in weeks, not months.

Security and Governance Considerations

For Malaysian enterprises, data governance is not optional. Fabric Real-Time Intelligence provides:

• Access control and least privilege — use Fabric workspace roles, item permissions, and governed sharing rather than giving broad workspace access to every dashboard user.
• Sensitive-field minimization — avoid exposing personally identifiable or commercially sensitive columns in operational dashboards. Use governed views/functions, semantic-model security, or separate curated datasets where appropriate.

// Example: expose a governed view that omits sensitive telemetry fields
.create-or-alter function ReadOnlyUserView () {
IoTReadings
| where timestamp > ago(7d)
| project timestamp, deviceId, status // Exclude temperature and humidity columns
}

• Row-level and object-level security where supported — apply the relevant Fabric/Power BI/Mirrored Database security feature for the item type being served.
• Audit logging and lineage — use Fabric audit and lineage features to understand who accessed data and how data moved from source to dashboard.

Common Pitfalls to Avoid

Pitfall 1: Treating Eventhouse like a data lake. Eventhouse is optimized for time-series queries over recent data. Do not use it as the primary storage for data older than your retention policy. Archive historical data to the Fabric Lakehouse and use Eventhouse for the hot path only.

Pitfall 2: Over-provisioning capacity upfront. Start with F2 or F4 capacity. Monitor CU consumption in the Capacity Metrics app for two weeks before scaling up. Most workloads consume less CU than expected.

Pitfall 3: Ignoring data structure for mirrored databases. Mirrored databases replicate the source schema exactly, including any messy column names, nullable fields, or missing indexes. Plan for data cleanup in your KQL queries rather than trying to enforce schema changes on the source.

Pitfall 4: Real-time for the sake of real-time. Not every dashboard needs sub-second refresh. If your business decisions are made daily, a scheduled Power BI report or batch dashboard is usually cheaper than a Real-Time Dashboard. Use Real-Time Intelligence only for the subset of dashboards that genuinely need it.

Key Takeaways

1. Real-Time Intelligence reduces the need to operate dedicated streaming infrastructure — no self-managed Kafka or Spark cluster for common dashboard scenarios, but production ownership and data governance remain necessary.
2. Eventhouse is the core engine — KQL databases with native time-series optimizations, capable of querying billions of events in seconds.
3. Fabric capabilities are improving, but validate item-level support — materialized views, source connectors, governance controls, and mirrored databases are useful patterns, but validate each feature in your target region, tenant, capacity, and source system.
4. Start with one use case — pick a single high-value dashboard (IoT monitoring, API performance, or real-time sales), build it completely, prove the value, then expand.
5. Monitor CU consumption — use the Fabric Capacity Metrics app to track costs. Use materialized views and retention policies to optimize.
6. Mirroring can simplify operational reporting — supported sources can replicate into Fabric with near real-time behaviour, but latency and source support must be validated for each workload.

Building real-time intelligence on Fabric is a capability I help Malaysian enterprises design and implement. If your organisation needs live dashboards without dedicated data engineering headcount, let's discuss on LinkedIn.