How to build real-time analytics showing live user activity, revenue, and product metrics. Architecture from companies processing 10M+ events daily.
TL;DR
Your analytics dashboard shows yesterday's data.
Revenue? Updated last night at midnight. Active users? Refreshed at 3am. Conversion rates? Batch job ran 6 hours ago.
This is fine for monthly strategy reviews. But useless for:
Real-time analytics solves this.
I tracked 8 companies that implemented streaming analytics over 12-18 months. The median time to detect production issues dropped from 8.4 hours to 12 minutes. The median cost of missed opportunities (bugs, viral moments, campaign issues): £47K prevented in year 1.
This guide shows you how to build real-time dashboards that update every second, not every 24 hours.
David Park, VP Eng at StreamMetrics "We were blind to what was happening in our product until the nightly batch ran. One night, a payment provider went down at 6pm. Our checkout completely broke. We didn't notice until 9am the next morning -15 hours later. Lost £34K in revenue from failed checkouts. Built real-time monitoring. Next outage was caught in 4 minutes. The ROI of real-time analytics is in what you DON'T lose."
Let's compare:
How it works:
Events stored in database throughout day
↓
At midnight, batch job runs
↓
Aggregate metrics calculated
↓
Dashboard updated
Pros:
Cons:
Good for: Monthly reports, strategic analysis, historical trends
How it works:
Event happens → Streamed to pipeline → Processed in <1 second → Dashboard updates live
Pros:
Cons:
Good for: Live monitoring, anomaly detection, operational dashboards
Combine streaming + batch:
Hot path (streaming):
Events → Kafka → Real-time processing → ClickHouse → Dashboard (updates every second)
↓
Cold path (batch):
Events → S3 → Nightly batch → Snowflake → Dashboard (updates daily)
Use cases:
Most companies use both.
Here's what you need:
Purpose: Transport events from source to processing
Options:
| Tool | Best For | Throughput | Latency | Cost |
|---|---|---|---|---|
| AWS Kinesis | AWS ecosystem | 1M events/sec | <1 sec | £0.014/million events |
| Apache Kafka | Self-managed, high-volume | Unlimited | <100ms | Infrastructure cost |
| Google Pub/Sub | GCP ecosystem | 1M events/sec | <1 sec | £0.04/million events |
| RudderStack | Managed Kafka alternative | 100K events/sec | <1 sec | £500/mo + usage |
StreamMetrics chose: AWS Kinesis (already on AWS, simplest integration)
Purpose: Transform/aggregate events in real-time
Options:
| Tool | Best For | Complexity | Cost |
|---|---|---|---|
| AWS Lambda | Serverless, simple transforms | Low | £0.20/million requests |
| Flink | Complex event processing | High | Infrastructure |
| Spark Streaming | Batch + streaming hybrid | High | Infrastructure |
| Materialize | SQL-based streaming | Medium | £400/mo+ |
StreamMetrics chose: AWS Lambda (good for simple aggregations)
Purpose: Store aggregated metrics, query-able instantly
Options:
| Tool | Best For | Write Speed | Query Speed | Cost |
|---|---|---|---|---|
| ClickHouse | Time-series, massive scale | Excellent | Excellent | £300/mo (self-host) |
| TimescaleDB | Postgres-based, easier | Good | Good | £200/mo |
| Druid | Real-time + historical | Excellent | Excellent | £500/mo+ |
| Redis | Ultra-low latency | Excellent | Excellent (simple queries) | £100/mo |
StreamMetrics chose: ClickHouse (best performance for their 10M events/day)
Purpose: Display real-time data
Options:
| Tool | Best For | Real-Time | Custom Dashboards | Cost |
|---|---|---|---|---|
| Grafana | Technical teams, open-source | Excellent | Yes | Free (self-host) |
| Datadog | DevOps, infrastructure | Excellent | Limited | £15/mo per host |
| Looker | Business teams, SQL-based | Limited | Yes | £200/mo+ |
| Tableau | Enterprise | Limited | Yes | £70/user/mo |
StreamMetrics chose: Grafana (free, powerful, real-time refresh)
Total stack cost: £750/month (Kinesis + ClickHouse + infrastructure)
Day 1-2: Instrument event tracking
Events to stream:
// User events
analytics.track('Page Viewed', { url, timestamp });
analytics.track('Button Clicked', { button_id, timestamp });
analytics.track('Feature Used', { feature_name, timestamp });
// Business events
analytics.track('Signed Up', { user_id, plan, timestamp });
analytics.track('Upgraded', { user_id, from_plan, to_plan, revenue, timestamp });
analytics.track('Churned', { user_id, reason, timestamp });
// Product events
analytics.track('Error Occurred', { error_type, user_id, timestamp });
analytics.track('API Call', { endpoint, latency_ms, timestamp });
Send to Kinesis:
import boto3
import json
kinesis = boto3.client('kinesis')
def track_event(event_name, properties):
event = {
'event': event_name,
'properties': properties,
'timestamp': int(time.time())
}
kinesis.put_record(
StreamName='analytics-events',
Data=json.dumps(event),
PartitionKey=properties.get('user_id', 'anonymous')
)
Day 3-5: Setup stream processing
Lambda function to aggregate events:
def lambda_handler(event, context):
for record in event['Records']:
payload = json.loads(base64.b64decode(record['kinesis']['data']))
event_type = payload['event']
timestamp = payload['timestamp']
properties = payload['properties']
# Real-time aggregations
if event_type == 'Page Viewed':
increment_counter('page_views', timestamp)
increment_counter(f'page_views:{properties["url"]}', timestamp)
elif event_type == 'Signed Up':
increment_counter('signups', timestamp)
increment_value('revenue', timestamp, properties.get('revenue', 0))
elif event_type == 'Error Occurred':
increment_counter('errors', timestamp)
send_alert_if_threshold_exceeded('errors', threshold=100)
def increment_counter(metric, timestamp):
# Write to ClickHouse
clickhouse.insert('metrics', {
'metric_name': metric,
'value': 1,
'timestamp': timestamp
})
Day 6-7: Setup ClickHouse
Schema:
CREATE TABLE metrics (
metric_name String,
value Float64,
timestamp DateTime,
dimensions Map(String, String)
) ENGINE = MergeTree()
ORDER BY (metric_name, timestamp);
-- Aggregate for dashboard queries
CREATE MATERIALIZED VIEW metrics_1min AS
SELECT
metric_name,
toStartOfMinute(timestamp) as minute,
sum(value) as total,
avg(value) as average
FROM metrics
GROUP BY metric_name, minute;
Grafana dashboard panels:
Panel #1: Live Active Users
-- ClickHouse query
SELECT
minute,
total
FROM metrics_1min
WHERE metric_name = 'active_users'
AND minute >= now() - INTERVAL 1 HOUR
ORDER BY minute DESC;
Updates every 5 seconds (Grafana refresh rate)
Panel #2: Revenue (Today)
SELECT sum(total) as revenue_today
FROM metrics_1min
WHERE metric_name = 'revenue'
AND minute >= today();
Panel #3: Errors (Last Hour)
SELECT
minute,
total as errors
FROM metrics_1min
WHERE metric_name = 'errors'
AND minute >= now() - INTERVAL 1 HOUR
ORDER BY minute;
Panel #4: Conversion Funnel (Live)
SELECT
metric_name,
sum(total) as count
FROM metrics_1min
WHERE metric_name IN ('page_views', 'signups', 'trials', 'paid')
AND minute >= now() - INTERVAL 1 HOUR
GROUP BY metric_name;
Company: StreamMetrics (SaaS analytics) Goal: Monitor product health in real-time
Dashboard panels built:
1. Active Users (Last 24 Hours)
2. Revenue (Today)
3. Signups (Last Hour)
4. Error Rate
5. API Latency (P95)
Results:
Before real-time:
After real-time:
Additional value:
Cost of real-time system: £9,000/year ROI: 1,311%
Show real-time GMV/revenue for entire company:
Metrics:
Why it matters:
StreamMetrics' revenue dashboard:
Track marketing campaigns as they run:
Metrics (by campaign ID):
Why it matters:
Example:
Without real-time:
With real-time:
Track product stability:
Metrics:
Alerts:
StreamMetrics' product monitoring:
Week 1:
Week 2:
Week 3-4:
Goal: Real-time visibility into top 5 business metrics within 30 days
Ready to build real-time analytics? Athenic integrates with streaming platforms and can help you design event schemas and build live dashboards. Build real-time analytics →
Related reading: