Data engineering · streaming · CDC
UX friction detection, in real time
A data platform that watches how people use an app and flags friction as it happens: rage clicks, abandoned sessions, screens people get stuck on. Every interaction is captured from the database with change data capture and processed as a stream.
Goal
Detect UX friction signals while sessions are happening.
What I built
A Docker stack: PostgreSQL with Debezium CDC, Kafka, Spark Structured Streaming, Delta Lake on MinIO.
Output
Friction events scored per interaction and per session, stored with ACID guarantees.
Friction rules
| Signal | Rule | Severity |
|---|---|---|
| Rage click | event_type = 'rage_click' or more than 5 clicks | High |
| Session abandonment | event_type = 'session_abandon' | High |
| Long screen time | more than 120 s on a screen | Medium |
| Form error | event_type = 'form_error' | Medium |
| Excessive scrolling | event_type = 'rapid_scroll' | Low |
Data model
| Table | What it holds |
|---|---|
| users | Profiles: device, OS, browser, country |
| screens | Each screen's category, UX goal, expected time and whether it sits on the critical path |
| sessions | Start and end, device, converted, abandoned, number of events, total friction score |
| ux_interactions | Each event: screen, type, time on screen, clicks, scroll depth, friction flag and score |
How it runs
Capture
A Python simulator writes realistic sessions to PostgreSQL. Debezium reads the logical replication log and publishes every insert and update to Kafka as JSON.
Process and store
Spark Structured Streaming reads from Kafka, applies the rules and aggregates per session in time windows. Results land in Delta Lake on MinIO, S3-compatible storage with ACID transactions.
Operate
One docker-compose up -d starts everything, with health checks, automatic connector registration and named volumes. Kafka UI and Spark UI make it observable.
A reusable template
The same pattern as a mini data platform: CSV-driven activity, Kafka in KRaft mode without Zookeeper, and bronze, silver and gold layers on MinIO, started with one script.