Kryptur Research presents RetailX — a fully operational, end-to-end data-engineering platform that transforms raw business data into certified analytics through a zero-install terminal wizard and secure REST API. Kafka streaming, DuckDB lakehouse, dbt transformations, and Airflow orchestration on a single commodity VPS — with multi-tenant isolation and 25 self-service jobs.
We present RetailX, a turnkey data-engineering platform on commodity hardware: Apache Kafka ingests CSV, API, CDC, and visitor-pixel events; a Python consumer writes Parquet to the data lake; DuckDB serves as the warehouse; dbt transforms and tests the data; and Airflow schedules the pipeline every 10 minutes — all exposed through a 25-job terminal wizard and FastAPI REST API.
| No. | Path | Kafka topic | Description |
|---|---|---|---|
| 1 | CSV Upload | retail-events | Structured file upload via /ingest endpoint |
| 2 | REST API Source | retail-events | Job 20 fetches JSON from external endpoints |
| 3 | CDC Simulation | cdc-events | Job 19 accepts change-event JSON arrays |
| 4 | Visitor Pixel | visitor-events | 1×1 transparent GIF pageview tracking |
| No. | Model | Purpose |
|---|---|---|
| 1 | stg_retail_events | Staging view: cast timestamps, compute total_amount, deduplicate |
| 2 | dim_product | Product dimension with surrogate keys and SCD attributes |
| 3 | fact_sales | Sales fact table joining cleaned events with product dimension |
| Metric | Value |
|---|---|
| Endpoints captured | 26 |
| JSON artefacts | 22 files |
| Chart outputs | 3 (ML, web analytics, visitor) |
| Tier-1 artefacts | 18 files (Jobs 24–25 report, dashboard, HTML) |
| dbt test result | PASS=17 WARN=0 ERROR=0 |
| Metric | Live value |
|---|---|
| Total revenue | $28,431.74 |
| Quantity sold | 386 |
| Average order value | $154.52 |
| Unique products | 11 |
| Total events | 184 |
| Data quality (dbt) | 17/17 tests (100%) |
| Metric | Value |
|---|---|
| CPU cores | 16 vCPUs (AMD EPYC) |
| Memory | 30 GiB RAM |
| CPU utilisation (steady state) | < 10% |
| Airflow DAG completion time | < 2 seconds |
Kryptur OU · doi:10.5281/zenodo.20944392 · Corresponding author Raja Ram M · Zius Quantum R&D Center
RetailX engineering report — Kafka ingestion, DuckDB lakehouse, dbt star schema, 25 wizard jobs, and tier-1 analytics deliverables.
View project →FastAPI backend exposing 27 endpoints — multi-tenant DuckDB schemas, REST ingestion, and interactive Bash wizard via /client.sh.
View project →Live analytics overview console — 13 Chart.js panels, KPI strip, GET /analytics_overview proxy, and dbt quality gauge.
View project →Complementary quantum-enhanced lead pipeline — UN Comtrade tensors and QSVM scoring from Kryptur OU.
View project →Quantum-optimised space-based solar power beam scheduling — tiered classical/QAOA solvers and ERA5 validation.
View project →