Real-Time Telecom CDR Pipeline

A production-minded telecom ingestion pipeline for streaming CDR events into operational analytics and monitoring.

Built an operational telecom ETL pipeline processing 1M+ daily CDRs for anomaly detection, SLA measurement, and automated alerts.

Improved visibility for operations teams with automated data quality monitoring and SLA reporting.

KafkaSparkAirflowDockerKubernetes

Real-time ingest and ETL for telecom CDRs

Auto-scaling Kubernetes deployment

Data quality monitoring and alerting

Architecture

Kafka receives CDR events, Spark transforms the stream, and Airflow coordinates scheduled checks and downstream reporting tasks.

Kafka topic for incoming CDRs

Spark ETL for cleaning and enrichment

Airflow DAGs for orchestration and alerts

Repository contents

The telecom repo now includes the Docker Compose stack, data generator, Spark ETL entrypoint, and setup documentation needed to run the pipeline locally.

Docker Compose stack

Python-based CDR generator

Spark ETL job scaffold

Run instructions

Project 1 is the most complete implementation and can be started directly from the telecom repo root with Docker Compose and the helper scripts.

docker-compose up -d

scripts/up.ps1 or scripts/up.sh

data generator helper scripts

Source folder: telecom-data-portfolio/Project_1_CDR_Pipeline