Airflow Data Pipeline & ML Foundation — O-Kidia

Situation

O-Kidia builds software that assesses children's cognitive skills through interactive tasks, primarily for education and healthcare professionals. As the products scaled, assessment data accumulated faster than the existing manual processes could handle.

Task

The company needed a reliable, monitored pipeline to ingest, clean, and expose clinical assessment data to the teams building the product and reporting to customers — plus ML models feeding product features.

Action

I owned the data & ML foundation:

• Airflow platform: production DAGs scheduling ingestion and transformation jobs, with retries, alerting, and a reproducible Docker-based deployment.
• ETL jobs: parsing raw assessment exports, cleaning them, and loading standardized tables into PostgreSQL.
• Data-quality monitoring: anomaly detection and drift checks over incoming data, so regressions surfaced quickly rather than silently corrupting reports.
• ML models: statistical models over the assessment data feeding product features (readiness scores, anomaly flags).

Result

• Processed ~1 TB of multimodal assessment data in near-real time — roughly 10x faster than manual annotation.
• Data-quality checks cut silent errors that previously required re-processing entire batches.
• The work also produced 2 peer-reviewed articles and multiple conference contributions.
• The pipeline pattern was reused for new assessment formats as the product line grew.

Confidentiality note

Work involved health-adjacent, regulated data. Names and precise technical specifics are kept general here; the approach — conservative access controls, minimal data exposure, clean separation between systems — is what I'd apply to any regulated engagement.

Stack highlights

• Airflow for orchestration and scheduling
• PostgreSQL as the analytical store
• Docker for reproducible pipeline environments
• Scikit-learn for the ML components