TECHSALOCASE STUDY / HEALTHTECH : 2023HEALTHCARE DATA.
HEALTHCARE DATA.
CONNECTED TO ACTION.
Engineering a high-scale healthcare data, interoperability and patient communication platform for continuously changing clinical and operational workloads.
- SCALE
- 2+ crore events / day
- INDUSTRY
- HealthTech / Analytics
- CLIENT
- Confidential
THE ENGINEERING MANDATEFOUR INFRASTRUCTURE LAYERS.
FOUR INFRASTRUCTURE LAYERS.
ONE OPERATING SYSTEM.
Healthcare events begin in heterogeneous hospital, clinical and business systems. They must become trustworthy data—and often trigger a time-sensitive patient interaction.
The program connected multi-cloud data engineering, healthcare interoperability and distributed communication rather than treating them as separate applications.
01 / MULTI-CLOUD DATA PLATFORMTWO CLOUD PATHS.
TWO CLOUD PATHS.
ONE ENGINEERING MODEL.
Workloads used services appropriate to each customer environment while preserving the same durable pattern: capture change, land immutable data, resolve state, publish analytical tables.
AWS IMPLEMENTATIONSERVERLESS-ORIENTED DATA LAKE
01Operational DB
02AWS DMS
03Kinesis
04Amazon S3
05Apache Iceberg
06Glue / Athena
AZURE IMPLEMENTATIONCDC-DRIVEN LAKEHOUSE
01MongoDB
02Kafka Connect
03Kafka
04ADLS
05Databricks
06Delta / SQL
02 / ENTERPRISE HEALTHCARE DATA LAKEFROM FRAGMENTED SOURCES TO
FROM FRAGMENTED SOURCES TO
A TRUSTED DATA FOUNDATION.
Operational systems remained focused on care and transactions while cloud data platforms handled transformation, analytics and downstream intelligence.
HIS / EMRLaboratoryBillingPharmacyCRMPatient apps
INGESTIONDMS · KINESIS · KAFKA · CDC
S3 / ADLSRAW → STANDARDIZED → CURATED
CONSUMPTIONATHENA · DATABRICKS SQL · AI
Continuous ingestion
Batch, CDC, APIs, files and events moved changing healthcare information without repeatedly extracting complete source datasets.
- Incremental capture
- Schema-aware ingestion
- Fresh downstream data
Immutable lakehouse
Source information was retained before standardization so historical records could be audited, investigated and reprocessed.
- Raw / never mutate
- Standardized layer
- Curated datasets
Distributed processing
Spark and Databricks patterns scaled transformation, validation and analytical preparation with the workload.
- Parallel processing
- Partitioned storage
- Distributed SQL
Quality & reconciliation
Controls compared source, snapshot, CDC and analytical records to expose silent loss and synchronization anomalies.
- Record and key checks
- Duplicate detection
- Historical consistency
03 / SNAPSHOT + CDCRECONSTRUCTING THE
RECONSTRUCTING THE
CURRENT BUSINESS STATE.
A complete dataset needed historical records and every subsequent insert, update and delete. The processing layer ordered change events, removed duplicates and resolved the authoritative latest version before merging it into Iceberg or Delta tables.
01 / ORDER02 / DEDUPLICATE03 / LATEST STATE04 / VALIDATE
OPEN TABLE FORMATICEBERG / DELTAMERGE
Exists → update
New → insert
Deleted → reconcile
Document ID · cluster timestamp · sequence · operation
Nested documents · arrays · schema normalization
Counts · keys · hashes · CDC gaps · field comparison
02 / HEALTHCARE INTEROPERABILITYEXTERNAL PROTOCOLS.
EXTERNAL PROTOCOLS.
ONE INTERNAL MODEL.
The HL7 integration layer insulated internal applications from the protocol and data-model differences of every hospital system.
HL7 message parsingSchema and content validationPatient and identifier mappingEvent transformationError and exception handling
EXTERNAL SYSTEMSHIS · EMR · LIS · CLINICAL
INTEROPERABILITY LAYERPARSE · VALIDATE · MAP · TRANSFORM
INTERNAL PLATFORMDATA · ANALYTICS · ENGAGEMENT
03 / PATIENT COMMUNICATION2+ CRORE EVENTS.
2+ CRORE EVENTS.
EVERY DAY.
At this volume, messaging is a distributed system—not a loop around a provider API. Sending, routing and delivery reporting each required independent, observable workloads.
01Healthcare event
02Orchestrator
03Message queue
04Routing engine
05Delivery workers
06SMS · RCS · WhatsApp
Intent stays separate from provider behavior.
A common orchestration layer selected channels, applied policies and allowed fallback from rich channels to SMS when delivery conditions changed.
Every send creates another event stream.
Submitted, accepted, delivered, failed, read and interaction events flowed through their own queue, consumers, operational store and analytics path.
DESIGNED FOR FAILUREABSORB THE PRESSURE.
ABSORB THE PRESSURE.
ISOLATE THE FAILURE.
Requests persisted intent and returned quickly. Queues absorbed traffic spikes, workers scaled with depth, and slow providers could not bring down the wider notification ecosystem.
Queue buffering
Rate limiting
Exponential backoff
Provider isolation
Dead-letter handling
Idempotent consumers
EVENT IDPROCESSING CHECKSEEN → IGNORE DUPLICATENEW → PROCESS
EVENT-DRIVEN HEALTHCAREWHEN THE DATA CHANGES,
WHEN THE DATA CHANGES,
THE WORKFLOW RESPONDS.
Healthcare and engagement data met at the event layer. An appointment, lab result or follow-up condition could become a governed communication workflow without waiting for a broad scheduled batch.
COMPLETE ENGINEERING ARCHITECTUREFROM SOURCE CHANGE TO
FROM SOURCE CHANGE TO
PATIENT ENGAGEMENT.
HEALTHCARE SOURCESHIS · EMR · LIS · MongoDB · PostgreSQL
INGESTION & CDCAWS DMS · Kinesis · Kafka Connect
CLOUD DATA LAKESAmazon S3 · Azure Data Lake Storage
OPEN TABLE FORMATSApache Iceberg · Delta Lake
DISTRIBUTED PROCESSINGApache Spark · Azure Databricks
QUALITY & INTEROPERABILITYReconciliation · HL7 · schema validation
ANALYTICS & EVENTSAthena · Databricks SQL · workflow triggers
PATIENT ENGAGEMENTSMS · RCS · WhatsApp · delivery analytics
ENGINEERING SCALEA FOUNDATION BUILT TO
A FOUNDATION BUILT TO
EXPLAIN WHAT IT IS DOING.
Communication events processed
AWS + Azure implementations
Continuously synchronized datasets
Apache Iceberg + Delta Lake
Amazon Kinesis + Kafka
Spark + Databricks compute
Healthcare interoperability
SMS + RCS + WhatsApp
CAPABILITIES DEMONSTRATED
Healthcare Data Lake · CDC · Spark · Databricks · HL7 · Data Quality · Event-Driven Architecture · SMS · RCS · WhatsApp · Observability · Cloud & DevOps