Architecture¶
Pipeline flow¶
flowchart TD
A[ccda_chunks.csv<br/>fhir_chunks.csv<br/>uuid_mapping.csv] --> B[Stage 1<br/>Reassembly and decoding]
P[inputs/fhir/<br/>inputs/documents/] --> C
B --> C[Stage 2<br/>Format detection]
C --> D{Format}
D -->|C-CDA XML| E[C-CDA parser]
D -->|FHIR JSON| F[Stage 3<br/>FHIR parser]
D -->|RTF, HTML, PDF, text| G[Text extraction]
E --> H[Stage 4<br/>Joining and assembly]
F --> H
G --> H
H --> I[Stage 5<br/>Display enrichment]
I --> J[Stage 6<br/>Test-patient filter]
J --> K[dashboard_data.json<br/>dashboard_data.xlsx<br/>csv_exports/]
K --> L[Stage 7<br/>Code inventory, patient master,<br/>note extraction]
K --> M[run_all.py add-on modules]
Files¶
| File | What it does |
|---|---|
run_pipeline.py |
The core pipeline, stages 1 to 7 |
note_extraction.py |
Note extraction, run by run_pipeline.py as stage 7c |
run_all.py |
Runs the core pipeline and the add-on modules, and sorts outputs into identified, restricted/ and reports/ |
cohort_eda.py, note_dashboard.py, device_*.py, exposure_*.py, drug_repurposing.py, phenopackets_etl.py, omop_etl.py, mondo_omop_bridge.py |
Add-on modules; each reads dashboard_data.json |
dashboard_common.py |
Shared styles, small-cell rules and excerpt masking for the dashboards |
dashboard.html |
The record viewer |
databricks_export.py, databricks_export.sql |
Warehouse-side export to chunked CSV |
Sections of run_pipeline.py¶
| Section | Contents |
|---|---|
| Part A | Imports and settings |
| Part B | Display-name lookup tables (LOINC, SNOMED CT) |
| Part C | Name cleaning and MRN normalization |
| Part D | Reading codes from C-CDA <translation> elements and FHIR coding lists |
| Part E | Format detection, decoding and text extraction |
| Part F | Stage 1: reassembly |
| Part G | Stage 2: C-CDA parsing |
| Part H | Stage 3: FHIR parsing |
| Part I and J | Stage 4: patient identity, field aliases and bundle assembly |
| Part K | Stage 5: display enrichment |
| Part L | Stage 6: test-patient filter |
| Part M | Output writers (JSON, XLSX, CSV, code inventory, patient master) |
| Part N | main() |
The bundle¶
dashboard_data.json is one JSON object:
| Key | Contents |
|---|---|
metadata |
Generation time, pipeline version, patient count and warnings |
patients |
Demographics, identifiers, vital status and num_documents |
documents |
One record per source document with plain_text |
notes |
One row per C-CDA section narrative |
encounters, problems, medications, procedures |
Clinical records |
labs, vitals, labs_vitals |
Observations; labs_vitals joins the other two |
allergies, immunizations, careplans, goals |
Clinical records |
diagnostic_reports, document_references |
FHIR report and document metadata |
Every clinical record has a patient_id that joins to patients[].patient_id, and a source_file naming the file it came from. Field definitions are on Output schema.
References across FHIR files¶
A MedicationRequest in one file can point to a Medication in another, and a document can be described by a DocumentReference in a third. Stage 3 reads every FHIR file once to index each Medication (by ID, full URL and Medication/<id>) and each DocumentReference subject, then reads the files again to build records. Without this step, many medications would have only a reference instead of their RxNorm code and name. See Stage 3.