Overview

run_pipeline.py turns electronic health record (EHR) exports, as C-CDA XML, FHIR R4 Bundles, HTML, RTF, PDF and plain text, into one bundle (dashboard_data.json) and the tables built from it. run_all.py runs the core pipeline and then the add-on modules, and sorts the outputs into identified, restricted and counts-only folders.

Inputs

The pipeline reads either of two layouts in a working directory.

Warehouse chunk exports, as databricks_export.py writes them:

File	Required	Contents
CCDA and FHIR data/ccda_chunks.csv	yes, or the FHIR file	Base64-encoded C-CDA XML and other documents, split into rows
CCDA and FHIR data/fhir_chunks.csv	yes, or the C-CDA file	FHIR Bundle JSON, split into rows
uuid_mapping.csv	yes	document_uuid to patient_id, plus optional demographic columns. The patient ID here decides who a document belongs to
test_patients.txt	optional	Exclusion rules, one per line
vital_status.csv	optional	Deaths and alive confirmations from outside the EHR, used by run_all.py

Each chunk file has three columns: id, chunk_index, chunk_data. Clinical documents often exceed the cell size a warehouse can export to CSV, so they are split; the pipeline joins the chunks in chunk_index order and then decodes them.

Files already split by participant: inputs/fhir/<patient_id>/ and inputs/documents/<patient_id>/, optionally with participant_files.csv. See Configuration.

Stages

Stage	Name	What it does
1	Decoding and reassembly	Joins chunks and decodes Base64 (warehouse route only)
2	Format detection and parsing	Detects C-CDA, RTF, PDF, HTML or text from the file's first bytes and parses each
3	FHIR resource extraction	Reads every coding on each resource, and the text of reports and notes delivered as attachments
4	Joining and assembly	One patient ID per participant, with the source file kept on every row
5	Display-name enrichment	Fills in missing LOINC and SNOMED CT display names
6	Test-patient filter	Removes records matching the exclusion rules
7	Tables and note extraction	Code inventory, patient master CSV, and note extraction as stage 7c

Outputs of the core pipeline

File	Contents
dashboard_data.json	The full bundle, read by the record viewer and every add-on module
dashboard_data.xlsx	One sheet per table
dashboard_data.prefilter.json	The bundle before stage 6, for checking what the filter removed
csv_exports/*.csv	One file per table
code_inventory.csv	Every vocabulary and code with record and patient counts (Code inventory)
patient_master.csv	One row per coded record with demographics joined on (Patient master)
note_extractions.csv	Candidate matches from notes (Note extraction)
pipeline_run.log	The run log

All of these are identified. The bundle structure is on Output schema, and what each file holds is on Participant identifiers.

Design choices

Chunked CSV files. CSV can be read without warehouse tools and moves through any approved transfer route; splitting documents into rows keeps large documents under export size limits.

Indexes across bundles. FHIR exports often put a Medication resource in a different bundle from the MedicationRequest that points to it. Stage 3 first indexes every Medication and DocumentReference across all bundles, so those references resolve.

Field aliases. Each record has canonical field names (effective_date, display_name) and aliases (start_date, authored_on, onset_datetime, allergen, vaccine), so later tools can use whichever they expect.

A record viewer that runs in the browser. The viewer is one HTML file that loads dashboard_data.json from a file picker, with no server and no network calls. It shows identified data, so it is used inside the approved environment.