Skip to content

Quality control

Every Registry Forge output comes from the same bundle, so a problem upstream appears everywhere downstream. Quality control has four parts: tests that the code does what it says, checks on each run, checks that outputs agree with each other, and review by people who know the clinical content.

The checks here are a starting set, not a full data quality program. For that, see the harmonized framework of Kahn et al. (2016), eGEMs 4(1):1244 and the OHDSI Book of OHDSI chapter on data quality.

Automated tests · Each run · Across outputs · Clinical review · Validators · Vocabulary versions

Automated tests

python -m pytest -q tests runs on every site build and should be run after installing and after any change. The tests use the invented cohort and the single-patient sample only.

Test file Checks
test_note_patterns.py Every note pattern finds its positive examples with the right value, stays silent on look-alike and negated text, and gives the expected context cue
test_pipeline_rules.py Decoding, identity rules, follow-up and censoring, Kaplan-Meier medians, suppression, one-to-many OMOP mappings, file reading, the LOINC, ATC and ICD-10-CM codes the modules rely on, and the Phenopacket seed tables
test_pipeline_end_to_end.py A full run_all.py over the invented cohort: patient IDs, deaths, decoded attachments, candidates in every note category, device and exposure findings, outputs in the right folders, and no IDs, names, record dates or internet resources in reports/
test_samples.py The single-patient chunk export runs, and its codes are the checked ones
test_phenopackets.py One Phenopacket per patient, no IDs or dates in de-identified files, schema fields only, every ontology prefix declared
test_notebook.py The run notebook is valid, every code cell compiles, and it starts on the invented cohort

When you fix a problem found in real data, add an invented example of it to the tests so it cannot return.

Back to top

Each run

Check Where to look
Input counts match what the warehouse export or the health systems delivered pipeline_run.log, "Input files"
Files that could not be read, XML refused, documents cut for length pipeline_run.log
Identity warnings: name and birth date matches left apart, files with more than one Patient resource pipeline_run.log; metadata.warnings in the bundle
Test patients removed pipeline_run.log, stage 6; compare with each health system's test accounts
New or unexpected codes and vocabularies code_inventory.csv; the Vocabularies tab of the cohort summary
Attachments not delivered and scanned PDFs with no text Documents with format external_attachment_not_retrieved or [PDF - no extractable text]
Codes with no standard mapping omop_etl.log; phenopackets_output/*/unmapped_codes.csv

Keep a few numbers from each run so changes stand out: the number of distinct source codes, the share of OMOP rows with concept_id other than 0, and the number of unmapped codes. A sudden change usually means the source data changed. "Unmapped" is not the same as "wrong": a code with no mapping is visible, while a code mapped to the wrong concept is not, which is why clinical review is still needed.

Back to top

Across outputs

The same patient should look the same in every output.

Check Expected
Patient count The same in patients, OMOP PERSON.csv and the Phenopacket cohort
Medications per patient patient_master_fhir.csv medications against OMOP DRUG_EXPOSURE.csv, allowing for codes with no standard mapping
Problems per patient patient_master.csv problems against OMOP CONDITION_OCCURRENCE.csv and Phenopacket diseases and phenotypic features

Differences usually come from a record filtered at one step and not another, a code mapped differently, or an identifier changed between steps. A short check in a restricted notebook:

import json, pandas as pd
pm = pd.read_csv('patient_master.csv', encoding='utf-8-sig', dtype=str)
person = pd.read_csv('restricted/omop_output_<release>/PERSON.csv', dtype=str)
xwalk = pd.read_csv('restricted/phenopackets_output/identified/crosswalk.csv', dtype=str)  # needs --also-write-identified

ids_master = set(pm.loc[pm['category'] == 'patient', 'patient_id'])
ids_omop = set(person['person_source_value'])
ids_pp = set(xwalk['participant_id'])
print('Missing from OMOP:', ids_master - ids_omop)
print('Missing from Phenopackets:', ids_master - ids_pp)

Run checks like this inside the approved environment; they use identified files.

Back to top

Clinical review

Automated checks find structural problems. Review by a clinician or trained abstractor finds codes mapped to the wrong term, wrong dates, and note patterns that captured the wrong number.

Review a stratified random sample of patients regularly during a pilot, then less often, and again after any change to a seed table, a pattern or an Athena vocabulary release. For each patient, compare the record viewer, the patient's rows in patient_master.csv, and the patient's Phenopacket.

Kind of problem Example Fix
Wrong mapping A problem code mapped to an unrelated HPO term Edit the seed table
Missed content Spasticity written in notes and not captured anywhere Add a pattern or a seed mapping
Wrong value An ALSFRS-R total taken from the wrong visit Tighten the pattern; check the document date
Date mismatch A medication start date that differs between outputs Trace it through stage 4
Missing patient A patient in the record viewer but not in OMOP Check identifiers between steps

Record each review and the edits that followed, so the registry has a trail from finding to fix. Candidates from notes are reviewed before any value is used; see Note extraction.

Back to top

Formal validators

Back to top

Vocabulary versions

Every output that depends on a vocabulary records the version it used: CDM_SOURCE.csv for the OMOP tables and metaData.resources in each Phenopacket. With those and the code version, anyone can reconstruct the mapping decisions behind an output, which an audit, a data sharing agreement or a vocabulary update may require. Keep each run's outputs in a folder named for the run date and vocabulary release, and compare runs after an update.

Back to top