Note extraction¶

note_extraction.py reads dashboard_data.json and searches every free-text source with a set of regular expressions written for ALS clinic documentation. Each match is written to note_extractions.csv as a candidate with review_status set to needs_review. A trained reviewer reads each candidate in its full document and confirms, corrects or rejects it; the module does not decide what is true.

The patterns are seed patterns for site-specific tuning, not validated clinical natural language processing (NLP). Phrasing differs between health systems and clinicians, so check every pattern against your own documents before any captured value is used in analysis.

Module source (note_extraction.py) Pattern catalog (CSV) Output from the sample data (CSV)

Text it reads · Patterns · Output · Running it · Tuning · Clinical NLP

Why notes are searched¶

Much of what an ALS natural history registry needs is written in notes rather than recorded as coded values:

Severity at a point in time. ALSFRS-R and ECAS scores are usually typed into the note. LOINC has a code for the ALSFRS-R total (82953-1), but many EHRs do not record it as a discrete observation, and LOINC has no codes for the four domain subscores.
Diagnostic certainty. El Escorial or Awaji-Shima categories ("clinically probable ALS") rarely reach a coded field.
Family history and genes. These are usually written in history sections or genetic test reports.
Phenotype. Site of onset and cognitive or behavioral involvement are usually free text.
Milestones. Gastrostomy (PEG) placement, start of non-invasive ventilation (NIV), tracheostomy and the start of riluzole or edaravone are sometimes coded and often only narrated.

Back to top

Text it reads¶

Source	`source_kind` in the output
C-CDA section narratives (the `notes` array), each section read once	`ccda_section`
Decoded RTF, PDF, HTML and plain-text documents	`document:<format>`, for example `document:rtf` or `document:pdf`
Text attached to FHIR DiagnosticReport, DocumentReference and Binary resources	`fhir_attachment:<format>`

The full text of a C-CDA document repeats its section narratives, so C-CDA documents are read section by section only and each passage produces one row. Scanned PDFs with no text layer are skipped; they need optical character recognition (OCR) first.

Back to top

Patterns¶

Category	Pattern	Captures
ALSFRS-R	`alsfrs_r_total`	Total score, 0 to 48
	`alsfrs_r_bulbar`, `alsfrs_r_fine_motor`, `alsfrs_r_gross_motor`, `alsfrs_r_respiratory`	Domain subscores, 0 to 12 each
ECAS and FTD	`ecas_total`	ECAS total, maximum 136
	`ecas_als_specific`, `ecas_non_als_specific`	ECAS subscores
	`ftd_spectrum`	Mentions of frontotemporal-spectrum involvement
Diagnosis	`el_escorial`	El Escorial or Awaji-Shima category (definite, probable, laboratory-supported probable, possible, suspected)
	`onset_region`	Site of onset (bulbar, limb, upper or lower limb or extremity, spinal, cervical, lumbar, axial, respiratory)
Family history and genetics	`family_history_negative`	Statements such as "no family history of ALS"
	`family_history_positive`	An affected relative or "family history of" ALS, motor neuron disease (MND) or FTD, not preceded by a negation
	`genetic_mutation`	An ALS-associated gene (C9orf72, SOD1, FUS, TARDBP and others) followed by a word such as variant, expansion or positive. "Negative for C9orf72 expansion" also matches, so read the context cue.
Pulmonary	`fvc_percent_predicted`	Forced vital capacity (FVC), percent predicted
Treatment milestones	`peg_placement_date`, `niv_initiation_date`, `tracheostomy_date`, `riluzole_start_date`, `edaravone_start_date`	The date written with the event

The pattern catalog lists every pattern with its regular expression, the documents it is usually found in, and a note for reviewers. You can write the same file from the code with python note_extraction.py --catalog pattern_catalog.csv.

Context cues¶

Each row has a context_hint read from the sentence around the match: negated_possible, family_history_possible, uncertain_possible, historical_possible or unspecified. The cues are approximate and exist to sort review work, for example to read negated candidates first. The words checked are listed on the note extraction dashboard page.

Back to top

Output columns¶

Column	Contents
`patient_id`	Participant identifier from the bundle
`source_kind`	Where the text came from (see above)
`source_id`	Document identifier, with the section title for C-CDA sections
`document_date`	Date of the document
`pattern`	Pattern name
`category`	Content category
`value`	Captured value
`context_hint`	Context cue
`snippet`	The match with 60 characters either side
`char_offset`	Position of the match in the source text
`description`	What the pattern captures
`review_status`	`needs_review` for every row

Rows from the single-patient sample data (generated by the site build; run the quickstart to reproduce):

patient_id,source_kind,source_id,document_date,pattern,category,value,context_hint,...
demo-patient-001,ccda_section,demo-ccda-001::Plan of Care,2024-04-15,alsfrs_r_total,ALSFRS-R,32,unspecified,...
demo-patient-001,ccda_section,demo-ccda-001::Plan of Care,2024-04-15,el_escorial,Diagnosis,clinically probable,unspecified,...
demo-patient-001,ccda_section,demo-ccda-001::Plan of Care,2024-04-15,fvc_percent_predicted,Pulmonary,62,unspecified,...
demo-patient-001,ccda_section,demo-ccda-001::Medications,2024-04-15,riluzole_start_date,Treatment milestones,June 2023,unspecified,...

The file is restricted: snippets are free text from the record and can hold names and other identifiers. run_all.py keeps it with the identified core outputs. Cells that begin with =, +, - or @ are prefixed with an apostrophe so a spreadsheet does not read them as formulas.

The same fact can appear in more than one document, for example a score copied from one visit note into the next. Keep every row for provenance, and group by (patient_id, pattern, value) when you want one row per value.

Back to top

Running it¶

run_pipeline.py runs note extraction as stage 7c, and run_all.py runs it as part of the full sequence. To run it on its own:

python pipeline/note_extraction.py --bundle dashboard_data.json --out note_extractions.csv

import note_extraction
note_extraction.main(bundle_path='dashboard_data.json', out_path='note_extractions.csv')

The log ends with the number of matches for each pattern, which is a quick check that the patterns found something.

Back to top

Tuning patterns for your documents¶

Sample first. Pull a few hundred documents from each contributing health system and search for each pattern's anchor word (ALSFRS, ECAS, family history, FVC, riluzole). Read the matches and the sentences around them.
Add the phrasing you find. "ALSFRS-R: 32", "ALSFRS-R 32/48" and "total score: 38/48" are handled; a score written in words ("thirty-two") would need an extension.
Watch for negation and plans. "Considering PEG", "discussed risks of tracheostomy" and "declined NIV" contain the trigger word without the event. The context cues flag many of these; clinical NLP libraries handle them more fully with assertion and temporality detection.
Add patterns for your own measures in the same form, with a description and a review note.
Measure precision before using values. Hand-review a stratified sample of rows for each pattern, compute precision, and revise any pattern that falls short of the threshold set in your protocol.

The pattern tests include positive and negative examples for every pattern; add your own examples there when you change a pattern.

Back to top

When regular expressions stop being enough¶

Regular expressions suit stereotyped phrasing (scores, structured headers, value grids), pilot work that needs first coverage quickly, and a domain small enough for a few dozen patterns. They fall short when negation, hedging and timing decide the meaning, when the text refers back to earlier sentences, when numbers are written as words, or when templates repeat stale auto-filled content.

At that point a clinical NLP pipeline is a better fit:

Apache cTAKES, backed by the Unified Medical Language System (UMLS), with assertion and temporal modules
medspaCy, a Python library with context and negation detection
Commercial clinical entity extraction with code linking, such as Spark NLP for Healthcare
Transformer models trained on clinical text, for relation extraction that patterns cannot reach

The output keeps the row shape these tools use (patient_id, source_id, pattern, value, char_offset), so later steps can read either.

Back to top