Stage 2: Format detection and parsing¶
EHR exports mix C-CDA XML, RTF notes, HTML fragments, PDF reports and plain text under one "document" label. Stage 2 identifies each file's format and sends it to the right parser.
Format detection¶
detect_format(data) reads the first 512 bytes and returns one of:
| Value | Detected by |
|---|---|
pdf |
the file starts with %PDF |
rtf |
the text starts with {\rtf after leading white space |
ccda_xml |
the text starts with <?xml, or <ClinicalDocument appears in the first 512 bytes |
html_fragment |
an <html>, <body>, <div>, <p>, <span>, <br> or <table> tag appears in the first 512 bytes (any case) |
unknown |
none of the above |
The checks run in that order: the binary PDF signature first, then RTF, then XML, then the looser HTML test.
Text from each format¶
| Format | Parser | Notes |
|---|---|---|
pdf |
extract_pdf_text, using pypdf |
A scanned PDF with no text layer gives [PDF - no extractable text] and needs optical character recognition (OCR) |
rtf |
strip_rtf |
Handles \'XX hex escapes and \uNNNN? Unicode escapes, removes font, color, style and list tables, keeps paragraphs |
html_fragment |
strip_html |
Removes <script> and <style> blocks and tags, and decodes HTML entities |
ccda_xml |
full structured parse | Coded sections (problems, medications, results and others) become records, and each section's narrative becomes a row in notes |
unknown |
UTF-8 decode | Treated as plain text |
Every document gets a plain_text field, which the record viewer searches and note extraction reads. XML documents that declare entities are refused rather than parsed, which protects against entity-expansion attacks.
The RTF parser¶
Python has no maintained pure-Python RTF parser; the available libraries need system software such as LibreOffice or fail on the RTF that EHR systems write. The built-in parser has no dependencies and tolerates malformed groups. Table layout is lost and paragraphs are kept, which suits keyword search and review.
Scanned PDFs¶
Many PDFs in EHR exports are scanned images with no text layer, which pypdf cannot read. These show [PDF - no extractable text] in the record viewer and produce no note extraction candidates. Run OCR on them before the pipeline if their content is needed.