Skip to content

Stage 1: Decoding and reassembly

Stage 1 runs only on the warehouse route. It joins the chunked CSV exports back into one file per document and decodes Base64 where needed.

Inputs

  • <base-dir>/CCDA and FHIR data/ccda_chunks.csv
  • <base-dir>/CCDA and FHIR data/fhir_chunks.csv

Each is a long, narrow CSV with three columns:

Column Type Contents
id string Document UUID; one document spans many rows
chunk_index integer Position of the chunk within the document, from 0
chunk_data string Up to 30,000 characters of the document

For C-CDA and other documents the content is Base64-encoded. For FHIR it is the JSON text itself, which is safe in a CSV cell when quoted.

Outputs

  • <base-dir>/ccda_assembled/<id>.xml: one decoded file per document
  • <base-dir>/fhir_assembled/<id>.json: one file per FHIR Bundle

Delete the assembled folders before a new export

When an assembled folder already holds more than 100 files, stage 1 skips reassembly and reuses them, which saves time on reruns of the same export. Before running a new export in the same working directory, delete ccda_assembled/ and fhir_assembled/ so no file from the earlier export is read.

How it works

Joining chunks

df = pd.read_csv(csv_path).sort_values(['id', 'chunk_index'])
for fid, chunks in df.groupby('id'):
    data = ''.join(chunks['chunk_data'].dropna().astype(str))

groupby keeps the sort order, so chunks are joined in chunk_index order. An empty chunk_data cell is skipped.

Base64 decoding

Some warehouses store documents as Base64 and then encode them again on export. robust_decode handles both cases:

  1. If the input already starts like a decoded document (PDF, RTF, XML or HTML), it is returned unchanged.
  2. Otherwise characters outside the Base64 alphabet are removed and the string is padded to a multiple of 4.
  3. It is decoded once; if the result still does not look like a document, it is decoded a second time.

The first check keeps binary files intact: passing PDF bytes through a UTF-8 decode drops bytes and corrupts the file.

Reading the CSV

pandas.read_csv handles quoted fields with commas and line breaks when the file follows RFC 4180, where a quote inside a field is written as two quotes. The Databricks export sets that escape option explicitly, because Spark's default writes a backslash before quotes instead.

Timing

Reassembly time grows with the size of the export, and most of it goes to reading the CSV. Check pipeline_run.log for the time stamps of each stage on your own data.