Stage 6 -- Test-patient filter

Purpose

Remove training, QA, and synthetic test patients from the analytic data set.

Two sources, merged

Exclusion rules come from two places, both always applied:

1. HARDCODED_TEST_EXCLUSIONS -- a list at the top of run_pipeline.py. Edit it once, commit it, never worry about it.
2. test_patients.txt -- a plain-text file at the working directory. Useful for site-specific overrides without touching the code.

When both are present, all rules are applied. The log line Total active exclusion rules: N tells you how many got loaded.

Rule syntax

One rule per line. Four match modes:

Type	Example	Match
id	id: a1b2c3d4-...	Exact patient_id UUID match
name	name: Test Patient	Exact case-insensitive full-name match
name_contains	name_contains: zzztest	Substring in first or last name (case-insensitive)
mrn_contains	mrn_contains: 99999	Substring in MRN (case-insensitive)

Lines without a type: prefix default to name_contains. Lines starting with # are comments. Blank lines are ignored.

What gets removed

When a patient matches any rule, the pipeline removes:

• The patient record from patients.
• All clinical records (problems, meds, observations, encounters, allergies, immunizations, care plans, reports, goals, sections) where patient_id matches.
• All documents where patient_id matches.

The pre-filter snapshot is preserved as dashboard_data.prefilter.json so you can audit what was removed without re-running the pipeline.

Hardcoded vs file-based: when to use which

Use hardcoded	Use file
The exclusion is permanent (e.g. a known test account)	The exclusion list changes between runs
You want it under version control	You want non-engineers to edit it
You want it to apply even if the file is missing	You're sharing rules across multiple deployments

In practice, most teams put long-lived rules in HARDCODED_TEST_EXCLUSIONS and use the file for ad-hoc additions.

Example

# Top of run_pipeline.py
HARDCODED_TEST_EXCLUSIONS = [
    'name: Zztest Patient',
    'name_contains: training',
    'mrn_contains: 99999',
]

# test_patients.txt
# Add temporary exclusions here
name: Demo User

Combined effect: 4 rules. Anyone whose name contains "training" or "zztest", anyone with MRN containing "99999", anyone named exactly "Demo User" (case-insensitive) gets filtered.