--- license: mit language: - en library_name: transformers pipeline_tag: text-classification tags: - medical - triage - emergency-medicine - esi - biomedbert - clinical-nlp - decision-support - not-for-clinical-use base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext model-index: - name: bert-esi-triage-v58 results: - task: type: text-classification name: ESI 1-5 triage classification dataset: type: mimic-iv-ed-holdout name: MIMIC-IV-ED holdout (n=7,917) metrics: - type: recall name: ESI 1 recall (ensembled) value: 0.821 - type: recall name: ESI 1 recall (calibrated) value: 0.802 - type: accuracy name: ESI exact (calibrated) value: 0.572 - task: type: text-classification name: ESI 1-5 triage classification dataset: type: mc-med-clean name: MC-MED Stanford clean (n=1,000) metrics: - type: recall name: ESI 1 recall (calibrated) value: 0.840 - type: accuracy name: ESI exact (calibrated) value: 0.602 - task: type: text-classification name: ESI 1-5 triage classification dataset: type: mietic-clean name: MIETIC narrative clean (n=200) metrics: - type: recall name: ESI 1 recall (ensembled) value: 0.967 - type: accuracy name: ESI exact (calibrated) value: 0.775 - task: type: text-classification name: ESI 1-5 triage classification dataset: type: lukina-v3 name: Lukina v3 (n=201) metrics: - type: recall name: ESI 1 recall (calibrated) value: 0.943 --- # BERT-ESI-Triage v58 — BiomedBERT multi-head triage classifier ## TL;DR `bert-esi-triage-v58` is a fine-tuned BiomedBERT classifier for the **Emergency Severity Index (ESI 1-5)** triage decision. v58 is the successor to v57 (epoch 3) on the same 18-eval suite. Headline improvements vs v57 e3: - MIMIC-IV-ED ESI 1 recall: **+4.0pp** ensemble (78.1% → 82.1%). - MC-MED Stanford ESI 1 recall: **+7.0pp** (77.0% → 84.0%). - MIETIC ensemble ESI 1 recall held at **96.7%**. - Lukina v3 ESI 1 recall held at **94.3%**. - Concept-density / frequent-flyer / polypharmacy / geriatric / multi-CC subgroup slices all win ESI 1 recall by +1.6 to +6.7pp. Trade-off: cal_exact regresses by 2-6pp on MIETIC, judgment_gap, cardiac_arrest, and sepsis slices (a recall-for-safety shift). ### Architecture differences vs v57 - **No engineered-feature fusion.** v57 trained with an 11-d MEWS / qSOFA / shock_index feature vector projected to 64-d and concatenated with CLS into `esi_head` Linear(832, 5). v58 drops the fusion path and uses `esi_head` Linear(768, 5) directly on CLS. The encoder learns vital patterns from raw text alone. - **Trainer.** v58 uses `train_bert_v44_bf16.py` (bf16 native on GB10); v57 used `train_bert_v44_v53_3.py` (feature-fusion variant). - **Symptom head dim.** v58 = 199, v57 = 203 (concept vocab pruning). ### Inference Load with PyTorch + transformers; CLS-pooled features feed each head: - `esi_head`: Linear(768, 5) — main output. - 19 auxiliary heads: symptom, flag, pain, arrival, gestalt, disposition, resource, vitals, etc. The companion Space `vadimbelsky/esi-triage-demo` ships the full calibration stack (demographic normalizer → per-dialect temperature → confidence-aware ESI 1 bias → deterministic engine `min` ensemble). ### License - **Model weights:** MIT. - **Training data:** private (MIMIC-IV-ED + Stanford MC-MED + MIETIC + Lukina v3 + curated synth). ⚠️ **Not a medical device. Not for clinical use.** This is a research artifact distributed for reproducibility and benchmarking. Real deployment requires institutional validation + IRB + local recalibration. ## Performance — v58 epoch 3 vs v57 epoch 3 Per-slice ESI 1 recall (calibrated): | Slice | n | v57 cal_ESI1_R | v58 cal_ESI1_R | Δ | |---|---:|---:|---:|---:| | mimic_holdout | 7,917 | 76.6% | 80.2% | +3.6pp | | mcmed_clean_holdout | 1,000 | 77.0% | 84.0% | +7.0pp | | mietic_clean_holdout | 200 | 90.0% | 86.7% | -3.3pp | | lukina_v3 | 201 | 94.3% | 94.3% | 0pp | | sepsis_eval_v1 | 93 | 95.0% | 95.0% | 0pp | | stroke_eval_v1 | 97 | 95.0% | 92.5% | -2.5pp | | anaphylaxis_eval_v1 | 90 | 97.1% | 97.1% | 0pp | | cardiac_arrest_eval_v1 | 103 | 93.3% | 90.0% | -3.3pp | | ob_emergency_eval_v1 | 98 | 95.0% | 90.0% | -5.0pp | | pediatric_n200_v2 | 200 | 95.0% | 95.0% | 0pp | | judgment_gap_v1 | 177 | 88.9% | 83.3% | -5.6pp | | geriatric_n200_v1 | 200 | 78.8% | 80.0% | +1.3pp | | polypharmacy_n200_v1 | 200 | 72.5% | 76.2% | +3.7pp | | frequent_flyer_n200_v1 | 135 | 80.0% | 86.7% | +6.7pp | | vital_completeness | 231 | 83.3% | 86.7% | +3.3pp | | multi_cc_complexity | 304 | 76.2% | 77.8% | +1.6pp | | concept_density | 260 | 80.0% | 86.7% | +6.7pp | Verdict: 8 wins / 5 losses / 5 neutral on ESI 1 recall (mean +0.78pp). MIMIC per-class recall (calibrated): | Class | n | v57 R | v58 R | Δ | |---|---:|---:|---:|---:| | ESI 1 | 475 | 76.6% | 80.2% | +3.6pp | | ESI 2 | 2,582 | 64.6% | 62.7% | -1.9pp | | ESI 3 | 4,286 | 54.4% | 50.1% | -4.4pp | | ESI 4 | 554 | 64.6% | 67.7% | +3.1pp | | ESI 5 | 20 | 40.0% | 40.0% | 0pp | ## Known gaps (carried to v59) - **SYMPTOM_LABELS index mismatch.** v58 symptom_head outputs 199 logits; the demo's SYMPTOM_LABELS list (inherited from v47/v49) has 178 canonical entries plus runtime padding for the extra positions. Specific concepts that grew between v57→v58 may be mislabeled in the Space UI. Tracked for v59 (task #463). - **ESI 5 recall ≤ 40% across slices** — sparse class, dialect-noisy labels. v58 holds the v57 floor but does not improve it. - **MIETIC / OB / cardiac_arrest ESI 1 R regressions** vs v57. Likely driven by dropping the engineered-feature signal on narrative slices where vitals are inline rather than parsed. - **Engineered features no longer in graph.** Downstream stacks that computed MEWS / qSOFA / shock_index and passed them to v57 must drop that path for v58. ## Companion Space `vadimbelsky/esi-triage-demo` — 19-head dashboard with engine ensemble, demographic normalizer, dialect detection, per-dialect calibration. The Space loads this model and exposes the full calibrated pipeline.