Raunak Agarwal

MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events

Multi-label classification where uncertainty quantification counts as much as accuracy.

· 5 min read

MADE (ACL 2026) is a benchmark for multi-label classification of medical device adverse event reports. In a domain where model output feeds human review, being right is not enough: the model also has to signal when it is likely wrong. MADE scores predictive performance and uncertainty quantification side by side.

The data: 488k FDA MAUDE reports, 1,154 IMDRF codes

488,273 reports from the FDA’s MAUDE adverse event database, 2015 to mid-2025, labelled with 1,154 IMDRF codes over three levels and up-propagated to parent codes. About 8.79 labels per report, roughly 370 tokens each, with a long tail of rare codes. Splits are strictly temporal: 298,825 training reports from 2015 to 2023, 71,271 for validation from the first half of 2024, and 10,288 for testing from July 2024 to June 2025. The benchmark is living, refreshed quarterly, so the test set always post-dates whatever a model saw in pretraining.

The setup: 20+ models, three paradigms, six uncertainty signals

20+ encoder and decoder models, three learning paradigms, and six uncertainty signals, evaluated as one grid.

01 · LANGUAGE MODELS 02 · PARADIGMS 03 · UNCERTAINTY discriminative Ettin-encoder 150m Ettin-encoder 400m / 1B Llama-3.2 1B / 3B ... generative Ettin-decoder-1B Llama-3.1 8B / 70B Qwen3-235B · DeepSeek-R1 GPT-4.1 · GPT-5 discriminative fine-tuning generative fine-tuning few-shot prompting entropy perplexity max token logprob mean token logprob Laplacian eigenvalues self-verbalized 04 · ONE REPORT IN Device Information: lavage, jet (fan spray kit; zimmer surgical, inc.) operated by health professional Event Type: malfunction Description of Event: it was reported that before surgery, there was an explosion of the battery in the packaging. the device was discovered like this in the surgery room on opening the box. there was no patient involvement. due diligence is complete, no further information is available. 05 · LABELS OUT, WITH UNCERTAINTY denser dither = less certain A02 A02 · manufacturing, packaging or shipping problem A0207 A0207 · device damaged prior to receipt by user A020701 A020701 · delivered as unsterile product A04 A04 · material integrity problem A0403 A0403 · explosion A07 A07 · electrical / electronic property problem A0705 A0705 · battery problem E24 E24 · other terms related to clinical signs, symptoms or conditions E2403 E2403 · no clinical signs, symptoms or conditions hover a label for its definition 1.0 0.5 0.0 per-label uncertainty
The benchmark in one picture: every model runs through every paradigm it supports and is scored on all six uncertainty signals. The report below is a real one from the test split (idx 19950511); its nine labels sit at three levels of the taxonomy, and the uncertainties shown are illustrative.

Classification and Uncertainty Quantification Results

Accuracy is macro-F1 and Jaccard, broken out by label frequency, because a single average hides how badly models do on rare codes. Uncertainty is scored on rejection ranking, per-label correlation, and calibration error.

Only the two best models per paradigm are shown, with the paradigm median above each pair; full results are in the paper. Hover over a column header to see what the metric means.

Paradigm / model F1 Head Tail ET J PRR ρ ECE+
Discriminative FT.51.72.50.12.59.46-.40.59
↳ Llama-3.1-8B-Base.54.74.53.12.62.47-.40.58
↳ Ettin-1B-Encoder.53.73.51.13.61.46-.40.56
Generative FT.48.67.46.11.58.57-.44.57
↳ Llama-3.1-8B-Base.50.70.48.12.59.63-.30.52
↳ Llama-3.1-70B-Base.53.73.51.16.61.55-.27.49
Prompting, instruct.29.49.25.08.43.49-.15.68
↳ Qwen3-235B-Instruct.44.60.42.24.49.56-.34.56
↳ GPT-4.1.43.59.42.22.57.45-.31.60
Prompting, reasoning.44.58.42.26.47.21-.07.59
↳ Qwen3-235B-Thinking.49.62.48.33.48.34-.09.45
↳ GPT-5 (reasoning).54.68.53.34.57NANANA

What we found

  • Small discriminatively fine-tuned models give the best head-to-tail accuracy at a fraction of the cost of the large ones.
  • Generative fine-tuning gives the most dependable uncertainty, with the lowest variance across runs.
  • Reasoning models are the only ones making real progress on the extreme tail, yet their uncertainty ordering collapses to near-random.
  • Asking a model how confident it is (self-verbalized confidence) turns out to be a bad signal, with rejection ranking near zero.
  • Token-level entropy is the mechanism that actually works for non-reasoning models.

The headline number is that the best model reaches 54% macro-F1. Multi-label classification here is far from solved, and the head-to-tail gap and systematic under-confidence leave plenty of room for better uncertainty quantification.

Appendix

MADE at a glance

Training set 2015–2023298,825
Validation 1–6/202471,271
Test set 7/2024–6/202510,288
Total samples488,273
Unique labels 3 levels1,154
Avg. labels / sample8.79
Avg. tokens / report~370

Long tail of labels

10^5 10^4 10^3 10^2 10^1 0 400 800 1200 labels, ranked by frequency
The head of the distribution is worth tens of thousands of examples per label, while hundreds of labels are seen fewer than a hundred times.

Citation

@inproceedings{agarwal-etal-2026-made,
    title = "{MADE}: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events",
    author = "Agarwal, Raunak  and
      Wenzel, Markus A.  and
      Baur, Simon  and
      Zimmer, Jonas  and
      Harvey, George  and
      Ma, Jackie",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.2148/",
    doi = "10.18653/v1/2026.acl-long.2148",
    pages = "46308--46328",
    ISBN = "979-8-89176-390-6",
    abstract = "Machine learning in high-stakes domains such as healthcare requires not only strong predictive performance but also reliable uncertainty quantification (UQ) to support human oversight. Multi-label text classification (MLTC) is a central task in this domain, yet remains challenging due to label imbalances, dependencies, and combinatorial complexity. Existing MLTC benchmarks are increasingly saturated and may be affected by training data contamination, making it difficult to distinguish genuine reasoning capabilities from memorization. We introduce MADE, a living MLTC benchmark derived from {m}edical device {ad}verse {e}vent reports and continuously updated with newly published reports to prevent contamination. MADE features a long-tailed distribution of hierarchical labels and enables reproducible evaluation with strict temporal splits. We establish baselines across more than 20 encoder- and decoder-only models under fine-tuning and few-shot settings (instruction-tuned/reasoning variants, local/API-accessible). We systematically assess entropy-/consistency-based and self-verbalized UQ methods. Results show clear trade-offs: smaller discriminatively fine-tuned decoders achieve the strongest head-to-tail accuracy while maintaining competitive UQ; generative fine-tuning delivers the most reliable UQ; large reasoning models improve performance on rare labels yet exhibit surprisingly weak UQ; and self-verbalized confidence is not a reliable proxy for uncertainty. Our work is publicly available at https://hhi.fraunhofer.de/aml-demonstrator/made-benchmark."
}