In our previous post, we described how Doximity Ask answers clinical questions, and our commitment to safety and oversight as part of building each one of its components.
Today we are releasing Bedside Bench, an open benchmark of 500 physician-validated clinical question-answering cases, together with the complete grading rubrics we use as part of our core internal evaluations for this question set. Bedside Bench covers a wide range of clinically specialized evaluation tasks covering both common clinical query scenarios, safety-focused evaluations, as well as areas where large language models are prone to error. The distribution of tasks was curated based on years of observation and data collection on real-world clinical AI deployments. We are releasing Bedside Bench so that anyone can now run the same evaluation against any model or system, and compare results directly.
The dataset is available on Hugging Face and GitHub under a CC BY-NC-SA 4.0 license.
Motivation
Clinical usefulness depends on several capabilities operating together. A system must understand the question, retrieve appropriate evidence, reason through the clinical context, use the right tools, perform exact calculations, communicate uncertainty, and avoid unsupported claims. Success on one capability cannot establish success across the whole workflow.
Bedside Bench was built around three priorities.
1. Questions that resemble clinical practice
The benchmark focuses on task-oriented questions that clinicians could plausibly ask during care. These include diagnostic and management decisions, medication questions, clinical calculations, and the use of validated risk scores. The prompts require systems to generate useful clinical answers in context, without the scaffolding of multiple-choice options.
2. Tasks where current models remain prone to consequential errors
Some clinical questions demand synthesis and judgment. Others contain an answer that can be calculated exactly. Clinical AI systems need to recognize that difference and use appropriate tools.
Bedside Bench includes medication concentration problems, weight-based dosing, unit-sensitive calculations, and validated clinical risk scores. These tasks test whether a system can select the right method, identify the necessary inputs, execute the calculation accurately, and interpret the result appropriately. A fluent approximation carries little value when the correct answer follows from a defined function.
The benchmark also tests medical hallucinations: claims that sound clinically plausible yet lack support or conflict with established evidence. In medicine, confidence and correctness can diverge. Evaluation needs to detect that divergence.
3. Dimensions of quality that medicine cannot leave invisible
Clinical performance has to be evaluated across the populations a system may serve. Bedside Bench includes health equity as a core dimension, creating a structured way to examine bias, representation, and clinically important differences in model behavior.
This reflects a broader principle behind the benchmark. Average performance can conceal concentrated risk. A model can improve its overall score while preserving a serious weakness in a specific task or patient population. Evaluations should make those weaknesses visible.
From external validation to open evaluation
Earlier this year, Doximity Ask achieved the highest score among commercially available products in the independent Stanford- and Harvard-led NOHARM v2 evaluation. NOHARM tested whether AI recommendations could contribute to patient harm across 1,100 realistic clinical scenarios. Its results showed the value of clinically focused model development and evaluation grounded in the conditions where AI will actually be used.
Bedside Bench extends that commitment across a broader range of everyday clinical tasks. We use evaluations like these to identify weaknesses in Doximity Ask, guide model training, and determine whether a new system is genuinely improving. Releasing Bedside Bench gives the medical and AI communities access to the same kinds of tests we use internally.
Trust in clinical AI requires visibility into how performance is defined, where a system succeeds, and where it still fails. Open evaluation allows claims about clinical capability to be examined, reproduced, and challenged. It also gives health systems, researchers, and developers a common foundation for testing models against the priorities of clinical practice.
What Bedside Bench measures
Bedside Bench contains 500 questions spanning ten dimensions of clinical AI performance. These dimensions include:
- Clinical safety
- Diagnostic and management accuracy
- Medical calculations
- Selection and use of calculator tools
- Application of validated clinical risk scores
- Medication safety and drug-related questions
- Hallucinations in the medical domain
- Health equity and bias
The remaining dimensions and full evaluation methodology are described in the accompanying benchmark documentation.
The dataset is divided evenly between a 250-question public development set and a 250-question private evaluation set. The public set allows clinicians, researchers, and developers to inspect the tasks, understand the evaluation criteria, reproduce results, and test their own systems. The held-out set preserves the value of independent evaluation by limiting direct optimization against the final test questions.
This structure supports transparency and credible comparison at the same time. The community can see what the benchmark values and how it measures performance, while the private set provides a stronger signal of how well a system generalizes to unseen clinical tasks.
Benchmark Design
Bedside Bench comprises 10 thematic benchmarks of 50 cases each (500 total), each organized around a failure mode:
| Benchmark | Grading | What it tests |
|---|---|---|
| Drug safety | Points-based rubric | Interactions, contraindications, renal/hepatic dosing, cross-reactivity |
| Guideline adherence | Points-based rubric | Guideline-concordant care and society recommendations |
| Landmark trials | Points-based rubric | Pivotal trial evidence and outcome interpretation |
| Medical hallucinations | Points-based rubric | Fictitious drugs, trials, and frameworks; appropriate refusal |
| Health equity | Points-based rubric | Bias-aware clinical reasoning across populations |
| Clinical calculations | Points-based rubric | Exact numeric answers: doses, scores, conversions |
| Numerical calculators | Points-based rubric | Calculator tasks with numeric rubric criteria |
| Conditional calculators | Points-based rubric | Calculator tasks with conditional or mixed scoring |
| Clinical safety (diagnostic setting) | F1-weighted rubric | Free-text diagnostic and follow-up management plans |
| Clinical safety (therapeutic setting) | F1-weighted rubric | Free-text medication and counseling management plans |
Two grading methods are used, one for discrete factual criteria and one for free-text management plans.
Points rubrics (8 benchmarks, 400 cases) grade a response against physician-authored criteria with signed weights. Most criteria award credit when satisfied; negative criteria deduct credit when a specific failure mode occurs.
F1-weighted rubrics (2 benchmarks, 100 cases) grade free-text management plans against a panel of roughly 50 candidate actions per case. Each action carries a signed severity weight: appropriate actions score +1 to +3 by importance of inclusion, inappropriate actions score −1 to −3 by harm if taken, and equivocal actions score 0. The final score is a weighted F1 over the actions mentioned in the response. Half of the candidate actions in these panels are inappropriate, so indiscriminate over-inclusion is penalized as heavily as omission of critical steps. A high score requires both restraint and completeness, consistent with the emphasis on omission-driven harm in recent clinical safety work (Wu et al., 2025).
Construction and Validation
Cases originate from question pools authored and validated by physicians for the internal evaluation of Doximity Ask. Before release, every case passed through a multi-stage pipeline: schema validation, near-duplicate detection within and across benchmarks, a model-assisted medical review pass that flagged potential factual errors, prompt–rubric mismatches, and unanswerable prompts, physician adjudication of all flagged items — including step-by-step recomputation of every calculator answer — and difficulty balancing to avoid saturating the easy end of the scale.
The released 500 cases are a randomized subsample of a larger physician-authored pool, and we retain the remainder as a private companion set. The public set enables full reproducibility, while the held-out set lets us detect score divergence indicating benchmark contamination as the public cases enter training corpora over time.
All cases were written expressly for evaluation purposes. They contain no protected health information. Some cases intentionally embed fictitious drugs, unsafe suggestions, or false premises designed to probe model behavior.
Descriptive Statistics
Across the 500 released cases:
- 6,085 rubric items in total: 2,943 positive (credit), 2,580 negative (penalty), and 562 equivocal.
- Points-rubric benchmarks contain 1,067 criteria (865 weighted +1, 152 weighted +2, 50 weighted −1), averaging 1.1 criteria per calculator case and up to 4.9 per health-equity case.
- The two management-plan benchmarks contain 5,018 candidate actions (mean 50.2 per case). By severity: 68 actions weighted +3 (critical to include), 675 at +2, 1,183 at +1, 562 at 0, 1,100 at −1, 1,237 at −2, and 193 at −3 (severe harm if taken).
- Prompts average 71 words (range 4–199). Calculator vignettes average over 100 words of clinical detail, while hallucination probes average 9 words — terse queries of the kind a physician would actually type, with no surrounding context to compensate for a fabricated premise.
Every case follows a uniform schema. Difficulty calibration was performed by randomly subsampling questions for each dimension (within a much larger pool of questions) for an average gpt-5.4-mini accuracy level of approximately 70%.
How Doximity Ask Performs
Limitations
The suite is sized for directional comparison between systems. Rubrics encode the judgment of the physician authors at a point in time, and criteria may lag the current standard of care as guidelines evolve. Errata will be made available publicly. Rubric grading in practice typically relies on LLM judges, which introduce their own variance and bias (Zheng et al., 2023); we recommend reporting the judge model and prompt alongside any scores. The cases are English-language and predominantly reflect U.S. guidelines and practice patterns. Finally, like every public benchmark, Bedside Bench will eventually appear in training data, which is why the held-out companion pool exists.
Conclusion
Reliability in clinical AI must be made observable, evaluated rigorously, and improved systematically. Releasing the evaluations themselves is part of that: Bedside Bench makes our grading criteria public, reproducible, and open to correction.
Bedside Bench measures a targeted slice of clinical behavior; establishing the safety of a clinical AI system requires broader evidence, including the system-level controls and physician review described in our previous post. The benchmark provides a common, physician-authored evaluation of the failure modes most relevant at the point of care, applicable to our system or any other.
No single benchmark can capture the full complexity of medicine. Clinical quality changes across specialties, patient populations, workflows, and settings. New model capabilities will reveal new failure modes. Strong evaluation will therefore require a growing suite of tests, informed by practicing clinicians and continuously updated as the technology evolves.
Bedside Bench is available at huggingface.co/datasets/doximity/bedside-bench.
References
Arora, R. K., Wei, J., Soskin Hicks, R., et al. (2025). HealthBench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775. https://doi.org/10.48550/arXiv.2505.08775
Wu, D., et al. (2025). First, do NOHARM: Towards clinically safe large language models. arXiv preprint arXiv:2512.01241. https://doi.org/10.48550/arXiv.2512.01241
Be sure to follow @Doximity_tech and @Doximity if you'd like to be notified about new blog posts.