A study published in February 2023 found that the first iteration of ChatGPT, built on the GPT-3.5 model OpenAI debuted in 2022, scored at or near the passing threshold on publicly available questions from all three steps of the United States Medical Licensing Examination (USMLE). That single benchmark has shaped two years of medical artificial intelligence research, vendor pitches and procurement decisions.

Large language models have posted passing-level scores on medical licensing exam questions since early 2023, according to Research & Development World. A new Google study on AMIE — the company's Articulate Medical Intelligence Explorer — now pushes the evaluation from test items to direct patient encounters, under licensed-physician supervision.

Why move from exam items to bedside encounters?

Standardized licensing questions isolate one skill: factual medical reasoning drawn from a fixed bank. Patient encounters add history-taking, follow-up questioning, ambiguous presentations and patient communication. R&D World frames the AMIE trial as evidence that conversational medical AI can operate in the latter setting, with doctors reviewing each step.

What does the AMIE study actually test?

The AMIE study evaluates the model in direct dialogue with patients inside a clinical workflow. The available excerpt does not state patient count, comparator performance, study site or supervising physician panel size. Those are the metrics research managers will request before any integration commitment.

How should labs and budget holders read this?

For research directors running medical AI programs, the AMIE work shifts evaluation weight from benchmark accuracy to supervised clinical performance. The February 2023 USMLE result remains the reference point for medical knowledge. AMIE adds the reference point for clinical interaction. Together they form a two-axis evaluation framework: knowledge plus bedside behavior.

Evaluation checklist for medical AI procurement

• Licensing-exam benchmark score (USMLE, all three steps)
• Supervised patient-encounter data with published methodology
• Error taxonomy by clinical category
• Multi-site replication outside the developer
• Comparator performance against a physician baseline

What limits apply to the evidence so far?

The original USMLE study used publicly available exam items, a method that does not measure patient outcomes. AMIE's bedside evidence, as R&D World describes it, runs under physician oversight — a control that limits the autonomy claim and also limits the autonomy risk. Labs considering integration should treat the vendor summary as data to verify rather than as a conclusion, and request the missing variables: sample size, patient demographic mix and the supervising physician protocol.

What changes in vendor evaluation now?

Before 2023, medical AI procurement centered on regulatory clearance and benchmark dataset performance. AMIE-style trials add a third column: supervised patient interaction data. R&D managers building evaluation rubrics should now score physician-supervised clinical encounter data alongside licensing-exam scores when comparing vendors.

What is the forward question for the field?

The combination of USMLE-level exam performance and supervised bedside evidence points toward the next evaluation phase: multi-site trials with measured patient outcomes rather than test scores. Research leaders should track whether Google publishes AMIE performance against physician baselines across hospital systems, and whether peer-reviewed replication emerges from independent groups before any clinical deployment budget moves.