Microsoft AI Outperforms Doctors in Diagnostic Study

Microsoft researchers built an AI diagnostic tool that solves complex medical cases more than twice as accurately as general doctors.

The study appeared Thursday in the journal Science under the title Sequential Diagnosis with Language Models. The system is called MAI-DxO. It reached an 83 percent accuracy rate on tough cases from the New England Journal of Medicine. General doctors scored less than half as well on the exact same cases.

Why Older Medical AI Tests Fell Short

Most medical AI models evaluate patients all at once. They ingest full medical charts, lab results, and scans in a single prompt. Earlier models became famous by acing multiple-choice medical board exams.

Dr. Dominic King leads the health unit at Microsoft AI. He pointed out that multiple-choice tests do not match daily hospital care.

“AI has become very good at getting a 100 percent pass rate on those. But of course, that’s not real-world. No patient ever came to see me with a nice summary.”

Dr. Dominic King, Corporate Vice President of Health, Microsoft AI

In a real clinic, doctors start with very little information. A patient arrives with a vague symptom like fatigue or chest pain. The doctor must ask questions, look for physical clues, order tests in stages, and update their working theory as new data arrives.

How MAI-DxO Copies Real-World Doctoring

Microsoft built MAI-DxO to follow that same clinical path. The system acts like a panel of medical specialists. A central coordinator directs specialized AI agents through five clear stages:

1. Patient History: An agent takes the chief complaint and asks targeted questions about symptoms, medications, and family history.
2. Physical Examination: A specialist agent reviews physical exam findings to narrow the list of possibilities.
3. Differential Hypotheses: The system forms an initial list of possible diagnoses and weighs probabilities.
4. Targeted Lab Work: The agent panel selects only required tests. It orders labs and scans to rule out specific illnesses.
5. Dynamic Updates: The system reviews new test results and updates its notes. It decides if more tests are needed before naming the final diagnosis.

Lowering the Cost of Healthcare Testing

This step-by-step approach also cut healthcare costs. Human doctors often order broad sets of blood tests and imaging out of caution. This habit, called shotgun testing, drives up hospital bills and wastes clinic time.

The AI coordinator ordered fewer unnecessary tests. That lowered the total cost of patient care while boosting accuracy.

Harsha Nori heads AI at Microsoft AI Health. He pointed to the practical side of the test.

“We’re not just measuring the accuracy of the systems. We’re also measuring how judiciously and carefully it goes through a diagnostic work-up.”

Harsha Nori, Head of AI, Microsoft AI Health

Tested on 300 Rare Cases and Emergency Rooms

The research team tested the system on a new benchmark called SDBench:

• Complex CPC Dilemmas: Researchers tested the model on 300 rare cases from the New England Journal of Medicine. Hospitals usually send these complex dilemmas to specialist boards. MAI-DxO solved 83 percent of these cases correctly. Human doctors with internet access scored less than half that well.
• Emergency Room Admissions: Researchers also tested the system on real emergency room visits from the MIMIC-IV-ED database. The AI matched or beat human doctors in emergency triage while keeping test costs down.

What Software Engineers Can Learn

For developers building AI systems, the Microsoft study offers a clear lesson.

Monolithic prompts often fail when tasks require multiple steps. Dumping an entire database into a large context window can overwhelm an LLM.

Instead, breaking tough workflows into small, coordinated steps delivers better results. By tracking hypotheses across multiple turns and ordering tools only when needed, multi-agent systems can tackle problems that single prompts cannot solve.

Formatting agent schemas, tool parameters, or system prompts? Validate and format your payloads with TeckScaler’s free client-side JSON to YAML Converter.

Primary Sources: Science: Sequential Diagnosis with Language Models | Reuters: Microsoft AI Diagnostic Tool Study