Can AI Interpret Medical Images and Converse Like a Doctor? AMIE’s Multimodal Breakthrough in Telemedicine

Doctoral student Marta Arbizu Gómez explores how AMIE, Google DeepMind’s multimodal AI, is transforming medical consultations and the future of telemedicine.

Telemedicine Is No Longer Just Text: The Challenge of Integrating Images and Clinical Documents

Remote medical care has changed radically in recent years. Today, many patients send photographs of skin lesions, electrocardiograms (ECGs), laboratory reports, or clinical documents directly from their phones during an online consultation.

However, the majority of artificial intelligence (AI) systems used in healthcare still operate primarily through text. This represents an important limitation: in real-world clinical practice, physicians continually combine conversation with visual and documentary information to make diagnostic decisions.

A study published in Nature Medicine in 2026 presents a new multimodal version of AMIE (Articulate Medical Intelligence Explorer), a system developed by Google DeepMind capable of:

• conducting clinical conversations,
• requesting relevant images or documents,
• interpreting that multimodal information,
• and engaging in diagnostic reasoning dynamically during the consultation.

The aim of the study was to assess whether this AI could perform in complex telemedicine consultations by directly comparing it with primary care physicians.

How Does the AMIE Multimodal System Work?

The system’s main innovation is a mechanism called “state-aware reasoning” (“clinical state-aware reasoning”).

Rather than simply answering one question at a time, AMIE structures the consultation into different phases, imitating the usual clinical reasoning process:

1. Medical history and patient history: The system collects symptoms, medical history, and possible risk factors.
2. Requesting and interpreting multimodal information: If it determines that it needs more information, it can request dermatological photographs, ECGs, or clinical documents.
3. Dynamic generation of diagnostic hypotheses: The AI continuously updates its possible diagnoses according to the new information received.
4. Diagnostic and treatment plan: Finally, it generates recommendations and answers the patient’s questions.

This approach seeks to reproduce the progressive reasoning physicians use during a real consultation.

How Was This Study on AI and Healthcare Conducted?

To assess the system’s performance, the researchers conducted an OSCE-type study (Objective Structured Clinical Examination), a methodology widely used to assess medical clinical competencies.

Participants included:

• 19 primary care physicians, 
• 25 trained actor-patients, 
• and 18 independent specialist evaluators.

The researchers designed 105 simulated clinical scenarios, including:

• dermatological photographs,
• ECG tracings,
• clinical documents,
• and clinical conversations via multimodal chat.

Each actor-patient had two consultations:

• one with a human physician,
• and another with multimodal AMIE,

all in randomized order and with blinded evaluation by specialists.

What Did the Study Results Show?

The results were particularly striking.

Greater Diagnostic Accuracy

AMIE achieved greater diagnostic accuracy than the primary care physicians across the different levels of differential diagnosis assessed.

In addition, its diagnostic lists were:

• more comprehensive,
• more consistent,
• and contained fewer relevant omissions.

Better Interpretation of Images and Documents

One of the most important findings was the system’s ability to correctly integrate visual information into the clinical conversation.

Specialists rated AMIE more highly for:

• image interpretation,
• reasoning based on visual artifacts,
• explaining findings to the patient,
• and handling questions related to medical images.

Interestingly, actor-patients perceived that the AI explained images better than many human physicians, probably because it explicitly verbalized what it observed and how this influenced the diagnosis.

Clinical Conversation Remains Essential

The study also demonstrated something highly relevant: analyzing isolated images was not enough. When the system only viewed images without maintaining a clinical conversation, diagnostic accuracy clearly decreased.

This reinforces a fundamental idea: clinical context and medical history remain essential even in the era of multimodal AI.

The combination of:

• clinical dialogue,
• structured reasoning,
• and visual interpretation

was what made it possible to achieve the best results.

Can AI Be Empathetic?

In addition to diagnostic accuracy, the researchers assessed communication and emotional factors.

Patients rated AMIE equally to or higher than physicians in areas such as:

• empathy,
• active listening,
• clarity of explanations,
• addressing concerns,
• and the confidence it conveyed.

Although this does not mean that AI will replace the physician-patient relationship, it demonstrates the potential of advanced conversational systems to improve the telemedicine experience.

What Limitations Does the Study on AMIE’s Multimodal Breakthrough in Telemedicine Have?

The authors highlight several important limitations:

• This is not yet a real clinical trial.
• All consultations were simulated.
• The interaction took place via written chat, not video consultation.
• The system remains experimental and is not ready for autonomous clinical use.

In addition, the researchers acknowledge that it is still necessary to validate:

• safety,
• robustness,
• and system fairness

in real populations and diverse care settings.

What Are the Implications for the Future of Telemedicine?

This work represents one of the first steps toward AI systems capable of integrating clinical conversation and multimodal reasoning in real time.

In the future, technologies like this could:

• streamline clinical screening,
• support remote consultations,
• improve access to healthcare in areas with few professionals,
• and help prioritize patients according to severity.

However, the authors stress that the goal is not to replace physicians, but to develop support tools capable of complementing clinical practice.

How Is This Breakthrough Related to NeuronUP?

At NeuronUP, technology and cognitive health are advancing in parallel toward increasingly personalized and digital models.

The integration of multimodal AI tools could facilitate:

• faster and more accessible clinical assessment,
• better detection of cognitive and neurological symptoms,
• and more continuous monitoring in remote settings.

Furthermore, the combination of AI-assisted diagnosis, digital biomarkers, and personalized cognitive rehabilitation programs opens the door to more comprehensive and adaptive care models.

Conclusion

The new AMIE multimodal system demonstrates that conversational artificial intelligence can move beyond text by integrating images and clinical documents into simulated medical consultations.

In this study, the AI achieved results comparable or superior to those of primary care physicians in diagnostic accuracy, multimodal interpretation, and communication quality.

Although there is still a long way to go before real-world clinical implementation, these results show the enormous potential of multimodal AI to transform telemedicine and support healthcare professionals in the future.

References

• Saab K, Park C, Strother T, Freyberg J, Barrett DGT, Cheng Y, Weng WH, Stutz D, Tomasev N, Palepu A, et al. Advancing conversational diagnostic AI with multimodal reasoning. Nature Medicine. 2026. doi:10.1038/s41591-026-04371-0.

If you enjoyed this article about AMIE’s multimodal breakthrough in telemedicine, you may also be interested in these NeuronUP articles: