DIAGNOSTIC AND THERAPEUTIC ACCURACY OF LARGE LANGUAGE MODELS IN EMERGENCY DIGESTIVE SURGERY: A PRELIMINARY EVALUATION USING THE ARTIFICIAL INTELLIGENCE PERFORMANCE INSTRUMENT (AIPI) SCORING SYSTEM marie.desonniaux@skynet.be

366-07

DIAGNOSTIC AND THERAPEUTIC ACCURACY OF LARGE LANGUAGE MODELS IN EMERGENCY DIGESTIVE SURGERY: A PRELIMINARY EVALUATION USING THE ARTIFICIAL INTELLIGENCE PERFORMANCE INSTRUMENT (AIPI) SCORING SYSTEM

4

Belgium

Oral or Poster

Large Language Models (LLMs) including ChatGPT-4, Claude Sonnet 4, and DeepSeek have gained interest as decision-support tools in medical education. Their performance in high-acuity contexts like emergency digestive surgery, however, had remained under-evaluated. This study presented a preliminary evaluation of the diagnostic and therapeutic performance of three LLMs in emergency digestive surgery using the Artificial Intelligence Performance Instrument (AIPI) and aimed to determine their potential to assist junior surgical trainees in diagnostic reasoning and management planning.

Data from twenty emergency digestive surgery cases were collected prospectively between May and July 2025. Each case was independently submitted to ChatGPT-4, Claude Sonnet 4, and DeepSeek using identical standardized prompts. Responses were scored independently using AIPI across four dimensions: primary diagnosis, differential diagnosis, investigations, and treatment. An independent blinded evaluation by expert surgeons is currently being conducted for comparative analysis.

All models correctly identified the primary diagnosis in 14/20 cases (70%). Claude Sonnet 4 and ChatGPT-4 achieved 70% accuracy in differential diagnoses, and DeepSeek slightly higher at 75%. Regarding complementary investigations, Claude 4 scored highest (90%), followed by ChatGPT-4 (85%) and DeepSeek (80%). Treatment recommendations were appropriate in 85% of cases for DeepSeek, and 80% for the other two models.

This preliminary evaluation suggests that LLMs provide clinically relevant support in emergency surgical decision-making. Their consistent performance across diagnostic and therapeutic tasks indicates potential value in medical education. Ongoing expert validation will clarify their role in complex cases.