The dominant narrative in artificial intelligence (AI) right now is seductively simple: scale up the model, add more general data, increase compute, performance will follow. This has worked remarkably well for language, image generation, and coding assistants, so it’s worth paying attention to the cases where it doesn’t work.
In collaboration with Surgical Data Science Collective (SDSC), X.Y. Han, Kirill Skobelev, and colleagues at the University of Chicago Booth School of Business recently published a timely reality check for medical AI. In “A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scaling,” we test whether generic out-of-the-box AI “foundation models” can reliably understand surgical video like a much smaller, specialized SDSC model.1
This blog explores five key experiments from the paper, which reveals a systematic failure of frontier AI through a seemingly tractable perception problem, and provides empirical evidence in favour of SDSC’s specialized approach to model building.
The promise of foundation models
We live in the age of “foundation models” – large neural networks trained on vast amounts of data. Vision Language Models (VLMs) are one type of foundation model, and these typically produce text outputs based on image, video, and text inputs. Examples include GPT by OpenAI, Claude by Anthropic, and Qwen by Alibaba.
These models have produced remarkable results on multipurpose AI benchmarks, but the prevailing assumption is that scaling is the answer to solving increasingly complex tasks. To test this, tool detection was selected as a relatively simple surgical benchmark, because if frontier AI cannot reliably identify a suction device in a surgical scene, more advanced surgical reasoning – action recognition, phase detection, anomaly flagging – remains far away.
However, considering that surgery is significantly more visually challenging than tasks like detecting pedestrians on a sidewalk, it is understandable that without the inclusion of surgical video in the training data, a model will undoubtedly be less successful.
Experiment 1: Zero-shot AI performs surprisingly poorly
20 open-weight VLMs were tested on Endoscopic Endonasal Approach (EEA) neurosurgery videos using zero-shot evaluation – meaning the models were asked to identify tools without task-specific training.
Every model performed no better than if it had simply guessed the most common surgical instrument, which achieved a baseline Micro-F1 score of 47.3% (Figure 1). The closer this score is to 100%, the higher the overlap between predicted tools and ground truth annotations.
This is particularly notable because these same models perform extremely well on many conventional AI benchmarks, and the findings suggest that competence on general-vision language tasks does not transfer cleanly into specialized medical domains.
Experiment 2: Fine-tuning somewhat helps
Next, one model, Gemma 3 27B, was fine-tuned using LoRA adapters (a method that allows task-specific adaptation without re-training the entire model) to output structured JavaScript Object Notation (JSON) predictions identifying surgical tools. Performance improved dramatically, with Micro-F1 increasing from near-baseline to 81.40%.
This demonstrates that targeted domain training matters enormously, however, the system was still making errors on roughly one in five predictions. Fine tuning can narrow the gap, but it does not close it.
Experiment 3: More compute leads to overfitting
One of the AI industry’s core assumptions was tested: would scaling solve the problem?
By increasing the number of trainable parameters nearly 1,000-fold, we got a result of classic overfitting (Figure 2). Training accuracy reached 98.6%, and validation performance on unseen surgical video also improved. However, there is a persistent gap between training and validation performance across datasets.
In other words, the model memorized the training data rather than developing robust surgical understanding. This is one of the paper’s most important findings and demonstrates how maximizing computational power alone cannot produce generalizable surgical intelligence.
Experiment 4: A small specialist model wins
Perhaps the most humbling results came from comparison with YOLOv12-m, a comparatively lightweight object detection model specifically designed for identifying objects in images.
Despite having roughly 1,000 times fewer parameters than the large VLMs (which scored below the 47.3% Micro-F1 score baseline in experiment 1), YOLOv12-m scored significantly better, at 80.48%.
At a time when AI discourse is dominated by scale, this finding reinforces a longstanding engineering principle: specialized task-specific tools can outperform fine-tuned general-purpose systems on tightly constrained tasks.
Experiment 5: The pattern holds across datasets
To ensure the findings weren’t specific to this neurosurgery dataset, key experiments were repeated on three additional surgical datasets: CholecT50 (laparoscopic gallbladder surgery), SurgVU (robotic-assisted training sessions on porcine tissue), and PitVis-2023 (endoscopic pituitary neurosurgery).
The pattern held, with the smaller fine-tuned open-weight models and specialized computer vision model YOLOv12-m both outperforming all zero-shot methods, including frontier proprietary models from GPT, Gemini, and Claude families.
Why is Med-AGI so hard?
It would be easy to frame this as a data problem, considering that surgical datasets are small, annotation is expensive, and domain shift is severe. But Experiment 3 showed us that more parameters and a better training-set fit doesn’t necessarily translate to validation gains. The bottleneck is more than just data volume and model capacity.
The paper’s results counsel caution and importantly demonstrate that modern AI can recognize many generic visual patterns and answer text-based questions on medical knowledge, but that alone will not lead to a generically useful surgical intelligence system. Moreover, the current performance gaps cannot be solved by scaling a vision transformer trained on internet images, even with extensive fine-tuning.
For machine learning researchers, the paper is a useful empirical counterweight to “scaling solves everything.”
For clinicians and the wider public, it is a reminder that genuine medical AI remains a profoundly difficult challenge, and that medical intelligence is not simply “internet intelligence applied to medicine.”
And for anyone who uses a chatbot: VLMs will always give you a confident answer, even if it is wrong. The chat format creates an implicit sense of authority that makes it easy to trust responses uncritically, so in medical contexts especially, it is imperative to maintain a healthy level of skepticism.
Read the full paper here: https://www.chicagobooth.edu/research/center-for-applied-artificial-intelligence/research/our-faculty-research/2026/a-comparative-study-in-surgical-ai
X.Y. Han’s Data Science Roundtables:
https://www.surgicalvideo.io/blog/xy-han-neural-collapse-in-deep-net-training
https://www.surgicalvideo.io/blog/x-y-han-phd-and-kirill-skobelev-what-will-get-us-to-surgical-agi
1. Skobelev K, Fithian E, Baranovski Y, Cook J, Angara S, Otto S, et al. A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scaling. 2026; doi:10.2139/ssrn.6476260
.png)
