%20FRC%20Headshots.png)
In our last benchmarking effort, we asked a deliberately simple question: can general-purpose AI identify the instruments visible in surgical video? Across multiple video-language models (VLMs) and frontier model benchmarks on surgical instruments/tools, the answer was clear.
Models trained on the breadth of the internet often struggled with the specialized visual vocabulary of surgery, while smaller systems improved dramatically when given task-specific supervision.
But identifying an instrument is only the beginning. A grasper may be visible without grasping. Scissors may enter the frame without cutting. A surgical action may continue even when the active instrument is partly hidden.
If tools are the nouns of surgical video, gestures are its verbs. But the relationship is not one-to-one: our expert-defined gesture vocabulary teaches a model to combine instruments, motion, tissue response, timing, and likely surgical intent into a clinically meaningful action. Understanding surgery requires both spatial evidence describing what is present and temporal context revealing what is happening.
This follow-up benchmark moves from nouns to verbs. We compare a purpose-built video model to surgically pretrained models and frontier generalists. Where full-video results are available, every model watches the same OB-GYN case and is judged against the same expert timeline. The question is no longer simply whether AI can see surgery. It is whether AI can read and understand it.
Why the verbs matter
A gesture is a short, purposeful interaction between an instrument and tissue. Grasp means hold or reposition tissue. Dissect means separate tissue planes. Coagulate means apply energy to seal tissue or control bleeding. Cut divides tissue; clip closes a structure; suture joins or secures it. The benchmark also includes aspirating, irrigating, removing a specimen, and moments when none of these actions is underway.
Recognizing these actions could make long surgical videos searchable, help educators find teaching moments, and support documentation or quality review. But those uses depend on timing. A system that correctly names an action yet places it twenty seconds early has not actually understood the operation.
The two tests, in one minute
We ran two tests. First, four surgery-focused setups—Multiscale Vision Transformer (MViT) Multi-task, two versions of SurgMotion, and one version of LEMON—used identical training, validation, and 382-clip test splits. This tested how well each setup could recognize an action in a prepared clip.
Next, six systems, including three frontier AI models, followed the same uninterrupted 21-minute, 42-second operation. This harder test asked whether they could keep track of an action as it began, continued, and ended.
In short: clips test recognition; a full operation tests following and real-world video understanding.
TWO QUESTIONS 1. Is broad surgical knowledge with surgical video enough, or does a model need to be built around this exact task? 2. Can the largest general-purpose AI systems replace the specialist model?
Comparison one: surgical experience versus task experience
SurgMotion and LEMON are two surgical foundation models selected for our evaluation because they have already learned rich representations from large collections of surgical videos. Evaluating these models therefore provides an important domain-specific benchmark for understanding how well the pretrained surgical representations transfer to downstream surgical tasks such as gesture recognition. That makes them valuable starting points: they have seen anatomy, instruments, smoke, blood, camera motion, and the unusual visual texture of surgical videos. Yet pretraining is not the same as knowing our local gesture vocabulary or when one action ends and the next begins.
Are these collections large enough for surgical pretrained models? We cannot call them saturated: neither project shows that more representative data has stopped helping. SurgMotion reports 3,658 hours and LEMON 938 hours, which are substantial collections, but still uneven in coverage. Hours are not equal units of learning: surgical content is relevant, but long operations repeat similar views. SurgMotion is broad, yet 2,860.85 hours (78%) are neurosurgery and only 23.12 hours are gynecology. LEMON spans 35 procedures, but its 4,194 endoscopy videos come from various online sources. They trade breadth for relevance; coverage remains uneven.
On the shared held-out gesture test set, exact matches can be compared directly across four configurations from three model families. A clip counts only when the model returns the complete correct gesture label set. MViT Multi-task reached 67.6%, SurgMotion attentive reached 59.2%, LEMON linear reached 29.6%, and SurgMotion linear reached 24.9%. SurgMotion’s attentive probe improved sharply over its simpler linear probe, showing how much the task-specific adaptation layer matters. But prepared clips do not require a model to discover where an action begins and ends across a full operation. The continuous runs below answer that harder question.
MViT is a video model designed to learn both what is visible and how a scene changes over time. In SDSC’s MViT Multi-task system, one shared MViT backbone supports separate outputs for three parts of surgical understanding: the tools that are present, the anatomy in view, and the gesture being performed. This benchmark focuses on its gesture output. The model begins with pretrained weights, then is fine-tuned using our labeled gesture data and trained to label an uninterrupted procedure. The benchmark video and clips derived from it were not included in fine-tuning. On the separate uninterrupted benchmark case, it agreed with the annotations 79% of the time. MViT remained the leader, but SurgMotion at 75.3% and LEMON at 69.2% show that surgical pretraining can transfer much further in continuous video than the prepared-clip result alone suggested.
Comparison two: when every model watches the whole case
This is the most direct comparison. All six continuous prediction tracks were judged against the same labeled moments. The three surgery-aligned systems occupied the top three positions: MViT Multi-task reached 79.0% agreement, SurgMotion 75.3%, and LEMON 69.2%. OpenAI’s SOL reached 58.0%, Moonshot AI’s Kimi K3 reached 56.6% after a documented label correction, and Anthropic’s Claude Opus 5 reached 45.8%. SurgMotion came within 3.7 points of MViT, while LEMON finished 11.2 points ahead of the strongest untouched frontier result.
That changes the interpretation. This is not simply one purpose-built model defeating every foundation model. Systems with direct surgical task supervision or surgical-video pretraining formed the top tier. MViT’s lead shows that close task alignment still matters. SurgMotion and LEMON show that surgical pretraining can transfer meaningfully. The frontier systems bring broad knowledge and reasoning, but internet-scale breadth did not replace surgical experience in this case.
The aggregate chart establishes the ranking, but not the quality of the timeline. MViT combined the highest agreement with the strongest balance across gesture classes. SurgMotion came close on the overall scores, but switched labels much more frequently during the active middle of the case. LEMON also outperformed the frontier models, although its lower class-balanced score shows that its strength was concentrated in the more common actions. Among the frontier systems, SOL maintained long states, K3 detected more activity but changed its mind frequently, and Opus remained more conservative.
The take-away message
SURGICAL EXPERIENCE MATTERS—AND TRAINING FOR THE EXACT JOB STILL LEADS. In this uninterrupted case, MViT Multi-task, SurgMotion, and LEMON were the three strongest systems. MViT remained first, but SurgMotion came within 3.7 points. Models that had learned directly from surgical data outperformed the general-purpose AI systems. In this benchmark, broad internet knowledge alone did not match surgical experience.
The models with surgical experience stood strongest when the test moved from selected clips to an uninterrupted case. MViT Multi-task remained the leader, SurgMotion came close, and LEMON also finished ahead of the frontier models. However, the frontier models did not fail in the same way. SOL followed long actions but missed brief ones. Kimi K3 noticed more activity but changed its answer too often. Claude Opus was more cautious and missed cutting entirely in this case. There is no single “AI error.” Different models lose the surgical thread in different ways, and those differences will shape how they can be used safely. Because this comparison comes from a single continuous case, it reveals how the models behaved under the same conditions, however, it does not intend to rank the specialist model universally superior.
Coming in Part 2: We will widen the test to more continuous surgical cases and compare MViT Multi-task directly with surgical foundation models and frontier AI. We will also introduce V-JEPA, a video model pretrained at internet scale. Can broad experience with motion and everyday video compete with models that have learned directly from surgery? The next chapter moves beyond who won one case to a more important question: which kind of experience travels best to new operations?
Sources
[1] Part Zero: Bigger Isn’t Better—Why Surgical AI Is Harder Than the Hype Suggests. View source
[2] Fan et al., Multiscale Vision Transformers. View source
[3] OpenAI, Previewing GPT-5.6 Sol. View source
[4] Moonshot AI, Kimi K3 official repository and report. View source
[5] Anthropic, model system-card catalog. View source
[6] Wu et al., SurgMotion. View source
[7] Che et al., LEMON (CVPR 2026). View source
[8] Morais et al., SAGES surgical gesture taxonomy consensus. View source
[9] Leibetseder et al., surgical actions in laparoscopic gynecology. View source
.png)
