Authors: Zhuang-Fang Yi, Constantin Venhoff, Sonia Joseph, Jack Cook, Margaux Masson-Forsythe, Neeraj Mainkar
A new way to review how video AI understands surgical motion
Surgical video contains far more information than a procedure name. It shows how a surgery progresses: the actions being performed, the instruments being used, and the anatomy involved.
Our ultimate goal is to turn that video into useful, reviewable information for surgical teams. In the near term, this could make video easier to search, annotate, review, and use for training and quality improvement.
To be useful, however, a model must do more than produce the right label. We also need to understand what it considered similar and whether it relied on clinically meaningful evidence rather than camera position, lighting, or another unintended shortcut.
That is the question behind this work.
We developed a way to connect each prediction back to the model's learned patterns and, most importantly, to the original surgical clips that experts can inspect and challenge.
Opening the black box
Our Multiscale Vision Transformer, or MViT, was fine-tuned on human experts' annotated surgical videos to predict gestures, bounding boxes of tools, and anatomy. We described tools as the nouns of surgery and gestures as its verbs in our last blog post. Together, they help describe what is visible and what is happening.
For this study, we focused on laparoscopic gynecologic surgery and asked whether we could look beyond a gesture prediction to examine the visual and motion patterns behind it.
We combined three methods to make those patterns reviewable:
We added supervised contrastive learning to the model’s training objective. Alongside learning to predict a gesture, the model was encouraged to place clips with the same gesture closer together in its learned representation and move clips with different gestures farther apart.
Recent work has shown that sparse mixtures of low-dimensional manifolds can capture interpretable features in vision models better than 'traditional' linear methods like Sparse Autoencoders. Therefore, we fit a Block Sparse Featurizer, or BSF, without directly using gesture labels. It reconstructs the MViT’s dense representation as a sparse combination of recurring patterns that reviewers can inspect. Because the MViT backbone had already been fine-tuned with gesture, tool, and anatomy supervision, these patterns may reflect both recurring visual and motion signals and structure shaped by the training labels. A BSF group is therefore not another gesture class: it is an internal pattern that may respond to motion, instruments, tissue appearance, camera view, or combinations of these signals across clips.
UMAP (Uniform Manifold Approximation and Projection) projects the model’s learned representation into an interactive three-dimensional map. Each point represents a video clip and is colored by the model’s predicted gesture. Reviewers can select a point to view its source clip, prediction, confidence, and timestamp. The map helps reviewers explore what the model considers similar, but it does not show which patterns caused a prediction.
What BSF helps us see
After fine-tuning the MViT model, BSF was fitted to representations from 1,781 private OB/GYN clips during MViT model fine-tuning. Each clip lasts from 3 to 10 seconds. The resulting dictionary contained 256 groups, of which 184 were active and 72 were inactive. We conducted an exploratory review of 16 selected high-activation groups per clip; however, this was not an independent surgeon-validation study.
These three layers need to remain separate:
A single clip activates only a small subset of the available BSF groups. One group may respond to visible suture and tissue closure, while others respond to the instrument, operative field, smoke, camera angle, or a motion shared across several gestures.
To interpret a group, we retrieve the clips that activate it most strongly and look for repeated evidence. In the figure below, the strongest clips repeatedly show visible suture material, needle holders, tissue approximation, and similar pelvic operative fields. A reviewer might therefore describe the group as suturing and tissue closure. That description is an interpretation; it is not a label learned by BSF, and it is not a second gesture prediction.
Other groups are less visually uniform. That can still be informative. A broader group may capture energy-device contact, tissue response, smoke, illumination, camera position, or a temporal movement that is difficult to see in one frame. This also explains why a clip predicted as cut can activate a group provisionally associated with dissect: the actions may share an instrument, tissue response, or lower-level movement.
BSF does not tell us why a model made a prediction with certainty. In this experiment, the sparse reconstruction achieved an R² of approximately 0.646, which measures how much of the representation was captured by the sparse reconstruction, leaving a substantial portion of the dense representation unexplained. BSF is therefore one possible deconstruction, not a complete account of the representation. It gives us a structured review process: identify the groups active for a clip, retrieve their strongest examples, look for repeated evidence, and ask whether that evidence is clinically meaningful or an unintended shortcut.
In short, the prediction is the model's answer. The BSF group is a recurring internal pattern. The group description can be the reviewer's interpretation.
Why this should matter beyond the model
The value of this work is not the three-dimensional map itself. It is the review workflow the map can support.
More efficient annotation
Long procedures are expensive to label second by second. Grouping similar moments could help reviewers prioritize representative clips, uncertain regions, and outliers.
Better model quality assurance
Aggregate accuracy can hide unreliable model behavior. Linked clips let reviewers check whether a pattern reflects surgical motion or an irrelevant cue such as smoke, lighting, or camera state.
More informative failure analysis
When cutting and dissection are confused, a score shows uncertainty. The representation map also shows which clips the model considered similar, giving clinical and engineering teams a more concrete place to begin.
A path toward data-driven surgical action vocabularies
Human gesture labels can vary and may combine several lower-level motions. These methods may help reveal recurring tool-tissue interactions or composite actions that are not captured by the original vocabulary.
Because our current representation was trained with gesture labels, some structures will naturally reflect those labels. Next, we need to test whether similar patterns emerge from models never trained on our gesture vocabulary. One hypothesis is that interpreting video foundation models could reveal lower-level and composite action structures and, over time, support more evidence-based ontologies for surgical video annotation.
What we still need to test
The discovered groups are not yet validated clinical concepts. Important questions remain:
- Do different surgeons describe the groups in similar ways?
- Are the patterns stable across hospitals, devices, procedures, and surgeons?
- Does the review interface reduce annotation time without reducing label quality?
- Are some groups driven by shortcuts rather than surgical motion?
A planned comparison with V-JEPA 2.1, a self-supervised video model, will help test whether the patterns come from the surgery itself or primarily from our chosen labels. We also plan to compare BSF with other disentangling methods and ask clinicians to describe groups without seeing their gesture enrichment.
Toward useful and reviewable surgical video
Our ultimate goal is to help surgical teams learn from the enormous amount of information already contained in their videos. Near-term tools could support search, annotation, case review, training, and quality improvement. Over time, they may also help clinicians study variation across procedures and investigate whether recurring patterns are associated with surgical technique, quality, or patient outcomes.
The goal is not to replace surgical judgment. It is to keep model outputs connected to the video evidence behind them, so experts can inspect, question, and challenge what the model appears to have learned.
We do not yet have a complete explanation of surgical intelligence. But each output can now lead back to related clips and reviewable video evidence. For clinical AI, what matters is performance and the ability to inspect and challenge what a model learned.
Acknowledgements
This work grew from a collaboration between Surgical Data Science Collective (SDSC) and World Mechanics, an AI research lab founded by Sonia Joseph and Constantin Venhoff that studies interpretable representations for physical AI and video world models. SDSC conducted the experiments internally using its own model and computing environment. World Mechanics contributed research guidance on interpretability methods but did not receive or access SDSC’s model, model weights, code, or private surgical datasets. Our shared question was whether interpretability methods could uncover recurring patterns already present in the model's learned representation and connect those patterns to source video for review.
References
Wang, Z., Lu, B., Long, Y., et al. “AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy.” MICCAI, 2022. Source
Yi, Z.-F., Angara, S., Otto, S., Zhu, J., Cook, J., & Masson-Forsythe, M. (2026). Can AI Understand a Surgery, Not Just Look at It? Surgical Data Science Collective. Source
Li, Y., Wu, C.-Y., Fan, H., et al. (2022). MViTv2: Improved Multiscale Vision Transformers for Classification and Detection. Proceedings of CVPR, 4804–4814. Source
Khosla, P., Teterwak, P., Wang, C., et al. (2020). Supervised Contrastive Learning. Advances in Neural Information Processing Systems, 33. Source
Fel, T., Kowal, M., Jacobs, M., et al. (2026). Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds. arXiv:2606.25234. Source
McInnes, L., Healy, J., & Melville, J. (2018). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv:1802.03426. Source
Mur-Labadia, L., Muckley, M., Bar, A., et al. (2026). V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning. arXiv:2603.14482. Source
Leike, J., Wu, J., Bills, S., Saunders, W., Gao, L., Tillman, H., & Mossing, D. (2023). Language Models Can Explain Neurons in Language Models. OpenAI. Source
