Edit this page

NA-MIC Project Weeks

Back to Projects List

Deploying ScribblePrompt and MultiverSeg for interactive segmentation as a 3D Slicer extension

Key Investigators

Project Description

We will develop a 3D slicer extension to deploy two interactive segmentation models aimed at helping researchers and clinicians perform new segmentation tasks:

ScribblePrompt is a deep learning model that enables users to interactively segment an image using scribbles, clicks, and bounding boxes. The model is designed to generalize to new labels and types of biomedical images and uses a lightweight UNet-based architecture so it runs quickly even without a GPU.

MultiverSeg extends this interactive approach to speed up the segmentation of sets of similar images. Using the same interaction types as ScribblePrompt (scribbles, clicks, bounding boxes), the system learns from each segmentation to improve subsequent predictions. Given enough similar example segmentations, MultiverSeg can also automatically segment new images without any user interaction.

Objective

  1. Implement a 3D slicer extension for interactive segmentation with ScribblePrompt using scribbles, clicks, and bounding boxes
  2. Add MultiverSeg to the extension to enable interactive and automatic segmentation of sets of images (or slices from 3D volumes)
  3. Compare to other interactive segmentation tools

Approach and Plan

  1. We will start by following the tutorial for developing a 3D slicer extension

Progress and Next Steps

  1. Describe specific steps you have actually done.

Illustrations

No response

Background and References

Relevant Publications:

Wong, H.E., Rakic, M., Guttag, J., & Dalca, A.V., (2024). ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Biomedical Image. In European Conference on Computer Vision. paper code

Wong, H.E., Ortiz, J.J.G., Guttag, J. & Dalca, A.V., (2024). MultiverSeg: Scalable Interactive Segmentation of Biomedical Imaging Datasets with In-Context Guidance. arXiv preprint arXiv:2412.15058. paper code

Related 3D Slicer extensions: