Edit this page

NA-MIC Project Weeks

Back to Projects List

Deploying ScribblePrompt and MultiverSeg for interactive segmentation as a 3D Slicer extension

Key Investigators

Project Description

We will develop a 3D slicer extension to deploy two interactive segmentation models aimed at helping researchers and clinicians perform new segmentation tasks:

ScribblePrompt is a deep learning model that enables users to interactively segment an image using scribbles, clicks, and bounding boxes. The model is designed to generalize to new labels and types of biomedical images and uses a lightweight UNet-based architecture so it runs quickly even without a GPU.

MultiverSeg extends this interactive approach to speed up the segmentation of sets of similar images. Using the same interaction types as ScribblePrompt (scribbles, clicks, bounding boxes), the system learns from each segmentation to improve subsequent predictions. Given enough similar example segmentations, MultiverSeg can also automatically segment new images without any user interaction.

Objective

  1. Implement a 3D slicer extension for interactive segmentation with ScribblePrompt using scribbles, clicks, and bounding boxes
  2. Add MultiverSeg to the extension to enable interactive and automatic segmentation of sets of images (or slices from 3D volumes)
  3. Compare to other interactive segmentation tools

Approach and Plan

  1. We will start by following the tutorial for developing a 3D slicer extension

Progress and Next Steps

  1. Adapted the SlicerSegmentWithSAM extension to use ScribblePrompt
  2. Added support for storing and retrieving the previous prediction for each slice to use as input when updating the segmentation
  3. Added support for positive and negative scribble inputs

Next Steps:

  1. Add support for bounding box inputs
  2. Test and debug MultiverSeg predictions for slice-by-slice segmentation of volumes
  3. Refactor the extension to integrate into SegmentEditor instead of being a standalone module

Illustrations

Background and References

Relevant Publications:

Wong, H.E., Rakic, M., Guttag, J., & Dalca, A.V., (2024). ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Biomedical Image. In European Conference on Computer Vision. paper code

Wong, H.E., Ortiz, J.J.G., Guttag, J. & Dalca, A.V., (2024). MultiverSeg: Scalable Interactive Segmentation of Biomedical Imaging Datasets with In-Context Guidance. arXiv preprint arXiv:2412.15058. paper code

Related 3D Slicer extensions: