Edit this page

NA-MIC Project Weeks

Back to Projects List

Real-time ultrasound AI segmentation using Tensorflow and PyTorch models

Key Investigators

Project Description

The module “Segmentation U-Net”, from extension SlicerAIGT, applies deep learning models on an ultrasound image stream to generate the predicted segmentation in real time. This is shown in the following example, where it is used to detect tumour tissue (highlighted in red) on breast images. That way, we can apply a live volume reconstruction on this prediction and visualize the complete region of interest (in this case, the area of the tumour). Another instance, using spine images, is shown in (Figure 1).


Currently, this module supports models trained with the TensorFlow ecosystem. However, in recent years, PyTorch has become an increasingly popular machine learning framework, especially in medical imaging applications (an example of this is the MONAI framework, which is based on PyTorch).

We have developed a separate module to run the inference of PyTorch model for the segmentation of breast ultrasound images: Breast Lesion Segmentation (Figure 2). However, our module does not integrate parallel processing to enable real-time image segmentation.

In this project, we aim to adapt the current “Segmentation U-Net” module to enable the use of models trained with both ecosystems, PyTorch and TensorFlow, for real-time ultrasound image segmentation.

In addition, we will discuss further improvements for this module. For instance, automatically visualize the prediction overlayed on the input ultrasound image and avoid changing to different modules to activate the visualization.


  1. Adapt the previous module to support models trained with PyTorch
  2. Automatically display the AI segmentation overlayed on the input ultrasound image

Approach and Plan

  1. Describe specific steps of what you plan to do to achieve the above described objectives.

Progress and Next Steps

  1. Describe specific steps you have actually done.



Figure 1. Real-time spine segmentation and volume reconstruction using the module “Segmentation U-Net”

Figure 2. Segmentation of breast ultrasound images using the module “Breast Lesion Segmentation”

Background and References

This project is based on the previous Segmentation Unet and Breast Lesion Segmentation modules:

Integration of PyTorch and Slicer: