NA-MIC Project Weeks

Back to Projects List

Real-time ultrasound AI segmentation using Tensorflow and PyTorch models

Key Investigators

• María Rosa Rodríguez Luque (Universidad de Las Palmas de Gran Canaria, Spain) [on site]
• Tamas Ungi (Queen’s University, Canada) [remote]
• David García Mato (Ebatinca S.L., Las Palmas de Gran Canaria, Spain) [on site]
• Chris Yeung (Queen’s University, Canada) [remote]

Project Description

The module “Segmentation U-Net”, from extension SlicerAIGT, applies deep learning models on an ultrasound image stream to generate the predicted segmentation in real time. This is shown in the following example, where it is used to detect tumour tissue (highlighted in red) on breast images. That way, we can apply a live volume reconstruction on this prediction and visualize the complete region of interest (in this case, the area of the tumour). Another instance, using spine images, is shown in (Figure 1).

      

Currently, this module supports models trained with the TensorFlow ecosystem. However, in recent years, PyTorch has become an increasingly popular machine learning framework, especially in medical imaging applications (an example of this is the MONAI framework, which is based on PyTorch).

We have developed a separate module to run the inference of PyTorch model for the segmentation of breast ultrasound images: Breast Lesion Segmentation (Figure 2). However, our module does not integrate parallel processing to enable real-time image segmentation.

In this project, we aim to adapt the current “Segmentation U-Net” module to enable the use of models trained with both ecosystems, PyTorch and TensorFlow, for real-time ultrasound image segmentation.

In addition, we will discuss further improvements for this module. For instance, automatically visualize the prediction overlayed on the input ultrasound image and avoid changing to different modules to activate the visualization.

Objective

1. Adapt the previous module to support models trained with PyTorch
2. Automatically display the AI segmentation overlayed on the input ultrasound image

Approach and Plan

1. Describe specific steps of what you plan to do to achieve the above described objectives.
2. …
3. …

Progress and Next Steps

1. Describe specific steps you have actually done.
2. …
3. …

Illustrations

          

Figure 1. Real-time spine segmentation and volume reconstruction using the module “Segmentation U-Net”

Figure 2. Segmentation of breast ultrasound images using the module “Breast Lesion Segmentation”

Background and References

This project is based on the previous Segmentation Unet and Breast Lesion Segmentation modules:

• Segmentation Unet admits Tensorflow models to develop the segmentation task on an ultrasound image stream:
  • GitHub repository: Segmentation Unet module.
  • The tutorial about how to use the previous module was shown during PerkLabBootcamp held virtually on May 24-26, 2022.
  • The video tutorials of the breast and spine segmentation are also available.
• Breast Lesion Segmentation deploy deep learning models trained in PyTorch for segmentation of 2D Ultrasound images:
  • GitHub Repository: Breast Lesion Segmentation module.

Integration of PyTorch and Slicer:

• To use a deep learning model trained in PyTorch inside Slicer, we install the PyTorch extension, presented during 35th Project Week held virtually June 28-July 2, 2021.