NA-MIC Project Weeks

Back to Projects List

3D and 2D Radiology Copilot Integration in 3D Slicer

Key Investigators

• Andres Diaz-Pinto (NVIDIA, UK)
• Andras Lasso (Queen's University, Canada)
• Vishwesh Nath (NVIDIA, USA)
• Nigel Nelson (NVIDIA, USA)
• Sean Huver (NVIDIA, USA)
• Mingxin Zheng (NVIDIA, China)
• Wenqi Li (NVIDIA, UK)
• Xueyan Mei (Mount Sinai, USA)
• Zelong Liu (Mount Sinai, USA)
• Tim Deyer (East River Imaging, USA)

Project Description

This project aims to create the first 3D and 2D radiology copilot in 3D Slicer by developing a module that can consume REST APIs of radiology copilots. The main goal is to demonstrate the benefits of using the NVIDIA Holoscan platform for deploying AI models in medical imaging applications.

Objective

1. Develop a 3D Slicer module that integrates with radiology copilots via REST APIs.
2. Showcase the capabilities of the NVIDIA Holoscan platform for AI model deployment in medical imaging.
3. Implement functionality to send 3D volumes to the copilot, ask questions, and receive insights.

Approach and Plan

1. Create a new 3D Slicer module that can communicate with REST APIs
2. Integrate the private model trained by East River Imaging and NVIDIA using the RadImageNet dataset.
3. Implement support for the MONAI VILA-M3 model.
4. Develop a user interface within the module to allow sending 3D volumes, asking questions, and displaying copilot insights.
5. Optimize the module’s performance using the NVIDIA Holoscan platform.

Progress and Next Steps

1. Initial project setup and team coordination completed.
2. Research on integrating REST APIs within 3D Slicer modules conducted.
3. Preliminary design of the user interface drafted.
4. Repository: https://github.com/Project-MONAI/VLM/tree/main/plugins/RadCoPilot_Slicer

Next Steps:

For RadViLLA server:

• In the server, create a session so loading volume and resizing doesn’t need to happen every time the user send a prompt
• Cache the volume so inference is faster

VILA-M3:

• Current server using NVCF doesn’t accept volumes in the cloud with HTTPS. Change this flag or make sure it also accepts volumes hosted using HTTP
• Get the current slice and send it to the prompt request rather than keeping it hard coded in the server

Illustrations

video

Background and References

• NVIDIA Holoscan SDK
• East River Imaging
• NVIDIA
• RadImageNet dataset
• MONAI VILA-M3 paper