NA-MIC Project Weeks

Back to Projects List

3D and 2D Radiology Copilot Integration in 3D Slicer

Key Investigators

• Andres Diaz-Pinto (NVIDIA, UK)
• Andras Lasso (Queen's University, Canada)
• Vishwesh Nath (NVIDIA, US)
• Nigel Nelson (NVIDIA, US)
• Sean Huver (NVIDIA, US)
• Mingxin Zheng (NVIDIA, China)
• Wenqi Li (NVIDIA, UK)

Project Description

This project aims to create the first 3D and 2D radiology copilot in 3D Slicer by developing a module that can consume REST APIs of radiology copilots. The main goal is to demonstrate the benefits of using the NVIDIA Holoscan platform for deploying AI models in medical imaging applications.

Objective

1. Develop a 3D Slicer module that integrates with radiology copilots via REST APIs.
2. Showcase the capabilities of the NVIDIA Holoscan platform for AI model deployment in medical imaging.
3. Implement functionality to send 3D volumes to the copilot, ask questions, and receive insights.

Approach and Plan

1. Create a new 3D Slicer module that can communicate with REST APIs.
2. Integrate the private model trained by East River Imaging and NVIDIA using the RadImageNet dataset.
3. Implement support for the MONAI VILA-M3 model.
4. Develop a user interface within the module to allow sending 3D volumes, asking questions, and displaying copilot insights.
5. Optimize the module’s performance using the NVIDIA Holoscan platform.

Progress and Next Steps

1. Initial project setup and team coordination completed.
2. Research on integrating REST APIs within 3D Slicer modules conducted.
3. Preliminary design of the user interface drafted.

Illustrations

https://github.com/user-attachments/assets/0d9fd2c6-ef26-4d14-851e-c761bb218ea7

Background and References

• NVIDIA Holoscan SDK: https://github.com/nvidia-holoscan/holoscan-sdk
• East River Imaging: https://eastriverimaging.com/
• NVIDIA: www.nvidia.com
• RadImageNet dataset: https://www.radimagenet.com/
• MONAI VILA-M3 paper: https://arxiv.org/pdf/2411.12915