Edit this page

NA-MIC Project Weeks

Back to Projects List

AR in Slicer

Key Investigators

Project Description

Augmented Reality has increase its adoption in many areas with exciting benefits. Universidad Carlos III de Madrid (Madrid, Spain) has already worked in several medical projects based on AR (see their progress in https://biig-igt.uc3m.es/augmented-reality/). On these studies, they usually export information from Slicer to an alternative software (Unity).

The ultimate goal of this project is to check if it is possible to incorporate this technology directly to 3D Slicer in order to centralize the working process, at the time of benefiting from all Slicer tools.


  1. Learn what can be done in AR using 3D Slicer.
  2. Explore possible paths to integrate AR in 3D Slicer in the future.
  3. Study if we can receive transformations in Unity from Slicer, so that we can transfer navigation information between the two softwares.

Some links of interest:

  1. Writing a Holographic Remoting remote app using the OpenXR API
  2. Slicer Documentation on Augmented Reality and Virtual Reality support

Approach and Plan

  1. Develop a new Slicer extension called SlicerAR that streams AR directly to HoloLens 2.

Progress and Next Steps

  1. AR Support (OpenXR) in VTK:

The WIP branch supporting Holographic remoting to stream VTK rendering inside the Hololens has been submitted here. The immediate actions to take are listed there in the TODOS section: 1. and 2. should be addressed to reuse this work in SlicerVR.

  1. Alicia and Lucas tried to replicate in Alicia’s computer the steps of the WIP branch Lucas’ shared above. They summarized them in this pdf document. Despite they were not able to complete the final step, they found out and fixed many new issues that improved the project.

  2. Alicia, Houssem and Étienne Léger also met to discuss some features related to pattern recognition in HoloLens with Vuforia and ArUco, latency and interconnectivity between Slicer and Unity.

  3. Alicia and Naghmeh additionally talked about connecting 3D Slicer and Unity in real time to send transformations between them. Here, here and here you can find two GitHub projects that explain how to achieve this connection via OpenIGTLink.

  4. Alicia and Étienne met again after Project Week to discuss further about Étienne’s work. He tracks a tool connected to Plus and Slicer via OpenIGTLink. He managed to send this information, in real-time, to a self-developed AR tablet application (MARIN) using IBIS. Everything is programmed in Qt (check out the Qt creator from Qt.io to start programming in Qt).


Here below you can find some AR implementations in health by Universidad Carlos III de Madrid (Madrid, Spain) in the past years:

HoloLens 2 in Orthopedic Oncological Surgeries:

HoloLens 2 in Orthopedic Oncological Surgeries

Smartphone app to communicate with the patient and help him/her understand his/her condition:

Smartphone app to communicate with the patient and help him/her understand his/her condition

Real-time guidance during Open Cranial Vault Remodeling using smartphone:

Smartphone app to guide open cranial vault remodeling

Needle Insertion Guidance for Sacral Nerve Stimulation using smartphone:

Smartphone app to guide needle insertion for sacral nerve stimulation

Background and References