Edit this page

NA-MIC Project Weeks

Back to Projects List

SlicerVR - Restore Interactions

Key Investigators

Project Description

The main controller interactions in SlicerVR have been broken for about a year, some interaction types even longer. It would be crucial for keeping SlicerVR usable to make the interactions work again.

Kitware and Robarts (Jean-Christophe Fillion Robin, Lucas Gandel, Sankhesh Jhaveri, Adam Rankin) have been investing resources and effort in rehauling the AR/VR backend in VTK for a while, thus now we have a new OpenXR backend and restructured libraries SlicerVR is built on. The goal is to give a small push to their efforts in terms of SlicerVR interactions during the project week, towards restoring at least the previous feature set.


  1. Fix the main controller interactions
    • Flying (joystick forward-backward)
    • Grab and move objects (trigger down and move)
    • Two-controller world move/zoom (i.e. 3D pinch)
  2. Customization of controller buttons. Either via the
    • Method in-place (functions integrated here, see also here)
    • Json manifest files (see here)

Approach and Plan

  1. Continuous coordination with Kitware / Robarts about our potential to help during the week
    • Make sure the absolutely necessary core VTK changes are in place by the start of the week (RecognizeComplexGesture virtual etc.)
  2. Set up VR workstations at the hotel to be able to test and develop
  3. Make sure event pipeline reaches the SlicerVR functions handling the events
  4. Fix interactions: fly, grab, pinch3D

Progress and Next Steps

  1. Restored support for complex gesture (grip) through SlicerVirtualReality#109 and VTK MR-9892
  2. Restored partial support for interactions like Select3DEvent and ViewerMovement3DEvent through SlicerVirtualReality#108
  3. Cherry-picked said commits, built, and confirmed that the fly and grab&move features worked
  4. Confirmed that the actions manifest works on the HP Reverb 2 headset
  5. Started work towards restoring the “Pinch 3D” complex gesture (saving initial physical to world transform and manipulate world as the controllers move)
  6. Created coarse UML model of the relationship between SlicerVR and VTK VR to help troubleshoot the module.

Class diagram SlicerVR vs VTK



SlicerVR example

SlicerVR collaborative example

In-VR widget example

Background and References

Past project week pages

Pinter, C., Lasso, A., Choueib, S., Asselin, M., Fillion-Robin, J. C., Vimort, J. B., Martin, K., Jolley, M. A. & Fichtinger, G. (2020). SlicerVR for Medical Intervention Training and Planning in Immersive Virtual Reality. IEEE Transactions on Medical Robotics and Bionics, vol. 2, no. 2, pp. 108-117, May 2020, doi: 10.1109/TMRB.2020.2983199.