NA-MIC Project Weeks

Back to Projects List

Integration of Slicer and ROS

Key Investigators

• Junichi Tokuda (BWH)
• Anton Deguet (JHU)
• Steve Pieper (Isomics)
• Mariana Bernardes (BWH)
• Laura Connolly (Queen's University)
• Simon Leonard (JHU)

Presenter location: In-person

Project Description

SlicerROS2 is an extension that enables direct communication between the Robot Operating System 2 (ROS2) and 3D Slicer. The ROS is a set of software libraries and tools for building robot applications. ROS2 has been developed and distrubted using an open-source model and widely used in the robotics community. The goal of SlicerROS2 is to facilitate the integration of 3D Slicer and ROS to build systems for image-guided robot-assisted intervention.

SlicerROS2 provides UI and API to communicate with other ROS nodes through Data Distribution Service (DDS), the publish-subscribe data transport middleware used in ROS2, allowing 3D Slicer to sychnonize its scen graph (MRML) with ROS’s tf. It also has an interface to load a visual model of the robot onto the Slicer scene from robot description data in the URDF format published on the ROS system.

Objective

The objectives of this project are as follows:

1. Improve existing implementation based on feedback from Slicer experts. We will discuss use-cases with current and potential users in the community.
2. Explore options for binary distribution. Since the module must be built against 3D Slicer with system SSL and specific versions of ROS2, it is not possible to be built as part of the nightly build process.

Approach and Plan

Questions discuss during the week:

• Questions of future use
  • Limited to robots or anything with a ROS2 interface (e.g. haptic devices, IMU, optical and magnetic trackers…)
  • Other software packages: Gazebo, AMBF, US/CT/MRI simulators…
• Detailed questions
  • Is multithreading possible? Any other way to trigger a periodic computing task (now using QtTimer)
  • vtkObjects all have a name and timestamp. Is the name used in Slicer? Can we have a timestamp that is not a counter?
  • Improving unit testing, we have some but implementation seems clunky
  • For binary distributions, is there a way to host build and/or tgz files? I can host build on JHU computers.
  • Is there an existing way to document code in modules, e.g. doxygen
  • Small issues: QLatinString in 2 places breaks compiling on older Qt and likely useless, some missing const

Progress and Next Steps

1. Coding
   1. Fixed a few issues based on Slicer experts in the room
      1. We use a Qt timer to “spin” the ROS node. The timer was instantiated in the qLogicWidget and didn’t spin until the user would select the ROS2 module. We moved it to qLogic so it’s always running.
      2. Fixed CMake project name conflicts between ROS and Slicer macros so the build is in slicer_ros2_module (snake_case for TROS) but the module name is ROS2
      3. Fixed issue with tests turning off errors and warnings for everything
   2. Code generation
      1. Finished CMake macros to call code generator as well as code generated by CMake itself
      2. Fixed code generation so vtk Python wrappers work (using simpler types int vs int32_t)
      3. Some simplification in code generator
   3. Usability
      1. Allow to create subscribers and publishers using short names, i.e. String vs vtkMRMLROS2PublisherStringNode
      2. Added method to list all existing publishers and subscribers
      3. When user tries to create a publisher or subscriber with invalid name, display list of option (good for typos)
      4. Publishers and subscribers now have method GetBlankMessage so it’s easier to create a payload in Python
      5. Create a self-contained ROS2 package for US simulator in Gazebo that builds a minimal version of PlusLib/IGSIO/vtkAddon that contains launch models and launch files to teleoperate a UR-5 mounted with a US probe.
2. Discussions
   1. Possible feature requests for Slicer
      1. Saved history in Python interpreter (like iPython)
      2. Time stamps in MRML nodes or vtk Object, as in std::chrono. Very useful for realtime applications or anything intro-operative. Most ROS payloads have timestamps. This could be either set automatically (in SetModified) or controlled by user to preserve time of data collection.
   2. Distribution(s), pros and cons from user perspective
      1. Continue as-is, i.e. source code and users have to compile the module. Pros: can add new messages. Cons: have to compile Slicer from scratch, extensions are not available to download from kitware servers.
      2. Binary distribution with ROS2 core libraries added to Slicer super build. Pros: ready to use, might even provide ROS2 support from Windows, MacOS, any linux instead of Ubuntu only. Cons: harder to add custom messages without a compiler
      3. Binary (bis). If we figure out how to “super build” ROS2 core libraries (rclcpp), why not provide the Python wrappers too (rclpy) as a pip build. At that point, could port existing features from C++ to Python. A ROS2 pip build could be used outside Slicer.
      4. Simulate and evaluate US/Robot hand-eye calibration by inserting probe in the simulated image and then compare results to ground-truth from the simulation. Adding reference 3D markers to make the simulation more relevant to clinical applications.
3. Next steps
   1. Investigate simple options to make source based distribution easier: provide Slicer and Slicer-SuperBuild, document how to compile and add extensions
   2. For any binary distribution, how hard would it be to compile all ROS2 dependencies from source for “super-building” or “piping” them

Illustrations

Here we can see the robot model loaded with SlicerROS2 into the Slicer Scene. In this example the we have a transperineal needle robot guide inserting a biopsy needle to sample the prostate of a pig model. The scene also shows the intraoperative MRI images of the pig pelvis and segmentations of main anatomical structures.

No response

Background and References

No response