NA-MIC Project Weeks

Back to Projects List

Simple custom GPU accelerated filtering and volume rendering pipeline

Key Investigators

• Simon Drouin (ETS Montreal, Canada)
• Rafael Palomar (Oslo University Hospital, Norway)

Project Description

3D Slicer has the potential to be a powerful dissemination platform for novel 3D rendering methods targeted at medical imaging. However, its complex architecture is challenging for most experienced graphics programmers. It is difficult to create extensions that implement novel graphics pipeline without mastering a large portion of the Slicer architecture.

Thibault Pelletier at Kitware France has recently contributed a powerful Slicer extension called Layer Displayable Manager. This extension simplifies the creation of new interactive VTK-based graphics rendering pipeline, both from Loadable or Scripted modules.

Kyle Sunderland has created another valuable tool: vtkGPUImageFilters. Each filter in this collection can be connected to the regular VTK pipeline, but use GPU for processing without having to bring the data back to the CPU memory between filters. The code is not yet integrated in VTK and is still a

The combination of Layer Displayable Manager and vtkGPUImageFilters has the potential to enable a new range of GPU accelerated filtering and volume rendering effects.

Objective

The goal of this projet is to generate a proof of concept of the potential provided by the integration of LayerDisplayableManader and vtkGPUImageFilters. The project aims at:

• Creating an experimental scripted module that can apply a gaussian blur to a volume in the GPU
• Render the resulting volume without bringing the filtered image back to the CPU memory.

Approach and Plan

1. Build a simple volume rendering scripted module that implements a custom pipeline using the Layer Displayable Manager extension and the existing vtkGPUVolumeRaycastMapper
2. Build Kyle Suntherland’s vtkGPUImageFilter as an external VTK module. This step may require major modifications in the code as this collection of filter was implemented for a version of VTK from 2019.
3. Link the vtkGPUImageFilter module to be usable in the scripted module developed in step 1.
4. In the loadable module, implement a class that derives from vtkGPUVolumeRaycastMapper but is able to take a volume already in the GPU as an input.
5. Modify the pipeline of the module developed in the first step to filter the input volume and pass it on to the volume rendering mapper.

Progress and Next Steps

The project to create a simple shader as an input to volume rendering using LayerDisplayableManager turned into a reflection on the future of rendering in Slicer.

• Solution explored are the following:
  • Simple: Create a GPU processing pipeline and make mapper inputs more flexible
    • Pros: simple. Kyle Sunderland’s vtkGPUImageFilter can be used as a basis. Minor changes can be made to each mapper to accept the output of GPU filters as their inputs, and make the number of inputs more flexible.
    • Cons: Many GPU mapper are extremely complicated because they package too much functionality in the same class. They are difficult to modify and it is difficult to convince maintainers to accept changes that can affect other projects.
  • Extreme: Replace VTK altogether
    • Pros: possibly a lighter rendering engine, possibility to fully embrace modern graphics software designs.
    • Cons: lots of work, risk of
  • Middle Ground: replace main VTK rendering classes (Surface, Volume) + GPU processing pipeline
    • Pros: Can start with existing code and simplify, introduces sufficient flexibility and has the potential to yield results shortly.
    • Cons: Rendering classes no longer supported by Kitware.

Illustrations

Proposed simple architecture for volume rendering in the Middle ground solution above

The new architecture opens the possibility of creating rendering graphs, like in many CGI software and game engines.

Background and References

• Layer Displayable Manager on GitHub
• vtkGPUImageFilters (branch on Kyle Sunderland’s VTK fork)