NA-MIC Project Weeks

Back to Projects List

Cinematic rendering in Slicer leveraging VTK Physically Based Rendering (PBR)

Key Investigators

• Shreeraj Jadhav, Kitware, USA
• Jiayi Xu, Kitware, USA
• Jean-Christophe Fillion-Robin Kitware, USA
• Andras Lasso (PerkLab, Queen’s University, Canada)

Project Description

• The goal of this project is to identify and evaluate Kitware’s previous efforts on cinematic rendering and how these can be leveraged in Slicer.

• Slicer internally uses VTK as its rendering engine to display the 2D and 3D viewports in its interface. Since release 9.0, VTK has more enhanced rendering capabilities such as the Physically Based Rendering (PBR) lighting model and the integration of ray tracing backends such as the Nvidia OptiX and intel’s OSPRay engine. These capabilities are already being leveraged in Paraview as discussed in the following blog posts here and here.

• Primary focus will be on VTK’s PBR capabilities and ray tracing backends (OptiX, OSPRay). Since Slicer already uses VTK rendering pipeline, and it will be significantly easier to incorporate/leverage these capabilities from within Slicer.

Objective

Approach and Plan

1. Review accessible offerings on cinematic rendering (VTK PBR, VTK backends: OSPRay and OptiX, omniverse, etc) that can be utilized inside Slicer.
2. Develop prototypes showing how these can be enabled and used in Slicer.
   • Start with surface rendering (PBR, ambient occlusion, global ilumination)
   • Investigate what capabilities can be enabled for volume rendering (perhaps OSPRay).
   • Tradeoffs: Performance vs Image Quality without degrading user experience.
   • Changes to user interface, parameters tuning, simplification.
3. Review existing Slicer modules (such as Light Module) that enhance Slicer’s rendering capabilities and evaluate how these can be included in the current effort.
4. Evaluate how integrating these in Slicer will affect other modules such as LookingGlass, OpenXR, etc.

Progress and Next Steps

Use of vtkSSAOPass class to generate ambient occlusion (AO) for volumes:

• Initial attempt encountered many OpenGL State errors
• Volume mapper cannot directly work when AO pass is enabled, need further investigations to understand how this could be done.

Adapt vtkSSAOPass to create local ambient occlusion (LAO) implementation for volumes:

• ComputeKernel() in vtkSSAOPass can be modified to adapted for LAO of volumes [Jiayi]
• Possibly, only compute an occlusion volume once (pre-compute), use this for shading in raycaster

Through discussions with the VTK team (Timothee Chabat, Mathieu Westphal), we identified another feature which could help improve shading in volume rendering:

• https://gitlab.kitware.com/vtk/vtk/-/merge_requests/9231
• an accurate ambient occlusion effect can be achieved with the current VTK master, setting the GlobalIllumationReach to 0 and setting VolumetricScatteringBlending to something >= 1.0

Image Courtesy: Gaspard Thevenon

Illustrations

Background and References

Related modules:

• Models: In recent Slicer versions, PBR interpolation can be selected and PBR material properties can be edited.
• Lights (in Sandbox extension): it can configure lighting, PBR, image based lighting, ambient shading (SSAO)

Slicer Discourse References:

1. https://discourse.slicer.org/t/how-to-perform-3d-cinematic-rendering/7313
2. https://discourse.slicer.org/t/is-there-interest-in-higher-quality-rendering-for-slicer/6862/5
3. https://discourse.slicer.org/t/2021-01-19-hangout/15585/2

VTK References:

1. VTK PBR https://www.kitware.com/vtk-pbr/
2. PBR integration in Paraview https://www.kitware.com/physically-based-rendering-improvements-in-paraview/
3. Related merge request for VTK https://gitlab.kitware.com/vtk/vtk/-/merge_requests/5584