Back to Projects List
The future of rendering in VTK and Slicer
- Simon Drouin (École de Technologie Supérieure)
- Steve Pieper (Isomics, Inc., USA)
- Murat Maga (University of Washington, USA)
- Andras Lasso (Queen's University, Canada)
- Sara Rolfe (Seattle Children's Research Institute, USA)
- Jean-Christophe Fillion-Robin (Kitware, Inc., USA)
- Stephen Aylward (Kitware, Inc., USA)
- Rafael Palomar (NTNU, Norway)
- Sankhesh Jhaveri (Kitware, Inc., USA)
- Matt McCormick (Kitware, Inc., USA)
- Forrest Li (Kitware, Inc., USA)
- Lucas Gandel (Kitware SAS, USA)
- Jaswant Panchumarti (Kitware, Inc., USA)
- Shreeraj Jadhav (Kitware, Inc., USA)
- Tom Birdsong (Kitware, Inc., USA)
The goal of this breakout session is to gather all parties interested in the future of rendering in VTK and Slicer, present ongoing development by Kitware and others and discuss potential future directions and clinical and biomedical needs.
During the Breakout Session
Links and notes are organized at https://hackmd.io/Mq81LxbYTfqrwOBRjxrb6Q. It is a markdown based document we can collaboratively & interactively edit.
Once the breakout session is over, we will contribute the information back to this page. See Notes below.
- VTK Evolution
- Status of the replacement for OpenGL (WebGPU)
- Integration of VTK and VTK.js (common shaders?)
- Global illumination
- Support for high resolution volumes
- Support for shared graphics contexts
- Support for GPU pre-processing of volumes and meshes
- Creation of an experimental rendering module:
- Support for multiple volumes/surfaces handled by the same pipeline
- Modifiable shaders
- Multipass rendering
- Arbitrary number of transfer functions
- Better support for animation and high-resolution rendering
- New Slicer Mixed-Reality module (for Hololens remoting)
- Support for OpenXR in Slicer Virtual Reality module
- Status of WebXR in vtk.js
VTK C++: WebGPU
Contact: Sankhesh Jhaveri @ Kitware
WebGPU effort in VTK aims to provide a future-proof rendering backend as an alternative to OpenGL.
- VTK’s data model and visualization pipeline will have no architectural changes.
- For the most part, there will be no frontend user-facing changes to the rendering classes either. Applications would still have to instantiate
- There will be changes with respect to how platform-specific windows are created. These windows no longer need to be tied to the graphics backend i.e. no need for
- For advanced users of VTK who modify rendering logic, use shader replacements, etc. there will be significant changes. This should be expected, IMO.
- RenderingWebGPU: https://docs.vtk.org/en/latest/modules/vtk-modules/Rendering/WebGPU/README.html
- Draft: WebGPU: Native graphics backend. See https://gitlab.kitware.com/vtk/vtk/-/merge_requests/10239
- multi-volume rendering
- Will it be the default ? In the OpenGL backend, there were two backends.
- vtkMultiVolume issues
UserMatrix not taken into account for the first volume (issue) (works in
- Only shows up in Composite w/ Shading, but nothing in MIP (Error: Shader failed to compile in
- context sharing ?
- Goal is to reduce the memory footprint by avoiding having duplicate of teture memory (e.g multiple 3D views of the same volume data)
- Availability of result of compute shader for rendering.
- Compute shaders
- moltengl: Applications built for OpenGL ES 2.0 can use MoltenGL to run on top of Metal,
- Unity allows to easily compose complex rendering pipeline.
- The shader is the center piece
- Possibly use SPIRV-Reflection to decipher shader inputs, their attributes/dimensionality and expose a VTK interface to populate shader bindings with vtkImageData, vtkDataArray, etc.
- Compute Shader
- Simon will share details about experiment for implementing compute shader
- AI and WebGPU for training ?
- Rendering into texture ?
- Idea would be to support Differential Rendering
- Review meeting ?
Contact: Jaswant Panchumarti @ Kitware
VTK C++: OpenXR
Contact: Lucas Gandel @ Kitware
Improvement roadmap for
OpenXR Remoting support in VTK:
- OpenXR controller model support:
vtkOpenXRManager already supports for querying the controller models in a buffer (see here)
- The buffer should be forwarded to vtkGLTFReader using vtkResourceStream/vtkResourceParser
- Need to add vtkResourceStream/vtkResourceParser support to vtkGLTFReader (see PLYReader )
- OpenXR Remoting hand interaction improvements
- The current interactor style does not allow for accessing different poses from a single Move3D event (see discussions in this MR)
- Possible options are:
- Add support for storing additional poses (maybe as additional TrackedDevice)
- Add support for hands joints tracking and implement basic gestures recognition
- Add support for scene understanding in OpenXR Remoting, to occlude the VTK scene with the real world.
- Are we ready to transition to OpenXR ?
Contact: Jean-Christophe Fillion-Robin @ Kitware
Question: What to do once we start working on adding
OpenXR support to
vtk.js: Rendering Non-uniform image series
Contact: Shreeraj Jadhav @ Kitware
vtk.js: Interactive, in-browser cinematic volume rendering of medical images
Contact: Stephen Aylward & Forrest Li @ Kitware
The diversity and utility of cinematic volume rendering (CVR) for medical image visualisation have grown rapidly in recent years. At the same time, volume rendering on augmented and virtual reality systems is attracting greater interest with the advance of the WebXR standard.
Contact: Tom Birdsong @ Kitware
VTK & ITK interoperability
- ITK IO are compiled to WASM and re-used in vtk.js based web application
- ITK-WASM is a building block for VTK-WASM
Slicer + Large Image Rendering:
- We currently do not have any readily available implementation in Slicer, from a high level, the idea would be to:
- support streaming dataset at different resolutions and updating the rendering accordingly.
- leverage ITK+Zarr integration and ensure information flow all the way through the VTK pipeline
- Consolidate ITKVTKGlue modules
- Improve ITK module: ITKIOOMEZarrNGFF. See https://github.com/InsightSoftwareConsortium/ITKIOOMEZarrNGFF
Background and References