Blood effects for Flinders Sinus Surgery Simulator Proposal presentation for honours project Lee-Ying Wu BEng(Software) Supervisor: Dr Greg Ruthenbeck

Outline: Endoscopic Sinus Surgery (ESS) Medical simulation Previous studies : what has been done What has been done at Flinders Aim of the project Tools and approaches used to implement blood effects Progress result of the project Summary

Endoscopic Sinus Surgery (ESS) The top two pictures illustrate the endoscopic sinus surgery. When a surgeon is performing the surgery, he inserts the endoscope into the nose of the patient. Endoscope is a very thin fibre-optic tube. And the surgeon can perform the direct visual examination of the openings of the sinuses. The visual feedback is sent from the fibre-optic camera and the visual feedback is displayed on the monitor. The surgeon can also receive the haptic feedback by using handpieces of the surgical instruments. The third picture shows two haptic devices are used in our simulator to simulate the sinus surgery.

Endoscopic Sinus Surgery (ESS) This picture shows the tissues in the sinus. The surgeon was using the handpiece of instrument backbiter to cut a piece of tissue from the interior. This picture shows the tissue started to bleed and the blood was accumulated in the concaves of the tissue surface. Once these concaves have been fully filled with blood, the blood started to spread and flow into bigger areas. This is the propagation of the blood flow that we would like to represent in Flinders Sinus Surgery Simulator.

Medical Simulations Medical simulator provides realistic user interactions with the simulated patient. Advantages: reduced risk to patients, access to rare scenarios and unlimited repeatability Challenge: increased computational processing when simulating interactive tissue Real-time simulations allow trainees to interact with the tissue and modify the tissue as if they were practising on real patients. The advantages of using medical simulations include reducing the risk of practising on real patients, facilitating self-directed learning, and repeating the same scenario many time to remedy deficiencies. One key challenge is the increased computational processing when simulating interactive tissue. So there are many studies have been done to find out good approaches to reduce the computational processing to improve the performance of medical simulations.

Previous studies: what has been done Halic et al. (2010) generate realistic bleeding and smoke in virtual-reality (VR) based surgical simulators. The approach they used is to outsource the computation to the graphic processing unit (GPU). Müller et al. (2004) simulate blood using the method based on Smoothed Particle Hydrodynamics (SPH) . They have illustrated that interactive modelling of fluids such as blood or water is an essential component in medical simulations. It also can help to improve treatment planning and decrease surgical risks. It can add substantial realism to variety of application scenarios They use CPU to do the detection of vein density and increment of the bleeding speed. Then they use GPU to calculates the pixel values for the blood colour. The method built by Muller is implemented in interactive surgical training systems which produce up to 3000 particles. Producing 3000 particles was pretty good job for the machine in 2004. However, due to the advance in hardware and software technology, we would like to produce more particles in our simulator.

What has been done at Flinders By using the simulator, the user can interact with the turbinate via using desktop haptic devices such as Sensable Phantom device. The force feedback is generated when the surgical tool is touching/intersecting with the turbinate model. The user can experience the sense-of-touching when holding the stylus of haptic device, which is represented as the handpieces of the surgical instrument on the screen. The turbinate model that is rendered in Flinders Sinus Surgery Simulator (Ruthenbeck 2012). The turbinate is part of the accurate 3D model of the sinus of a real patient.

What has been done at Flinders The triangle mesh view of turbinate shown in the previous slide (Ruthenbeck 2012).

What has been done at Flinders Tissues have been cut by the backbiter and the see-through view is created (Ruthenbeck 2012).

What has been done at Flinders Voxels (red triangles) are generated to form the surface for the cut volumes where tissues have been ablated by the backbiter (Ruthenbeck 2012).

Aim of the project Create realistic blood effects for Flinders Sinus Surgery Simulator By creating realistic bleeding effects, trainees/surgeons can practice making appropriate decisions such as the timing of clearing the lens of the endoscope. And also if the patient bleeds to much, the surgeon might need to apply more vasoconstriction drug to the patient in order to reduce the bleeding.

The tools and strategies used to implement blood effects DirectX 11 (Direct3D) Parallel computing architecture – CUDA C++ and Microsoft Visual studio 2010 C# Versioning control – Subversion Direct3D 11 is a rendering library for producing high performance 3D graphics application using modern graphics hardware on the Windows platform. The user can write C++ code to use Direct3D. Direct3D is a low-level library so its API can closely model the underlying graphics hardware.. CUDA is a C-like language. It supports many languages such as C++, C, Java, Fortran, etc. You can write a CUDA code and it in parallel using GPU. GPU is especially designed to perform parallel computation. The same code can be executed by many threads in parallel. For example, in graphics rendering, large sets of pixels and vertices are mapped to parallel threads. Using CUDA code, you can specify the number of threads in a block of memory and number of blocks per grid in the memory. Hence, I will use Direct3D 11 to perform the rendering pipeline. The rendering pipeline refers to the entire sequence of steps necessary to generate a 2D image based on what the virtual camera sees. Then I will use CUDA code to perform more complicate calculations which will done in parallel on GPU. GPUs have been architected to be massively parallel. GPUs have been optimised to process a large amount of memory from a single operation. This is in contract to a CPU which is designed for random memory access. GPUs architecture is designed for graphics. So vertices and pixels can be processed independently in parallel on GPUs. 3. All the final-version applications are built or to be built in C++ using IDE Visual Studio 2010. 4. C# is very similar to Java. It is used to implement or test new ideas/thoughts for the project. For example, 2D particle application was created by Greg and myself using C# to use particles to simulate the water-flow behaviour. (I will run the application to demonstrate shortly). It is pretty quick to build this type of application using C# than using Direct3D. So it’s a good idea to build the working application in C# and make sure it working properly as expected and then migrate to Direct3D. 5. Subversion is an open source version control system. The system maintains versioning for directories, renames and file metadata. Users can move or copy entire directory tree quickly while retaining full version history.

The tools and strategies used to implement blood effects Approach Texturing-based blood simulation technique Particle systems Smoothed Particle Hydrodynamics (SPH) For example, using texturing, we can combine two textures together to create a new texture with special effects. This enables us to increase the details and realism of our model significantly. We are using particle systems to simulate the behaviour of blood flow. This means that a huge amount of particles will be created to make them look like fluid. If these particles are small and close to each other and moving in the similar direction and have similar velocities, these particle will look like blood flows. Smoothed Particle Hydrodynamics is a computational method used to simulate fluid flows. It is a mesh-free Lagrangian method where coordinates move with the fluid. SPH works by dividing fluid into a set of particles. These particles have spatial distance known as smoothing length. Over the smoothing length, the particles are smoothed by a kernel function. This means that the physical property of a particle can be obtained by summing the relevant properties of all particles that lie within the range of the smoothing length.

Progress result of the project A particle system application is created using C#. We use C# to implement the design of the application. If the design works correctly and then we will implement the same thing in C++ using Direct3D and CUDA. By keeping pressing the mouses, many particles are generated. These particles are flowing down to the bottom of the panel due to the applied gravity. And these particles are accumulated at the bottom of the panel. We would like to do the same thing using DirectX 11 and CUDA. We will be using DirectX 11 to perform the rendering of the particles and using CUDA to calculate to position of the particle.

Summary Creating bleeding effects on the medical simulation provides important cues to trainees/surgeons. These cues are important factors in surgical skill assessments (Halic et al. 2010). The project aims to create realistic blood effects for Flinders Sinus Surgery Simulator.

Thank you for listening! Questions?

Reference HALIC, T., SANKARANARAYANAN, G. & DE, S. 2010. GPU- based efficient realistic techniques for bleeding and smoke generation in surgical simulators. International Journal of Medical Robotics and Computer Assisted Surgery, 6, 431-443. MÜLLER, M., SCHIRM, S. & TESCHNER, M. 2004. Interactive blood simulation for virtual surgery based on smoothed particle hydrodynamics. Technology and Health Care, 12, 25-31. RUTHENBECK, G. S. 2010. Interactive Soft Tissue for Surgical Simulation. Doctorate of Philosophy, Flinders University, South Australia. RUTHENBECK, G. S., CARNEY, A. S., HOBSON, J. C., SACKS, R., AND REYNOLDS, K. J., 2011, the GPRWMF Virtual Reality Endoscopic Sinus Surgery Simulator.

Gantt Chart