Development of immersive real-time applications Miroslav Andel miroslav.andel@liu.se

Miroslav Andel - Development of immersive real-time applications Lectures Development of immersive real-time applications Online/direct rendering Hardware Programming Video production Offline rendering Hybrid solutions 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Outline Hardware Understanding the technology Clusters and GPUs Warping, color matching & blending Immersion in a virtual environment (VE) Software design Desktop application vs. immersive cluster application Scalability & configuration of VEs Frameworks Scene graph 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Hardware 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Projectors 6x Digital Projection Titan 1920 x 1200 @ 120 Hz Frame doubling 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Image Generators (IGs) Workstation high-end graphics cards Frame sync External house sync Genlock (video signal sync) Swaplock (swap buffer sync) 3D sync (for 3D glasses) Supports active stereoscopy using OpenGL Nvidia Quadro or AMD/ATI FirePro No gaming cards Gaming cards: Optimized for DirectX, Proffesional cards are optimized for OpenGL. OpenGL and DirectX are getting more and more similar. Professional cards: Better floating point pressiton and more memory (less optimization) Gaming graphics cards focus on speed, and trade off by providing a much lower precision than their workstation counterparts. Professional cards are optimized for 3D modelling, CAD and video editing software. A lot of memory -> raytracing on the GPU. Our problem is that we are using gaming techniques but need the synchronizing and 3D features. Less performance… 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Gaming vs. Workstation GPU GPU is pretty much the same but different driver software and firmware Gaming cards are optimized for speed and DirectX Hi frame-rate in games, highly optimized Workstation cards are optimized for precision, OpenGL & OpenCL Designed for 3D modeling, CAD, video editing and computation Higher floating point and sub-pixel precision Better quality and larger memory (12 GB in Quadro K6000, 16 GB AMD FirePro 9100) Ray tracing on the GPU Our problem: We use gaming techniques but need workstation features like synchronization and stereoscopy (less performance) 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Cluster of IGs Often consists of workstations with the same specification Master – Slave In our case the master has an additional surround sound card (live encoding to Dolby Digital or DTS) Miroslav Andel - Development of immersive real-time applications

Clusters (at the visualization center) Dome Alpha (primary real-time cluster used for shows) Beta (secondary development cluster) Delta (movie and presentations playback cluster) Gamma (single node eyefinity machine) VR-Arena Theta Videowall (just a single node with 6 outputs) used for movie and presentations playback Mention the number of nodes for each cluster Miroslav Andel - Development of immersive real-time applications

OpenGL/DirectX projections One viewplane per projector Curved surface? 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Color/gammut matching What is white? 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Color matching White points (CIE standard illuminant) D65 and D75 D65 = 6500 K = (0.3127, 0.3290) D75 = 7500 K = (0.2990, 0.3149) Other white points: DCI = (0.3140, 0.3510) DICOM clear base = (0.280, 0.305) DICOM blue base = (0.250, 0.304) DCI = Digital Cinema Initiative 2012-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Color matching All display devices and projectors have a different gamut Projector lamp color temperature changes over time Must be done before blending Can be done on each IG but with less precision (8 or 10 bits per color) Better to do that in the projector (12-16 bit precision) 2012-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Color matching 2012-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Blending Digital blending – good bright levels Not a linear gradient (gamma) Done after color and brightness matching 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Blending Optical blending for better dark levels Projected black is not black, still projected light 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Blending Optical blending for better dark levels Projected black is not black, still projected light 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Blending Blending in the dome Blue = Projection borders Red = Blend zones 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Warping/geometric correction Correct the rendered frame to match the geometry of the screen, lens and projector’s position Correct overlaps between adjacent channels Spatial 2D transform 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Warping and blending hardware Processing unit with input & outputs cards (FPGA) with minimal latency 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Warping and blending using OpenGL No latency but uses a bit of the GPU 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Calibration process Manual warping and blending calibration takes many hours Few reference points in the dome Moving mesh points by hand causes linearity problems Semi-automated calibration Automated camera-based systems Current systems don’t handle optical blends 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Hardware (end) 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Immersion To give the audience the impression that they have stepped into a synthetic world 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Immersion To give the audience the impression that they have stepped into a synthetic world Enhance immersion by: Large field of view High resolution Stereoscopy (can also reduce immersion) Surround sound Smooth navigation (based on a physical model) Realistic rendering (shadows etc., to improve depth cue) 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Immersion Stereoscopy in the dome? The paradox Field of view wearing 3D glasses Use stereoscopy wisely 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Immersion Stereoscopy in the dome? The paradox Field of view wearing 3D glasses Use stereoscopy wisely Cybersickness in immersive environments Nausea Eye strain Headache Vomiting (worst case) 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Software design 2014-09-03 Miroslav Andel - Development of immersive real-time applications

The difference (from normal desktop application development) Several projectors/screens A lot of pixels Alignment, blending & warping Color & Brightness matching Several computers (cluster) Synchronization Frame Buffers swap Data Video signal (genlock) Multi-threading GENLOCK: Genlock refers to the process of synchronizing the pixel scanning of one or more displays to an external synchronization source. NVIDIA Genlock requires the external signal to be either TTL or composite, such as used for NTSC, PAL, or HDTV. It should be noted that the NVIDIA Genlock implementation is guaranteed only to be frame-synchronized, and not necessarily pixel-synchronized. FRAME LOCK: Frame Lock involves the use of hardware to synchronize the frames on each display in a connected system. When graphics and video are displayed across multiple monitors, frame locked systems help maintain image continuity to create a virtual canvas. Frame lock is especially critical for stereo viewing, where the left and right fields must be in sync across all displays. In short, to enable genlock means to sync to an external signal. To enable frame lock means to sync 2 or more display devices to a signal generated internally by the hardware, and to use both means to sync 2 or more display devices to an external signal. SWAP SYNC: Swap sync refers to the synchronization of buffer swaps of multiple application windows. By means of swap sync, applications running on multiple systems can synchronize the application buffer swaps between all the systems. In order to work across multiple systems, swap sync requires that the systems are frame locked. 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Stereoscopy How does it work? 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Stereoscopy 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Stereoscopy 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Stereoscopy 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Stereoscopy Co-location Camera position = eye position Projection plane = real physical screen 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Stereoscopy Don’t change eye separation, change the scale of your scene ~15 meters limit 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Stereoscopy Object in focus in negative parallax 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Stereoscopy Object in focus in positive parallax 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Stereoscopy Problems with a fixed head setup (no head tracking) 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Scalability One application can run in several virtual environments without the need of changing the code In the dome In the VR-Arena In the VR-Lab At home using your computer At the bus using a laptop If the platform is the same then there is no need for recompiling the application (just copy & paste) 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Scalability The application uses a configuration file that describes the hardware in terms like: Number of IGs Number of displays Number of viewports Physical location of displays/projection planes Physical location of the user Eye separation of the user Type if Stereoscopy Etc… 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Configuration “3d drawing” 2014-09-03

Configuration Monoscopic projection Immersive applications Often uses non-symmetric frustums/projections, while normal OpenGL applications don’t (left FOV = right FOV, top FOV = bottom FOV) Cyclops solution. Normally we have one dominant eye but for multi-person use lets just pick a point between the eyes. Use projection plane coordinates to calculate the projection frustum 2014-09-03

Configuration Stereoscopic projection 2014-09-03

Configuration Configuration - scalability

Configuration All users except the target user will perceive the rendered object incorrectly. Cyclops solution. Normally we have one dominant eye but for multi-person use lets just pick a point between the eyes.

Configuration All users except the target user will perceive the rendered object incorrectly. One solution would be using HMDs or a multi-view screen. Cyclops solution. Normally we have one dominant eye but for multi-person use lets just pick a point between the eyes.

Miroslav Andel - Development of immersive real-time applications Navigation In a desktop application a user navigates by moving the camera 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Navigation In a desktop application a user navigates by moving the camera In a scalable application the number of cameras/frustums is not known 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Navigation In a desktop application a user navigates by moving the camera In a scalable application the number of cameras/frustums is not known Instead of moving the user and all “cameras” in the same direction the whole scene is moved in the opposite direction 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Frameworks (the good stuff) Will help you with Cluster setup & synchronization Frustums & viewports Input Tracking 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Frameworks VR Juggler CAVElib (commercial) Equalizer 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Frameworks (the bad stuff) Massive and have a lot of dependencies Big initial effort to get started Often in legacy OpenGL Takes usually several days to get started Getting all dependencies Compile all dependencies Compile the whole framework Read the documentation 2014-09-03 Miroslav Andel - Development of immersive real-time applications

SGCT Simple Graphics Cluster Toolkit 2014-09-03 Miroslav Andel - Development of immersive real-time applications

SGCT Simple Graphics Cluster Toolkit Static library based on GLFW GLEW GLM Freetype TinyThread++ TinyXML PNG Z VRPN TurboJPEG 2014-09-03 Miroslav Andel - Development of immersive real-time applications

SGCT Simple Graphics Cluster Toolkit Developed here and is free for everybody to use Designed to get started immediately (rapid development) Cross-platform C++ Modern and Legacy OpenGL support (OGL 2.2 – 4+) Single static library (no dll-hell) Using callbacks rather than inheritance/polymorphism Easy porting application from GLUT, GLFW or similar 2014-09-03 Miroslav Andel - Development of immersive real-time applications

SGCT Simple Graphics Cluster Toolkit Wiki at: c-student.itn.liu.se Documentation Tutorials Configuration files Downloads (installers) Support Forum (on it’s way) 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications SGCT – How it works Has an engine which handles Initiation of OpenGL Rendering Synchronization Network communication Configuration handling Input The user can bind callbacks (function pointers) to the engine to customize the functionality 2014-09-03 Miroslav Andel - Development of immersive real-time applications

2014-09-03

Miroslav Andel - Development of immersive real-time applications SGCT – How it works Set only the callbacks you need A minimal cluster application contains: Draw callback: Render your stuff Pre Sync callback: generate the data to be shared Encode callback: Send the data to the slave(s) Decode callback: Receive the data from the master Other callbacks to handle input devices and external GUIs 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications SGCT - Features Frame-, swapbuffer- and application data-synchronization 3D/Stereoscopic rendering Active/Quadbuffer stereo Side-by-side stereo Top-bottom stereo DLP/Checkerboard stereo Line interleaved stereo Anaglyph stereo 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications SGCT - Features Fisheye rendering PNG texture management GLSL shader management Freetype font management Error messaging across the cluster External TCP control interface Tracking support Joystick, gamepad, Kinect, steering wheel etc. input Antialiasing 3D screenshots in maximum texture resolution Overlays Statistics and performance graphs 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications SGCT - Configuration XML configuration All nodes are using the same configuration and can automatically determine if they are a slave or a master 2014-09-03 Miroslav Andel - Development of immersive real-time applications

<?xml version="1.0" ?> <Cluster masterAddress="127.0.0.1"> <Node ip="127.0.0.1" port="20401"> <Window fullscreen="false"> <Size x="640" y="360" /> <Viewport> <Pos x="0.0" y="0.0" /> <Size x="1.0" y="1.0" /> <Viewplane> <!-- Lower left --> <Pos x="-1.778" y="-1.0" z="0.0" /> <!-- Upper left --> <Pos x="-1.778" y="1.0" z="0.0" /> <!-- Upper right --> <Pos x="1.778" y="1.0" z="0.0" /> </Viewplane> </Viewport> </Window> </Node> <User eyeSeparation="0.065"> <Pos x="0.0" y="0.0" z="4.0" /> </User> </Cluster>

Miroslav Andel - Development of immersive real-time applications SGCT arguments Normal use: app.exe –config “config.xml” Running cluster application locally as master: app.exe –config “cluster_config.xml” –local 0 Running cluster application locally as slave: app.exe –config “cluster_config.xml” –local 0 --slave 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Running SGCT Command prompt/terminal Create a Batch file (.bat) Create a Shortcut From IDE 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Running cluster applications The application folder needs to be accessible from all IGs/nodes Copy the whole folder Use Microsoft’s robocopy tool for mirroring folders across a cluster Use a network share DemoManager tool for launching (and killing) processes across a cluster Or manually start the application on all nodes 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications SGCT – Example Basic spinning triangle synchronized across a cluster using less then 80 lines of code (including comments and spaces) 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Data synchronization What to share? Time or dt Run simulations only on master or on whole cluster? Random numbers? Minimize traffic Compression Bit-masking 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Math OpenGL Mathematics (GLM) is included in SGCT Computer graphics is linear algebra! GLM makes life easy Vectors Matrices Quaternions Similar syntax to glsl 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Testing anywhere Local cluster on a single computer View 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Testing anywhere 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications SGCT + OSG Open Scene Graph (OSG) www.openscenegraph.org Feature rich SGCT manages Cluster sharing & synchronization Window, frustum and viewport creation OSG manages Content OSGMaxExporter (3Ds Max plugin) 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Scene graph Tree structure of nodes Intelligent structure Visibility & occlusion culling 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications OSG example 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Audio Positional 3D audio OpenAL Fmod Ex 2014-09-03 Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications Next time Video pipeline Offline rendering for the dome 2014-09-03 Miroslav Andel - Development of immersive real-time applications