Development of immersive real-time applications

Development of immersive real-time applications
Miroslav Andel

Miroslav Andel - Development of immersive real-time applications
Lectures Development of immersive real-time applications Online/direct rendering Hardware Programming Video production Offline rendering Hybrid solutions Miroslav Andel - Development of immersive real-time applications

Outline Hardware Understanding the technology Clusters and GPUs Warping, color matching & blending Immersion in a virtual environment (VE) Software design Desktop application vs. immersive cluster application Scalability & configuration of VEs Frameworks Scene graph Miroslav Andel - Development of immersive real-time applications

Hardware Miroslav Andel - Development of immersive real-time applications

Projectors 6x Digital Projection Titan 1920 x 120 Hz Frame doubling Miroslav Andel - Development of immersive real-time applications

Image Generators (IGs)
Workstation high-end graphics cards Frame sync External house sync Genlock (video signal sync) Swaplock (swap buffer sync) 3D sync (for 3D glasses) Supports active stereoscopy using OpenGL Nvidia Quadro or AMD/ATI FirePro No gaming cards Gaming cards: Optimized for DirectX, Proffesional cards are optimized for OpenGL. OpenGL and DirectX are getting more and more similar. Professional cards: Better floating point pressiton and more memory (less optimization) Gaming graphics cards focus on speed, and trade off by providing a much lower precision than their workstation counterparts. Professional cards are optimized for 3D modelling, CAD and video editing software. A lot of memory -> raytracing on the GPU. Our problem is that we are using gaming techniques but need the synchronizing and 3D features. Less performance… Miroslav Andel - Development of immersive real-time applications

Gaming vs. Workstation GPU
GPU is pretty much the same but different driver software and firmware Gaming cards are optimized for speed and DirectX Hi frame-rate in games, highly optimized Workstation cards are optimized for precision, OpenGL & OpenCL Designed for 3D modeling, CAD, video editing and computation Higher floating point and sub-pixel precision Better quality and larger memory (12 GB in Quadro K6000, 16 GB AMD FirePro 9100) Ray tracing on the GPU Our problem: We use gaming techniques but need workstation features like synchronization and stereoscopy (less performance) Miroslav Andel - Development of immersive real-time applications

Cluster of IGs Often consists of workstations with the same specification Master – Slave In our case the master has an additional surround sound card (live encoding to Dolby Digital or DTS) Miroslav Andel - Development of immersive real-time applications

Clusters (at the visualization center)
Dome Alpha (primary real-time cluster used for shows) Beta (secondary development cluster) Delta (movie and presentations playback cluster) Gamma (single node eyefinity machine) VR-Arena Theta Videowall (just a single node with 6 outputs) used for movie and presentations playback Mention the number of nodes for each cluster Miroslav Andel - Development of immersive real-time applications

OpenGL/DirectX projections
One viewplane per projector Curved surface? Miroslav Andel - Development of immersive real-time applications

Color/gammut matching
What is white? Miroslav Andel - Development of immersive real-time applications

Color matching White points (CIE standard illuminant) D65 and D75 D65 = 6500 K = (0.3127, ) D75 = 7500 K = (0.2990, ) Other white points: DCI = (0.3140, ) DICOM clear base = (0.280, 0.305) DICOM blue base = (0.250, 0.304) DCI = Digital Cinema Initiative Miroslav Andel - Development of immersive real-time applications

Color matching All display devices and projectors have a different gamut Projector lamp color temperature changes over time Must be done before blending Can be done on each IG but with less precision (8 or 10 bits per color) Better to do that in the projector (12-16 bit precision) Miroslav Andel - Development of immersive real-time applications

Color matching Miroslav Andel - Development of immersive real-time applications

Blending Digital blending – good bright levels Not a linear gradient (gamma) Done after color and brightness matching Miroslav Andel - Development of immersive real-time applications

Blending Optical blending for better dark levels Projected black is not black, still projected light Miroslav Andel - Development of immersive real-time applications

Blending Blending in the dome Blue = Projection borders Red = Blend zones Miroslav Andel - Development of immersive real-time applications

Warping/geometric correction
Correct the rendered frame to match the geometry of the screen, lens and projector’s position Correct overlaps between adjacent channels Spatial 2D transform Miroslav Andel - Development of immersive real-time applications

Warping and blending hardware
Processing unit with input & outputs cards (FPGA) with minimal latency Miroslav Andel - Development of immersive real-time applications

Warping and blending using OpenGL
No latency but uses a bit of the GPU Miroslav Andel - Development of immersive real-time applications

Calibration process Manual warping and blending calibration takes many hours Few reference points in the dome Moving mesh points by hand causes linearity problems Semi-automated calibration Automated camera-based systems Current systems don’t handle optical blends Miroslav Andel - Development of immersive real-time applications

Hardware (end) Miroslav Andel - Development of immersive real-time applications

Immersion To give the audience the impression that they have stepped into a synthetic world Miroslav Andel - Development of immersive real-time applications

Immersion To give the audience the impression that they have stepped into a synthetic world Enhance immersion by: Large field of view High resolution Stereoscopy (can also reduce immersion) Surround sound Smooth navigation (based on a physical model) Realistic rendering (shadows etc., to improve depth cue) Miroslav Andel - Development of immersive real-time applications

Immersion Stereoscopy in the dome? The paradox Field of view wearing 3D glasses Use stereoscopy wisely Miroslav Andel - Development of immersive real-time applications

Immersion Stereoscopy in the dome? The paradox Field of view wearing 3D glasses Use stereoscopy wisely Cybersickness in immersive environments Nausea Eye strain Headache Vomiting (worst case) Miroslav Andel - Development of immersive real-time applications

Software design Miroslav Andel - Development of immersive real-time applications

The difference (from normal desktop application development)
Several projectors/screens A lot of pixels Alignment, blending & warping Color & Brightness matching Several computers (cluster) Synchronization Frame Buffers swap Data Video signal (genlock) Multi-threading GENLOCK: Genlock refers to the process of synchronizing the pixel scanning of one or more displays to an external synchronization source. NVIDIA Genlock requires the external signal to be either TTL or composite, such as used for NTSC, PAL, or HDTV. It should be noted that the NVIDIA Genlock implementation is guaranteed only to be frame-synchronized, and not necessarily pixel-synchronized. FRAME LOCK: Frame Lock involves the use of hardware to synchronize the frames on each display in a connected system. When graphics and video are displayed across multiple monitors, frame locked systems help maintain image continuity to create a virtual canvas. Frame lock is especially critical for stereo viewing, where the left and right fields must be in sync across all displays. In short, to enable genlock means to sync to an external signal. To enable frame lock means to sync 2 or more display devices to a signal generated internally by the hardware, and to use both means to sync 2 or more display devices to an external signal. SWAP SYNC: Swap sync refers to the synchronization of buffer swaps of multiple application windows. By means of swap sync, applications running on multiple systems can synchronize the application buffer swaps between all the systems. In order to work across multiple systems, swap sync requires that the systems are frame locked. Miroslav Andel - Development of immersive real-time applications

Stereoscopy How does it work? Miroslav Andel - Development of immersive real-time applications

Stereoscopy Miroslav Andel - Development of immersive real-time applications

Stereoscopy Co-location Camera position = eye position Projection plane = real physical screen Miroslav Andel - Development of immersive real-time applications

Stereoscopy Don’t change eye separation, change the scale of your scene ~15 meters limit Miroslav Andel - Development of immersive real-time applications

Miroslav Andel - Development of immersive real-time applications

Stereoscopy Object in focus in negative parallax Miroslav Andel - Development of immersive real-time applications

Stereoscopy Object in focus in positive parallax Miroslav Andel - Development of immersive real-time applications

Stereoscopy Problems with a fixed head setup (no head tracking) Miroslav Andel - Development of immersive real-time applications

Scalability One application can run in several virtual environments without the need of changing the code In the dome In the VR-Arena In the VR-Lab At home using your computer At the bus using a laptop If the platform is the same then there is no need for recompiling the application (just copy & paste) Miroslav Andel - Development of immersive real-time applications

Scalability The application uses a configuration file that describes the hardware in terms like: Number of IGs Number of displays Number of viewports Physical location of displays/projection planes Physical location of the user Eye separation of the user Type if Stereoscopy Etc… Miroslav Andel - Development of immersive real-time applications

Configuration “3d drawing”

Configuration Monoscopic projection Immersive applications
Often uses non-symmetric frustums/projections, while normal OpenGL applications don’t (left FOV = right FOV, top FOV = bottom FOV) Cyclops solution. Normally we have one dominant eye but for multi-person use lets just pick a point between the eyes. Use projection plane coordinates to calculate the projection frustum

Configuration Stereoscopic projection

Configuration Configuration - scalability

Configuration All users except the target user will perceive the rendered object incorrectly. Cyclops solution. Normally we have one dominant eye but for multi-person use lets just pick a point between the eyes.

Configuration All users except the target user will perceive the rendered object incorrectly. One solution would be using HMDs or a multi-view screen. Cyclops solution. Normally we have one dominant eye but for multi-person use lets just pick a point between the eyes.

Navigation In a desktop application a user navigates by moving the camera Miroslav Andel - Development of immersive real-time applications

Navigation In a desktop application a user navigates by moving the camera In a scalable application the number of cameras/frustums is not known Miroslav Andel - Development of immersive real-time applications

Navigation In a desktop application a user navigates by moving the camera In a scalable application the number of cameras/frustums is not known Instead of moving the user and all “cameras” in the same direction the whole scene is moved in the opposite direction Miroslav Andel - Development of immersive real-time applications

Frameworks (the good stuff)
Will help you with Cluster setup & synchronization Frustums & viewports Input Tracking Miroslav Andel - Development of immersive real-time applications

Frameworks VR Juggler CAVElib (commercial) Equalizer Miroslav Andel - Development of immersive real-time applications

Frameworks (the bad stuff)
Massive and have a lot of dependencies Big initial effort to get started Often in legacy OpenGL Takes usually several days to get started Getting all dependencies Compile all dependencies Compile the whole framework Read the documentation Miroslav Andel - Development of immersive real-time applications

SGCT Simple Graphics Cluster Toolkit
Miroslav Andel - Development of immersive real-time applications

Static library based on GLFW GLEW GLM Freetype TinyThread++ TinyXML PNG Z VRPN TurboJPEG Miroslav Andel - Development of immersive real-time applications

Developed here and is free for everybody to use Designed to get started immediately (rapid development) Cross-platform C++ Modern and Legacy OpenGL support (OGL 2.2 – 4+) Single static library (no dll-hell) Using callbacks rather than inheritance/polymorphism Easy porting application from GLUT, GLFW or similar Miroslav Andel - Development of immersive real-time applications

Wiki at: c-student.itn.liu.se Documentation Tutorials Configuration files Downloads (installers) Support Forum (on it’s way) Miroslav Andel - Development of immersive real-time applications

SGCT – How it works Has an engine which handles Initiation of OpenGL Rendering Synchronization Network communication Configuration handling Input The user can bind callbacks (function pointers) to the engine to customize the functionality Miroslav Andel - Development of immersive real-time applications

SGCT – How it works Set only the callbacks you need A minimal cluster application contains: Draw callback: Render your stuff Pre Sync callback: generate the data to be shared Encode callback: Send the data to the slave(s) Decode callback: Receive the data from the master Other callbacks to handle input devices and external GUIs Miroslav Andel - Development of immersive real-time applications

SGCT - Features Frame-, swapbuffer- and application data-synchronization 3D/Stereoscopic rendering Active/Quadbuffer stereo Side-by-side stereo Top-bottom stereo DLP/Checkerboard stereo Line interleaved stereo Anaglyph stereo Miroslav Andel - Development of immersive real-time applications

SGCT - Features Fisheye rendering PNG texture management GLSL shader management Freetype font management Error messaging across the cluster External TCP control interface Tracking support Joystick, gamepad, Kinect, steering wheel etc. input Antialiasing 3D screenshots in maximum texture resolution Overlays Statistics and performance graphs Miroslav Andel - Development of immersive real-time applications

SGCT - Configuration XML configuration All nodes are using the same configuration and can automatically determine if they are a slave or a master Miroslav Andel - Development of immersive real-time applications

<?xml version="1.0" ?> <Cluster masterAddress=" "> <Node ip=" " port="20401"> <Window fullscreen="false"> <Size x="640" y="360" /> <Viewport> <Pos x="0.0" y="0.0" /> <Size x="1.0" y="1.0" /> <Viewplane>  <Pos x="-1.778" y="-1.0" z="0.0" />  <Pos x="-1.778" y="1.0" z="0.0" />  <Pos x="1.778" y="1.0" z="0.0" /> </Viewplane> </Viewport> </Window> </Node> <User eyeSeparation="0.065"> <Pos x="0.0" y="0.0" z="4.0" /> </User> </Cluster>

SGCT arguments Normal use: app.exe –config “config.xml” Running cluster application locally as master: app.exe –config “cluster_config.xml” –local 0 Running cluster application locally as slave: app.exe –config “cluster_config.xml” –local 0 --slave Miroslav Andel - Development of immersive real-time applications

Running SGCT Command prompt/terminal Create a Batch file (.bat) Create a Shortcut From IDE Miroslav Andel - Development of immersive real-time applications

Running cluster applications
The application folder needs to be accessible from all IGs/nodes Copy the whole folder Use Microsoft’s robocopy tool for mirroring folders across a cluster Use a network share DemoManager tool for launching (and killing) processes across a cluster Or manually start the application on all nodes Miroslav Andel - Development of immersive real-time applications

SGCT – Example Basic spinning triangle synchronized across a cluster using less then 80 lines of code (including comments and spaces) Miroslav Andel - Development of immersive real-time applications

Data synchronization What to share? Time or dt Run simulations only on master or on whole cluster? Random numbers? Minimize traffic Compression Bit-masking Miroslav Andel - Development of immersive real-time applications

Math OpenGL Mathematics (GLM) is included in SGCT Computer graphics is linear algebra! GLM makes life easy Vectors Matrices Quaternions Similar syntax to glsl Miroslav Andel - Development of immersive real-time applications

Testing anywhere Local cluster on a single computer View Miroslav Andel - Development of immersive real-time applications

Testing anywhere Miroslav Andel - Development of immersive real-time applications

SGCT + OSG Open Scene Graph (OSG) Feature rich SGCT manages Cluster sharing & synchronization Window, frustum and viewport creation OSG manages Content OSGMaxExporter (3Ds Max plugin) Miroslav Andel - Development of immersive real-time applications

Scene graph Tree structure of nodes Intelligent structure Visibility & occlusion culling Miroslav Andel - Development of immersive real-time applications

OSG example Miroslav Andel - Development of immersive real-time applications

Audio Positional 3D audio OpenAL Fmod Ex Miroslav Andel - Development of immersive real-time applications

Next time Video pipeline Offline rendering for the dome Miroslav Andel - Development of immersive real-time applications

Development of immersive real-time applications

Similar presentations

Presentation on theme: "Development of immersive real-time applications"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Development of immersive real-time applications

Similar presentations

Presentation on theme: "Development of immersive real-time applications"— Presentation transcript:

Similar presentations

About project

Feedback