Presentation is loading. Please wait.

Presentation is loading. Please wait.

Development of immersive real- time applications Miroslav Andel

Similar presentations


Presentation on theme: "Development of immersive real- time applications Miroslav Andel"— Presentation transcript:

1 Development of immersive real- time applications Miroslav Andel

2 Development of immersive real-time applications –Online/direct rendering –Hardware –Programming Video production –Offline rendering –Hybrid solutions Lectures Miroslav Andel - Development of immersive real-time applications 2

3 Hardware –Understanding the technology –Clusters and GPUs –Warping, color matching & blending Immersion in a virtual environment (VE) Software design –Desktop application vs. immersive cluster application –Scalability & configuration of VEs –Frameworks –Scene graph Outline Miroslav Andel - Development of immersive real-time applications 3

4 Hardware Miroslav Andel - Development of immersive real-time applications 4

5 6x Digital Projection Titan 1920 x 120 Hz Frame doubling Projectors Miroslav Andel - Development of immersive real-time applications 5

6 Image Generators (IGs) Miroslav Andel - Development of immersive real-time applications 6 –Workstation high-end graphics cards Frame sync External house sync Genlock (video signal sync) Swaplock (swap buffer sync) 3D sync (for 3D glasses) –Supports active stereoscopy using OpenGL –Nvidia Quadro or AMD/ATI FirePro –No gaming cards

7 GPU is pretty much the same but different driver software and firmware Gaming cards are optimized for speed and DirectX –Hi frame-rate in games, highly optimized Workstation cards are optimized for precision, OpenGL & OpenCL –Designed for 3D modeling, CAD, video editing and computation –Higher floating point and sub-pixel precision –Better quality and larger memory (12 GB in Quadro K6000, 16 GB AMD FirePro 9100) Ray tracing on the GPU Our problem: We use gaming techniques but need workstation features like synchronization and stereoscopy (less performance) Gaming vs. Workstation GPU Miroslav Andel - Development of immersive real-time applications 7

8 Cluster of IGs Often consists of workstations with the same specification Master – Slave In our case the master has an additional surround sound card (live encoding to Dolby Digital or DTS) Miroslav Andel - Development of immersive real-time applications 8

9 Clusters (at the visualization center) Dome –Alpha (primary real-time cluster used for shows) –Beta (secondary development cluster) –Delta (movie and presentations playback cluster) –Gamma (single node eyefinity machine) VR-Arena –Theta –Videowall (just a single node with 6 outputs) used for movie and presentations playback Miroslav Andel - Development of immersive real-time applications 9

10 OpenGL/DirectX projections Miroslav Andel - Development of immersive real-time applications 10 One viewplane per projector Curved surface?

11 What is white? Color/gammut matching Miroslav Andel - Development of immersive real-time applications 11

12 Color matching White points (CIE standard illuminant) D65 and D Miroslav Andel - Development of immersive real-time applications 12 D65 = 6500 K = (0.3127, ) D75 = 7500 K = (0.2990, ) Other white points: DCI = (0.3140, ) DICOM clear base = (0.280, 0.305) DICOM blue base = (0.250, 0.304)

13 Color matching All display devices and projectors have a different gamut Projector lamp color temperature changes over time Must be done before blending Can be done on each IG but with less precision (8 or 10 bits per color) Better to do that in the projector (12-16 bit precision) Miroslav Andel - Development of immersive real-time applications 13

14 Color matching Miroslav Andel - Development of immersive real-time applications 14

15 Digital blending – good bright levels –Not a linear gradient (gamma) –Done after color and brightness matching Blending Miroslav Andel - Development of immersive real-time applications 15

16 Optical blending for better dark levels –Projected black is not black, still projected light Blending Miroslav Andel - Development of immersive real-time applications 16

17 Optical blending for better dark levels –Projected black is not black, still projected light Blending Miroslav Andel - Development of immersive real-time applications 17

18 Blending in the dome –Blue = Projection borders –Red = Blend zones Blending Miroslav Andel - Development of immersive real-time applications 18

19 Correct the rendered frame to match the geometry of the screen, lens and projector’s position Correct overlaps between adjacent channels Spatial 2D transform Warping/geometric correction Miroslav Andel - Development of immersive real-time applications 19

20 Processing unit with input & outputs cards (FPGA) with minimal latency Warping and blending hardware Miroslav Andel - Development of immersive real-time applications 20

21 No latency but uses a bit of the GPU Warping and blending using OpenGL Miroslav Andel - Development of immersive real-time applications 21

22 Manual warping and blending calibration takes many hours –Few reference points in the dome –Moving mesh points by hand causes linearity problems Semi-automated calibration Automated camera-based systems –Current systems don’t handle optical blends Calibration process Miroslav Andel - Development of immersive real-time applications 22

23 Hardware (end) Miroslav Andel - Development of immersive real-time applications 23

24 To give the audience the impression that they have stepped into a synthetic world Immersion Miroslav Andel - Development of immersive real-time applications 24

25 To give the audience the impression that they have stepped into a synthetic world Enhance immersion by: –Large field of view –High resolution –Stereoscopy (can also reduce immersion) –Surround sound –Smooth navigation (based on a physical model) –Realistic rendering (shadows etc., to improve depth cue) Immersion Miroslav Andel - Development of immersive real-time applications 25

26 Stereoscopy in the dome? –The paradox –Field of view wearing 3D glasses –Use stereoscopy wisely Immersion Miroslav Andel - Development of immersive real-time applications 26

27 Stereoscopy in the dome? –The paradox –Field of view wearing 3D glasses –Use stereoscopy wisely Cybersickness in immersive environments –Nausea –Eye strain –Headache –Vomiting (worst case) Immersion Miroslav Andel - Development of immersive real-time applications 27

28 Software design Miroslav Andel - Development of immersive real-time applications 28

29 Several projectors/screens –A lot of pixels –Alignment, blending & warping –Color & Brightness matching Several computers (cluster) Synchronization –Frame –Buffers swap –Data –Video signal (genlock) Multi-threading The difference (from normal desktop application development) Miroslav Andel - Development of immersive real-time applications 29

30 How does it work? Stereoscopy Miroslav Andel - Development of immersive real-time applications 30

31 Stereoscopy Miroslav Andel - Development of immersive real-time applications 31

32 Stereoscopy Miroslav Andel - Development of immersive real-time applications 32

33 Stereoscopy Miroslav Andel - Development of immersive real-time applications 33

34 Co-location –Camera position = eye position –Projection plane = real physical screen Stereoscopy Miroslav Andel - Development of immersive real-time applications 34

35 Don’t change eye separation, change the scale of your scene ~15 meters limit Stereoscopy Miroslav Andel - Development of immersive real-time applications 35

36 Miroslav Andel - Development of immersive real-time applications 36

37 Miroslav Andel - Development of immersive real-time applications 37

38 Miroslav Andel - Development of immersive real-time applications 38

39 Object in focus in negative parallax Stereoscopy Miroslav Andel - Development of immersive real-time applications 39

40 Stereoscopy Miroslav Andel - Development of immersive real-time applications 40 Object in focus in positive parallax

41 Problems with a fixed head setup (no head tracking) Stereoscopy Miroslav Andel - Development of immersive real-time applications 41

42 One application can run in several virtual environments without the need of changing the code –In the dome –In the VR-Arena –In the VR-Lab –At home using your computer –At the bus using a laptop If the platform is the same then there is no need for recompiling the application (just copy & paste) Scalability Miroslav Andel - Development of immersive real-time applications 42

43 The application uses a configuration file that describes the hardware in terms like: Number of IGs Number of displays Number of viewports Physical location of displays/projection planes Physical location of the user Eye separation of the user Type if Stereoscopy Etc… Scalability Miroslav Andel - Development of immersive real-time applications 43

44 “3d drawing” Configuration

45 Configuration Immersive applications Often uses non-symmetric frustums/projections, while normal OpenGL applications don’t (left FOV = right FOV, top FOV = bottom FOV) Monoscopic projection

46 Configuration Stereoscopic projection

47 Configuration 47

48 Configuration 48 All users except the target user will perceive the rendered object incorrectly.

49 Configuration 49 All users except the target user will perceive the rendered object incorrectly. One solution would be using HMDs or a multi-view screen.

50 In a desktop application a user navigates by moving the camera Navigation Miroslav Andel - Development of immersive real-time applications 50

51 In a desktop application a user navigates by moving the camera In a scalable application the number of cameras/frustums is not known Navigation Miroslav Andel - Development of immersive real-time applications 51

52 In a desktop application a user navigates by moving the camera In a scalable application the number of cameras/frustums is not known Instead of moving the user and all “cameras” in the same direction the whole scene is moved in the opposite direction Navigation Miroslav Andel - Development of immersive real-time applications 52

53 Will help you with –Cluster setup & synchronization –Frustums & viewports –Input –Tracking Frameworks (the good stuff) Miroslav Andel - Development of immersive real-time applications 53

54 Frameworks Miroslav Andel - Development of immersive real-time applications 54 VR Juggler CAVElib (commercial) Equalizer

55 Massive and have a lot of dependencies Big initial effort to get started Often in legacy OpenGL Takes usually several days to get started –Getting all dependencies –Compile all dependencies –Compile the whole framework –Read the documentation Frameworks (the bad stuff) Miroslav Andel - Development of immersive real-time applications 55

56 SGCT Simple Graphics Cluster Toolkit Miroslav Andel - Development of immersive real-time applications 56

57 Static library based on –GLFW –GLEW –GLM –Freetype –TinyThread++ –TinyXML –PNG –Z –VRPN –TurboJPEG SGCT Simple Graphics Cluster Toolkit Miroslav Andel - Development of immersive real-time applications 57

58 Developed here and is free for everybody to use Designed to get started immediately (rapid development) Cross-platform C++ Modern and Legacy OpenGL support (OGL 2.2 – 4+) Single static library (no dll-hell) Using callbacks rather than inheritance/polymorphism –Easy porting application from GLUT, GLFW or similar SGCT Simple Graphics Cluster Toolkit Miroslav Andel - Development of immersive real-time applications 58

59 Wiki at: c-student.itn.liu.se –Documentation –Tutorials –Configuration files –Downloads (installers) –Support Forum (on it’s way) SGCT Simple Graphics Cluster Toolkit Miroslav Andel - Development of immersive real-time applications 59

60 Has an engine which handles –Initiation of OpenGL –Rendering –Synchronization –Network communication –Configuration handling –Input The user can bind callbacks (function pointers) to the engine to customize the functionality SGCT – How it works Miroslav Andel - Development of immersive real-time applications 60

61

62 Set only the callbacks you need A minimal cluster application contains: –Draw callback: Render your stuff –Pre Sync callback: generate the data to be shared –Encode callback: Send the data to the slave(s) –Decode callback: Receive the data from the master Other callbacks to handle input devices and external GUIs SGCT – How it works Miroslav Andel - Development of immersive real-time applications 62

63 Frame-, swapbuffer- and application data- synchronization 3D/Stereoscopic rendering –Active/Quadbuffer stereo –Side-by-side stereo –Top-bottom stereo –DLP/Checkerboard stereo –Line interleaved stereo –Anaglyph stereo SGCT - Features Miroslav Andel - Development of immersive real-time applications 63

64 Fisheye rendering PNG texture management GLSL shader management Freetype font management Error messaging across the cluster External TCP control interface Tracking support Joystick, gamepad, Kinect, steering wheel etc. input Antialiasing 3D screenshots in maximum texture resolution Overlays Statistics and performance graphs SGCT - Features Miroslav Andel - Development of immersive real-time applications 64

65 XML configuration All nodes are using the same configuration and can automatically determine if they are a slave or a master SGCT - Configuration Miroslav Andel - Development of immersive real-time applications 65

66 66

67 Normal use: app.exe –config “config.xml” Running cluster application locally as master: app.exe –config “cluster_config.xml” –local 0 Running cluster application locally as slave: app.exe –config “cluster_config.xml” –local 0 -- slave SGCT arguments Miroslav Andel - Development of immersive real-time applications 67

68 Command prompt/terminal Create a Batch file (.bat) Create a Shortcut From IDE Running SGCT Miroslav Andel - Development of immersive real-time applications 68

69 The application folder needs to be accessible from all IGs/nodes 1.Copy the whole folder Use Microsoft’s robocopy tool for mirroring folders across a cluster 2.Use a network share DemoManager tool for launching (and killing) processes across a cluster Or manually start the application on all nodes Running cluster applications Miroslav Andel - Development of immersive real-time applications 69

70 SGCT – Example Miroslav Andel - Development of immersive real-time applications 70 Basic spinning triangle synchronized across a cluster using less then 80 lines of code (including comments and spaces)

71 What to share? –Time or dt –Run simulations only on master or on whole cluster? –Random numbers? –Minimize traffic –Compression –Bit-masking Data synchronization Miroslav Andel - Development of immersive real-time applications 71

72 OpenGL Mathematics (GLM) is included in SGCT Computer graphics is linear algebra! GLM makes life easy –Vectors –Matrices –Quaternions Similar syntax to glsl Math Miroslav Andel - Development of immersive real-time applications 72

73 Local cluster on a single computer View Testing anywhere Miroslav Andel - Development of immersive real-time applications 73

74 Testing anywhere Miroslav Andel - Development of immersive real-time applications 74

75 Open Scene Graph (OSG) Feature rich SGCT manages –Cluster sharing & synchronization –Window, frustum and viewport creation OSG manages –Content OSGMaxExporter (3Ds Max plugin) SGCT + OSG Miroslav Andel - Development of immersive real-time applications 75

76 Scene graph Miroslav Andel - Development of immersive real-time applications 76 Tree structure of nodes Intelligent structure –Visibility & occlusion culling

77 OSG example Miroslav Andel - Development of immersive real-time applications 77

78 Positional 3D audio –OpenAL –Fmod Ex Audio Miroslav Andel - Development of immersive real-time applications 78

79 Video pipeline Offline rendering for the dome Next time Miroslav Andel - Development of immersive real-time applications 79


Download ppt "Development of immersive real- time applications Miroslav Andel"

Similar presentations


Ads by Google