1. The Visual Display Transform for Virtual Reality Cyrus Moon Computer Integrated Surgery II (600.446)

2. Presentation Outline Synopsis of Current Project Concepts introduced in the Reading –VQS Representation –Coordinate System Graph –Object-to-Screen Transform Relevancy of concepts to the current project

3. Current Project: Image/Video Overlay Image Overlay: -the merging of relevant computer generated information with the user’s actual view of the real world Video Overlay: -the merging of relevant computer generated information with display output from a video source

4. Relevant Concepts Registration Tracking in 3D real space 3D modeling/rendering Frame transformations

5. Background Reading Robinett Warren, Halloway Richard. The Visual Display Transformation for Virtual Reality. Technical Report TR94-031 (1994), Dept. of Computer Science, University of North Carolina at Chapel Hill.

6. VQS - Vector Quaternion Scalar Representation (for frame transformations) Vector(v) (3 terms)  displacement Quaternion(q) (4 terms)  rotation Scalar(s) (1 term)  uniform scaling [ (v x, v y, v z ), (q x, q y, q z, q w ), s ]

7. Advantages of VQS (over the Euler 4x4 homogeneous matrix) Translation, rotation, and scaling components are separated Renormalizing the rotation is simpler (since rotation and scaling are separate) Uniform scaling in the virtual world a useful operation Quaternions a better method of manipulating 3D rotations than Euler rotations

8. Advantages of the Quaternion (over the Euler 3x3 Rotation Matrix) Fewer components (4 instead of 9); fewer redundant parameters More elegant, numerically robust Explicit representation of the angle and axis of rotation Allows easy interpolation between two orientations

9. The Quaternion [(q x, q y, q z ), q w ] (q x, q y, q z )  axis of rotation q w  angle of rotation: –q w takes values from –1 to 1

10. Quaternion Math Addition: Multiplication: Multiplication by Scalar:

11. Quaternion Math (cont’d) Taking the norm: Normalization: Inversion: Interpolation:

12. Using Quaternions Rotation of a vector p by quaternion q: p new = q*p*q -1 -the vector p is treated as a quaternion w/ 0 as the 4 th (scalar) term -the result will always have a 4 th term of 0, as well

13. VQS Math Conversion to 4x4 Matrix: [v,q,s] = M translate * M rotate * M scale =

14. Using VQS Transforms Transformation of a vector (p) with VQS: p ’ = [v, q, s]*p = s(q*p*q -1 ) + v Composition of two VQS transforms: T A_B *T B_C = [v A_B,q A_B,s A_B ]*[v B_C,q B_C,s B_C ] = [(s A_B *(q A_B *v B_C *q A_B ) -1 ) + v A_B, q A_B *q B_C, s A_B *s B_C ] Inverse of a VQS Transform: T A_B -1 = [v A_B, q A_B, s A_B ] -1 = [1/s A_B *(q A_B -1 *(-v A_B )*q A_B ), q A_B -1, 1/s A_B ]

15. The Coordinate System Graph Robinett & Halloway

16. The Coordinate System Graph Representation: –Each Node represents a coordinate system –Each line represents some kind of independent transformation Properties: –Connected: each node is connected with every other node –Acyclic: there is only one pathway between any two given nodes

17. Transformations Independent: characterized by being independent variables within the software –Measured by tracker –Constant (rigid) Dependent: calculated from independent transforms The Coordinate System Graph: -intuitively organizes all independent transforms and coordinate systems; easily expandable -allows easy calculation of any dependent transform present within the VR system

18. The Object-to-Screen Transform T S_O = T S_US * T US_N * T N_E * T E_H * T H_HS * T HS_TB * T TB_R * T R_W * T W_O -T B_A is defined as the transformation from frame A to B S = ScreenHS = Head Sensor US = Undistorted ScreenTB = Tracker Base N = NormalizedR = Room E = EyeW = World H = HeadO = Object

19. T W_O : World_Object – object in the virtual world. –v - position –q - orientation –s - size T R_W : Room_World – the user position in the virtual world –v - position –q - tilt of the world –s - user's size (shrinking or expanding of the world) T TB_R : TrackerBase_Room – position of tracker base (stored in calibration file) –s - must always be one (both cs’s in real-space) –Pre-calculated VQS Transforms

20. T HS_TB : HeadSensor_TrackerBase – inverse of the head position and orientation read from tracker –s = 1 T TH_HS : Head_HeadSensor – position and orientation of the HMD sensor w/ respect to the head (center of the eyes) –s = 1 –Pre-calculated T E_H : Eye_Head – position/orientation of head coordinate system w/ respect to each eye –v – different for each user (though a default value can be used –q – dependent on orientation of HMD displays –s = 1 –Pre-calculated VQS Transforms (cont’d)

21. T Eye_Head Robinett & Halloway

22. T N_E : Normalized_Eye – perspective projection, normalization –Three to two dimensions (projection of world onto viewing plane) T US_N : UndistortedScreen_Normalized– conversion to pixel coordinates –Simple scaling process T S_US : Screen_UndistortedScreen – correction of image distortion Non-VQS Transforms

23. Applicable Concepts VQS Representation –Tracker Data  Transform calculations –Registration Transforms (non-deformable objects) Elimination of the possibility of warping Coordinate System Graph Other concepts discussed in the individual transforms –Example: perspective transform

24. Coordinate Graph (Video Overlay) Patient markers Model (from Imaging) World (Tracker) Tool 1Tool k.... Camera Lens Screen

25. Coordinate Graph (Image Overlay) Patient markers Model (from Imaging) World (Tracker) Tool 1Tool k.... Silvered Glass Head Virtual Projection Plane Screen