Concordia University: Hui Wang Amin Hammad Sudhir P. Mudur Design of a Distributed Mixed Reality System for Construction Simulation Concordia University: Hui Wang Amin Hammad Sudhir P. Mudur ASCE2007

Content Motivation Proposed Approach Implementation & Case Study Conclusion ASCE2007

Motivation One crane upset occurs during every 10,000 hours of crane use. Nearly 80 percent of these upsets can be attributed to predictable human error. ASCE2007

Motivation (cont’d) We need a system to simulate crane actions and construction progress in outdoor environment. We also want to visualize the multi-users interactive construction simulation while checking workspace conflicts and engineering constraints within a real construction site. Simulation software: to train crane operators using virtual reality (VR) (Simlog 2006); and to visualize the results of construction simulation (Kamat and Martinez 2001) ASCE2007

Motivation (cont’d) However, typical environment modeling and visualization methods is time consuming and difficult to apply for real outdoor construction site. Mixed Reality (MR) is a relatively new approach of human-computer interaction which provides advanced user immersion and experience. ASCE2007

Motivation (cont’d) ASCE2007

Motivation (cont’d) The advantages of MR have been discussed in many engineering applications such as design perception (Dunston et al., 2002). Using a variety of 3D modeling, tracking, user interaction, rendering and display techniques, MR enhances users’ immersion by allowing them to view the MR environment while moving in the real world. ASCE2007

Objectives Design a Distributed Interaction System of Construction Simulation (DISCS) with accurate interaction of users and virtual objects in outdoor MR environment of construction simulation. Develop a prototype system that can be used to test the potential of these approaches and methods. ASCE2007

Content Motivation Proposed Approach Implementation & Case Study Conclusion ASCE2007

Design overview of DMR system ASCE2007

Tracking and Registration To track a user’s viewpoint and movement, we need two important pieces of equipments. There are the GPS receiver and a 3-DOF orientation tracker. The GPS receiver provides the DMR system with real-time position tracking in the global space. Trimble 5700 RTK GPS ASCE2007

Tracking and Registration (cont’d) 3D Orientation Sensor: It is a system combining an accelerometer and magnetic sensor with a gyro. It provides yaw-pitch-roll angles as output data. Wireless InertiaCube3 and Receiver ASCE2007

Dynamic Object Modeling and Engineering Constraints Dynamically assemble CAD models of the simulation objects (i.e. cranes) based on the requirement of construction tasks. 3D CAD Model of a Crane ASCE2007

Dynamic Object Modeling and Engineering Constraints Kinematics animation is essential for producing the interaction needed to operate a crane. The DoF for the movements of the boom and the hook ASCE2007

Dynamic Object Modeling and Engineering Constraints (cont’d) For better effects, we have to consider the constraints of rendering or animation. For example, the simulated operation of a crane should respect the constraints imposed by the working ranges and load charts of the crane (Hammad et al. 2006). Example of load chart and working range (Groove Crane 2006) ASCE2007

User Interaction Display For a mobile person in the outdoor environment, the method of displaying information decides the quality of the user experience and immersion. A head-mounted display (HMD) provides the view of a computer-generated image imposed upon a see-through or video-captured real world view. User Control In order to allow the user to operate the virtual crane, the input device should provide all the controllers needed to control the DoFs of the crane. ASCE2007

Mobile Computing and Communication Mobile computing is critical for DMR in outdoor environment. There are several factors that limit the choices of a mobile computing platform, including the size and weight, graphics and multimedia capabilities, batteries, etc. Panasonic Toughbook CF-18 Tablet PC ASCE2007

Mobile Computing and Communication (cont’d) It is essential for developing a multi-user collaboration because most collaborative works need real-time communication. For the wireless connection in the outdoor environment, ad-hoc technology provides potential solution for point-to-point connection without base stations. It reduces the difficulty of setting wireless stations and provides suitable wireless communication. ASCE2007

Mobile Computing and Communication (cont’d) ASCE2007

Conceptual Configuration ASCE2007

Content Motivation Proposed Approach Implementation & Case Study Conclusion ASCE2007

Implementation tools Java & Java 3D Java Database Connectivity MapObjects Java Edition JNI DirectInput Java Communication API ASCE2007

Equipment specifications Device Type Specifications HMD nVisor SX Resolution: 1280×1024 pixels Vertical frequency: 60 Hz Joysticks Saitek Cybord evo wireless Joystick Range: 30 feet Frequency: 2.4GHz Tablet PC Panasonic ToughBook CF-18 CPU: 1GHz RAM: 512MB GPS Receiver Trimble 5700 RTK GPS Frequency: 10Hz Horizontal accuracy: ±(10 mm + 1 ppm) (×baseline length) RMS Vertical accuracy: ±(20 mm + 1 ppm) RMS 3D Motion Sensor InertiaCube3 Degrees of Freedom: 3 (Yaw, Pitch and Roll) Angular Range: Full 360° (All Axes) Maximum Angular Rate: 1200° per second Minimum Angular Rate: 0° per second RMS Accuracy: 1° in yaw, 0.25° in pitch & roll at 25°C RMS Angular Resolution: 0.03° ASCE2007

Case Study Jacques Cartier Bridge ASCE2007

Deck rehabilitation project ASCE2007

Case Study ASCE2007

Content Motivation Proposed Approach Implementation & Case Study Conclusion ASCE2007

Conclusion This paper presented design issues in integrating a variety of high-end equipment to build a construction simulation system in outdoor DMR environment. The paper provided a proposed method of visualization of crane operation and simulation of crane related construction works on the construction site considering engineering constraints. Wireless techniques for multi-user communication and collaboration for advanced user interaction in real-time outdoor environment were discussed. ASCE2007

Summary of the DMR System Complete—tracking all six degrees of freedom (position and orientation); Accurate—with resolution of 1 0 mm in position and 0.03 degree in orientation; Fast—running at 10 Hz; Robust—resisting performance degradation from magnetic fields, network communication error. Wireless—running with real-time communication ASCE2007

Future work Improving the modeling of the construction site and adding more constraints to achieve more realistic simulation. Investigating multi-user collaboration methods for advanced construction simulation. Improving the accuracy of tracking and registration in outdoor environment. ASCE2007

Thank You! ASCE2007

Thank You! ASCE2007

Rendering Pipeline ASCE2007

Motivation One crane upset occurs during every 10,000 hours of crane use. Nearly 80 percent of these upsets can be attributed to predictable human error. We need a system to simulate crane actions and construction progress in outdoor environment. ASCE2007

Motivation (cont’d) We also want to visualize the multi-users interactive construction simulation while checking workspace conflicts and engineering constraints within a real construction site. Mixed Reality (MR) is a relatively new approach of human-computer interaction which provides advanced user immersion and experience. ASCE2007

Example of Mixed Reality System (a) Concept of Mixed Reality (b) A crane at Guy Street in a outdoor test. ASCE2007

Objectives Design a Distributed Interaction System of Construction Simulation (DISCS) with accurate interaction of users and virtual objects in outdoor MR environment of construction simulation. Develop a prototype system that can be used to test the potential of these approaches and methods ASCE2007

Work Progress of DMR ASCE2007

Design of Distribute Mixed Reality ASCE2007

Tracking and Registration One of the most basic problems currently limiting Augmented Reality application is the registration problem. (Azuma, 1997) How to correctly and accurately track a user’s viewpoint? ASCE2007

Tracking and Registration (cont’d) To track a user’s viewpoint and movement, we need two important pieces of equipments. There are the GPS receiver and a 3-DOF orientation tracker. The GPS receiver provides the DMR system with real-time position tracking in the global space. Trimble 5700 RTK GPS ASCE2007

Tracking and Registration (cont’d) 3D Orientation Sensor: It is a system combining an accelerometer and magnetic sensor with a gyro. It provides yaw-pitch-roll angles as output data. ASCE2007

Rendering We use CAD models of the simulation objects (i.e. cranes) and video of real world to render combined video and display it in HMD. ASCE2007 3D CAD Model of a Crane

The DoF for the movements of the boom and the hook Rendering (cont’d) Kinematics animation is essential for producing the interaction needed to operate a crane in the outdoor environment. The DoF for the movements of the boom and the hook ASCE2007

Rendering-Constraints For better effects, we have to consider the constraints of rendering or animation. For example, the simulated operation of a crane should respect the constraints imposed by the working ranges and load charts of the crane (Hammad et al. 2006). Example of load chart and working range (Groove Crane 2006) ASCE2007

Another important constraint of outdoor augmented reality is the environment modeling. The environment modeling can solve the problems such as occlusion, terrain following, etc. ASCE2007

User Interaction For better user immersion and experience in outdoor environment, we want to use HMD and joystick to display and operate. ASCE2007

Mobile Network Communication Many constructions need advanced user collaboration. To present multi-users interaction and collaboration, the mobile network communication is critical for outdoor MR system. ASCE2007

Mobile Network Communication (cont’d) Deck-replacement project of Jacques Cartier Bridge ASCE2007

Mobile Network Communication (cont’d) Client - Server ASCE2007

ASCE2007

A Framework of Distributed Mixed Reality for Construction Simulation Distributed Mixed Reality (DMR) is a relatively new approach of human-computer interaction which provides better user immersion and experience. DMR has some new characteristics such as distributed environment, user collaboration, etc. ASCE2007

System Framework ASCE2007

Overview of System ASCE2007

AR Work Flow ASCE2007

Content Mobile Network Communication Tracking Registration Rendering User Interaction ASCE2007

Mobile Network Communication Wireless network is necessary for user communication and collaboration. Connect methods: Ad-hoc Wireless Router ASCE2007

Content Mobile Network Communication Tracking Registration Rendering User Interaction ASCE2007

Tracking-GPS The Global Positioning System (GPS), is the only fully-functional satellite navigation system. Twenty-four GPS satellites orbit the Earth, transmitting radio signals which allow GPS receivers to determine their location, speed and direction. ASCE2007

4 satellites used in calculating the location of receiver Tracking-GPS GPS provides 4D position information based on World Geodetic System 84 (WGS84) coordinates system. The information includes X,Y,Z position and time stamp. 24 satellites 4 satellites used in calculating the location of receiver ASCE2007

Tracking 3D Motion Sensor: It is a system combining an accelerometer and magnetic sensor with a gyro. It provides Z-Y-X Euler's Angle as output data. We can use the output result to calculate the current orientation. ASCE2007

Content Mobile Network Communication Tracking Registration Rendering User Interaction ASCE2007

Registration One of the most basic problems currently limiting Augmented Reality application is the registration problem. (Azuma, 1997) ASCE2007

Registration Problem Static errors Optical distortion Errors in the tracking system Mechanical misalignments Incorrect viewing parameters ( e.g., field of view, tracker-to eye position and orientation) ASCE2007

Registration Problem Dynamic errors Dynamic errors occur because of system delays, or lags. End-to end delays of 100 ms are fairly typical on existing system. Simpler systems can have less delay, but other systems have more. ASCE2007

Registration Problem We can use the methods to reduce the dynamic errors (Azuma, 1997). Reduce system lag Reduce apparent lag Match temporal streams (with video-based systems) Predict future locations ASCE2007

Content Mobile Network Communication Tracking Registration Rendering User Interaction ASCE2007

Rendering-GIS The proposed method of terrain constraint for outdoor augmented reality is loading terrain data from Geographic Information System (GIS) based on the GPS position. Terrain from GIS ASCE2007

Content Mobile Network Communication Tracking Registration Rendering User Interaction ASCE2007

User Interaction Portable computer Head-Mounted display Joystick ASCE2007

Reference Azuma, R., 1997, A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6(4): pp. 355-385. Azuma, R., 1999, The challenge of making augmented reality work outdoors. In ohta, Y. and Tamura, H., editors, Mixed Reality, Merging Real and Virtual Worlds, pp. 379-390. Ohmsha/Springer, Tokyo/New York. Hammad, A., Wang, H., Zhang, C. and Al-Hussein, M. (2006). Visualizing Crane Selection and Operation in Virtual Environment. In proceedings of the 6th Visualizing Crane Selection and Operation in Virtual Environment in Orlando, Florida, August, 2006 Kalagnanam, A and Balu, G (2002). Merlin brings nonblocking I/O to the Java platform. [Online]. Available:http://www-128.ibm.com/developerworks/java/library/j-javaio/ [2006, November 15] Lea, D (2002).Scalable IO in Java. [Online]. Available:http://gee.cs.oswego.edu/dl/cpjslides/nio.pdf [2006, November 15] Lewandowski, S.M. (1998) Frameworks for component-based client/server computing, ACM Computing Surveys, Volume 30 , Issue 1,pp3-27 Schmidt, D.C. (1995). Reactor -- An Object Behavioral Pattern for Event Demultiplexing and Event Handler Dispatching. In proceedings of the First Pattern Languages of Programs conference in Monticello, Illinois, August, 1994 Schmidt, D.C. (1996). Asynchronous Completion Token -- An Object Behavioral Pattern for Efficient Asynchronous Event Handling. In proceeding of the 3rd annual Pattern Languages of Programming conference in Allerton Park, Illinois, September 4-6, 1996 Shirazi, J and Pepperdine, K (2004).Eye on performance: MegaJogos scales up with NIO. [Online]. Available:http://www-128.ibm.com/developerworks/java/library/j-perf03174.html [2006, October 30] You, S., Neumann, U, and Azuma, R. (1999) Orientation Tracking for Outdoor Augmented Reality Registration. IEEE Computer Graphics and Applications, 19(6):36--42, Nov/Dec 1999. Zaki, A. R., and Mailhot, G. (2003). Deck Reconstruction of Jacques Cartier Bridge Using Precast Prestressed High Performance Concrete Panels, PCI Journal. pp20-33. ASCE2007

ASCE2007

ceramic gyro A ceramic gyro is a sensor that measures rotational (angular) velocity by oscillating a piezoelectric ceramic resonator with alternating voltage. The NECTokin Ceramic Gyro uses an extremely simple construction: piezoelectric ceramic rods, to which electrodes are attached during the printing process. This gives the sensor consistent quality and a wide range of applications, including shake detectors for hand-held video cameras. ASCE2007

Other ASCE2007

Mobile Network Communication (cont’d) h1 l r w θ h2 α ASCE2007

Optical see-through HMD Graphic Images Head Tracker Head Locations Scene Generator Monitors Real World Optical Combiners ASCE2007

Euler Angle Euler angles are used extensively in the classical mechanics of rigid bodies, and in the quantum mechanics of angular momentum. When studying rigid bodies, one calls the xyz system space coordinates, and the XYZ system body coordinates. The space coordinates are treated as unmoving, while the body coordinates are considered embedded in the moving body. Calculations involving kinetic energy are usually easiest in body coordinates, because then the moment of inertia tensor does not change in time. If one also diagonalizes the rigid body's moment of inertia tensor (with nine components, six of which are independent), then one has a set of coordinates (called the principal axes) in which the moment of inertia tensor has only three components. ASCE2007

Euler Angle (contd.) The angular velocity, in body coordinates, of a rigid body takes a simple form using Euler angles: where IJK are unit vectors for XYZ. ASCE2007

Euler Angle (contd.) In quantum mechanics, explicit descriptions of the representations of SO(3) are very important for calculations, and almost all the work has been done using Euler angles. In the early history of quantum mechanics, when physicists and chemists had a sharply negative reaction towards abstract group theoretic methods (called the Gruppenpest), reliance on Euler angles was also essential for basic theoretical work. ASCE2007

Euler Angle (contd.) Haar measure for Euler angles has the simple form sin(β)dαdβdγ, usually normalized by a factor of 1/8π2. For example, to generate uniformly randomized orientations, let α and γ be uniform from 0 to 2π, let z be uniform from −1 to 1, and let β = arccos(z). Unit quaternions, also known as Euler-Rodrigues parameters, provide another mechanism for representing 3D rotations. This is equivalent to the special unitary group description. Quaternions are generally quicker for most calculations, conceptually simpler to interpolate, and are not subject to gimbal lock. Much high speed 3D graphics programming (gaming, for example) uses quaternions because of this. ASCE2007

Converting Euler Angles to a Matrix Euler angles define a sequence of rotations. We have compute the matrix for each individual rotation and then concatenate these matrices into one matrix that defines the total angular displacement. Let us assume we use the heading, pitch, and bank system of Euler Angles. ASCE2007

Converting Euler Angles to a Matrix M=HPB Question: How we use the orientation based on Euler’s angle? ASCE2007

3D Orientation Sensor ASCE2007

Yaw-Pitch-Roll ASCE2007

Design Challenge Mixed Reality needs accurate tracking of augmented space in real time. The outdoor environment brings difficulties and challenges for the tracking because the outdoor is unprepared and dynamic. A major difference between indoor and outdoor MR system is user’s capacity. The size, weight, power and convince of distributed mixed reality (DMR) system will affect the efficient and user interaction of the system. Registration of virtual objects in DMR system have to use constraints for reality. Wireless networking is critical for dynamic and flexible DMR because real-time information cannot be stored or calculated on the portable computer before application run-time. ASCE2007