Download presentation
Presentation is loading. Please wait.
Published byCarmella Fields Modified over 7 years ago
1
Concordia University: Hui Wang Amin Hammad Sudhir P. Mudur
Design of a Distributed Mixed Reality System for Construction Simulation Concordia University: Hui Wang Amin Hammad Sudhir P. Mudur ASCE2007
2
Content Motivation Proposed Approach Implementation & Case Study
Conclusion ASCE2007
3
Motivation One crane upset occurs during every 10,000 hours of crane use. Nearly 80 percent of these upsets can be attributed to predictable human error. ASCE2007
4
Motivation (cont’d) We need a system to simulate crane actions and construction progress in outdoor environment. We also want to visualize the multi-users interactive construction simulation while checking workspace conflicts and engineering constraints within a real construction site. Simulation software: to train crane operators using virtual reality (VR) (Simlog 2006); and to visualize the results of construction simulation (Kamat and Martinez 2001) ASCE2007
5
Motivation (cont’d) However, typical environment modeling and visualization methods is time consuming and difficult to apply for real outdoor construction site. Mixed Reality (MR) is a relatively new approach of human-computer interaction which provides advanced user immersion and experience. ASCE2007
6
Motivation (cont’d) ASCE2007
7
Motivation (cont’d) The advantages of MR have been discussed in many engineering applications such as design perception (Dunston et al., 2002). Using a variety of 3D modeling, tracking, user interaction, rendering and display techniques, MR enhances users’ immersion by allowing them to view the MR environment while moving in the real world. ASCE2007
8
Objectives Design a Distributed Interaction System of Construction Simulation (DISCS) with accurate interaction of users and virtual objects in outdoor MR environment of construction simulation. Develop a prototype system that can be used to test the potential of these approaches and methods. ASCE2007
9
Content Motivation Proposed Approach Implementation & Case Study
Conclusion ASCE2007
10
Design overview of DMR system
ASCE2007
11
Tracking and Registration
To track a user’s viewpoint and movement, we need two important pieces of equipments. There are the GPS receiver and a 3-DOF orientation tracker. The GPS receiver provides the DMR system with real-time position tracking in the global space. Trimble 5700 RTK GPS ASCE2007
12
Tracking and Registration (cont’d)
3D Orientation Sensor: It is a system combining an accelerometer and magnetic sensor with a gyro. It provides yaw-pitch-roll angles as output data. Wireless InertiaCube3 and Receiver ASCE2007
13
Dynamic Object Modeling and Engineering Constraints
Dynamically assemble CAD models of the simulation objects (i.e. cranes) based on the requirement of construction tasks. 3D CAD Model of a Crane ASCE2007
14
Dynamic Object Modeling and Engineering Constraints
Kinematics animation is essential for producing the interaction needed to operate a crane. The DoF for the movements of the boom and the hook ASCE2007
15
Dynamic Object Modeling and Engineering Constraints (cont’d)
For better effects, we have to consider the constraints of rendering or animation. For example, the simulated operation of a crane should respect the constraints imposed by the working ranges and load charts of the crane (Hammad et al. 2006). Example of load chart and working range (Groove Crane 2006) ASCE2007
16
User Interaction Display
For a mobile person in the outdoor environment, the method of displaying information decides the quality of the user experience and immersion. A head-mounted display (HMD) provides the view of a computer-generated image imposed upon a see-through or video-captured real world view. User Control In order to allow the user to operate the virtual crane, the input device should provide all the controllers needed to control the DoFs of the crane. ASCE2007
17
Mobile Computing and Communication
Mobile computing is critical for DMR in outdoor environment. There are several factors that limit the choices of a mobile computing platform, including the size and weight, graphics and multimedia capabilities, batteries, etc. Panasonic Toughbook CF-18 Tablet PC ASCE2007
18
Mobile Computing and Communication (cont’d)
It is essential for developing a multi-user collaboration because most collaborative works need real-time communication. For the wireless connection in the outdoor environment, ad-hoc technology provides potential solution for point-to-point connection without base stations. It reduces the difficulty of setting wireless stations and provides suitable wireless communication. ASCE2007
19
Mobile Computing and Communication (cont’d)
ASCE2007
20
Conceptual Configuration
ASCE2007
21
Content Motivation Proposed Approach Implementation & Case Study
Conclusion ASCE2007
22
Implementation tools Java & Java 3D Java Database Connectivity
MapObjects Java Edition JNI DirectInput Java Communication API ASCE2007
23
Equipment specifications
Device Type Specifications HMD nVisor SX Resolution: 1280×1024 pixels Vertical frequency: 60 Hz Joysticks Saitek Cybord evo wireless Joystick Range: 30 feet Frequency: 2.4GHz Tablet PC Panasonic ToughBook CF-18 CPU: 1GHz RAM: 512MB GPS Receiver Trimble 5700 RTK GPS Frequency: 10Hz Horizontal accuracy: ±(10 mm + 1 ppm) (×baseline length) RMS Vertical accuracy: ±(20 mm + 1 ppm) RMS 3D Motion Sensor InertiaCube3 Degrees of Freedom: 3 (Yaw, Pitch and Roll) Angular Range: Full 360° (All Axes) Maximum Angular Rate: 1200° per second Minimum Angular Rate: 0° per second RMS Accuracy: 1° in yaw, 0.25° in pitch & roll at 25°C RMS Angular Resolution: 0.03° ASCE2007
24
Case Study Jacques Cartier Bridge ASCE2007
25
Deck rehabilitation project
ASCE2007
26
Case Study ASCE2007
27
Content Motivation Proposed Approach Implementation & Case Study
Conclusion ASCE2007
28
Conclusion This paper presented design issues in integrating a variety of high-end equipment to build a construction simulation system in outdoor DMR environment. The paper provided a proposed method of visualization of crane operation and simulation of crane related construction works on the construction site considering engineering constraints. Wireless techniques for multi-user communication and collaboration for advanced user interaction in real-time outdoor environment were discussed. ASCE2007
29
Summary of the DMR System
Complete—tracking all six degrees of freedom (position and orientation); Accurate—with resolution of 1 0 mm in position and 0.03 degree in orientation; Fast—running at 10 Hz; Robust—resisting performance degradation from magnetic fields, network communication error. Wireless—running with real-time communication ASCE2007
30
Future work Improving the modeling of the construction site and adding more constraints to achieve more realistic simulation. Investigating multi-user collaboration methods for advanced construction simulation. Improving the accuracy of tracking and registration in outdoor environment. ASCE2007
31
Thank You! ASCE2007
32
Thank You! ASCE2007
33
Rendering Pipeline ASCE2007
34
Motivation One crane upset occurs during every 10,000 hours of crane use. Nearly 80 percent of these upsets can be attributed to predictable human error. We need a system to simulate crane actions and construction progress in outdoor environment. ASCE2007
35
Motivation (cont’d) We also want to visualize the multi-users interactive construction simulation while checking workspace conflicts and engineering constraints within a real construction site. Mixed Reality (MR) is a relatively new approach of human-computer interaction which provides advanced user immersion and experience. ASCE2007
36
Example of Mixed Reality System
(a) Concept of Mixed Reality (b) A crane at Guy Street in a outdoor test. ASCE2007
37
Objectives Design a Distributed Interaction System of Construction Simulation (DISCS) with accurate interaction of users and virtual objects in outdoor MR environment of construction simulation. Develop a prototype system that can be used to test the potential of these approaches and methods ASCE2007
38
Work Progress of DMR ASCE2007
39
Design of Distribute Mixed Reality
ASCE2007
40
Tracking and Registration
One of the most basic problems currently limiting Augmented Reality application is the registration problem. (Azuma, 1997) How to correctly and accurately track a user’s viewpoint? ASCE2007
41
Tracking and Registration (cont’d)
To track a user’s viewpoint and movement, we need two important pieces of equipments. There are the GPS receiver and a 3-DOF orientation tracker. The GPS receiver provides the DMR system with real-time position tracking in the global space. Trimble 5700 RTK GPS ASCE2007
42
Tracking and Registration (cont’d)
3D Orientation Sensor: It is a system combining an accelerometer and magnetic sensor with a gyro. It provides yaw-pitch-roll angles as output data. ASCE2007
43
Rendering We use CAD models of the simulation objects (i.e. cranes) and video of real world to render combined video and display it in HMD. ASCE2007 3D CAD Model of a Crane
44
The DoF for the movements of the boom and the hook
Rendering (cont’d) Kinematics animation is essential for producing the interaction needed to operate a crane in the outdoor environment. The DoF for the movements of the boom and the hook ASCE2007
45
Rendering-Constraints
For better effects, we have to consider the constraints of rendering or animation. For example, the simulated operation of a crane should respect the constraints imposed by the working ranges and load charts of the crane (Hammad et al. 2006). Example of load chart and working range (Groove Crane 2006) ASCE2007
46
Another important constraint of outdoor augmented reality is the environment modeling. The environment modeling can solve the problems such as occlusion, terrain following, etc. ASCE2007
47
User Interaction For better user immersion and experience in outdoor environment, we want to use HMD and joystick to display and operate. ASCE2007
48
Mobile Network Communication
Many constructions need advanced user collaboration. To present multi-users interaction and collaboration, the mobile network communication is critical for outdoor MR system. ASCE2007
49
Mobile Network Communication (cont’d)
Deck-replacement project of Jacques Cartier Bridge ASCE2007
50
Mobile Network Communication (cont’d)
Client - Server ASCE2007
51
ASCE2007
52
A Framework of Distributed Mixed Reality for Construction Simulation
Distributed Mixed Reality (DMR) is a relatively new approach of human-computer interaction which provides better user immersion and experience. DMR has some new characteristics such as distributed environment, user collaboration, etc. ASCE2007
53
System Framework ASCE2007
54
Overview of System ASCE2007
55
AR Work Flow ASCE2007
56
Content Mobile Network Communication Tracking Registration Rendering
User Interaction ASCE2007
57
Mobile Network Communication
Wireless network is necessary for user communication and collaboration. Connect methods: Ad-hoc Wireless Router ASCE2007
58
Content Mobile Network Communication Tracking Registration Rendering
User Interaction ASCE2007
59
Tracking-GPS The Global Positioning System (GPS), is the only fully-functional satellite navigation system. Twenty-four GPS satellites orbit the Earth, transmitting radio signals which allow GPS receivers to determine their location, speed and direction. ASCE2007
60
4 satellites used in calculating the location of receiver
Tracking-GPS GPS provides 4D position information based on World Geodetic System 84 (WGS84) coordinates system. The information includes X,Y,Z position and time stamp. 24 satellites 4 satellites used in calculating the location of receiver ASCE2007
61
Tracking 3D Motion Sensor:
It is a system combining an accelerometer and magnetic sensor with a gyro. It provides Z-Y-X Euler's Angle as output data. We can use the output result to calculate the current orientation. ASCE2007
62
Content Mobile Network Communication Tracking Registration Rendering
User Interaction ASCE2007
63
Registration One of the most basic problems currently limiting Augmented Reality application is the registration problem. (Azuma, 1997) ASCE2007
64
Registration Problem Static errors Optical distortion
Errors in the tracking system Mechanical misalignments Incorrect viewing parameters ( e.g., field of view, tracker-to eye position and orientation) ASCE2007
65
Registration Problem Dynamic errors
Dynamic errors occur because of system delays, or lags. End-to end delays of 100 ms are fairly typical on existing system. Simpler systems can have less delay, but other systems have more. ASCE2007
66
Registration Problem We can use the methods to reduce the dynamic errors (Azuma, 1997). Reduce system lag Reduce apparent lag Match temporal streams (with video-based systems) Predict future locations ASCE2007
67
Content Mobile Network Communication Tracking Registration Rendering
User Interaction ASCE2007
68
Rendering-GIS The proposed method of terrain constraint for outdoor augmented reality is loading terrain data from Geographic Information System (GIS) based on the GPS position. Terrain from GIS ASCE2007
69
Content Mobile Network Communication Tracking Registration Rendering
User Interaction ASCE2007
70
User Interaction Portable computer Head-Mounted display Joystick
ASCE2007
71
Reference Azuma, R., 1997, A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6(4): pp Azuma, R., 1999, The challenge of making augmented reality work outdoors. In ohta, Y. and Tamura, H., editors, Mixed Reality, Merging Real and Virtual Worlds, pp Ohmsha/Springer, Tokyo/New York. Hammad, A., Wang, H., Zhang, C. and Al-Hussein, M. (2006). Visualizing Crane Selection and Operation in Virtual Environment. In proceedings of the 6th Visualizing Crane Selection and Operation in Virtual Environment in Orlando, Florida, August, 2006 Kalagnanam, A and Balu, G (2002). Merlin brings nonblocking I/O to the Java platform. [Online]. Available: [2006, November 15] Lea, D (2002).Scalable IO in Java. [Online]. Available: [2006, November 15] Lewandowski, S.M. (1998) Frameworks for component-based client/server computing, ACM Computing Surveys, Volume 30 , Issue 1,pp3-27 Schmidt, D.C. (1995). Reactor -- An Object Behavioral Pattern for Event Demultiplexing and Event Handler Dispatching. In proceedings of the First Pattern Languages of Programs conference in Monticello, Illinois, August, 1994 Schmidt, D.C. (1996). Asynchronous Completion Token -- An Object Behavioral Pattern for Efficient Asynchronous Event Handling. In proceeding of the 3rd annual Pattern Languages of Programming conference in Allerton Park, Illinois, September 4-6, 1996 Shirazi, J and Pepperdine, K (2004).Eye on performance: MegaJogos scales up with NIO. [Online]. Available: [2006, October 30] You, S., Neumann, U, and Azuma, R. (1999) Orientation Tracking for Outdoor Augmented Reality Registration. IEEE Computer Graphics and Applications, 19(6):36--42, Nov/Dec 1999. Zaki, A. R., and Mailhot, G. (2003). Deck Reconstruction of Jacques Cartier Bridge Using Precast Prestressed High Performance Concrete Panels, PCI Journal. pp20-33. ASCE2007
72
ASCE2007
73
ceramic gyro A ceramic gyro is a sensor that measures rotational (angular) velocity by oscillating a piezoelectric ceramic resonator with alternating voltage. The NECTokin Ceramic Gyro uses an extremely simple construction: piezoelectric ceramic rods, to which electrodes are attached during the printing process. This gives the sensor consistent quality and a wide range of applications, including shake detectors for hand-held video cameras. ASCE2007
74
Other ASCE2007
75
Mobile Network Communication (cont’d)
h1 l r w θ h2 α ASCE2007
76
Optical see-through HMD
Graphic Images Head Tracker Head Locations Scene Generator Monitors Real World Optical Combiners ASCE2007
77
Euler Angle Euler angles are used extensively in the classical mechanics of rigid bodies, and in the quantum mechanics of angular momentum. When studying rigid bodies, one calls the xyz system space coordinates, and the XYZ system body coordinates. The space coordinates are treated as unmoving, while the body coordinates are considered embedded in the moving body. Calculations involving kinetic energy are usually easiest in body coordinates, because then the moment of inertia tensor does not change in time. If one also diagonalizes the rigid body's moment of inertia tensor (with nine components, six of which are independent), then one has a set of coordinates (called the principal axes) in which the moment of inertia tensor has only three components. ASCE2007
78
Euler Angle (contd.) The angular velocity, in body coordinates, of a rigid body takes a simple form using Euler angles: where IJK are unit vectors for XYZ. ASCE2007
79
Euler Angle (contd.) In quantum mechanics, explicit descriptions of the representations of SO(3) are very important for calculations, and almost all the work has been done using Euler angles. In the early history of quantum mechanics, when physicists and chemists had a sharply negative reaction towards abstract group theoretic methods (called the Gruppenpest), reliance on Euler angles was also essential for basic theoretical work. ASCE2007
80
Euler Angle (contd.) Haar measure for Euler angles has the simple form sin(β)dαdβdγ, usually normalized by a factor of 1/8π2. For example, to generate uniformly randomized orientations, let α and γ be uniform from 0 to 2π, let z be uniform from −1 to 1, and let β = arccos(z). Unit quaternions, also known as Euler-Rodrigues parameters, provide another mechanism for representing 3D rotations. This is equivalent to the special unitary group description. Quaternions are generally quicker for most calculations, conceptually simpler to interpolate, and are not subject to gimbal lock. Much high speed 3D graphics programming (gaming, for example) uses quaternions because of this. ASCE2007
81
Converting Euler Angles to a Matrix
Euler angles define a sequence of rotations. We have compute the matrix for each individual rotation and then concatenate these matrices into one matrix that defines the total angular displacement. Let us assume we use the heading, pitch, and bank system of Euler Angles. ASCE2007
82
Converting Euler Angles to a Matrix
M=HPB Question: How we use the orientation based on Euler’s angle? ASCE2007
83
3D Orientation Sensor ASCE2007
84
Yaw-Pitch-Roll ASCE2007
85
Design Challenge Mixed Reality needs accurate tracking of augmented space in real time. The outdoor environment brings difficulties and challenges for the tracking because the outdoor is unprepared and dynamic. A major difference between indoor and outdoor MR system is user’s capacity. The size, weight, power and convince of distributed mixed reality (DMR) system will affect the efficient and user interaction of the system. Registration of virtual objects in DMR system have to use constraints for reality. Wireless networking is critical for dynamic and flexible DMR because real-time information cannot be stored or calculated on the portable computer before application run-time. ASCE2007
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.