1. Concordia University: Hui Wang Amin Hammad Sudhir P. Mudur
Design of a Distributed Mixed Reality System for Construction Simulation
Concordia University: Hui Wang
Amin Hammad
Sudhir P. Mudur
ASCE2007

2. Content Motivation Proposed Approach Implementation & Case Study
Conclusion
ASCE2007

3. Motivation
One crane upset occurs during every 10,000 hours of crane use.
Nearly 80 percent of these upsets can be attributed to predictable human error.
ASCE2007

4. Motivation (cont’d)
We need a system to simulate crane actions and construction progress in outdoor environment.
We also want to visualize the multi-users interactive construction simulation while checking workspace conflicts and engineering constraints within a real construction site.
Simulation software: to train crane operators using virtual reality (VR) (Simlog 2006); and to visualize the results of construction simulation (Kamat and Martinez 2001)
ASCE2007

5. Motivation (cont’d)
However, typical environment modeling and visualization methods is time consuming and difficult to apply for real outdoor construction site.
Mixed Reality (MR) is a relatively new approach of human-computer interaction which provides advanced user immersion and experience.
ASCE2007

6. Motivation (cont’d)
ASCE2007

7. Motivation (cont’d)
The advantages of MR have been discussed in many engineering applications such as design perception (Dunston et al., 2002).
Using a variety of 3D modeling, tracking, user interaction, rendering and display techniques, MR enhances users’ immersion by allowing them to view the MR environment while moving in the real world.
ASCE2007

8. Objectives
Design a Distributed Interaction System of Construction Simulation (DISCS) with accurate interaction of users and virtual objects in outdoor MR environment of construction simulation.
Develop a prototype system that can be used to test the potential of these approaches and methods.
ASCE2007

9. Content Motivation Proposed Approach Implementation & Case Study
Conclusion
ASCE2007

10. Design overview of DMR system
ASCE2007

11. Tracking and Registration
To track a user’s viewpoint and movement, we need two important pieces of equipments.
There are the GPS receiver and a 3-DOF orientation tracker.
The GPS receiver provides the DMR system with real-time position tracking in the global space.
Trimble 5700
RTK GPS
ASCE2007

12. Tracking and Registration (cont’d)
3D Orientation Sensor:
It is a system combining an accelerometer and magnetic sensor with a gyro. It provides yaw-pitch-roll angles as output data.
Wireless InertiaCube3 and Receiver
ASCE2007

13. Dynamic Object Modeling and Engineering Constraints
Dynamically assemble CAD models of the simulation objects (i.e. cranes) based on the requirement of construction tasks.
3D CAD Model of a Crane
ASCE2007

14. Dynamic Object Modeling and Engineering Constraints
Kinematics animation is essential for producing the interaction needed to operate a crane.
The DoF for the movements of the boom and the hook
ASCE2007

15. Dynamic Object Modeling and Engineering Constraints (cont’d)
For better effects, we have to consider the constraints of rendering or animation. For example, the simulated operation of a crane should respect the constraints imposed by the working ranges and load charts of the crane (Hammad et al. 2006).
Example of load chart and working range (Groove Crane 2006)
ASCE2007

16. User Interaction Display
For a mobile person in the outdoor environment, the method of displaying information decides the quality of the user experience and immersion. A head-mounted display (HMD) provides the view of a computer-generated image imposed upon a see-through or video-captured real world view.
User Control
In order to allow the user to operate the virtual crane, the input device should provide all the controllers needed to control the DoFs of the crane.
ASCE2007

17. Mobile Computing and Communication
Mobile computing is critical for DMR in outdoor environment. There are several factors that limit the choices of a mobile computing platform, including the size and weight, graphics and multimedia capabilities, batteries, etc.
Panasonic Toughbook CF-18 Tablet PC
ASCE2007

18. Mobile Computing and Communication (cont’d)
It is essential for developing a multi-user collaboration because most collaborative works need real-time communication.
For the wireless connection in the outdoor environment, ad-hoc technology provides potential solution for point-to-point connection without base stations.
It reduces the difficulty of setting wireless stations and provides suitable wireless communication.
ASCE2007

19. Mobile Computing and Communication (cont’d)
ASCE2007

20. Conceptual Configuration
ASCE2007

21. Content Motivation Proposed Approach Implementation & Case Study
Conclusion
ASCE2007

22. Implementation tools Java & Java 3D Java Database Connectivity
MapObjects Java Edition
JNI
DirectInput
Java Communication API
ASCE2007

23. Equipment specifications
Device
Type
Specifications
HMD
nVisor SX
Resolution: 1280×1024 pixels
Vertical frequency: 60 Hz
Joysticks
Saitek Cybord evo wireless Joystick
Range: 30 feet
Frequency: 2.4GHz
Tablet PC
Panasonic ToughBook CF-18
CPU: 1GHz
RAM: 512MB
GPS Receiver
Trimble 5700 RTK GPS
Frequency: 10Hz
Horizontal accuracy: ±(10 mm + 1 ppm) (×baseline length) RMS
Vertical accuracy: ±(20 mm + 1 ppm) RMS
3D Motion Sensor
InertiaCube3
Degrees of Freedom: 3 (Yaw, Pitch and Roll)
Angular Range: Full 360° (All Axes)
Maximum Angular Rate: 1200° per second
Minimum Angular Rate: 0° per second
RMS Accuracy: 1° in yaw, 0.25° in pitch & roll at 25°C
RMS Angular Resolution: 0.03°
ASCE2007

24. Case Study
Jacques Cartier Bridge
ASCE2007

25. Deck rehabilitation project
ASCE2007

26. Case Study
ASCE2007

27. Content Motivation Proposed Approach Implementation & Case Study
Conclusion
ASCE2007

28. Conclusion
This paper presented design issues in integrating a variety of high-end equipment to build a construction simulation system in outdoor DMR environment.
The paper provided a proposed method of visualization of crane operation and simulation of crane related construction works on the construction site considering engineering constraints.
Wireless techniques for multi-user communication and collaboration for advanced user interaction in real-time outdoor environment were discussed.
ASCE2007

29. Summary of the DMR System
Complete—tracking all six degrees of freedom (position and orientation);
Accurate—with resolution of 1 0 mm in position and 0.03 degree in orientation;
Fast—running at 10 Hz;
Robust—resisting performance degradation from magnetic fields, network communication error.
Wireless—running with real-time communication
ASCE2007

30. Future work
Improving the modeling of the construction site and adding more constraints to achieve more realistic simulation.
Investigating multi-user collaboration methods for advanced construction simulation.
Improving the accuracy of tracking and registration in outdoor environment.
ASCE2007

31. Thank You!
ASCE2007

32. Thank You!
ASCE2007

33. Rendering Pipeline
ASCE2007

34. Motivation
One crane upset occurs during every 10,000 hours of crane use.
Nearly 80 percent of these upsets can be attributed to predictable human error.
We need a system to simulate crane actions and construction progress in outdoor environment.
ASCE2007

35. Motivation (cont’d)
We also want to visualize the multi-users interactive construction simulation while checking workspace conflicts and engineering constraints within a real construction site.
Mixed Reality (MR) is a relatively new approach of human-computer interaction which provides advanced user immersion and experience.
ASCE2007

36. Example of Mixed Reality System
(a) Concept of Mixed Reality
(b) A crane at Guy Street in a outdoor test.
ASCE2007

37. Objectives
Design a Distributed Interaction System of Construction Simulation (DISCS) with accurate interaction of users and virtual objects in outdoor MR environment of construction simulation.
Develop a prototype system that can be used to test the potential of these approaches and methods
ASCE2007

38. Work Progress of DMR
ASCE2007

39. Design of Distribute Mixed Reality
ASCE2007

40. Tracking and Registration
One of the most basic problems currently limiting Augmented Reality application is the registration problem. (Azuma, 1997)
How to correctly and accurately track a user’s viewpoint?
ASCE2007

41. Tracking and Registration (cont’d)
To track a user’s viewpoint and movement, we need two important pieces of equipments.
There are the GPS receiver and a 3-DOF orientation tracker.
The GPS receiver provides the DMR system with real-time position tracking in the global space.
Trimble 5700
RTK GPS
ASCE2007

42. Tracking and Registration (cont’d)
3D Orientation Sensor:
It is a system combining an accelerometer and magnetic sensor with a gyro. It provides yaw-pitch-roll angles as output data.
ASCE2007

43. Rendering
We use CAD models of the simulation objects (i.e. cranes) and video of real world to render combined video and display it in HMD.
ASCE2007
3D CAD Model of a Crane

44. The DoF for the movements of the boom and the hook
Rendering (cont’d)
Kinematics animation is essential for producing the interaction needed to operate a crane in the outdoor environment.
The DoF for the movements of the boom and the hook
ASCE2007

45. Rendering-Constraints
For better effects, we have to consider the constraints of rendering or animation. For example, the simulated operation of a crane should respect the constraints imposed by the working ranges and load charts of the crane (Hammad et al. 2006).
Example of load chart and working range (Groove Crane 2006)
ASCE2007

46. Another important constraint of outdoor augmented reality is the environment modeling. The environment modeling can solve the problems such as occlusion, terrain following, etc.
ASCE2007

47. User Interaction
For better user immersion and experience in outdoor environment, we want to use HMD and joystick to display and operate.
ASCE2007

48. Mobile Network Communication
Many constructions need advanced user collaboration. To present multi-users interaction and collaboration, the mobile network communication is critical for outdoor MR system.
ASCE2007

49. Mobile Network Communication (cont’d)
Deck-replacement project of Jacques Cartier Bridge
ASCE2007

50. Mobile Network Communication (cont’d)
Client - Server
ASCE2007

51. ASCE2007

52. A Framework of Distributed Mixed Reality for Construction Simulation
Distributed Mixed Reality (DMR) is a relatively new approach of human-computer interaction which provides better user immersion and experience.
DMR has some new characteristics such as distributed environment, user collaboration, etc.
ASCE2007

53. System Framework
ASCE2007

54. Overview of System
ASCE2007

55. AR Work Flow
ASCE2007

56. Content Mobile Network Communication Tracking Registration Rendering
User Interaction
ASCE2007

57. Mobile Network Communication
Wireless network is necessary for user communication and collaboration.
Connect methods:
Ad-hoc
Wireless Router
ASCE2007

58. Content Mobile Network Communication Tracking Registration Rendering
User Interaction
ASCE2007

59. Tracking-GPS
The Global Positioning System (GPS), is the only fully-functional satellite navigation system. Twenty-four GPS satellites orbit the Earth, transmitting radio signals which allow GPS receivers to determine their location, speed and direction.
ASCE2007

60. 4 satellites used in calculating the location of receiver
Tracking-GPS
GPS provides 4D position information based on World Geodetic System 84 (WGS84) coordinates system. The information includes X,Y,Z position and time stamp.
24 satellites
4 satellites used in calculating the location of receiver
ASCE2007

61. Tracking 3D Motion Sensor:
It is a system combining an accelerometer and magnetic sensor with a gyro. It provides Z-Y-X Euler's Angle as output data. We can use the output result to calculate the current orientation.
ASCE2007

62. Content Mobile Network Communication Tracking Registration Rendering
User Interaction
ASCE2007

63. Registration
One of the most basic problems currently limiting Augmented Reality application is the registration problem. (Azuma, 1997)
ASCE2007

64. Registration Problem Static errors Optical distortion
Errors in the tracking system
Mechanical misalignments
Incorrect viewing parameters ( e.g., field of view, tracker-to eye position and orientation)
ASCE2007

65. Registration Problem Dynamic errors
Dynamic errors occur because of system delays, or lags. End-to end delays of 100 ms are fairly typical on existing system. Simpler systems can have less delay, but other systems have more.
ASCE2007

66. Registration Problem
We can use the methods to reduce the dynamic errors (Azuma, 1997).
Reduce system lag
Reduce apparent lag
Match temporal streams (with video-based systems)
Predict future locations
ASCE2007

67. Content Mobile Network Communication Tracking Registration Rendering
User Interaction
ASCE2007

68. Rendering-GIS
The proposed method of terrain constraint for outdoor augmented reality is loading terrain data from Geographic Information System (GIS) based on the GPS position.
Terrain from GIS
ASCE2007

69. Content Mobile Network Communication Tracking Registration Rendering
User Interaction
ASCE2007

70. User Interaction Portable computer Head-Mounted display Joystick
ASCE2007

71. Reference
Azuma, R., 1997, A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6(4): pp
Azuma, R., 1999, The challenge of making augmented reality work outdoors. In ohta, Y. and Tamura, H., editors, Mixed Reality, Merging Real and Virtual Worlds, pp Ohmsha/Springer, Tokyo/New York.
Hammad, A., Wang, H., Zhang, C. and Al-Hussein, M. (2006). Visualizing Crane Selection and Operation in Virtual Environment. In proceedings of the 6th Visualizing Crane Selection and Operation in Virtual Environment in Orlando, Florida, August, 2006
Kalagnanam, A and Balu, G (2002). Merlin brings nonblocking I/O to the Java platform. [Online]. Available: [2006, November 15]
Lea, D (2002).Scalable IO in Java. [Online]. Available: [2006, November 15]
Lewandowski, S.M. (1998) Frameworks for component-based client/server computing, ACM Computing Surveys, Volume 30 , Issue 1,pp3-27
Schmidt, D.C. (1995). Reactor -- An Object Behavioral Pattern for Event Demultiplexing and Event Handler Dispatching. In proceedings of the First Pattern Languages of Programs conference in Monticello, Illinois, August, 1994
Schmidt, D.C. (1996). Asynchronous Completion Token -- An Object Behavioral Pattern for Efficient Asynchronous Event Handling. In proceeding of the 3rd annual Pattern Languages of Programming conference in Allerton Park, Illinois, September 4-6, 1996
Shirazi, J and Pepperdine, K (2004).Eye on performance: MegaJogos scales up with NIO. [Online]. Available: [2006, October 30]
You, S., Neumann, U, and Azuma, R. (1999) Orientation Tracking for Outdoor Augmented Reality Registration. IEEE Computer Graphics and Applications, 19(6):36--42, Nov/Dec 1999.
Zaki, A. R., and Mailhot, G. (2003). Deck Reconstruction of Jacques Cartier Bridge Using Precast Prestressed High Performance Concrete Panels, PCI Journal. pp20-33.
ASCE2007

72. ASCE2007

73. ceramic gyro
A ceramic gyro is a sensor that measures rotational (angular) velocity by oscillating a piezoelectric ceramic resonator with alternating voltage. The NECTokin Ceramic Gyro uses an extremely simple construction: piezoelectric ceramic rods, to which electrodes are attached during the printing process. This gives the sensor consistent quality and a wide range of applications, including shake detectors for hand-held video cameras.
ASCE2007

74. Other
ASCE2007

75. Mobile Network Communication (cont’d)h1 l r w θ h2 α
ASCE2007

76. Optical see-through HMD
Graphic
Images
Head
Tracker
Head
Locations
Scene
Generator
Monitors
Real
World
Optical
Combiners
ASCE2007

77. Euler Angle
Euler angles are used extensively in the classical mechanics of rigid bodies, and in the quantum mechanics of angular momentum.
When studying rigid bodies, one calls the xyz system space coordinates, and the XYZ system body coordinates. The space coordinates are treated as unmoving, while the body coordinates are considered embedded in the moving body. Calculations involving kinetic energy are usually easiest in body coordinates, because then the moment of inertia tensor does not change in time. If one also diagonalizes the rigid body's moment of inertia tensor (with nine components, six of which are independent), then one has a set of coordinates (called the principal axes) in which the moment of inertia tensor has only three components.
ASCE2007

78. Euler Angle (contd.)
The angular velocity, in body coordinates, of a rigid body takes a simple form using Euler angles:
where IJK are unit vectors for XYZ.
ASCE2007

79. Euler Angle (contd.)
In quantum mechanics, explicit descriptions of the representations of SO(3) are very important for calculations, and almost all the work has been done using Euler angles. In the early history of quantum mechanics, when physicists and chemists had a sharply negative reaction towards abstract group theoretic methods (called the Gruppenpest), reliance on Euler angles was also essential for basic theoretical work.
ASCE2007

80. Euler Angle (contd.)
Haar measure for Euler angles has the simple form sin(β)dαdβdγ, usually normalized by a factor of 1/8π2. For example, to generate uniformly randomized orientations, let α and γ be uniform from 0 to 2π, let z be uniform from −1 to 1, and let β = arccos(z).
Unit quaternions, also known as Euler-Rodrigues parameters, provide another mechanism for representing 3D rotations. This is equivalent to the special unitary group description. Quaternions are generally quicker for most calculations, conceptually simpler to interpolate, and are not subject to gimbal lock. Much high speed 3D graphics programming (gaming, for example) uses quaternions because of this.
ASCE2007

81. Converting Euler Angles to a Matrix
Euler angles define a sequence of rotations. We have compute the matrix for each individual rotation and then concatenate these matrices into one matrix that defines the total angular displacement. Let us assume we use the heading, pitch, and bank system of Euler Angles.
ASCE2007

82. Converting Euler Angles to a Matrix
M=HPB Question: How we use the orientation based on Euler’s angle?
ASCE2007

83. 3D Orientation Sensor
ASCE2007

84. Yaw-Pitch-Roll
ASCE2007

85. Design Challenge
Mixed Reality needs accurate tracking of augmented space in real time. The outdoor environment brings difficulties and challenges for the tracking because the outdoor is unprepared and dynamic.
A major difference between indoor and outdoor MR system is user’s capacity. The size, weight, power and convince of distributed mixed reality (DMR) system will affect the efficient and user interaction of the system.
Registration of virtual objects in DMR system have to use constraints for reality.
Wireless networking is critical for dynamic and flexible DMR because real-time information cannot be stored or calculated on the portable computer before application run-time.
ASCE2007