InerVis Mobile Robotics Laboratory Institute of Systems and Robotics ISR – Coimbra Contact Person: Jorge Lobo Human inertial sensor: The vestibular system Within the inner ear, the vestibular system measures tilt and angular acceleration Integration of Vision and Inertial Sensing PhD 2003 – 2007 selected publications:  Jorge Lobo and Jorge Dias, "Relative Pose Calibration Between Visual and Inertial Sensors", International Journal of Robotics Research, Special InerVis Issue, in press.  Peter Corke, Jorge Lobo and Jorge Dias, "An introduction to inertial and visual sensing", International Journal of Robotics Research, Special InerVis Issue, in press.  Luiz G. B. Mirisola, Jorge Lobo, and Jorge Dias, "Stereo Vision 3D Map Registration for Airships using Vision-Inertial Sensing", In The 12th IASTED Int. Conf. on Robotics and Applications, Honolulu, USA, August  Jorge Lobo, João Filipe Ferreira and Jorge Dias, "Bioinspired Visuo-vestibular Artificial Perception System for Independent Motion Segmentation", In Second Inernational Cognitive Vision Workshop, ECCV 9th European Conference on Computer Vision, Graz, Austria, May  Jorge Lobo and Jorge Dias, "Relative Pose Calibration Between Visual and Inertial Sensors", Proceedings of the ICRA 2005 Workshop on Integration of Vision and Inertial Sensors - 2nd InerVis, Barcelona, Spain, April 18, Jorge Lobo, Jorge Dias, “Inertial Sensed Ego-motion for 3D Vision”, in Journal of Robotic Systems Volume 21, Issue 1, pp. 3-12, January Jorge Lobo and Jorge Dias, “Vision and Inertial Sensor Cooperation, Using Gravity as a Vertical Reference”, in IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI, 25(12), pp , December Jorge Lobo, Carlos Queiroz, Jorge Dias, “World Feature Detection and Mapping using Stereovision and Inertial Sensors”, in Robotics and Autonomous Systems, Elsevier Science, vol. 44, Issue 1, pp. 69–81, July selected publications:  Jorge Lobo and Jorge Dias, "Relative Pose Calibration Between Visual and Inertial Sensors", International Journal of Robotics Research, Special InerVis Issue, in press.  Peter Corke, Jorge Lobo and Jorge Dias, "An introduction to inertial and visual sensing", International Journal of Robotics Research, Special InerVis Issue, in press.  Luiz G. B. Mirisola, Jorge Lobo, and Jorge Dias, "Stereo Vision 3D Map Registration for Airships using Vision-Inertial Sensing", In The 12th IASTED Int. Conf. on Robotics and Applications, Honolulu, USA, August  Jorge Lobo, João Filipe Ferreira and Jorge Dias, "Bioinspired Visuo-vestibular Artificial Perception System for Independent Motion Segmentation", In Second Inernational Cognitive Vision Workshop, ECCV 9th European Conference on Computer Vision, Graz, Austria, May  Jorge Lobo and Jorge Dias, "Relative Pose Calibration Between Visual and Inertial Sensors", Proceedings of the ICRA 2005 Workshop on Integration of Vision and Inertial Sensors - 2nd InerVis, Barcelona, Spain, April 18, Jorge Lobo, Jorge Dias, “Inertial Sensed Ego-motion for 3D Vision”, in Journal of Robotic Systems Volume 21, Issue 1, pp. 3-12, January Jorge Lobo and Jorge Dias, “Vision and Inertial Sensor Cooperation, Using Gravity as a Vertical Reference”, in IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI, 25(12), pp , December Jorge Lobo, Carlos Queiroz, Jorge Dias, “World Feature Detection and Mapping using Stereovision and Inertial Sensors”, in Robotics and Autonomous Systems, Elsevier Science, vol. 44, Issue 1, pp. 69–81, July Integrating Inertial Sensors with Artificial Vision key contributions:  a common framework for inertial-vision sensor integration;  calibration methods for integrated inertial and vision systems;  vertical feature segmentation and 3D mapping;  ground plane segmentation;  3D depth map registration;  independent motion segmentation. MEMs Inertial sensors Analog devices ADXL202 dual axis ±2g accelerometer. Analog devices ADXRS150 angular rate sensor (gyroscope). (October 2002) Xsens MTi IMU Data From Inertial Sensors Independent Motion Segmentation Registering Stereo Depth Maps Inertial sensors Camera Images INS calcs Image processing Match features across images Dynamic 3D(t) reconstructured world model real world time sensor data fusion N static poses observing vertical target –Full camera calibration –IMU↔CAM Rotation estimated Sensor Calibration Both sensors used to measure the vertical direction N observations at different camera positions Unknown rotation determined 2N static poses with N rotations about IMU –IMU↔CAM Translation estimated Swinging pendulum sequence Seg. motion voxelsRaw voxelsbackground voxels -= 1)Image optical flow (LK) 2)Estimate optical flow from 3D data and reconstructed camera motion assuming static scene 3)Subtract and threshold to segment independent motion Optical flow consistency segmentation Images Camera motion Depth Map Background subtraction 1)Quantise registered point cloud to voxel space and accumulate occupancy votes for all frames 2)Threshold to obtain background voxels (apply thinning and growing transformation for noise filtering) 3)Intersect current frame voxels with complement of background voxels to have voxels from moving objects