EU funded FP7: Oct 11 – Sep 14 Co-evolution of Future AR Mobile Platforms Paul Chippendale, Bruno Kessler Foundation FBK, Italy

Move away from the Augmented Keyhole

User centric, not device centric HMDs lock displays to the viewer But what about handheld displays?

Device-World registration What is the devices real-world location? Which direction is it pointing?

Device-World registration What is the devices real-world location? GPS, Cell/WiFi tower triangulation (~10m)

Device-World registration Which direction is it pointing? Magnetometer, Gyros, Accelerometers (~5-20 º) Mems production variability Sensors age Soft/Hard iron influences vary across devices, environments and camera pose

Is +/- 10m and +/- 20 º sufficient for nailed-down AR?

But what about hand-held AR? Devices becomes an augmented window

User-Device-World registration What is the devices real-world location? Which direction is it pointing? Where is the user with respect to the screen?

Surely if we wait sensor errors will disappear? Unlikely! O Sensor errors are tolerable for non-AR application, handset manufacturers focus on price, power and form- factor Cant we just model the error in software? Not really! O Platform diversity and swift evolution make error modelling expensive and quickly obsolete Just wait for better AR devices!

So what can we do? The AR comunity should work with handset manufacturers and make recomendations Use computer vision to work with sensors

VENTURI project... o Match AR requirements to platform o Efficiently exploit CPUs & GPUs o Improving sensor-camera fusion by creating a common clock (traditionally only audio/video considered) o Applying smart power management policies o Optimizing AR chain, by exploiting both on-board and cloud processing/storage

Seeing the world o Improve device-world pose by: Matching visual features to 3D models of the world Matching camera feed to visual appearance of the world Fusing camera and sensors for ambiguity reasoning and tracking o Use front facing camera to estimate user-device pose via face tracking

Urban 3D Model matching o Use high resolution building models (e.g. laser scanned) and globally registered to geo-referenced coordinate system o Use 3D marker-less tracking to correlate distinctive features to 3D building models. Subsequent tracking using inertial sensors and visual optical flow

Terrain 3D Model matching o Synthetic model of world rendered from Digital Elevation Models. Salient features from camera feed (depth discontinuities) matched to similar synthetic features.

16 Use approximate location to gather nearby images from the cloud Exploit sensor data to provide a clue for orientation alignment Computer vision algorithms match feature descriptors from the camera feed to similar features in the cloud images Appearance matching

SLAM + Matching O Simultaneous Localization And Mapping - build a map of an unknown environment while at the same time navigating the environment using the map. o Mapped environment has no real-world scale nor absolute geo-coordinates. Exploit prior approaches to complete registration.

Mobile context understanding o User/environment context estimation: o PDR enriched with vision o User activity modelling o Sensing geo-objects o Harvest/create geo-social content

Context sensitive AR delivery o Inject AR data in a natural manner according to: o environment o occlusions o lighting and shadows o user activity o Exploit user and environment context to select best delivery modality (text, graphics, audio, etc.), i.e. scalable/simplify-able audio-visual content

User Interactions o Explore evolving AR delivery and interaction o In-air interfaces: device, hand and face tracking o 3D audio o Pico-projection for multi-user, social-AR o HMDs

Prototypes One consolidated prototype at the end of each year to be evaluated through Use-cases o Gaming - VeDi 1.0 o Blind assistant - VeDi 2.0 o Tourism - VeDi 3.0 Constraints relaxed

VeDi 1.0 Objective: Stimulate software and hardware cross- partner integration and showcase state-of- the-art indoor AR registration Scenario: Multi-player, table-top AR Game resembling a miniature city. Players must accomplish a set of AR missions in the city, that adhere to physical constraints. Software: Sensor-aided marker-less 3D feature tracking. City geometrically reconstructed offline correctly occlusion handling and model registration. Hardware: Demo runs on experimental ST Ericsson prototype mobile platform.

FP7-ICT Networked Media and Search Systems End-to-end Immersive and Interactive Media Technologies creating a pervasive Augmented Reality paradigm, where information is presented in a user rather than a device centric way Co-ordinated by Paul Chippendale, Fondazione Bruno Kessler