9th E3 Concertation Meeting, Brussels, September 10th, 2002 ARIS Augmented Reality Image Synthesis through illumination reconstruction and its integration in interactive and shared mobile AR-systems for E-(motion)-commerce applications Mr. Ioannis Koufakis, INTRACOM S.A. Ikouf@intracom.gr 9th E3 Concertation Meeting, Brussels, September 10th, 2002

Introduction Goals of ARIS: Provide new technologies for seamless integration of virtual objects in an augmented environment. Develop new visualization and interaction paradigms for novel collaborative AR-applications. Two application scenarios will be developed: Interactive desktop system. Mobile AR-unit,that could be used on-site.

Introduction

Introduction Collaboration Realism enhancements Between participants on-site With a remote consultant (interior designer) in the store Realism enhancements Physically correct placement of furniture (scaling, occlusion/collision detection etc.) Modeling of lighting effects and correct visualization of virtual objects within the real environment (shading, illumination etc.)

Consortium House Market S.A.(IKEA) A7 Athens Technology Center P6 University of Bristol P5 University of Manchester P4 Inria-Loria P3 Intracom S.A P2 Fraunhofer Institute C1 C - Coordinator P - Principal Contractor A - Assistant Contractor

Contract Information Contract No.: IST-2000-28707 Duration: 36 months Commencement: Oct. 2001 Current Stage: 12th month Budget: ~4.9M Euro ( ~2,9M funded by EU) Web: aris-ist.intranet.gr

Geometry and Illumination Geometry Reconstruction Scene and camera reconstruction, both from still images and videosequences. Real-time camera tracking. Illumination Reconstruction Reconstructing illumination and material properties from a sequence of images and 3D scene models. Camera calibration:computation of camera response function. Geometry INRIA: calibration and reconstruction from a single image using a reference plane defined by the user. Real-time camera tracking using multiplanar structures; ~15frames/sec. IGD: Automatic reconstruction using multiple images or videostream. Marker-based, using disparity maps calculated from image correspondences. Illumination: Camera calibration: IGD: computation of camera response function using multiple images with different exposure times, based on HDR images Illumination Reconstruction: Mention here what a light probe is: an aluminum perfect sphere IGD: estimating illumination using a light probe; user has to identify light sources on HDR images of light probe. UoM: Similar to above. Result is a 3D mesh with radiance info projected onto it.

Combined Lighting Simulation Algorithms for lighting simulation and image synthesis techniques for integration of virtual and real environments. IGD rendering based on pixel shaft hierarchy; first picture requires time but subsequent ones are rendered within seconds. UoM Rendering based on a GeForce 3 card; 2-10 frames/sec. Shading, shadowing and self-shadowing can be handled. Suitable for all types of lighting (except direct sunlight). ActiveX renderer, demonstrating preliminary results; suitable for web scenario.

Perceptual Evaluation Development of a psychophysical framework measuring the perceptual realism in mixed/augmented reality scenes Tone mapping operators Comparison of different tone mapping operators for both low and high dynamic range images. Development of perceptual tone mapping operators. Development of a psychophysical framework measuring the perceptual realism mixed/augmented reality scenes Comparing real scenes, photographs and rendered images.

Application Scenarios Specification of realistic application scenarios in an interior design application environment. User Requirements definition spanned two phases: Initial phase, in order to guide research activities Questionnaires on ARIS web site were filled by visitors of the site Use case diagrams defined in UML notation Final phase (due now), in order to capture the work situation in which the system will be deployed Special attention to HCI part A users’ workshop was organized internally in INTRACOM to evaluate the usability and acceptance of various tools.

Users Workshop

Application Scenarios Survey of related systems and applications (deliverable D5.2) Results of the previous two were used to define the application scenarios and provide an initial specification of the system.

WP6: Application System Leaded by INTRACOM. Initial prototype, simulating AR on a computer screen will be operational in month 25. Final prototype at end of project will be a completed AR system. Current stage: architecture specification and system design (due to Oct.2002).

Application System 3D e-commerce server: repository of furniture models along with e-commerce functionality. Database was designed and implemented in Oracle 9i (ICOM,ATC). First approach of AR mobile unit was designed (ZGDV) Marker based approach, for placement of furniture items within a scene Active X component for visualization (AR browser), displaying VRML and live video. Wireless network, based on IEEE 802.11b. 3D e-commerce server: repository of furniture models along with e-commerce functionality. Database was designed and implemented in Oracle 9i (ICOM,ATC). First approach of AR mobile unit was designed (ZGDV) Marker based approach, for placement of furniture items within a scene Active X component for visualization (AR browser), displaying VRML and live video. Communication between client/server via a Context Manager, based on servlets and JSPs.

Architecture Overview Interactive (web-based) application. Participants on-site 3D e-Commerce Server Authorization module Presentation Component Communication component Collaboration Gateway Repository DIM 3D models of furniture, textures etc. User profiles Wearable Computer (coordinator) Tracking Visualization Communication Collaboration Wearable Computer (participant) Communication Broadband modem Access Point Broadband Network Local Server 3D Geometry Reconstruction module Illumination Reconstruction module Collaboration Module

Demos Image synthesis using illumination reconstruction AR browser Interaction with Joystick Interaction with Voice Recognition

Problems so far User Interaction required for calibration and 3D reconstruction is too much (esp. for one of the techniques) Vision based techniques should be investigated to automate the process. Illumination reconstruction with the light probe is a strict procedure and requires knowledge of digital cameras and photography Automating detection of light sources should be pursued.

Problems so far Extension of those techniques to mobile AR-unit is not easy, especially due to limitations of mobile devices(small display, lack of keyboard, not enough computing power,etc.) Introduction of local server on-site, where an expert could operate the geometry and illumination modules. New interaction paradigms e.g voice,gesture recognition VRML not capable of handling illumination data and perform rendering of shadows. Extension of those techniques to mobile AR-unit is not easy, especially due to limitations of mobile devices(small display, lack of keyboard,etc.) Introduction of local server on-site, where an expert could operate the geometry and illumination modules. Investigation of new interaction paradigms e.g voice and gesture recognition VRML not capable of handling illumination data and perform rendering of shadows Our own rendering technique should be developed.

Future Work Finalization of the system architecture and design. Investigate on how to minimize /improve user interaction. Collaboration paradigms: Platforms to be used: NetMeeting, VoIP, Whiteboards, chat, Shared 3D Worlds etc. Investigate how wearable machines could be exploited. X-3D standardization effort.

Cross-project collaboration Exchange of experiences, especially for the mobile AR unit, with other projects LifePlus Archeoguide Arvika (funded by German Ministry of Education and Research) Interaction paradigms within 3D worlds for mixed/augmented environments Clustering