1. 9th E3 Concertation Meeting, Brussels, September 10th, 2002
ARIS Augmented Reality Image Synthesis through illumination reconstruction and its integration in interactive and shared mobile AR-systems for E-(motion)-commerce applications
Mr. Ioannis Koufakis, INTRACOM S.A.
9th E3 Concertation Meeting,
Brussels, September 10th, 2002

2. Introduction Goals of ARIS:
Provide new technologies for seamless integration of virtual objects in an augmented environment.
Develop new visualization and interaction paradigms for novel collaborative AR-applications.
Two application scenarios will be developed:
Interactive desktop system.
Mobile AR-unit,that could be used on-site.

3. Introduction

4. Introduction Collaboration Realism enhancements
Between participants on-site
With a remote consultant (interior designer) in the store
Realism enhancements
Physically correct placement of furniture (scaling, occlusion/collision detection etc.)
Modeling of lighting effects and correct visualization of virtual objects within the real environment (shading, illumination etc.)

5. Consortium House Market S.A.(IKEA) A7 Athens Technology Center P6
University of Bristol P5
University of Manchester P4
Inria-Loria P3
Intracom S.A P2
Fraunhofer Institute C1
C - Coordinator
P - Principal Contractor
A - Assistant Contractor

6. Contract Information Contract No.: IST-2000-28707 Duration: 36 months
Commencement: Oct. 2001
Current Stage: 12th month
Budget: ~4.9M Euro ( ~2,9M funded by EU)
Web: aris-ist.intranet.gr

7. Geometry and Illumination
Geometry Reconstruction
Scene and camera reconstruction, both from still images and videosequences.
Real-time camera tracking.
Illumination Reconstruction
Reconstructing illumination and material properties from a sequence of images and 3D scene models.
Camera calibration:computation of camera response function.
Geometry
INRIA:
calibration and reconstruction from a single image using a reference plane defined by the user.
Real-time camera tracking using multiplanar structures; ~15frames/sec.
IGD:
Automatic reconstruction using multiple images or videostream.
Marker-based, using disparity maps calculated from image correspondences.
Illumination:
Camera calibration:
IGD: computation of camera response function using multiple images with different exposure times, based on HDR images
Illumination Reconstruction:
Mention here what a light probe is: an aluminum perfect sphere
IGD: estimating illumination using a light probe; user has to identify light sources on HDR images of light probe.
UoM: Similar to above. Result is a 3D mesh with radiance info projected onto it.

8. Combined Lighting Simulation
Algorithms for lighting simulation and image synthesis techniques for integration of virtual and real environments.
IGD
rendering based on pixel shaft hierarchy; first picture requires time but subsequent ones are rendered within seconds.
UoM
Rendering based on a GeForce 3 card; 2-10 frames/sec. Shading, shadowing and self-shadowing can be handled. Suitable for all types of lighting (except direct sunlight).
ActiveX renderer, demonstrating preliminary results; suitable for web scenario.

9. Perceptual Evaluation
Development of a psychophysical framework measuring the perceptual realism in mixed/augmented reality scenes
Tone mapping operators
Comparison of different tone mapping operators for both low and high dynamic range images.
Development of perceptual tone mapping operators.
Development of a psychophysical framework measuring the perceptual realism mixed/augmented reality scenes
Comparing real scenes, photographs and rendered images.

10. Application Scenarios
Specification of realistic application scenarios in an interior design application environment.
User Requirements definition spanned two phases:
Initial phase, in order to guide research activities
Questionnaires on ARIS web site were filled by visitors of the site
Use case diagrams defined in UML notation
Final phase (due now), in order to capture the work situation in which the system will be deployed
Special attention to HCI part
A users’ workshop was organized internally in INTRACOM to evaluate the usability and acceptance of various tools.

11. Users Workshop

12. Application Scenarios
Survey of related systems and applications (deliverable D5.2)
Results of the previous two were used to define the application scenarios and provide an initial specification of the system.

13. WP6: Application System
Leaded by INTRACOM.
Initial prototype, simulating AR on a computer screen will be operational in month 25. Final prototype at end of project will be a completed AR system.
Current stage: architecture specification and system design (due to Oct.2002).

14. Application System
3D e-commerce server: repository of furniture models along with e-commerce functionality.
Database was designed and implemented in Oracle 9i (ICOM,ATC).
First approach of AR mobile unit was designed (ZGDV)
Marker based approach, for placement of furniture items within a scene
Active X component for visualization (AR browser), displaying VRML and live video.
Wireless network, based on IEEE b.
3D e-commerce server: repository of furniture models along with e-commerce functionality.
Database was designed and implemented in Oracle 9i (ICOM,ATC).
First approach of AR mobile unit was designed (ZGDV)
Marker based approach, for placement of furniture items within a scene
Active X component for visualization (AR browser), displaying VRML and live video.
Communication between client/server via a Context Manager, based on servlets and JSPs.

15. Architecture Overview
Interactive (web-based) application.
Participants on-site
3D e-Commerce Server
Authorization module
Presentation Component
Communication component
Collaboration Gateway
Repository
DIM
3D models of furniture, textures etc.
User profiles
Wearable Computer (coordinator)
Tracking
Visualization
Communication Collaboration
Wearable Computer (participant)
Communication
Broadband modem
Access Point
Broadband Network
Local Server
3D Geometry Reconstruction module
Illumination Reconstruction module
Collaboration Module

16. Demos Image synthesis using illumination reconstruction AR browser
Interaction with Joystick
Interaction with Voice Recognition

17. Problems so far
User Interaction required for calibration and 3D reconstruction is too much (esp. for one of the techniques)
Vision based techniques should be investigated to automate the process.
Illumination reconstruction with the light probe is a strict procedure and requires knowledge of digital cameras and photography
Automating detection of light sources should be pursued.

18. Problems so far
Extension of those techniques to mobile AR-unit is not easy, especially due to limitations of mobile devices(small display, lack of keyboard, not enough computing power,etc.)
Introduction of local server on-site, where an expert could operate the geometry and illumination modules.
New interaction paradigms e.g voice,gesture recognition
VRML not capable of handling illumination data and perform rendering of shadows.
Extension of those techniques to mobile AR-unit is not easy, especially due to limitations of mobile devices(small display, lack of keyboard,etc.)
Introduction of local server on-site, where an expert could operate the geometry and illumination modules.
Investigation of new interaction paradigms e.g voice and gesture recognition
VRML not capable of handling illumination data and perform rendering of shadows
Our own rendering technique should be developed.

19. Future Work Finalization of the system architecture and design.
Investigate on how to minimize /improve user interaction.
Collaboration paradigms:
Platforms to be used: NetMeeting, VoIP, Whiteboards, chat, Shared 3D Worlds etc.
Investigate how wearable machines could be exploited.
X-3D standardization effort.

20. Cross-project collaboration
Exchange of experiences, especially for the mobile AR unit, with other projects
LifePlus
Archeoguide
Arvika (funded by German Ministry of Education and Research)
Interaction paradigms within 3D worlds for mixed/augmented environments
Clustering