Presentation on theme: "9th E3 Concertation Meeting, Brussels, September 10th, 2002"— Presentation transcript:
19th E3 Concertation Meeting, Brussels, September 10th, 2002 ARIS Augmented Reality Image Synthesis through illumination reconstruction and its integration in interactive and shared mobile AR-systems for E-(motion)-commerce applicationsMr. Ioannis Koufakis, INTRACOM S.A.9th E3 Concertation Meeting,Brussels, September 10th, 2002
2Introduction Goals of ARIS: Provide new technologies for seamless integration of virtual objects in an augmented environment.Develop new visualization and interaction paradigms for novel collaborative AR-applications.Two application scenarios will be developed:Interactive desktop system.Mobile AR-unit,that could be used on-site.
4Introduction Collaboration Realism enhancements Between participants on-siteWith a remote consultant (interior designer) in the storeRealism enhancementsPhysically correct placement of furniture (scaling, occlusion/collision detection etc.)Modeling of lighting effects and correct visualization of virtual objects within the real environment (shading, illumination etc.)
5Consortium House Market S.A.(IKEA) A7 Athens Technology Center P6 University of BristolP5University of ManchesterP4Inria-LoriaP3Intracom S.AP2Fraunhofer InstituteC1C - CoordinatorP - Principal ContractorA - Assistant Contractor
6Contract Information Contract No.: IST-2000-28707 Duration: 36 months Commencement: Oct. 2001Current Stage: 12th monthBudget: ~4.9M Euro ( ~2,9M funded by EU)Web: aris-ist.intranet.gr
7Geometry and Illumination Geometry ReconstructionScene and camera reconstruction, both from still images and videosequences.Real-time camera tracking.Illumination ReconstructionReconstructing illumination and material properties from a sequence of images and 3D scene models.Camera calibration:computation of camera response function.GeometryINRIA:calibration and reconstruction from a single image using a reference plane defined by the user.Real-time camera tracking using multiplanar structures; ~15frames/sec.IGD:Automatic reconstruction using multiple images or videostream.Marker-based, using disparity maps calculated from image correspondences.Illumination:Camera calibration:IGD: computation of camera response function using multiple images with different exposure times, based on HDR imagesIllumination Reconstruction:Mention here what a light probe is: an aluminum perfect sphereIGD: estimating illumination using a light probe; user has to identify light sources on HDR images of light probe.UoM: Similar to above. Result is a 3D mesh with radiance info projected onto it.
8Combined Lighting Simulation Algorithms for lighting simulation and image synthesis techniques for integration of virtual and real environments.IGDrendering based on pixel shaft hierarchy; first picture requires time but subsequent ones are rendered within seconds.UoMRendering based on a GeForce 3 card; 2-10 frames/sec. Shading, shadowing and self-shadowing can be handled. Suitable for all types of lighting (except direct sunlight).ActiveX renderer, demonstrating preliminary results; suitable for web scenario.
9Perceptual Evaluation Development of a psychophysical framework measuring the perceptual realism in mixed/augmented reality scenesTone mapping operatorsComparison of different tone mapping operators for both low and high dynamic range images.Development of perceptual tone mapping operators.Development of a psychophysical framework measuring the perceptual realism mixed/augmented reality scenesComparing real scenes, photographs and rendered images.
10Application Scenarios Specification of realistic application scenarios in an interior design application environment.User Requirements definition spanned two phases:Initial phase, in order to guide research activitiesQuestionnaires on ARIS web site were filled by visitors of the siteUse case diagrams defined in UML notationFinal phase (due now), in order to capture the work situation in which the system will be deployedSpecial attention to HCI partA users’ workshop was organized internally in INTRACOM to evaluate the usability and acceptance of various tools.
12Application Scenarios Survey of related systems and applications (deliverable D5.2)Results of the previous two were used to define the application scenarios and provide an initial specification of the system.
13WP6: Application System Leaded by INTRACOM.Initial prototype, simulating AR on a computer screen will be operational in month 25. Final prototype at end of project will be a completed AR system.Current stage: architecture specification and system design (due to Oct.2002).
14Application System3D e-commerce server: repository of furniture models along with e-commerce functionality.Database was designed and implemented in Oracle 9i (ICOM,ATC).First approach of AR mobile unit was designed (ZGDV)Marker based approach, for placement of furniture items within a sceneActive X component for visualization (AR browser), displaying VRML and live video.Wireless network, based on IEEE b.3D e-commerce server: repository of furniture models along with e-commerce functionality.Database was designed and implemented in Oracle 9i (ICOM,ATC).First approach of AR mobile unit was designed (ZGDV)Marker based approach, for placement of furniture items within a sceneActive X component for visualization (AR browser), displaying VRML and live video.Communication between client/server via a Context Manager, based on servlets and JSPs.
16Demos Image synthesis using illumination reconstruction AR browser Interaction with JoystickInteraction with Voice Recognition
17Problems so farUser Interaction required for calibration and 3D reconstruction is too much (esp. for one of the techniques)Vision based techniques should be investigated to automate the process.Illumination reconstruction with the light probe is a strict procedure and requires knowledge of digital cameras and photographyAutomating detection of light sources should be pursued.
18Problems so farExtension of those techniques to mobile AR-unit is not easy, especially due to limitations of mobile devices(small display, lack of keyboard, not enough computing power,etc.)Introduction of local server on-site, where an expert could operate the geometry and illumination modules.New interaction paradigms e.g voice,gesture recognitionVRML not capable of handling illumination data and perform rendering of shadows.Extension of those techniques to mobile AR-unit is not easy, especially due to limitations of mobile devices(small display, lack of keyboard,etc.)Introduction of local server on-site, where an expert could operate the geometry and illumination modules.Investigation of new interaction paradigms e.g voice and gesture recognitionVRML not capable of handling illumination data and perform rendering of shadowsOur own rendering technique should be developed.
19Future Work Finalization of the system architecture and design. Investigate on how to minimize /improve user interaction.Collaboration paradigms:Platforms to be used: NetMeeting, VoIP, Whiteboards, chat, Shared 3D Worlds etc.Investigate how wearable machines could be exploited.X-3D standardization effort.
20Cross-project collaboration Exchange of experiences, especially for the mobile AR unit, with other projectsLifePlusArcheoguideArvika (funded by German Ministry of Education and Research)Interaction paradigms within 3D worlds for mixed/augmented environmentsClustering