1. JASS `04 Benjamin Fingerle, Christian Wachinger1 2nd Joined Advanced Student School Ubiquitous Tracking based AR-Setups Benjamin Fingerle Christian Wachinger

2. JASS `04 Benjamin Fingerle, Christian Wachinger2 Procedure We will present a scenario in which a user - Gerhard - gets into the benefits of an AR-enabled Environment First the scenario will be presented from the user’s perspective In a 2nd stage we’ll look behind the surface and see how the AR-environment can be modeled using the DWARF Service concept Finally we’ll go even deeper into detail and present how spatial relationships of objects can be represented using the Ubiquitous Tracking Framework

3. JASS `04 Benjamin Fingerle, Christian Wachinger3 The User’s Equipment Gerhard wears an Optical See Through Head Mounted Display (OST-HMD) Up on this HMD two 6DOF markers - one optical, one magnetic - are rigidly mounted Additionally a stereo-vision camera is installed on top of his head Gerhard wears special gloves with attached optical 6DOF markers

4. JASS `04 Benjamin Fingerle, Christian Wachinger4 A Day in Gerhard’s Life … Gerhard strolls down the TUM hallway greeting colleagues of his through closed doors Eventually he reaches his own office and steps in Sitting down in front of the desk he reads a virtual message of his Russian friend Vladimir from St. Petersburg asking for a chess game He starts the RemoteChess application whereupon a virtual chess board appears aligned on the desk and the virtual counterpart of Vladimir takes a seat across from him

5. JASS `04 Benjamin Fingerle, Christian Wachinger5 … A Day in Gerhard’s Life Suddenly an emergency request for instant help by his befriended Spanish colleague José shows up José asks Gerhard - a renowned surgeon - for advises regarding a difficult surgery José is currently conducting Just where a minute ago the chess board was visible a 3D - Model showing the patient crops up Gerhard studies the patient while having a look from different sides and getting computer tomography images registered on the patient on demand In vivid discussion with José a life is saved

6. JASS `04 Benjamin Fingerle, Christian Wachinger6 The Scenario Can Be Modeled With Services Services –Have needs Refined by predicates –Offer abilities Refined by attributes Connectors –Offer interfaces for data exchange Service Managers –One for each network node –Detect mutually satisfying services –Provide services with connectors <attribute name=“location” value=“$(marker.location)”> <attribute name=“identity” value=“$(marker.identity)”> Context = {location, identity, activity, time} can be modeled using Predicates and Attributes

7. JASS `04 Benjamin Fingerle, Christian Wachinger7 Services are Specified Using XML <need name=“... <attribute name=“location” value=“$(landmark.location)”> <attribute name=“identity” value=“Gerhard”>...

8. JASS `04 Benjamin Fingerle, Christian Wachinger8 Hallway Services Service NameNeedsAbilities GerhardConfigData--landmarkDescription HallwayConfigData--landmarkDescription HMDCamera--videostream HMDOpticalTrackervideostreammarkerPose, HMDPose HMDVideoShowvideostream-- WhatsBehindposevideostream ContextEstimatorlandmarkcontext

9. JASS `04 Benjamin Fingerle, Christian Wachinger9 Office Services Service NameNeedsAbilities GerhardConfigData--landmarkDescription HMDCamera--videostream HMDOpticalTrackervideostreammarkerPose, HMDPose HMDVideoShowvideostream-- ContextEstimatorlandmarkcontext RoomConfigData--landmarkDescription RoomCamera--videostream roomMagneticTrackerlandmarkmarkerPose roomOpticalTrackervideostreammarkerPose Table3D-content-- VirtualChesschessPartner3D-content VirtualSurgeryhandPose3D-content VirtualCommunicationcommunicationPartner3D-content

10. JASS `04 Benjamin Fingerle, Christian Wachinger10 Mutually Satisfying Services are Found by ServiceManager For each network node (e.g. Room, Hall, …) one ServiceManager exists This ServiceManager observes needs and abilities of those services belonging to his particular network node When a match is found the ServiceManager provides all involved parties with connector-objects These connectors offer push-, pull-, or shared memory access to abilities via various interfaces

11. JASS `04 Benjamin Fingerle, Christian Wachinger11 Matching of Mutually Satisfying Services

12. JASS `04 Benjamin Fingerle, Christian Wachinger12 Ubiquitous Tracking - The Formal Layer The crucial problem of Augmented Reality applications is the correct tracking of objects It is common to integrate the tracking procedure into the application Different tracking technologies are combined to get better results To release the AR application of tracking and to enable a seamless integration of new tracking devices a formal layer is introduced The formal framework, called UbiquitousTracking, forms the formal layer

13. JASS `04 Benjamin Fingerle, Christian Wachinger13 Ubiquitous Tracking – The Formal Framework Requests to the framework about the spatial relationship of objects can be send The answer delivers the optimal relationship available Graph-model: nodes == objects edges == spatial relationships

14. JASS `04 Benjamin Fingerle, Christian Wachinger14 Ubiquitous Tracking - The Underlying Graph Model Properties of a spatial relationship –Represents the transformation and translation of the source coordinate system to the target coordinate system –Attributes characterizing the quality of the relationship Three different types of graphs –Real relationship graph –Measured relationship graph –Inferred Relationship

15. JASS `04 Benjamin Fingerle, Christian Wachinger15 Real Relationship Graph Each pair of objects has at every point of time a geometric relationship

16. JASS `04 Benjamin Fingerle, Christian Wachinger16 Measured Relationship Graph Estimates of relationships are just available for certain objects for discrete points of time Additionally Attributes characterizing the quality of the measurement exist Error function describing the quality of a relationship

17. JASS `04 Benjamin Fingerle, Christian Wachinger17 Inferred Relationship Graph Knowledge about spatial relationships not just for discrete points of time is necessary Knowledge has to be inferred about the relationship of objects Error functions help to find optimal inferences

18. JASS `04 Benjamin Fingerle, Christian Wachinger18 A Possible Graph in our Scenario Landmark RoomCam HeadCam Drawing on th flipchart Add. Edges for magnetic tracker Drawing on th flipchart Add. Edges for magnetic tracker

19. JASS `04 Benjamin Fingerle, Christian Wachinger19 What Is a Good Error Function? Aim: Finding optimal paths in the graph! = { Latency, Update frequency Confidence value Pose accuray Time to live }

20. JASS `04 Benjamin Fingerle, Christian Wachinger20 Optimisation of Graph Algorithms Precomputing the Data Flow Graphs –Infrequently changing structure of the spatial relationships –Infrequently changing Attributes –No dependency on the pose measurements Grouping Nodes –Several nodes can be represented by a single supernode –Enabling level of detail hierarchies  Faster graph search What is the coordinate system of the new supernode?

21. JASS `04 Benjamin Fingerle, Christian Wachinger21 Security & Safety Aspects What if the “WhatsBehind” Service can be used in front of every door? What if your boss knows what you have done this weekend? What if other users know about every place where you have been this week? What if Indra the intern uses the AR setup of Gerhard? What if Eve corrupts the data showing you the false distance to the oncoming bus? What if you can’t see the car because the latest sports news occluded it? Privacy Restrictions Authenticity Privacy Restrictions Authenticity

22. JASS `04 Benjamin Fingerle, Christian Wachinger22 Finding Matches Forms Crucial Performance Issue Matching problem of high computational complexity One Service Manager for each Network Node Number of potential matchings grows exponentially with the number of services available inhibiting scalability Different strategies for coping with this issue could include Heuristics, based on context information, realized as graph searches on the spatial relationship graph Interpretation of matching problem as predicate logic formula, applying specialized algorithms known from model checking …

23. JASS `04 Benjamin Fingerle, Christian Wachinger23 Issues Concerning Graph Representation Possible ways of storing the spatial relationship graph include One graph-service holding the complete Graph + all graph algorithms applicable -contradicts the distributed computing paradigm Each Service knows its adjacency +complies with distributed computing paradigm -not all graph algorithms applicable

24. JASS `04 Benjamin Fingerle, Christian Wachinger24 Issues Concerning Access to the Graph Information Possible ways of accessing spatial relationships include Request formulated as need for spatialRelationship with predicates source and target, abilities offered by –Certain GraphInformationServices +Graph can be split up between different such services -different graph search strategies for the same relation cannot coexist –One service for each relationship instantiated by special relationShipGenerators +relationships instantly available -different graph search strategies for the same relation cannot coexist -number of services raises dramatically Requesting a graphInformerObject providing an interface getSpatialRelationShip(source, target) +Objects implementing different graph searches can coexist -centralization

25. JASS `04 Benjamin Fingerle, Christian Wachinger25 Open Questions How much information about the world do we have to put in the error function? Which graph algorithms have to be applicable? Which graph representation allows theses algorithms? How to perform context changes? How to enable new users to enter AR environments? What has then to be calibrated?

26. JASS `04 Benjamin Fingerle, Christian Wachinger26 Critical success factors Number of AR ready buildings Development and acceptance of standards Prices and convenience Legal issues Products and applications urging the user to buy