X3D Extension for (Mobile) AR Contents International AR Standards Workshop Seoul, Korea Oct 11-12, 2010 Gerard J. Kim (WG 6 AR Standards Study Group Coordinator) Korea University

Approach Extensibility to existing frameworks X3D (Scene graph) Because AR is implemented as VR! KML, OpenGIS, … We need location representation Generality/Flexibility to accommodate Different AR platforms (~Platform independence) Mobile, Desktop, HMD, … Sensors and devices Vision based, Marker based, Location based, … Focused on file format (Scene graph based?) vs. Contents representation Machine consumption

Various display types and platforms Video Combiner Optical Combiner Display Camera Display Camera Video Combiner [R. Azuma, 1997]

AR/MR Implementation

Various sensing

MR/AR Contents Context: Condition or situation that triggers an augmentation and mixing of real and virtual objects Resource: Raw data or information used for augmentation Content: One or more pairings of contexts and Resources + behaviors (that uses the resources) context resource

Related work Jung et al. (InstantReality Suite) Extension of Sensor nodes – Physical contexts Extension of Viewpoint nodes – Specification of camera parameters Layers: One layer served as background video Extension of X3DLightNode: Lighting effects SFImageSenosr : X3DDirectSensorNode { SFImage[in/out]value… SFBool[]outFalse SFString[]label … } DEF frame SFImageSensor { label “Video Frames” } ROUTE frame.value_changed TO surfaceTex.set_image

Major proposals Extend “View” node: Resolution between “live” camera and virtual Define “Live” camera node (G. Lee / ETRI) – Not necessarily for “AR” contents (e.g. Video textures) – Parameters set by user More detailed parameter specification for “View” Set by user Routed from “Live” camera node – With possibility of behavioral manipulation “Routed” from sensor – Camera could be tracked separately Default: same as the world – Note that view can be relative to anything LC, VC LC VC LC, VC

Major proposal Extending movie texture node (for AR background) Also proposed by G. Lee / Instant Reality Extend existing virtual “Sensor” nodes New X3DARNodes for target real object description ImagePatch, 3DObject, GPSLocation, SingleValue, … Existing: E.g. Visibility, Proximity, Touch sensor … New: RangeSensor, UIClickSensor, …

Not included in this proposal Lighting and Rendering issue Depth sensing and occlusion effects Extended point of interest (e.g. path, hierarchical POI) Platform type specification e.g. Resolution difference

AR contents (Real/Physical) X3D (Virtual) WORLD View (Virtual Camera) Other X3D Nodes AR Node + Sensor ROUTE* Live Camera Movie Texture* Virtualized Physical Contexts

Abstraction of MR/AR contents as a collection of context and resources connected by “Event in”’s and “Event out”’s.

<ROUTE fromField='touchTime' fromNode='TOUCH' toField='startTime' toNode='TIME'/> <ROUTE fromField='fraction_changed' fromNode='TIME' toField='set_fraction' toNode='INTERP_POS'/> <ROUTE fromField='value_changed' fromNode='INTERP_POS' toField='translation' toNode='BALL'/>

X3DNode X3DARNode ImagePatch 3DObject GPSLocation SingleValue UIDevice … X3DChildNode X3DSensorNode X3DEnvironmental SensorNode VisibilitySensor ProximitySensor RangeSensor … UIConfigNode UIClickSensor UIScrollSensor …

Vision based feature recognition and tracking (e.g. fiducials, markers, 3D points) Non-vision based env. sensor events and values (e.g. RFID, GPS, distance) User interaction devices events and values (e.g. buttons, touch screen, jog dial) Context information (e.g. user age) Real ObjectX3DARNodeMain AttributesSensor used MarkerImagePatchID, Position, Orientation Visibility 3D point3DObjectID, Type, Position, Orientation Visibility GPS Location ID, CoordinateRange RFIDSingleValueValue (Boolean)Existence Ultrasonic sensorSingleValueDistance (Integer)Proximity ButtonUIDeviceValue (Boolean)UIClickSensor User AgeSingleValueAge (Integer)Range

X3DARNode Placeholders for physical objects within AR/MR world “implementation” X3DARNode : X3DNode { SFNode[in, out]metadata SFNode[in, out]parent SFString[in, out]description SFBool[in, out]enabled } X3DARNode is the base type for the Marker, Location and General Event, …

ImagePatch (Marker) & VisibilitySensor ImagePatch : X3DARNode { SFNode[in, out]metadata SFNode[in, out]parent SFString[in, out]description SFBool[in, out]enabled SFString[in, out]filename SFVec3f[in, out]position SFRotation[in, out]orientation } VisibilitySensor : X3DEnvironmentalSensorNode { SFVec3f[in, out]center SFBool[in, out]enabled SFNode[in, out]metadata SFVec3f[in, out]size SFTime[out]enterTime SFTime[out]exitTime SFBool[out]isActive }

Location & RangeSensor GPSLocation : X3DSensorNode { SFNode[in, out]metadata SFNode[in, out]parent SFString[in, out]description SFBool[in, out]enabled SFInt32[in, out]device_description SFBool[out]status MFString[out]values } RangeSensor : X3DEnvironmentalSensorNode { SFVec3f[in, out]center SFBool[in, out]enabled SFNode[in, out]metadata SFVec3f[in, out]size SFTime[out]enterTime SFTime[out]exitTime SFBool[out]isActive SFInt32[in, out]sequence SFString[in, out]lBound SFString[in, out]uBound SFString[in, out]value }

Live camera LiveCamera = MR/AR Capture Camera Within the Scene node Image field is the out value Camera internal parameter  projmat field Camera external parameter  Set to World but can be tracked Live Camera { SFString[in, out]label "default“ SFString[out]parent SFImage[out]image SFMatrix4f[out]projmat "1 0 0 … " SFBool[out]on FALSE SFBool[out]tracking FALSE SFVec3f[out]position SFRotation [out]orientation }

Routing from LiveCam From Live Camera node “image” field To Background (LiveURL field) Shape (MovieTexture field)

Live video  Background <Background groundAngle=' ' groundColor=' ' skyAngle=' ' skyColor=' ' backUrl='mountns.png' frontUrl='mountns.png' leftUrl='mountns.png' rightUrl='mountns.png'/>

MovieTexture Node Add MovieTexture to X3DTextureNode hierarchy Used for TextureBackground Fix TextureBackground relative to camera Allow connection to live camera (not just through streaming server)

MovieTexture Node <MovieTexture loop='true' url=' "wrlpool.mpg" " '/> <Coordinate point=' '/>

Live Camera  Movie Texture

Live Camera and Virtual Camera Calibrating the virtual camera according to the parameters of live capture camera Internal parameter = projection matrix External parameter = camera pose Manual Direct specification Routing From the Live camera From the Sensor W0W0 T

Method 1 Viewpoint : X3DViewpointNode{ SFMatrix4f[in]projmat SFVec3f[in,out]position SFRotation [in,out]orientation SFNode[in,out]liveCamera Add distortion parameters here } …

Method 2 …

Other Activities Draft document Teleconferences with Web3D Implementation: k-MART Domestic workshop April, POSTECH, Pohang, Korea June, KIST, Seoul, Korea

Future More Extensions Examples Implementations International consensus