1. 1 Adding semantics to multimedia - European projects Franco Mastroddi European Commission DG Information Society Imagina 2002

2. 2 “Ambient Intelligence” tomorrow Nano-scale Information devices - many embedded.. Use all senses, touch, vision, hearing... Content- and context-based systems Infinite bandwidth, cross-media... Mobile/Wireless full multimedia Wide adoption (eHealth, eLearning...) >70% of world-wide population on line ICT today Micro scale…………………… PC based …………………… “Writing and reading”…….…. “Word” based metadata…… Low bandwidth, separate networks. Mobile telephony (voice)…….. eServices just emerging……… Only 5% of global population on-line.. ‘Mega-trends (ISTAG expert Group)’

3. 3 v In early 1990s the typical Web user was an American male student v Today 3many users are ‘normal’ consumers (Insight Express) 3female users overtaken males in US (Nielsen) 3worldwide, more than 65 million youths (5-17) have Internet access at home or school (Datamonitor). 3Web is used increasingly for trade and professional services, e- commerce etc v vast new opportunities… 3broader market 3economies of scale Multimedia demographics have changed

4. 4  from scribe to screen. Over 90 % of text / data is digital origin. Soon also for audio-visual.  From linear to non-linear content management  from mass broadcast to mass personalisation  from single media to cross-media.  from static to dynamic multimedia resources. Self-managing digital objects, links, embedded semantic metadata / ontologies.  from geographic branding to virtual community-building  from disintermediation to re-mediation through new content middleware systems. New forms of interactivity

5. 5  Example of consumer photographs  annual output: billions of photos / year.  Present storage method: albums & shoeboxes  “people want to do more than print a few pictures and store the rest on their hard disk”  but the average consumer is not a good indexer…  Professionals do not fare much better...  “1,5 hours to scan 42 photos...but 43 hours to index them…” (Washington State 2000 Diglib project) Is there too much content?

6. 6 Is there enough technology?  Increasing processing power  Data crunching content management  with new telecomms networks  mobile (UMTS still in the future - 4% global penetration by 2004 - UMTS Forum)  Bluetooth, ambient systems  and new a/v devices  over 17 mio digi-cams shipped in US in 2001 (Hallmark Card Store)  can integrate into flexible content platforms  e.g. digital camera as front-end of an integrated home image system via wireless/USB to PC or set- top-box - thence to Internet.

7. 7 Does the Web really help?  The Web paradigm does not necessarily help in multimedia searching...  “ Size of the database, frequency of update, search capability and design, and speed may lead to amazingly different results” (Kansas City Public Library)  “search engines are undergoing a profound evolution... such as... ways of automatically classifying information” (B. Grossan, Lawrence Berkely Library)  “a very small number of commercially available products exist which perform content-based image retrieval.” (Manchester Visualisation Centre, 2001) Which one is the criminal? Attrasoft

8. 8 DocumentsDatabasesEmailWebPeopleProcesses Multi-modal / visual interfaces Information Agents Semantic middleware : RDF, Ontologies, Semantic-based systems and services. Human Machine-Machine Towards a Semantic Middleware.

9. 9 Content description is the key Source: Gemini Consulting CreateHostConnectUse The linear content chain: The non-linear chain:

10. 10 Projects: Automated multimedia content analysis Other projects: personalised views of a broadcast archive (Primavera) advert analysis from TV and Web (Aramis) automated music editing & retrieval by content (CUIDADO) FAETHON- INTELLIGENT ACCESS TO HETEROGENEOUS AUDIOVISUAL CONTENT: extracting high level semantic out of syntactic and low level semantic information filter responses according to user’s up-dated profile Developing new search algorithms to work on MPEG 4/7 metadata.

11. 11 Projects: The visual interface Other projects: stereographic visualisation and analysis of astronomy data (Cosmo.Lab) Visual Data Navigators or "Collaboratories" scaleale to the Web (Smartdoc) MPEG 7 / 21 multimedia retrieval across mobile (UMTS), TV and Web (Perseo) 3D Search : next generation knowledge retrieval & analysis  selects objects from ontologies  3D visualisation  A.I. for updated filtering adjustments Visualising a semantic net in 3D. IICM.

12. 12 Projects: Still Image Retrieval Cobweb: Retrieval of images - based on their content (colour, texture, similarity etc) using algorithms and semantic analysis. http://www.imageQuery.com Other Projects: Automatic retrieval of video clips (CIWOS) real-time matching of video and database images (LIVE@Web.Com) retrieval of textile / fashion images by emotive values (FoundIt)

13. 13 Projects: Adding semantics to 3D objects Describing photos with RDF - W3C Lafon / Boss Project 1: combine 3D graphics in industrial product catalogues with semantic tags: the 3D model can be directly manipulated via the tags it can interact smartly with other 3D models Other Projects: add a semantic layer to a dig-lib to support 3D image processing (Project 2) enhancement of a video sequence captured on a real scene with real-time rendered 3D objects, incl indexing and retrieval (ENREVI)

14. 14 Projects - Semantic Web technologies Project S-E: progressing the Semantic Web, to “bring structure to the meaningful content of Web pages, creating an environment where software agents roaming from page to page can readily carry out sophisticated tasks for users”. Tim Berners-Lee, Scientific American. Projects: application service provision of semantic tools for “deep Web” multimedia content (Project E-S) a spatially-aware search engine that finds documents and datasets relating exactly or approximately to places or regions referred to in a query (Project SP)

15. 15 Widespread mark-up of digital resources (using RDF, XML) by their semantics e.g. “this picture is me”, not their labels, e.g. “this picture is JPG 99- 325687”. Develop flexible, interoperable ontologies for sharing metadata across different archives e.g. to allow comparisons, context, and classification of existing and new terms. agent-based techniques and cogitive systems which can act on the improved metadata structure to help with user profiling and collaborative filtering. SemWeb services e.g. in e-commerce, information retrieval etc. Main ingredients of the Semantic Web

16. 16 Anywhere anytime natural and enjoyable access to IST services for ALL Miniaturised, low cost low power components & µsytems Natural interactions with ‘ knowledge ’ Pervasive, mobile, wireless, trustful infrastructures Communication & networking Software µ, nano & opto electronics µ and nano systems Knowledge technologies interfaces Applied IST for major societal and economic challenges Trust & Security IST for societal challenges IST for economic challenges Demanding applications Sixth Framework Programme: Proposed structure

17. 17 Keywords Context - based Semantic-based Agent based Scaleable Sixth Framework Programme: “Knowledge Technologies” Research challenges: usability of knowledge systems content as knowledge standards and interoperability Knowledge communities & portals Http://www.cordis.lu/ist/ Technologies: Web resource description Content-based multimedia indexing Knowledge Enginering / AI / agents Natural Language processing (text / speech) Interactivity and visualisation machine learning