1 The Strategic Role of Vision in Cognitive Systems and Robotics - A Perspective from the EU Research Programme ECCV 2010, Special session on "Research Funding for Vision", September 9, 2010 Cécile Huet – Unit E5 - Cognitive Systems, Interaction, Robotics DG Information Society and Media European Commission

2 Outline  History  Projects portfolio  Perspectives and future opportunities  Conclusion

3  Limitation of computer vision systems:  Lack of ROBUSTNESS/VERSATILITY/SPEED  Cognitive vision  combine state-of-the-art image processing with cognitive reasoning: abstract representation, categorisation, memory organisation, self-adaptivity, contextual knowledge, behavioural/scene interpretation,… It all started with VISION: “Cognitive Vision” Initiative in 1998 (FP5): Rationale

4 Projects portfolio  >140 projects - €480 million EU funding.  FP5 ( ) Cognitive vision - 9 projects  FP6 ( ) Cognitive Systems + Advanced robotics - 45 projects  FP7 ( ) Cognitive systems, Interaction, Robotics 68 running projects + 22 projects under negotiation

5 FP5 Cognitive Vision - achievements  FP5 – DETECT Real Time Detection of Motion Picture Content in Live Broadcasts -> detect/track brand appearance in sport videos

6 FP5 Cognitive Vision - achievements  FP5 – ACTIPRET Interpreting and Understanding Activities of Expert Operators for Teaching and Education Recognition of hand and arm gestures: original image with overlay and 3D representation Long term goal: giving hints on how to clear the paper jam in a copy machine

7 FP5 Cognitive Vision - achievements  LAVA - Learning for adaptable visual assistants  Categorization and interpretation of large numbers of objects, scenes and events, in real settings. Recognition/Categorization: Occlusion – cluttered background – viewpoint-rotation-scale,… invariance, lighting conditions,… Categorization - Cope with different forms of the same object (shape, colours, size, angle etc) and an open set of instances of the object Scene understanding – disambiguation –> multi cue/information fusion – context,… Source: www

8 FP6: Cognitive systems research - motivation Beyond vision…. Achieving naturalness, versatility, robustness at system level real-world  from constrained to real-world environments general solutions  from application-specific to general solutions systems approach  from component methods to a systems approach working across disciplines  from a ‘monodisciplinary’ research effort to working across disciplines RESULTS: Integration of perception (attention mechanisms, affordances, multi-sensory/cue…), representation, learning, reasoning, decision-making, action and communication, cooperation (HRI/RRI), anticipation,… Strong multidisciplinarity (neurosc., cog sci., engineering, AI,…)

9  Robotics (navigation – manipulation) - action-perception(COSPAL- SPARK), affordances (MACS-PACO-PLUS), anticipation(MINDRACES), reasoning - understanding (COSY), attention, knowledge-reasoning (GNOSYS), observation (MATHESIS), decision-making (DECISIONS-IN- MOTION), reasoning & self-preservation (ICEA), Joint Action (JAST), Active haptic(SENSOPAC), developmental (ROBOT-CUB)  Interfaces - audio-visual- HHI/HRI(POP), HRI (PACO-PLUS, COSY), HRI- RRI (JAST) – ECAS (RASCALLI)  Assistive technologies - Blind (DECISIONS-IN-MOTION- CASBLIP) – driving assistance (BACS) – Augmented human action (PACO-PLUS)  Scene analysis/digital content - audio-visual + attention (POP-DIRAC), language & information discovery (RASCALLI), description image/video + text (CLASS), man-made scenes interpretation (eTrims), description of human behaviour(movement-facial expression) from video sequences (HERMES)  + euCognition / PASCAL FP6: Cognitive systems projects - classification

10 HRI FEELIX GROWING: FEEL, Interact, eXpress: a Global appRoach to develOpment With INterdisciplinary Grounding INDIGO: Interaction with Personality and Dialogue Enabled Robots COMMROB: Advanced Behaviour and High-level Multimodal Communication With and Among Robots PHRIENDS: Physical Human-Robot Interaction: DepENDability and Safety ROBOT – FOR HUMAN ASSISTANCE (environments: Firefighter/Urban pedestrian area/home) VIEWFINDER: Vision and Chemiresistor Equipped Web-connected Finding Robots / GUARDIANS URUS: Ubiquitous Networking Robotics in Urban Settings An Open Platform for Home Robotics (platform-perception-navigation) SWARM – for human assistance or autonomous tasks IWARD: Intelligent Robot Swarm for Attendance, Recognition, Cleaning and Delivery (+HRI) ROBOSWARM: Knowledge Environment for Interacting ROBOt SWARMs (IT infrastr./Cleaning Hospital) GUARDIANS: Group of Unmanned Assistant Robots Deployed In Aggregative Navigation supported by Scent detection Network of Excellence / Coordination and Support Actions EURON: European Robotics Research Network CARE: Coordination Actions for Robotics in Europe RoSta: Robot Standards and Reference Architectures (+benchmarking) RAWSEEDS: Robotics Advancement through Web-publishing of Data Sets FP6: Advanced Robotics projects - classification

11 FP6 – Vision: Examples  CASBLIP: Cognitive Aid System for Blind People  Sensory substitution: from vision to audio information  DECISIONS-IN-MOTION: Neural Decision-Making in Motion  understand the neural mechanisms underlying fast target selection and sensory-guided decisions and actions in humans in motion  transfer this knowledge into an artificial visual system for automated robotic navigation.

12 Challenges  greatly improving robustness etc. requires rethinking the way systems are engineered -> theories are needed  Scientific approach: understanding of what both natural and artificial systems can and cannot do, and how and why  integration of disciplines: robotics, AI, computer vision, natural language, cognitive science, psychology,… mathematics,… philosophy of mind,… Equally importantly  we need to be able to support claims that progress is being made  identify real applications that would greatly benefit from such systems FP7 - Challenge 2: Cognitive Systems, Robotics, Interaction

13 HMI FP7 – CALL1 – PROJECTS CLASSIFICATION Cognition & Language emergence in robots Autonomous Robots Dexterous Manipulation HRI Perception & Sensors Scene understanding PROMETHEUS CoFRIEND SCOVIS Sensor network Spoken language Multimodal interaction Cognition for Human Assistance Machine Learning Medical – Rehab./Surgery SF SPARK II REPLICATOR CHIROPING EYESHOTS SEARISE DIPLECS CogX LIREC SEMAINE PINVIEW ALEAR ITALK ROSSI CLASSIC EMIME ROBOCAST MIMICS GRASP CHRIS DEXMART POETICON PASCAL 2

14 FP7 – CALL3 – PROJECTS CLASSIFICATION: SCENE UNDERSTANDING SCENE UNDERSTANDING NAVIGATION CA in Cog Sys URBAN MULTI-ROBOT / SWARM ACOUSTIC SCENE ANALYSIS UNDERWATERAERIAL HRI SCANDLE Co3AUVs SHOAL FILOSE EUROPA SFLY EUCOGII

15 FP7 – CALL3 – PROJECTS CLASSIFICATION FROM AN APPLICATION ORIENTED PERSPECTIVE IM_CLEVER HANDLE ECHORD ECCEROBOT MOBILE MANIPULATION SORTING OBJECTS ASSEMBLY SERVICE ROBOTICS & HRI EUROPA GUIDANCE DELIVERY TRANSPORT HUMOUR REHABILITATION MOTOR SKILL LEARNING STIFF PROSTHETICS ROBOSKIN PROGRAMMING BY DEMO/ TOUCH-BASED SOCIAL INTERACTION HUMANOBS SOCIAL- COMMUNICATIVE SKILLS (multimodal interaction – TV-Host) Co3AUVs SCANDLE MONITORING SURVEILLANCE SECURITY FILOSE SHOAL SFLY ENVIRONMENTAL MONITORING SEARCH & RESCUE BRICS ROSETTA INDUSTRY / MANUFACTURING

16 FP7 – CALL4 – PROJECTS CLASSIFICATION HRI AUDIO- VISUAL NAVIGATION BIO-INSPIRED LOCOMOTION MOTOR CONTROL DEXTEROUS MANIPULATION SENSOR NETWORK/ HUMAN ACTION RECO TRIDENT PERCEPTION: VISION ROBOSOM FIRST-MM COGNITO euROBOTICS CA in ROBOTICS VANAHEIM TACO GERT THE HUMAVIPS MULTIMODAL LANGUAGE NIFTI ALIZ-e IURO AMARSI WWW SHARED DB: Robots actions MASH ROBOEARTH HAPTICS + VISION AIROBOTS RADHAR GARNICS

17 Shared resources: Contribute to and benefit from: RoboEarth – Robots sharing a knowledge base for world modelling and learning of actions - Shared resources: Contribute to and benefit from: RoboEarth – Robots sharing a knowledge base for world modelling and learning of actions -  World Wide Web for robots: a giant network and database repository where robots around the world access and continually update to share information and learn from each other about their behaviour and their environment.  Improve any robot’s 3D sensing, acting and learning capabilities.

18 Shared resources: Contribute to and benefit from: MASH - Massive Sets of Heuristics for Machine Learning  Design of complex learning systems integrating large and heterogeneous families of feature extraction modules.feature extraction  Open web-based platform where contributors will find tools to interact and communicate with each other, and the means to integrate their feature extractors in continuously running experiments.  Quantitative evaluation of submitted modules  Performance will be measured on action selection with a robotic arm and in a simulated 3d environment, and object detection.action selectionobject detection 

19 FP7 – CALL FoF – Smart Factories: ICT for agile and environmentally friendly manufacturing – Target outcome c) Robotics-enabled production processes tested and validated in real-world environment MANIPULATION/ HANDLING SORTING OBJECTS PRECISION FLEXIBILITY/SCALABILITY RELIABILITY ROBUSTNESS VERSATILITY MANIPULATION NON-RIGID / HEAVY OBJECTS PRECISION FLEXIBILITY/SCALABILITY RELIABILITY ROBUSTNESS VERSATILITY MANIPULATION NON-RIGID / HEAVY OBJECTS PACKAGING CustomPacker HRI + HR COOP.: WORKER DETECTION + INTENTION RECO- SKILL LEARNING PACKAGING ELECTRONIC CONSUMER GOODS CUSTOMPACKER NAVIGATION / PLANNING: MOBILE MANIPULATION / MOBILE ASSISANT: LOGISTICS + ASSEMBLY TAPAS VISUAL INSPECTION VISUAL SERVOING SHOE ROBOFOOT

20 VISION + Object/scene understanding FP7 – CALL6 – PROJECTS CLASSIFICATION (! Subject to successful negotiation) CA in ACS NAVIGATION DEXTEROUS MANIPULATION DEXTEROUS MANIPULATION HRI/SOCIAL INTERACT GOAL-DIRECTED/ANTICIPATION SENSOR NETWORK

21 INDICATIVE - BASED ON DRAFT TEXT ! INFORMATION TO BE CONFIRMED  TARGET a): Research projects on Robotic systems operating in real-world environments: + functionalities + autonomy, safety, robustness, efficiency, and ease of use AS APPROPRIATE: + new materials and advanced sensor, actuator, effector, memory and control technologies. ->> validation through REAL-WORLD scenarios  TYPE OF PROJECTS : STReP (Specific Targeted Research Projects) High risk/Focussing on specific research topic/component IP (Integrated Projects) System oriented  BUDGET: 70M€  DEADLINE: What next? The 2010 Call for Proposals – FP7 - ICT Call 7 -

22 INDICATIVE - BASED ON DRAFT TEXT ! INFORMATION TO BE CONFIRMED  TARGET d) Coordinated Action(CA) Fostering communication and co-operation between robotics and cognitive systems research communities: + knowledge sharing: EU, national, and international; + open source; R&D roadmaps, market potential, user acceptance, standardisation, education, ethics, socioeconomic impacts; outreach to relevant professional and general audiences.  TYPE OF PROJECTS : CA  BUDGET: 3M€  DEADLINE: What next? The 2010 Call for Proposals – FP7 - ICT Call 7 -

23 INDICATIVE - BASED ON DRAFT TEXT ! INFORMATION TO BE CONFIRMED  Cognition and control in complex systems (IP and STReP)  Cross-fertilisation between academic and industrial robotics research (IP)  Targeted competitions for smarter robots (CA)  INDICATIVE DEADLINE: What next? The 2011/12 Call for Proposals – FP7 - ICT Call 9 -

24 Perspectives  Outcome so far and next steps  New research questions -> need for further scientific progress  Also technological progress and application motivated research, building on previous results – innovation in integration and implementation of results to improve robotics functionalities and features.  Real world scenarios: source of research questions + validate research results  Demonstrate progress: Benchmarking

25 Role of vision? CONCLUSION FP7 – Workprogramme text « Research and development […] will be guided by demanding, yet pragmatic application scenarios »

26 Critical!! CONCLUSION FP7 – Workprogramme text « Research and development […] will be guided by demanding, yet pragmatic application scenarios »

27 THANK YOU!