1 Robots Unlimited PHD Summerschool July 2006 Alexander Brändle Marco Combetto

2 Agenda Who we are? Why robotics? Activity overview Future ideas

3 Intelligent Environments Team Alexander Braendle Marco Combetto Pierre-Louis Xech Andreas Heil (PHD Student)

4 Why robotics? How will Future Environments look like? –Challenges/Needs –Personal Devices, Embedded Devices, Sensors, Robots, Appliances? –Enabling technologies,Programming paradigms,Interaction paradigms? How will future applications look like? –Wide range of user types? –Mixing real and virtual world? Robots could form a part of it!

5 The Robotics Wave Service and consumer markets just emerging –Remote assistance/presence –Assistive –Facilities maintenance –Security –Education –Entertainment Academic Research –Complex topic –Moving from 8/16 to 32 bit –Lots of hand-coded solutions –Education and Hobbyist channel –Still mostly 8 bit, starting to shift Strong governmental interest worldwide EURON roadmap: > Its all about software

6 Robots! Wide range of applications –Commercial use –Academic use –Personal use How to program them? Challenges –Complexity –Reusability –Reliability –Resources –Tools –Technologies –Choice –Sharing –Transference of skills/experience

7 Links European Robotics Network (EURON) European Robotics Platform (EUROP) IEEE RAS Technical Commitee on Programming Environments in Robotics and Automation

8 Robots, too? Software –Internet Explorer –Media Player –MS Agent –Mobile Phones –… Definition: A robot is a device, hard- or software with the capability of sensing and (re-)acting.

9 Robotics at Microsoft [Research (Cambridge)] Robots in human environments End-user programming –Developer Tools (Fischertechnik- Demo, Scatterweb-Demo) –Visual Programming (.FUN-Demo) –End-user debugging –Microsoft Robotics Studio –Coding4Fun Enabling new applications –Emotional oriented computing –Personal robotics –Self-reconfigurable structures –Learn from interaction with animals

10 The Fischertechnik ROBO Interface –The in- and outputs 8 digital inputs 2 digital and analog distance sensors 4 analog sensors for resistance and voltage 4 motors with 8 different speeds The board Serial port, COM, RS232 USB Infrared R/F module available Ethernet for the next hardware revision planned TS%SystemRoot%\System32\mstsc.exe

11 Adding Control –Now we want to control the robot –Lets take another off the shelf product –Ordinary Joystick +

12 Coding4Fun

13 Integrating Sensors –Sensors are they eyes and ears of robots –Increasing Demand for easy robust sensor platform from biology, ecology, civil engineering et al. How to get easy access to Sensor Nodes systems A challenge on its own: Where are sensor node platforms heading? Dealing with vast amounts of real-time data becoming available

14 Sensor Networks in the real world Robustness … –WSNs have to work 24/7 –WSNs have to offer uniform APIs –Long-lived, autonomous networks –Self-organizing and energy efficient –Routing –Data aggregation –Deployment & Setup …Better development support –Realistic simulation tools –Heterogeneous testbeds –Useful traces –Tool integration (in well know platforms) –Development for heterogeneous systems ScatterWeb/.NET SN GW SN GW Bluetooth, … Ethernet SN GPRS WLAN

15 Sensors (ScatterWeb) An open and flexible platform for rapid protoyping & implementation of wireless sensor networks Nodes –with/without sensors Sensors –Luminosity, noise detection, vibration, PIR movement detection, Microphone/speaker IR sender/receiver –stand-alone/modular Acceleration, humidity, temperature, luminosity, noise detection, vibration, PIR movement detection on demand –Stand-by: 7.6µA, 5 years life-time with AA battery and 1% duty-cycle Gateways –WLAN, Ethernet, Bluetooth, GPS, GSM/GPRS, USB, RS485, serial… Software –Management, flashing, routing, ns-2 simulation models –TCP/IP, web server ( Contiki, TinyOS, … –Basic functions for energy management, routing Jochen Schiller,Free University Berlin

16.NET for ScatterWeb Sensor world available to every developer Easy access to sensor values, events, and functions Understandable namespaces and interfaces Support for IntelliSense and dynamic help Well known programming model (events, methods, properties) Extensibility of the nodes logic (Tiny C#) and instant IntelliSense-Update Development, deployment and debugging within Visual Studio Extend the.NET tools and architecture to small devices Jochen Schiller,Free University Berlin

17 A first step Attractive for developers End-users?

18.FUN A compelling & engaging programmable environment to play & learn for children (introduce children to Computing in new ways) Make technologies of tomorrow accessible to non technical market (children, nurse, elderly, machine operator) Linking real and virtual world Wider market potential of Robotics (Industrial, Assistive technology, new consumer products, health) Technical University Berlin

19 Next steps Keep graphical representation Using context data Domain specific language Changing the paradigm …

20 Integrating a Context Server 1/2 Context ServerApplication(s)Sensors DB Raw Sensor DataContext Events Context Requests Torben Weis, University of Stuttgart

21 Integrating a Context Server 2/2 Context ServerApplication(s)Context Simulator DB Raw Sensor DataContext Events Context Requests3D Data Torben Weis, University of Stuttgart

22 Torben Weis, University of Stuttgart

23 Domain-specific Languages Visual N# Domain-specific Graphical language Extension of VRDK C# / VB.NET General purpose programming language N# Domain-specific Textual language Users can switch between both notations Code translator Torben Weis, University of Stuttgart

24 N# - A Textual Notation for Visual N# ambient Person where filter ($1.Age < 40), filter($1.Location == "Stuttgart") { discover Light where distance(a), filter($1.Color == "red"); process l = Lichter.Added { l.On(); } process l = Lichter.Removed { l.Off(); } a = 100; } Torben Weis, University of Stuttgart

25 Microsoft Robotics Studio Concurrency Concurrency Services infrastructure Services infrastructure Samples and tutorials Samples and tutorials Robot services Robot services Robot models Robot models Technology servicesTechnology services Microsoft Robotics Studio Simulation Tool Simulation Tool Visual Programming Language Visual Programming Language A development platform for robotics community, supporting a wide variety of users, hardware, and application scenarios. Runtime AuthoringTools Authoring Tools Services and Samples Make it easy to manage asynchronous components Avoid need to understand manual threading, semaphores, etc. Provide a scalable programming model Make state observable, easily accessible Provide for reusability and failure Support component discovery and composition Support remote/distributed execution

26 Support standalone and distributed processing scenarios Connected operation (remote execution on PC) Disconnected autonomous operation (with optional networked monitoring) Distributed execution (execution across compute units) Microsoft Robotics Studio Key Runtime Features

27 Microsoft Robotics Studio Extensible to a wide variety of hardware

28 Microsoft Robotics Studio Services and Samples Over 15 tutorials –VB.Net, C#, JScript Support for –LEGO® Mindstorms® RCX –LEGO® Mindstorms® NXT –fischertechnik® –MobileRobots Pioneer P3

29 Microsoft Robotics Studio Other University Support Bryn Mawr College Carnegie Mellon University Cornell University Georgia Tech Massachusetts Institute of Technology Stanford University University of Pennsylvania University of Pisa University of Southern California University of Washington

30 Imagine a world … where Paper is able to understand, what you are doing …

31 Robotics at Microsoft [Research (Cambridge)] Robots in human environments End-user programming –Developer Tools (Fischertechnik- Demo, Scatterweb-Demo) –Visual Programming (.FUN-Demo) –End-user debugging –Microsoft Robotics Studio –Coding4Fun Enabling new applications –Emotional oriented computing –Personal robotics –Self-reconfigurable structures –Learn from interaction with animals

32 Emotion-Oriented Computing General Goal: Make interaction between human and machine more natural for the humans Machine should be able to: –To register human emotions –To convey and comunicate emotion –To understand the emotional relevance of the event 1/29/201432Microsoft Internal Only

33 Human Centred, Affects and Emotions Enhance communication through compelling, fun and emotional interactions with computing (Mobile devices, Robotics) Seeking to make the process more intuitive, interactive and appealing to a wider group of people. Experimenting, evolving, evaluate emotional models Support privacy, intimacy and different level of information sharing Extending the software as a new medium –Affective Media and Mobile media (affective loop) –Participatory creation, exchange, authoring and fruition –Storing the intimacy, the private value of the things –Fusion of different kind of data sensors, embodiments, 1/29/2014Microsoft Internal Only 33

34 Human Centred, Affects and Emotions 1/29/201434Microsoft Internal Only Enhance communication through compelling, fun and emotional interactions with computing. Seeking to make the process more intuitive, interactive and appealing to a wider group of people. Experimenting, evolving, evaluate emotional models Support privacy, intimacy and different level of information sharing Extending the software as a new medium –Affective Media and Mobile media (affective loop) –Participatory creation, exchange, authoring and fruition –Storing the intimacy, the private value of the things –Fusion of different kind of data sensors, embodiments,

35 Sensing Interaction with Environments Some examples

36 Affective Diary Designing for bodily expressiveness and self-reflection Collecting memories – including body memorabilia mingled with mobile materials (SMS, MMS, photographs, music listened to, video,.. Offering a diary medium in which those memories can be mirrored and organised Empowering the user to create meaning and alter those representations Prototype –build on TabletPC and Smartphone –Sensordata (movement, arousal) 1/29/2014Microsoft Internal Only 36

37 BSP: Interactive storytelling in vast location based Pervasive computing: Location based games provide navigational challenges e.g. Chasing Mobile Media: A flexible media and framework for interaction Interactive storytelling: Balance linearity with user control, believable characters The concept of believable environments, and our implementation, put a research focus on possibilities to: –Enrich pervasive games with location dependent narratives –Design the stage set to improve interactive storytelling 1/29/2014Microsoft Internal Only 37

38 Sensing Interaction with Robots An example

39 Why Robotics? Interest about robots is spreading fast Reinventing Personal Computing –New way of interact because the usual ways are not enough (keyboard, mouse, etc.) –We look for new interaction modality? Does Human size matter? Social aspects.. Robots as an embodiment How to discover contexts and adapting? Technology –Technology seems to be mature enough to produce robots not only for industry –Software for controlling social robots should be robust and should adapt to the changing environment –It seems that control systems theory doesnt scale well in this setup

40 The mind and the body Neurophysiologists suggest that the body plays a crucial role in our cognitive process The brain experiences the world through the body: –The environment is well known to the brain –Outside environment is a set of states of inside Agents with body (embodied agents) try to mimic this schema to produce intelligent behavior Dautenhan and Jakobi show that bodys presence influences the behavior of software (anything already heard?)

41 Component structure Knowledge Base Body Map WiFi Roblet MotionRoblet Vision Roblet Sensors Internal perception Network perception Arm Roblet

42 Robotics4.net Realization of a platform for define the body of a robot as a set of agents (Roblets) that acts as intermediaries between the devices and the reasoning software The robot provides a physical body and a set of core services to support the development of highly autonomous agent situated in both cyberspace (e.g Internet, virtual entities) and real world Definition of an Object Model, an API, a set of abstraction that allows: –Portability (on different type of platforms) –Scalability from small (e.g. Lego) to complex systems –Integration with the MS Development Platform and MS OS and Application suites

43 Functions implemented The platform provide the following base services to support the software implementing the roblets: –General purpose functions (I/O) –Basic motion abilities –Collision avoidance software –Vision software for face recognition –Speech and gesturing recognition –Positioning and navigation aided by video input –Network perception –IM interface –The Architecture is available as a simple Software Development Kit freely downloadable from

44 Perception 1/29/2014Microsoft Internal Only 44

45 Evolving the framework Reduce the programming model of Robotics4.NET to the one adopted by the Robotics Studio CTP Extending the Mind/Body model with self-developing tools, model and methods to verify the behavior of systems imposing very high level constrains and how that can be both formally and programmatically verified Testing interaction in real scenarios

46 Questions