National Aeronautics and Space Administration Practices for Improving Robotic Software Reliability in Flight and Research Projects Khaled S. Ali and Issa A.D. Nesnas Jet Propulsion Laboratory, California Institute of Technology The Sixth IARP-IEEE/RAS-EURON Joint Workshop on Technical Challenges for Dependable Robots in Human Environments, Pasadena, California, May 17-18, 2008

National Aeronautics and Space Administration May 17-18, 2008 Workshop on Robot Dependability 2 Presentation Overview The Need for Dependability in Robots for Space Applications The Flight Side: Mars Exploration Rovers –MER FSW Architecture Design Decisions for Reliability –MER FSW Development Procedures for Reliability The Research Side: CLARAty –What is CLARAty? –Techniques used by CLARAty to Improve Software Reliability

National Aeronautics and Space Administration The Need for Dependability in Space Applications If a robot fails after leaving Earth, repair may be difficult or impossible If a robot fails during a critical time, such as descent and landing, the results can be disastrous Research efforts need to consider dependability to address scalability and facilitate integration into flight applications May 17-18, 2008 Workshop on Robot Dependability 3

National Aeronautics and Space Administration The Flight Side: Mars Exploration Rovers (MER)

National Aeronautics and Space Administration MER Flight Software Design Decisions Overview Software Decomposition –Object Oriented –Hierarchical state machines Runtime Model –Multiple tasks –Asynchronous message passing –Multi-layered fault protection –Automation of critical behaviors Software Implementation Principles –Coding standards –Static memory allocation –Design patterns May 17-18, 2008 Workshop on Robot Dependability 5

National Aeronautics and Space Administration MER Flight Software Decomposition Object-oriented style design, with emphasis placed on interfaces, encapsulation, and modularity Objects implemented as hierarchical state machines May 17-18, 2008 Workshop on Robot Dependability 6

National Aeronautics and Space Administration MER Flight Software Runtime Model (Part 1) Multiple tasks on a RTOS, with task priorities reflecting criticality and required response time Asynchronous message passing as the principle means of communication between objects, and the messages drive the heierarchical state machines Multi-layered fault protection –Hardware –Software handling Minor faults handled locally Major faults handled by separate Health and Fault Protection tasks –Command sequences May 17-18, 2008 Workshop on Robot Dependability 7

National Aeronautics and Space Administration MER Flight Software Runtime Model (Part 2) Critical mission behaviors are automated –Entry, Descent, and Landing –Communication –Fault Handling –Attitude Acquisition –Wakeup and Shutdown Allows faster handling than ground-in-the-loop and richer handling than either hardware or command sequences May 17-18, 2008 Workshop on Robot Dependability 8

National Aeronautics and Space Administration MER Flight Software Implementation Principles Dynamic memory allocation is severely limited to prevent heap fragmentation and allow memory problems to be discovered during development Coding conventions, rules, and guidelines. Design patterns (message passing, command handling, hardware device objects) Reference: –Glenn E. Reeves & Joseph F. Snyder "A Overview of the Mars Exploration Rovers' Flight Software" 2005 IEEE International Conference on Systems, Man and Cybernetics Waikoloa, Hawaii, October 10-12, 2005 May 17-18, 2008 Workshop on Robot Dependability 9

National Aeronautics and Space Administration May 17-18, 2008 Workshop on Robot Dependability 10 Flight Software Development Procedures Use only mission-proven or thoroughly tested technologies Formal design reviews –Inheritance reviews –Design reviews –Implementation reviews Code reviews –Independent developer reviews code –Static code analyzers used

National Aeronautics and Space Administration Rigorous Flight Software Testing Unit Testing: –Extensive testing of each module in isolation by the developer Regression Testing: –Integrated module testing by a dedicated test team after new modules are integrated System Testing: –Project wide rehearsals of expected mission scenarios –Can last several days where several different activities would be tested in the manner they would be used in the mission –All communication is done during communication passes May 17-18, 2008 Workshop on Robot Dependability 11

National Aeronautics and Space Administration The Research Side: CLARAty Coupled Layer Architecture for Robotic Autonomy

National Aeronautics and Space Administration May 17-18, 2008 Workshop on Robot Dependability 13 What is CLARAty? CLARAty is a unified and reusable software that provides robotic functionality and simplifies the integration of new technologies on robotic platforms A research tool for technology development and maturation

National Aeronautics and Space Administration May 17-18, 2008 Workshop on Robot Dependability 14 Problem and Approach Problem: –Difficult to share software/algorithms across systems –Different hardware/software infrastructure –No standard protocols and APIs –No flexible code base of robotic capabilities Objectives –Improve software reliability by enabling and encouraging reuse –Provide mature and dependable robotic infrastructure Simplifies creation of robotic applications Simplified integration of new technologies Enables research efforts to concentrate on the algorithm being investigated –Mature software used on heterogeneous robots, enabling more reuse Same interfaces to high-level components

National Aeronautics and Space Administration Interoperability: Software & Hardware May 17-18, 2008 Workshop on Robot Dependability 15 Rover Acquire Image Goto Target 1 Swappable Algorithm or Robot Adaptation ATRV Jr. Rocky 7 ROAMS Functional Layer Decision Layer Declarative Activity Functional Abstraction Rocky 8 Explore Site Goto Target 3 Deploy Instrument Acquire & Analyze Navigator Morphin Locomotor R8_Model Motor R8_Motor Pose Estimator SAPP Pt Cloud IMU ISIS Target Tracker Falcon Camera 1394 Cam Stereovision JPLV

National Aeronautics and Space Administration May 17-18, 2008 Workshop on Robot Dependability 16 Technology Tasks Technology Development, Integration and Validation CLARAty Jet Propulsion Lab CMU NASA ARC U. Minnesota R&TD, MDS, DRDF Competed Mars Technology Program Other NASA Programs Rover Simulation ROAMS Rover Hardware JPL Internal Programs Flight Focused Technology Programs Science Instruments Simulation Operator Interface Legacy Algorithms Flight Algorithms NASA Centers and Universities Technology Tasks NASA Centers and Universities Technology Tasks NASA Centers and Universities Technology Tasks NASA Centers and Universities Technology Tasks Technology Validation Tasks Technology Tasks Technology Tasks

National Aeronautics and Space Administration Techniques We Use in CLARAty Some of the techniques that we have explored to improving software reliability are: –Improved processes and procedures for software development –Unified coding conventions –Static code analysis and validation tools –Increased software reliability through reuse –Formal technology validation –Automated nightly regression testing (to a limited extent) –Fault-tolerant software May 17-18, 2008 Workshop on Robot Dependability 17

National Aeronautics and Space Administration Back-up Slides

National Aeronautics and Space Administration May 17-18, 2008 Workshop on Robot Dependability 19 Acknowledgements CLARAty Team (multi-center) Jet Propulsion Laboratory Ames Research Center Carnegie Mellon University University of Minnesota

National Aeronautics and Space Administration May 17-18, 2008 Workshop on Robot Dependability 20 Current CLARAty Core Team NASA Ames Research Center –Lorenzo Flueckiger –Hans Utz Carnegie Mellon University –Reid Simmons –David Apelfaum –Nick Melchior University of Minnesota –Stergios Roumeliotis –Nikolas Trawny –Anastasios I. Mourikis Jet Propulsion Laboratory –Issa A.D. Nesnas –Hari Das Nayar –Tara Estlin –Richard Petras –Daniel Gaines –Robert Steele –Daniel Clouse –Michael McHenry –Khaled S. Ali –Mihail Pivtoraiko –Kelly Breed Affiliates –Jeffrey Edlund Full Credits for all Developers and Contributors at: