1. 1 Dr. Vijay Raghavan Defense Advanced Research Projects Agency Information Exploitation Office Network Embedded Systems Technology (NEST) December 17, 2003 Extreme Scaling Program Plan

2. 2 Topics Extreme Scaling Overview Workshop Action Items Project Plan

3. 3 Extreme Scaling Overview [Insert background/overview information here…]

4. 4 Workshop Action Items 1.Concept of Operations 2.Experiment Site Selection 3.System Design and Tier II Architecture 4.Xtreme Scaling Mote (XSM) Design 5.Xtreme Scaling Mote Sensor Board Design 6.Super Node Design 7.Application Level Fusion 8.Remote Programming 9.Localization 10.Power Management 11.Group Formation 12.Simulation 13.Support Infrastructure

5. 5 1. Concept of Operations Primary Effort Surveillance of Long, Linear, Static Structures Pipeline Problem:  Too vast an area for limited personnel resources (mobile guards)  Hostile actions: Destruction (explosives) Damage to pumps and transformers Stripping of copper power lines (for pumps) Operational Need:  Reliable automated surveillance to detect movement in security zone FY04 Experiment:  Sense movement of personnel and/or vehicles toward the pipeline  Track the movement and the stop/start of movement Damage in Iraq

6. 6 Pipeline Pump Station Security Zone 20 km 1 km Detection and tracking of personnel Detection and tracking of vehicles Detection of unknowns ? Guard Force Alerted Mobile Patrol Reaction Force Pipeline 1. Concept of Operations (cont.) Primary Effort

7. 7 IED Enemy Observation Point (OP) Similar Long, Linear, Static Structures Surveillance of Supply Routes:  Detect potential ambush sites:  Personnel w/shoulder fired weapons  Improvised Explosive Devices (IEDs) FY04 Experiment:  Sense movement of personnel/vehicles toward supply route and then:  They remain near a point  They remain for a while and then leave  Sense suspicious movement on the road Wire to OP More Related Efforts: Border Patrol, surveillance around air base/ammo point 1. Concept of Operations (cont.) Related Efforts IED

8. 8 2. Experiment Site Selection Characteristics Relatively flat, open area –Easier to survey/mark off 1 km x 20 km site –Easier to deploy/recover sensors –Easier for observers to see large section of experiment site No forests No large physical obstructions (e.g., buildings) to line-of- site communications –Small obstructions (e.g., small rocks) okay Relatively good weather (little rain, light winds, etc.) –Sensors can stay out for days Military base –Site can be guarded –Sensors deployed on day 1 and remain in place until end of experiment (days later) –Potential for personnel to support deployment/recovery of sensors

9. 9 Being used for DARPA SensIT effort (Feb 2004) 150 miles NE of Los Angeles Encompasses 1.1 million acres of land in California's upper Mojave Desert, ranging in altitude from 2,100 to 8,900 feet Varies from flat dry lake beds to rugged piñon pine covered mountains Weather should be consistent Summer will be hot 2. Experiment Site Selection (cont.) Primary Candidate Site Naval Air Weapons Facility, China Lake

10. 10 2. Experiment Site Selection (cont.) Other Candidate Sites Fort Bliss, TX (near El Paso, TX) Nellis AFB (near Las Vegas, NV) NAS Fallon (near Reno, NV) Marine Corps Air to Ground Combat Center, 29Palms, CA Eglin AFB (near Pensacola, FL)

11. 11 3. System Design and Tier II Architecture RoutingRouting Matched filter Group management (for multilateration) Group management (for multilateration) LocalizationClusteringLocalizationClustering Time sync Power management

12. 12 3. System Design and Tier II Architecture (cont.) Localization, time sync, and multihop reprogramming design/testing to be joint for Tier I & II –e.g., localization design to include how to associate XSMs with super nodes; to maintain clusters in stable way; etc. Reliable communication needed for exfil unicasts and for localization/multihop reprogramming broadcasts –because hops are many, comms are almost-always-off, & latency requirement is tight Testing 802.11 indoors problematic for some environments

13. 13 THREADED HALVES BATTERIES BALL SIMILAR TO PET TOYS, SIZE OF GRAPEFRUIT ANTENNA PIR SENSOR MICROPHONE 4. Xtreme Scaling Mote Design Design Concept #1

14. 14 STAKE ANTENNA BATTERIES PIR SENSOR MICROPHONE STAKE IN THE GROUND, SENSOR ON TOP 4. Xtreme Scaling Mote Design (cont.) Design Concept #2

15. 15 MICROPHONE PIR SENSOR ANTENNA SODA CAN WITH SENSOR ON TOP 4. Xtreme Scaling Mote Design (cont.) Design Concept #3

16. 16 4. Xtreme Scaling Mote Design (cont.) Proposed Changes Keep Daylight and Temperature Sensor New Mag Circuit - 2-Axis, Amplifier, Filter, and Set/Reset Circuit using HMR1052 or HMR1022 Anti-alias filter on Microphone, no tone-detector 1 PIR sensor with as big a FOV as possible - either Perkin Elmer or Kube No Adxl202 but the pads will be there so a few could be populated Loud Buzzer –Needs research on size, voltage requirements

17. 17 4. Xtreme Scaling Mote Design (cont.) Known Issues Loud (> 90dB) Sounders –Voltage Requirements – 9-12V –Size (1” x 1”) –What Frequency – 2, 4KHz –Tone Detection? PIR Field of View –Daylight detection circuit Standardize Battery Selection –Will improve battery voltage accuracy Watchdog Timer / Remote Programming –Needs significant testing –Preload Mote with Stable TinyOS+Watchdog+XNP

18. 18 4. Xtreme Scaling Mote Design (cont.) Proposed Phase 1 Build 20-30 New Sensor Boards and Distribute to Group for use with existing MICA2 –Late January In parallel, review package design

19. 19 5. Xtreme Scaling Mote Sensor Board Design Candidate sensor suite at Tier I PrimarySecondary MagnetometerTemperature PIR Seismic AcousticHumidity Buzzer Barometer LEDs Infrared LED Issues –What analog electronics to include to reduce sampling rates, (e.g., tunable LPF, integrator) –A to D lines –Packaging to address wind noise/eddies, and actuator visibility –Early API design for sensors and their TinyOS (driver) support –Early testing of sensor interference issues –Sensors at Tier II: GPS and ?

20. 20 6. Super Node Design Candidates (Crossbow is doing fine grain comparison) –Stargate –iPAQ –Instrinsyc Cerfcube μPDA –AppliedData Bitsy, BitsyX –Inhand Fingertip3 –Medusa MK-2 Evaluation Criteria –802.11 wireless range (need several hundred meters) –networking with motes –development environment –programming methodology support, simulation tool support –availability of network/middleware services –platform familiarity within NEST Issues –PDA wakeup times longer?

21. 21 7. Application Level Fusion Features to include in application data from Tier I  Tier II –energy content –signal duration –signal amplitude and signal max/min ratio –angular velocity –angle of arrival Issues –Tradeoffs Tier I XSM density of active nodes Tier II detection accuracy (to minimize communication power requirement) Tier III detection latency –Early validation of environment noise and intruder models –Early validation of influence field statistic w/ acoustic sensors and PIR –Might need CFAR in space and time

22. 22 8. Remote Programming The many levels of reprogramming: –In order of increasing cost and decreasing frequency: Re-configuration –Highly parameterized modules a big win for midterm demo Scripting –Good for rapid prototyping of top-level algorithmic approaches Page-level diffs for small changes to binary code –Pages are unit of loss, recovery, & change; acks for reliability –Many possible design choices for repair / loss recovery protocol –Space-efficient diffs will require some thought, compiler support Loading a whole image of binary code –Optimizations: pipelining a big win; but beware: many optimizations that sound good don’t help as much in practice as you might think (see, e.g., Deluge measurements) –Claim: All levels should use epidemic, anti-entropy mechanisms Good for extreme reliability: deals with partitions, new nodes, version sync Good for extreme scalability: avoids need for global state –Tradeoff: flexibility of reprogramming vs. reliability of reprogramming Want a minimal fail-safe bootloader that speaks reprogramming protocol Good for reliability: if you’re really sick, blow away everything but bootloader Discussion topic: How much do we hard-code in the bootloader?

23. 23 9. Localization Motes with the UIUC customized sounder Distance Estimation with the Custom Sounder Distance estimates based on Time Difference of Arrival Sound and Radio signals used Median value of repeated measurements used to eliminate random errors

24. 24 9. Localization (cont.) Experimental Validation Demo Sensor Network Deployment Ft. Benning localization experiments Localization based on trilateration. Use more than three anchors for correction of systematic errors. Pick largest consistent cluster of different anchors’ distance intersections. Minimize sum of least squares Gradient descent search used. Fort Benning localization results: actual location versus estimated location Results Error correction is effective Mean location errors of 30cm (median error lower) Computations can be done entirely on the motes.

25. 25 9. Localization (cont.) Plans for Extreme Scaling Problem Localize nodes in 100x100m2 UIUC customized motes reliably measure only up to 20m using acoustic ranging Proposed solution: Multi-Hop ranging Algorithm Measure the distances between nodes Find a relative coordinate system for each node for nodes within acoustic range Find transformations between coordinate systems Find distance to an anchor node or find position in the anchor’s coordinate system Simulation Results Error accumulates slowly with more transformations. 100 nodes in a100x100m2 area Acoustic signal range: 18m 0 mean, 3σ=2m normal error hop count = number of transformations

26. 26 10. Power Management Super Node Design –Power management at least as important at Tier II as at Tier I –Key evaluation criterion for device selection Tier II Power Management Needs –Exploit mote to PDA interrupt wakeup –Low Pfa in detection traffic from supernodes to support almost- always-off communication –TDMA?

27. 27 11. Group Formation Service(s) to Support –Multilateration with gradient descent for distributed tracking and classification (at Tier II) –Reliable broadcast of information from Super Nodes to Xtreme Scaling Motes –Power managed, persistent(?) hierarchical routing (at Tier II) Issues –Stability of persistent clusters e.g., in mapping XSM motes to supernodes use unison/hysteresis –Stabilization of clusters tolerance to failures, displacement, layout non-uniformity

28. 28 12. Simulation EmStar Simulation Environment UCLA (GALORE Project) Software for StarGate and other linux-based microserver nodes for hierarchical networks –EmStar: seamless simulation to deployment and, EmView: extensible visualizer http://cvs.cens.ucla.edu/viewcvs/viewcvs.cgi/emstar/ –CVS repository of Linux and bootloader code-base for StarGate http://cvs.cens.ucla.edu/viewcvs/viewcvs.cgi/stargate/ –Stargate users mailing list http://www.cens.ucla.edu/mailman/listinfo/stargate-users

29. 29 12. Simulation (cont.) Programming Microservers: EmStar What is it? –Application development framework for microserver nodes –Defines standard set of interfaces –Simulation, emulation, and deployment with same code –Reusable modules, configurable wiring –Event-driven reactive model –Support for robustness, visibility for debugging, network visualization –Supported on StarGate, iPAQs, Linux PCs, and pretty much anything that runs Linux 2.4.x kernel Where are we using it? –NEST GALORE system: sensing hierarchy –CENS Seismic network: time distribution, s/w upgrade –NIMS robotics application –Acoustic sensing using StarGate + acoustic hardware Note: EmStar co-funded by NSF CENS, main architect Jeremy Elson

30. 30 12. Simulation (cont.) From {Sim,Em}ulation to Deployment EmStar code runs transparently at many degrees of “reality”: high visibility debugging before low-visibility deployment Scalability Reality

31. 31 12. Simulation (cont.) Real System each node is autonomous; they communicate via the real environment Real Node 1 Radio Topology Discovery Collaborative Sensor Processing Application Neighbor Discovery Reliable Unicast Sensors Leader Election 3d Multi- Lateration Audio Time Sync Acoustic Ranging State Sync Real Node n Radio Topology Discovery Collaborative Sensor Processing Application Neighbor Discovery Reliable Unicast Sensors Leader Election 3d Multi- Lateration Audio Time Sync Acoustic Ranging State Sync...

32. 32 12. Simulation (cont.) Simulated System the real software runs in a synthetic environment (radio, sensors, acoustics) Simulated Node 1 Radio Topology Discovery Collaborative Sensor Processing Application Neighbor Discovery Reliable Unicast Sensors Leader Election 3d Multi- Lateration Audio Time Sync Acoustic Ranging State Sync Simulated Node n Radio Topology Discovery Collaborative Sensor Processing Application Neighbor Discovery Reliable Unicast Sensors Leader Election 3d Multi- Lateration Audio Time Sync Acoustic Ranging State Sync... Very Simple Radio Channel Model Very Simple Acoustic Channel Model EMULATOR/SIMULATOR

33. 33 12. Simulation (cont.) Hybrid System real software runs centrally, interfaced to hardware distributed in the real world Simulated Node 1 Radio Topology Discovery Collaborative Sensor Processing Application Neighbor Discovery Reliable Unicast Sensors Leader Election 3d Multi- Lateration Audio Time Sync Acoustic Ranging State Sync Simulated Node n Radio Topology Discovery Collaborative Sensor Processing Application Neighbor Discovery Reliable Unicast Sensors Leader Election 3d Multi- Lateration Audio Time Sync Acoustic Ranging State Sync... EMULATOR/SIMULATOR Radio

34. 34 12. Simulation (cont.) Interacting with EmStar Text/Binary on same device file –Text mode enables interaction from shell and scripts –Binary mode enables easy programmatic access to data as C structures, etc. EmStar device patterns support multiple concurrent clients –IPC channels used internally can be viewed concurrently for debugging –“Live” state can be viewed in the shell (“echocat –w”) or using emview

35. 35 13. Support Infrastructure Important techniques for monitoring, fault detection, and recovery: –System monitoring: big antenna was invaluable during midterm demo –Network health monitoring: e.g., min, max transmission rates –Node health monitoring: e.g., ping; query version, battery voltage, sensor failures; reset/sleep commands –Program integrity checks: e.g., stack overflow –Watchdog timer: e.g., tests timers, task queues, basic system liveness –Graceful handling of partial faults: e.g., flash/eeprom low voltage conditions –Log everything: use Matchbox flash filesystem + high-speed log extraction –Simulation at scale: tractable to simulate 1000’s of nodes; use it!

36. 36 13. Support Infrastructure (cont.) A possible network architecture: –Claim: the key to extreme scaling is hierarchy: 100 networks of 100 motes (+ a network of 100 Stargates?), not a network of 10,000 motes –“Everything runs TinyOS”: enables simulation of all levels of hierarchy –Consider adding high-speed backchannel (e.g., 802.11) to a subset of nodes for debugging, monitoring, log extraction Topics for discussion: –What is the role of end-to-end fault recovery? (e.g., watchdog timers) –What can we learn from theory? (e.g., Byzantine fault toler., self- stabilization) –Logging and replay mechanisms, for after-the-fact debugging? –Quantity vs. quality tradeoff? (Choice between focusing on making individual nodes more reliable, vs. adding more nodes for redundancy)

37. 37 Project Plan [Insert Project Plan slides here…]

38. 38 BACKUP / MISCELLANEOUS SLIDES

39. 39 Preliminary Program Plan Roles and Responsibilities Technology Development Middleware Services  Clock Sync (UCLA,OSU)  Group Formation (OSU, UCB)  Localization (UIUC)  Remote Programming (UCB)  Routing (OSU, UCB)  Sensor Fusion (OSU)  Power Management (UCB)  Relay Node Services (UCLA) Middleware Services  Clock Sync (UCLA,OSU)  Group Formation (OSU, UCB)  Localization (UIUC)  Remote Programming (UCB)  Routing (OSU, UCB)  Sensor Fusion (OSU)  Power Management (UCB)  Relay Node Services (UCLA) Display Unit Ohio State Display Unit Ohio State Application Layer Ohio State UC Berkeley Application Layer Ohio State UC Berkeley MAC Layer UC Berkeley MAC Layer UC Berkeley Xtreme Scaling Mote Crossbow Technology Xtreme Scaling Mote Crossbow Technology Relay Node Crossbow Technology Relay Node Crossbow Technology Systems Integration Ohio State Systems Integration Ohio State Transition Partners USSOUTHCOM, U.S. Customs & Border Protection, USSOCOM, AFRL Transition Partners USSOUTHCOM, U.S. Customs & Border Protection, USSOCOM, AFRL Sensors CrossbowSensors Operating System UC Berkeley Operating System UC Berkeley Application Tools Ohio State UCLA UC Berkeley Application Tools Ohio State UCLA UC Berkeley Auxiliary Services  Testing (OSU, MITRE, CNS Technologies)  Monitoring, logging, and testing infrastructure (UCB, OSU)  Evaluation (MITRE)  Logistics, site planning (CNS Technologies, OSU)  ConOps development (Puritan Research, CNS Technologies, SouthCom, US Customs & Border Protection, MITRE, OSU) Auxiliary Services  Testing (OSU, MITRE, CNS Technologies)  Monitoring, logging, and testing infrastructure (UCB, OSU)  Evaluation (MITRE)  Logistics, site planning (CNS Technologies, OSU)  ConOps development (Puritan Research, CNS Technologies, SouthCom, US Customs & Border Protection, MITRE, OSU) GUI Ohio StateGUI Simulation tools (UCB, UCLA, Vanderbilt, OSU)