1. Dr. Vijay Raghavan Defense Advanced Research Projects Agency Information Exploitation Office Network Embedded Systems Technology (NEST) November 12, 2003 Workshop on Extreme Scaling Arlington, VA

2. 2 UNCLASSIFED – FOR OFFICIAL USE ONLY Meeting Purpose  To discuss the NEST plan to create a robust 10,000 node autonomous ad hoc sensor network in an operationally relevant environment for demonstrating the capabilities in the monitoring and protection of long linear structures, by the end of FY ’04  Analysis: To discuss the NEST plan  Evaluate platform, identify technological challenges, identify crucial tasks, formulate metrics, and create a schedule Robust 10,000 node autonomous ad hoc sensor network  Not a toy demonstration, two orders of magnitude larger than before Operationally relevant environment  Not a laboratory demonstration, not a simulation, it will be in a real system Demonstrating the capabilities  Okay, not a real fieldable system, but having all the capabilities for the mission (TRL 4.5) Monitoring and Protection  The network detects, tracks, and classifies intruders Long linear structures  Pipelines, borders, … End of FY ’04  Okay, we can extend this a little, but …

3. 3 UNCLASSIFED – FOR OFFICIAL USE ONLY Deliverables of the meeting 1.Initial specification of hardware 2.Identification of tasks to be performed 3.Schedule of activities for FY ’04

4. 4 UNCLASSIFED – FOR OFFICIAL USE ONLY Agenda TIMETOPIC 0830 – 0900Breakfast 0900 – 0930Welcome / Introductions / Meeting Purpose (DARPA) 0930 – 1000Extreme Scaling Overview (Puritan Research) Background Operational Challenges A 0 th Order NEST Solution 1000 – 1045Technical Challenges in Extreme Scaling (UC Berkeley) Lessons learned Preliminary Analysis of Challenges 1045 – 1100Break 1100 – 1145Preliminary Technical Approach (The Ohio State University) Overview of “A Line in the Sand” Sensor Requirements for the Extreme Scaling Experiment System Architecture, Process 1200 – 1300Lunch 1300 – 1330Preliminary Program Plan (DARPA) 1330 – 1430Extreme Scaling Planning Session I (DARPA) Review Operational Scenario Discussion of Technical Challenges 1430 – 1500Break 1500 – 1630Extreme Scaling Planning Session II (DARPA) Refine Program Plan Identify Issues/Action Items Wrap Up / Next Steps (DARPA)

5. 5 UNCLASSIFED – FOR OFFICIAL USE ONLY 7/9/2003 Extreme Scaling-2004 Securing a Large Distributed Perimeter Norman Whitaker NEST Extreme Scaling Workshop November 12, 2003 Cano-Limon Pipeline Colombia

6. 6 UNCLASSIFED – FOR OFFICIAL USE ONLY N sensors Unit cost (N) Information flow (N) Unit Memory( N) Network Bandwidth(N) Exfiltration ~ N log N Information Flow Scaling

7. 7 UNCLASSIFED – FOR OFFICIAL USE ONLY

8. 8 300 km border IRAQ SYRIA AL QAIM IRAQ / SYRIA Border October 2003

9. 9 UNCLASSIFED – FOR OFFICIAL USE ONLY 25 km dirt road 500m Sensor node with 50m range Exfiltration node (PDA) 400 nodes/km 2 Both sides 1% of Total Laydown Schedule 3 month 6 month 9 month 12 month 1000 3000 6000 Full system demonstration Specs Dismounts – armed and unarmed Vehicles 2 second latency < $150/node Pfa < 1 alarm/day Pd = 0.95 at walking speed “Kansas Pipeline” – Canonical 10,000 Node Problem

10. 10 UNCLASSIFED – FOR OFFICIAL USE ONLY The Big Three Problems - MITRE  Sensor Range Require a magnetometer The sensor range needs to be ~1.3x the node spacing.  In this case ~65 m. Magnetometer ranges of 65 m are beyond the state of the art.  Earth’s magnetic “noise” exceeds the signal by ~10x.  Over-the-Air Loading This is a complex service:  Reliable multicast (i.e., with local retry).  May need group service. Very strong motivation towards incremental loading.  Some application may need to run while others load.  Implies dynamic linking.  Requires OS-style security

11. 11 UNCLASSIFED – FOR OFFICIAL USE ONLY The Big Three Problems - MITRE  Sentry Service (power) The required “on” subset of the network will be too large.  Alternative drives up required sensor range.  Extending system life by raising node density is inferior to just using bigger battery Need temporal sampling. Need power savings of ~100x. Need the almost always off communication model.  Maybe comm. deadline and comm. scheduling.

12. 12 UNCLASSIFED – FOR OFFICIAL USE ONLY 1.Timeline is too aggressive June 04100 December 041000 June 0510,000 2. Sensor mix: a. Magnetics unlikely to detect weapons at ranges > 1m b. PIR unlikely to be a cure-all c. Consider McKuen radars 3. Comms range seen as problematic 4. Need 4 X more PDA’s – Pfa / lifetime / latency very aggressive Feedback from 9-22-03 Conference Call

13. 13 UNCLASSIFED – FOR OFFICIAL USE ONLY 5.Need for rock-solid over air loading capability needed to demonstrate scalability – and this is not currently in hand. Architecture work needed. 6.Power is a huge problem for a demonstration -Always on needed to achieve Pfa -Debugging and software loads are battery intensive -Lifetimes are currently far too short 7.Solid packaging needed (“one touch only”) 8.Need a separate contractor to actually camp in desert and deploy sensors 9. Next tier PDA work needed, as well as highest tier (GUI) Feedback from 9-22-03 Conference Call (cont.)

14. 14 UNCLASSIFED – FOR OFFICIAL USE ONLY 10.Dynamic tagging seen as an alternate approach that demonstrates scalability without problems in static sensor laydown 11. Rush to production seen as a costly mistake. A disciplined, Step-by-step process with formalized acceptance criteria at each Stage was recommended to avoid a failure. Feedback from 9-22-03 Conference Call (cont.)

15. 15 UNCLASSIFED – FOR OFFICIAL USE ONLY 1. Self-contained sensors 2. Self-contained static sensor clusters 3. Sensor clusters with handoff interactions Systems that Scale Well

16. 16 UNCLASSIFED – FOR OFFICIAL USE ONLY 4. Sensor clusters with dynamic assignments 5. Robust systems that adjust to sensor dropouts, etc. 6. Systems which maintain only local state Systems that Scale Well (cont.)

17. 17 UNCLASSIFED – FOR OFFICIAL USE ONLY 1. Any system with a non-zero P fa per sensor node 2. Any non-robust system that requires multiple touches per sensor 3. Any system with performance that can be degraded by a malfunctioning, poorly placed, or imperfect node. 4. Any system that creates a disproportionate response to stimuli 5. Any system that does not plug into umbrella information systems Systems That Do Not Scale Well

18. 18 UNCLASSIFED – FOR OFFICIAL USE ONLY Conclusion  For most applications sensor nets are inherently scalable, but their implementations are often not  The path to practical scalable systems lies through careful extensible architecture work and bulletproof hardware/software design  It may be possible to demonstrate (but not fully test) a scalable system with fewer than 10,000 nodes

19. 19 UNCLASSIFED – FOR OFFICIAL USE ONLY Lessons from the UCB Wireless OEP Demo and Issues in Scaling Sensor Webs Cory Sharp, Robert Szewczyk, Massimo Franceschetti, Prabal Dutta, David Culler, Shankar Sastry NEST Extreme Scaling Workshop, Washington, DC 11/12/2003

20. 20 UNCLASSIFED – FOR OFFICIAL USE ONLY Lessons from UCB Demo  Scaling Issues we set out to tackle  Things that proved very valuable  Things that proved problematic  Things we wished we had  Monitoring and Restart from the Beginning  Scalable Network Reprogramming  Foundations of Scale  Recommendations

21. 21 UNCLASSIFED – FOR OFFICIAL USE ONLY Scaling Issues We Designed For

22. 22 UNCLASSIFED – FOR OFFICIAL USE ONLY Key Pragmatic Issues  Solve the specific problem within time and $ budget  Reliably  Time, Time, Time, Time  Time to design, prototype and test what you think is a working hardware & software solution Not at scale  Time to manufacture a large number of nodes Only get one shot (be conservative)  Time to test-fix-test-integrate-test-fix-… Don’t see scaling issues until late in the process ~1000 person-hours in last two weeks  Time to assemble, reassemble, … Human scaling limits

23. 23 UNCLASSIFED – FOR OFFICIAL USE ONLY Hardware  Driven by specific application plan Not generic experimental like Mica Usage factors  outside, robots running over it, in-sun, possibly wet, … Specific sensing modalities  Board designs, Geometric orientation, …  Mechanical Design is central (co-design) Appn => package => boards  Foresee the needs of dev, debug, deployment process Field UI, reset Assembly, re-assembly, recharging process  Minimize exposure  Lift radio off the ground

24. 24 UNCLASSIFED – FOR OFFICIAL USE ONLY PEG Mechanical Design Exposed components Watertight compartment ultrasound Mica2 dot mag sense power battery reflector Collision absorption O-ring Seal Good thermal characteristics

25. 25 UNCLASSIFED – FOR OFFICIAL USE ONLY Software  NEST service architecture as a beginning  Greater number => simpler function  Complete modularity of each subsystem Node positioning Sensing and report Mobile-to-mobile routing Pursuer hear and navigate  Not just source concept, but full interfaces to allow testing in the field of each without the others to blame  Delay Binding Wherever Possible Complete parameterization  Thresholds, update rates, ranges, …  Management services  Assume irregular, lossy connectivity

26. 26 UNCLASSIFED – FOR OFFICIAL USE ONLY  Lowest-tier nodes only performed entity detection Local processing, threshold detection, announce Simple Leader Election Simple aggregation for position estimation  Little calibration  Simple mobile-mobile reliable routing Solid tree rooted at landmark  Complexity consolidated in higher tier Disambiguation Trajectory Estimation Navigation Noise-adaptive control loop  Minimalism pays Nice to have solid theory to support doing very simple things Algorithmic Simplicity by Decomposition

27. 27 UNCLASSIFED – FOR OFFICIAL USE ONLY Steps that proved invaluable

28. 28 UNCLASSIFED – FOR OFFICIAL USE ONLY Management Services  Ident Ping, Version, Compile time  Command line scripting Check all report in Check config values  Config Set/Get each potential parameter  Automated behaviors on update Save/Restore to Flash Human override  Reset  Sleep / Wakeup

29. 29 UNCLASSIFED – FOR OFFICIAL USE ONLY Simple and Solid  Once a node is assembled, if it responds to Ping, it should work Reliability of quick test was key  Complete testing of each major component at near-scale in isolation Clean stimulus, observable response Combining was trivial  Powerful receiver to see what was going on in much of network

30. 30 UNCLASSIFED – FOR OFFICIAL USE ONLY Multiple Fall Backs  Localization: fix the position based on network address Transform of address Explicit Set Ranging  Routing Single hop to base 802.11 back channel to pursuer Evader-pursuer back channel Landmark routing  Excellent signal-strength based slow tree build  Sensing  Used most of them in testing Kept them in the code  Reduced stress along the way

31. 31 UNCLASSIFED – FOR OFFICIAL USE ONLY Things that Proved Problematic

32. 32 UNCLASSIFED – FOR OFFICIAL USE ONLY Hardware  Biggest Risk Factor Getting boards done and software working in time Budget allowed only one production shot Scale was limited by slowest production step  Antenna Internal coiled antenna was dismal => external whip Proved to be time consuming and problematic in assembly weak mount point (10% loss)  Recharging – required disassembly (see above)  Power conditioning introduced noise issues & leakage  Unforeseen interactions Radio + magnetometer Used the wrong kind of wire (duh!)  No self-guided assembly  Modularity, while has many values, has a significant cost  Too hard to see LEDs  no physical reset  Multiple antennas on pioneers interfered  Differential GPS black magic  TIME, TIME, TIME You don’t know what is wrong till late in the development

33. 33 UNCLASSIFED – FOR OFFICIAL USE ONLY Software  Network programming fell apart for large app on many nodes Dictated the development cycle Config was life saver Got solid single hop just after it was needed Current design has deep shortcomings (below)  Software Knowledge bottleneck Nice abstractions that too few people knew how to use  Underestimated the amount of glue code between the tiers  Wished we had Better Regression suites Logged Everything Much better operational deployment protocol and bookkeeping

34. 34 UNCLASSIFED – FOR OFFICIAL USE ONLY Towards the Extreme Scaling Effort

35. 35 UNCLASSIFED – FOR OFFICIAL USE ONLY Monitoring and Management of the Network  Determine what went wrong and take action Extremely simple and reliable Ensure liveness, preserve access to network nodes, help with fault diagnosis  Networking monitoring Enforce minimum and maximum transmission rates Verify minimum reception rate  Node health monitoring Beyond battery voltage – sensor data monitoring Failure of a sensor an indicator of node health Detectable failures impacting lifetime– GDI humidity and clock skew experience  Program integrity checks Stack overflow  Watchdog / deadman timer Require attention from different parts of the system, reset if abandoned Address many different time scales Test low-level system components – timers, and task queue to ensure basic system liveness First use: min reception rate  Partial system shutdown Flash/EEPROM writing under low voltage conditions, broken sensors, etc.  Fault handling Error logging and/or reporting

36. 36 UNCLASSIFED – FOR OFFICIAL USE ONLY Scalable Network Programming  Out of core algorithms, epidemic algorithms  Basic primitive – reliable page transfer Both broadcast and point-to-point transfer Fragment and transmit packets per page, vector ack, selective fix up, snoop, consistency checking Constraints: fit in ram, match with flash transfer size  Dissemination algorithms Fast push, pipeline series of pages in space Epidemic maintenance  Eliminate the “wrong image” problem  New version: propagate page diffs Potentially, multiple levels of pages to organize code and metadata System image: version number + vector of page hashes Organize image to reduce #pages that change  Linearization of functions, holes  Blue sky ideas default TinyOS network reprogramming kernel as the system restore checkpoint

37. 37 UNCLASSIFED – FOR OFFICIAL USE ONLY Conceptual Issues in Scaling

38. 38 UNCLASSIFED – FOR OFFICIAL USE ONLY  Percolation theory  Random graphs  Distributed computing  Distributed control  Channel physics  Distributed sampling  Network information theory  Network coding  Connectivity  Routing  Storage  Failures  Packet loss  Malicious behavior  Remote operation Theory Practice

39. 39 UNCLASSIFED – FOR OFFICIAL USE ONLY  Beyond the disc abstraction  Random connection model  Small world networks  Effect of interference  What is connectivity ? Example: Percolation theory Prob(correct reception)

40. 40 UNCLASSIFED – FOR OFFICIAL USE ONLY Connectivity in Practice Effective Region Transitional Region Clear Region Determines Node spacing many

41. 41 UNCLASSIFED – FOR OFFICIAL USE ONLY 1 Connection probability d Continuum percolation 2r New model New model d 1 Connection probability Beyond the disc abstraction

42. 42 UNCLASSIFED – FOR OFFICIAL USE ONLY CNP Squishing and squashing Shifting and squeezing (sporadic) long links help the connectivity process CNP=average number of connections per node needed for percolation What can we prove

43. 43 UNCLASSIFED – FOR OFFICIAL USE ONLY CNP Is the disc the hardest shape to connect overall? What about non-circular shapes?

44. 44 UNCLASSIFED – FOR OFFICIAL USE ONLY  Routing with occasional long links  Navigation in the small world  Connection with social science ! What about routing ? Prob(correct reception)

45. 45 UNCLASSIFED – FOR OFFICIAL USE ONLY Regular Random Small World Watts Strogatz (1998) can we use the few long links for routing? Small World Networks

46. 46 UNCLASSIFED – FOR OFFICIAL USE ONLY Kleinberg (2000)New model (2003) Nodes on the grid Fixed number of contacts Probability scales with distance Nodes at random on the plane Random number contacts in a given region Density scales with distance each node has only local information of the network connectivity Routing in a small world

47. 47 UNCLASSIFED – FOR OFFICIAL USE ONLY Long, Good Links Valuable

48. 48 UNCLASSIFED – FOR OFFICIAL USE ONLY  S T d Connections of z are Poisson points of density  - delivery occurs when msg is delivered within  to target Theorem about Routing in a Small World

49. 49 UNCLASSIFED – FOR OFFICIAL USE ONLY  S T d Build networks where the number of neighbors scales as 1/x 2 to obtain efficient routing Bottom line

50. 50 UNCLASSIFED – FOR OFFICIAL USE ONLY Red, Yellow, Green Lights: The Systems View  Define the application in levels Minimal demo Intermediate demo Ideal demo  Technical challenges required to reach a level determine the severity of the issue

51. 51 UNCLASSIFED – FOR OFFICIAL USE ONLY Red Light Challenges  Application definition Unchanging, realizable specifications Subject to sensor limitations (if any)  Hardware design For the specific application requirements Must have minimal human per-node interaction Requires multiple revisions – get it all done it time  Network reprogramming  Any suitable power source Sufficient for development and debugging  (De)composable services and architecture  Timeline for scalable solution  Manufacturing 10,000 nodes

52. 52 UNCLASSIFED – FOR OFFICIAL USE ONLY Yellow light challenges  Sensor limitations, sensing at scale Better sensors If no sensors: use active tagging  At-scale simulation  Enclosure with conveniences  Integration of heterogeneity  Watchdog timers

53. 53 UNCLASSIFED – FOR OFFICIAL USE ONLY Green light challenges  Security and demonstration  Possible renewable / rechargeable energy source  Regression suites  Logging of all demo data

54. 54 UNCLASSIFED – FOR OFFICIAL USE ONLY Long Poles in the Tent: Conceptual View  Architecture of the Networks: need to find stable routing trees for low latency comms using long links and short links. Heterogeneity with some less power constrained nodes, and many  Algorithms for sensor web localization poor  Theory based on “one shot” random networks. Networks are time varying and need stable features for routing.  Tracking of multiple entities  Distributed Control based on the quality of information

55. 55 UNCLASSIFED – FOR OFFICIAL USE ONLY Anatomy of a Sensor Node  Mechanical: Spike to ensure positive coupling with ground  Recharge in situ w/ solar cell and/or provide contacts?  Antenna placement > 3” off ground.  SMA antenna connector and real antenna (or maybe planar)  Crazy idea: why not stick a Yagi on there (Szewczyk & Culler) to ensure a long link (but will it be bi-directional?)  Power switch puts into ultra low power mode  Upside down camera and conical optics shields sensor from rain

56. 56 UNCLASSIFED – FOR OFFICIAL USE ONLY MobiLoc: Mobility Enhanced Localization  Some localization schemes Need special hardware support (Cricket, Calamari, AHLoS/Medusa) Do not need special hardware support (RSSI, hop count) Use mobile beacon(s) (NCSU)  Different idea: use range/angle to target, as measured by default sensors, to localize nodes

57. 57 UNCLASSIFED – FOR OFFICIAL USE ONLY Thinking Outside the Convex Lens: Catadioptrics  Most cameras have thousands of pixels – far too many for efficient processing on motes  Use optics to focus light and SW on area of interest  Research: Design a catadioptric mirror and lens (“magic”) that allows motes to focus on a very small number of pixels and follow tracks emerging from and terminating at those pixels  Use simple differencing type algorithms for detection  Only monitor pixels adjacent to current target position

58. 58 UNCLASSIFED – FOR OFFICIAL USE ONLY PIR: Cheap, Small, Long Range, Low Current  Use 4 PIR sensors, each with 90 degree FOV  Cost: < $5 each in 1K quantity  Range: ~ 10m on most sensitive setting  Use for passive vigilance? Duty cycle? Settling time? Draws 0.5mA @ 3V-9V  Problem: since it is a differential sensor, likely experiences washout in bright sunlight

59. 59 UNCLASSIFED – FOR OFFICIAL USE ONLY Lessons from A Line In the Sand for Project Echelon Anish Arora Mohamed Gouda Ted Herman Sandeep Kulkarni Mikhail Nesterenko November 12, 2003

60. 60 UNCLASSIFED – FOR OFFICIAL USE ONLY Brief Recap of A Line In The Sand

61. 61 UNCLASSIFED – FOR OFFICIAL USE ONLY Put tripwires anywhere—in deserts, other areas where physical terrain does not constrain troop or vehicle movement—to detect, classify & track intruders ALineInTheSand: ConOps

62. 62 UNCLASSIFED – FOR OFFICIAL USE ONLY  Hand-placed 90 pre-configured nodes at known locations, containing magnetometer & MIR sensors  System detects each intruder  System classifies each intruder as one or none of known intruder types, using distributed influence field concept  System tracks intruder as it moves past different nodes  System visualizes intruder on computer attached to relay Scenarios:  Soldier Carrying “Gun” Walks/runs across line on trails With another soldier on different trail & after other soldier on same trail  Car Runs across line on trails at 5-20mph  Civilian Walks across line on trail  The Line under Attack: Reliability of Middleware Services Several motes in a region are turned off & then on again Some motes are moved around Timing & Routing services are shown to self-heal 25’ 90 motes Trails 65’ Observation Tent Field Experiment Layout & Scenarios

63. 63 UNCLASSIFED – FOR OFFICIAL USE ONLY Our Focus Co-design of NEST-middleware & sensing/application components that is  accurate (i.e., low false negatives & low false positives)  robust (i.e., tolerates uncertain environment)  cost effective (i.e., using middleware on dense set of cheap sensors vs. sparse set of resource rich sensors)

64. 64 UNCLASSIFED – FOR OFFICIAL USE ONLY Lessons Learned from A Line In The Sand

65. 65 UNCLASSIFED – FOR OFFICIAL USE ONLY 1. The Network Is Unreliable  The application generates relatively high message traffic  Message loss is a dominant problem channel fading interference (interference range much larger than communication range) Problem: low end-to-end reliability  e.g.: if each link has 90% reliability, success rate over 6-hop routes is 50%  Message corruption is also an issue packets got corrupted in the air, even if CRC checking is used Problem: corrupt state of key middleware services, e.g. time sync  focus on dealing with problems resulting from network unreliability

66. 66 UNCLASSIFED – FOR OFFICIAL USE ONLY Impact on Application  Influence field concept for classification easily yields accuracy when network is reliable  As network unreliability increases, accurate classification becomes harder fewer messages  lesser separation between target classes lost messages  object can be classified as multiple smaller objects  application design must handle network unreliability late, out-of-order messages, e.g. by maintaining network time at classifier network jitter, e.g. by tuning latency and classification thresholds false positives and outliers

67. 67 UNCLASSIFED – FOR OFFICIAL USE ONLY Impact on Network Protocols  Added implicit-ACK based communication protocol to increase reliability each message has ACK field, which presents “aggregated picture” of local queue ACK-based timeout & retransmit used to deal with loss ACK-based flow control used to reduce contention implicit ACKs save bandwidth & reduce contention as well as delay  increased message reliability yields tradeoff with increased delay parameter tuning for sweetspot was challenging: 50% delivery within 13s protocol choice still under study: TDMA experiments underway

68. 68 UNCLASSIFED – FOR OFFICIAL USE ONLY Aside: TDMA  Developed TDMA schemes for grid based networks  Schemes are optimized for broadcast, convergecast, & local gossip  Simulation results show : 100% end-to-end delivery Slightly increased delay over CSMA based algorithms that suffer significant end-to-end loss

69. 69 UNCLASSIFED – FOR OFFICIAL USE ONLY Impact on Network Protocols (contd.)  Redesigned time synch to be robust wrt periods of poor connectivity used a structure-less approach, to :  avoid dealing with structure partitioning  exploit path diversity (converges fast too) used message redundancy & application level reasoning to deal with corruption side effect: can handle mobility of motes  increased synchronization robustness yields tradeoff with decreased accuracy (~100us/hop) statistical skew estimates improve accuracy (~15us/hop) but computation is costly some timestamps still get corrupted in spite of replication

70. 70 UNCLASSIFED – FOR OFFICIAL USE ONLY 2. Unanticipated Bad Behaviors Occur  Unpredictable behavior of some nodes occurred (& was hard to avoid), due to various reasons: incorrect program download depleted battery levels debonded or overheated sensors …  Effect is often serious, e.g., drowns out “good” traffic or compromises key services  detect & contain/correct Byzantine & transient misbehavior based on local invariants or contracts at node/service levels; e.g. blocked nodes that communicated events at a rate higher than normal

71. 71 UNCLASSIFED – FOR OFFICIAL USE ONLY 3. Testing should be at Scale  Sevices/apps designed & tested for n nodes do not work for 10n nodes !  E.g., reliable communication (scaling in network traffic)  95% reliability when tested with 16 nodes  reduced to 50% at 100 nodes with increased background & application traffic  E.g., time synchronization (scaling in processing complexity)  improved accuracy when tested with 8-10 nodes, by handling in radio stack layer the offset calculations for timesync messages  failed at higher scale : event loss → state corruption → arbitrary jumps in timesync  at design time, simulate at target scale develop traffic model generator, for each service, at target scale  btw, event loss is fundamental to platform ! both node & network capacity pushed to limits → source of bad behavior

72. 72 UNCLASSIFED – FOR OFFICIAL USE ONLY 4. System Tuning should be Easy  Achieving co-design objectives involved tuning ~15 service parameters transmission power level, number of retransmissions, periodic service frequencies, application thresholds… reprogramming modes for each parameter change is overkill  Scalable, distributed service for over-the-network monitoring/controlling of application parameters e.g., our walkabout ‘Commander/Query’ service automate addition of monitoring interfaces for parameters of each component

73. 73 UNCLASSIFED – FOR OFFICIAL USE ONLY 5. Reprogramming  During reprogramming, loss of program capsules is random, not bursty some capsules are lost due to source errors  all receivers miss them most other capsules are lost due to receiver errors  only a small subset of receivers miss them retransmission requests from motes often lost due to collision  Reduce reprogramming requests, especially when some capsules are lost only by a small subset of motes e.g., FEC-based components for reducing retransmission → most lost messages are recovered even while programming several motes at once currently evaluating several approaches for multihop case

74. 74 UNCLASSIFED – FOR OFFICIAL USE ONLY 6. Other Challenges  Micropower Impulse Radar (MIR) interfere with other MIRs no dithering on RF pulse timing or use of pseudo-random codes interference at close range (a few feet)  Non-deterministic interference between magnetometer & MIR magnetometer failed to detect when MIR physically present (but not on) MIR eventually failed to detect in the presence of magnetometer did not use a schedule to coordinate different sensors  Mechanical thermal effects severely reduced performance heavy rain took a mote’s life (and left a smoke trail) two screws is not enough  Operating System felt the lack of static analysis tools for determining if tasks are “schedulable”

75. 75 UNCLASSIFED – FOR OFFICIAL USE ONLY 8. Signal Propagation  Radios experience ground absorption effects when low-lying  Poor antenna orientation/polarization causes significant problems  Poor antenna choice (tinned copper wire)  Transmission line effects: impedance mismatched (Notre Dame)  Surface topography has serious effects (flat vs. peaks & valleys)  Network connectivity varies  Attenuation exponent is not constant  Magnetometer sensitivity decreases in the presence of ferro- magnetic surroundings (e.g. in a parking garage around rebar)  non-isotropic, non-deterministic, and time-varying

76. 76 UNCLASSIFED – FOR OFFICIAL USE ONLY 8. Signal Processing  Noise is not Gaussian, Laplacian, or IID  Noise is correlated  Noise mean and variance is time-varying (SNR is time-varying)  Signal is not deterministic  Signal has many nuisance parameters  Signal processing capability severely limited on motes  Difficult to distinguish signal from correlated noise  Sensors were not calibrated prior to deployment  Sensors drifted and became desensitized over time  Signal bias and scale unknown  MIR digital sensor outputs just plain don’t work and required statistical signal processing on analog signal to make usable   P FA   P D ; conversely  P D   P FA  sensing and signal processing require serious effort  node capacity is already being taxed

77. 77 UNCLASSIFED – FOR OFFICIAL USE ONLY Need CFAR and Hysteresis to  P FA,  P D  Bottom Line: Need adaptive signal processing.

78. 78 UNCLASSIFED – FOR OFFICIAL USE ONLY Echelon

79. 79 UNCLASSIFED – FOR OFFICIAL USE ONLY Sensing  Sensors Magnetometer Passive Infrared Acoustic Seismic Radar

80. 80 UNCLASSIFED – FOR OFFICIAL USE ONLY Sensing Options

81. 81 UNCLASSIFED – FOR OFFICIAL USE ONLY 2-Tier Architecture: Section  Line Lower Tier: Section  Size: 100 motes multihop connected (upto 5-7 hops) to 1-2 aggregation devices (iPAQ)  Sensing range:10m  Comm. range: 20m or more (for mote) 1km (for iPAQ, 102.11g or a class protocol)  Capability: detect & classify targets at line periperhy possibly in conjunction with neighboring sections  iPAQ role in signal processing / application tasks a small number (e.g., 2) of individual targets  Robust wrt: mote (&sensing ) loss path loss to ideal aggregation station

82. 82 UNCLASSIFED – FOR OFFICIAL USE ONLY Higher Tier: PDA network  Multi-hop PDA network glues together sections into a line it’s convenient to logically group 1-hop PDA-connected sections into a segment segment ~1km long segments participate in classification & tracking (of a somewhat larger number of targets) exfil to demonstrator node

83. 83 UNCLASSIFED – FOR OFFICIAL USE ONLY Configuring the Line  Higher density at perimeter  Simplest: Density only at perimeter  Grid: Sparse/Dense coverage  Lattice:

84. 84 UNCLASSIFED – FOR OFFICIAL USE ONLY Milestones

85. 85 UNCLASSIFED – FOR OFFICIAL USE ONLY Process  Always have a working prototype (continuous refinement)  Flexible, friendly, & organic collaboration  Regular interactions by phone & on some occasions in person  Open process, especially since zero-maintenance during project lifetime  design of packaging, sensing, & architecture  Single place to access & share materials  Liasons per team

86. 86 UNCLASSIFED – FOR OFFICIAL USE ONLY Concerns  Volunteers for signal processing/application development ?  Underprovisioning of sensor density of cost for infrastructure, storing, transporting, testing, controlling access, maintenance, space, programming, management of time  Overprovisioning of communication range current generation of motes will be pushed to its limits of sensing & communication capabilities

87. 87 UNCLASSIFED – FOR OFFICIAL USE ONLY Preliminary Program Plan  Schedule/Milestones  Roles and Responsibilities  Deployment Considerations for Extreme Scaling Experiment  System Metrics  Technology Transition

88. 88 UNCLASSIFED – FOR OFFICIAL USE ONLY Preliminary Program Plan Schedule and Performers Activity Primary Performer System Analysis Define Operational Scenario Define Operational/Technical Rqmts. Document System Architecture Document Performance Metrics DARPA OSU System Design Design System Hardware Design System Software Crossbow OSU System Development Enhance XSM Algorithms/Software Enhance Relay Node Software Enhance Display Unit Software Developers System Production XSM Production Relay Node Production Display Unit Production Crossbow OSU System Integration Integrate XSM / Relay Node / Display Unit Components OSU System Testing/Demonstration System Component Testing Preliminary Integration Testing Final Integration Testing System Demonstrations Developers OSU Program Management Extreme Scaling Workshop IPT Meetings Status Conference Calls DARPA Nov. ’03 Dec. ’03 Jan. ’04 Feb. ’04 Mar. ’04 Apr. ’04 May ’04 Jun. ’04 Jul. ’04 Aug. ’04 Sep. ‘04 Build 1 Delivered Build 2 Delivered Build 3 Delivered Component Demo Integrated Demo #1 Final Demo Integrated Demo #2

89. 89 UNCLASSIFED – FOR OFFICIAL USE ONLY Technology Development Preliminary Program Plan Roles and Responsibilities Middleware Services  Clock Sync (OSU, UCLA)  Group Formation (OSU, UCB)  Localization (UIUC)  Remote Programming (UCB)  Routing (OSU, UCB, UIUC)  Sensor Fusion (OSU)  Sentry (UCB, UVA) Middleware Services  Clock Sync (OSU, UCLA)  Group Formation (OSU, UCB)  Localization (UIUC)  Remote Programming (UCB)  Routing (OSU, UCB, UIUC)  Sensor Fusion (OSU)  Sentry (UCB, UVA) Display Unit Ohio State Display Unit Ohio State GUI/Application Layer Ohio State UC Berkeley GUI/Application Layer Ohio State UC Berkeley MAC Layer UC Berkeley MAC Layer UC Berkeley Xtreme Scaling Mote Crossbow Technology Xtreme Scaling Mote Crossbow Technology Relay Node Crossbow Technology Relay Node Crossbow Technology Systems Integration Ohio State Systems Integration Ohio State Transition Partners USSOUTHCOM, U.S. Customs & Border Protection, USSOCOM, AFRL Transition Partners USSOUTHCOM, U.S. Customs & Border Protection, USSOCOM, AFRL Military Consultants CNS Technologies, DBBL, Select Graybeards Military Consultants CNS Technologies, DBBL, Select Graybeards Sensors VariousSensors Operating System UC Berkeley Operating System UC Berkeley Application Tools Ohio State UCLA UC Berkeley Application Tools Ohio State UCLA UC Berkeley Testing and Evaluation Ohio State, MITRE, CNS Technologies Testing and Evaluation Ohio State, MITRE, CNS Technologies

90. 90 UNCLASSIFED – FOR OFFICIAL USE ONLY Preliminary Program Plan Roles and Responsibilities  Systems Integration Define system architecture and hardware/middleware services required Build and maintain integration testbed Integrate system components from Technology Developers Test, verify, and validate system components and overall system Manage field testing/evaluation  Technology Development Develop system components (both hardware and software) Fabricate system hardware components (including packaging) Develop and test algorithms and middleware services  Testing and Evaluation Plan field testing/evaluation Schedule testing facilities (e.g., MOUT site, other DoD facility) Support system deployment at testing facilities Conduct system training as necessary Define system metrics and document system performance  Transition Partners Support CONOP and TTP development Facilitate meetings between DARPA and User Communities Support technology transition activities (e.g., development of MOAs/MOUs)  Military Consultants Support CONOP and TTP development Interact with System Integrator, Technology Developers, and Transition Partners

91. Deployment Considerations for Extreme Scaling Experiment Al Sciarretta CNS Technologies, Inc. 703-517-2143 asciarretta@unconventional-inc.com

92. 92 Nominal Assumptions Objective experiment: –Approximately 10,000 sensors –Sensors can be placed about 50m apart on average (not a uniform distribution) –Sensor field: Approximately 1 km x 20 km –Hand emplacement There will be two smaller scale (proof of concept) experiments leading up to the objective experiment

93. 93 Characteristics of Experiment Site Relatively flat area –Able to survey/mark off 1 km x 20 km site –Allow for observers to see large section of experiment site No large physical obstructions (e.g., buildings) to line-of- site communications –Small obstructions (e.g., small rocks) okay Should be easy to walk around area and set sensors on ground Consistently good weather (no rain, no strong winds, etc.) –Allowing sensors to stay out for days Located on military base –Site can be guarded Sensors deployed on day 1 and remain in place until end of experiment (days later)

94. 94 Emplacement Rate One person: assume emplacement of 40 sensors/hour –Assuming each person can carry 40 sensors –Walking time of 4 km/hr (about 2.5 mph)  1.11 m/s Movement time ~ 1 min from sensor to sensor –56 sec to move 50 m distance between sensors –Add a few seconds to set sensor –Allow approximately 20 minutes per hour for person to receive sensors, get to the area, get oriented throughout the emplacement area, change direction, etc.

95. 95 Emplacement Time** & Number of Personnel Given: –440 sensors per 1 km x 1 km square –40 sensors/hour *Add 1 person for every 10 to manage the emplacement effort and distribute sensors ** These numbers and the ones on the previous page can be improved by taking into account that the average inter-sensor distance is much less (Vijay) Emplacement Time (hours) Number of Personnel (1 km x 1 km) Number of Personnel (1 km x 20 km) 111220* 26110* 3474* 4355*

96. 96 Additional Considerations Site should be out of the way of normal base operations –No interference from base vehicles/personnel Environmental considerations (i.e., batteries) will dictate that every sensor be picked up at end of experiment –Sensor should have a bright, fluorescent color (e.g., orange) to assist in sensor pick-up. Can color the case or add a small colored flag –Flag could increase emplacement time On-off switch must be simple –Shouldn’t depend on emplacement personnel to turn on the sensor Sensors will have to self-localize

97. 97 Additional Considerations (continued) Site should have personnel available to assist in emplacement of sensors and guard site –Tends toward active military base Experiment site will need some surveying –Sensor Field Mark outer boundaries Mark 50 m intervals along boundaries Mark boundary/intervals with tape, stakes, etc. Survey in the “hub” sensors Individual sensor site locations shouldn’t need to be surveyed –Assumes self-localization –Intruder paths need to be surveyed Provides “ground truth” Sensor batteries should not require replacement for duration of experiment

98. 98 Potential Sites Naval Air Weapons Facility, China Lake –150 miles NE of Los Angeles –Encompasses 1.1 million acres of land in California's upper Mojave Desert, ranging in altitude from 2,100 to 8,900 feet Varies from flat dry lake beds to rugged piñon pine covered mountains. –Weather should be consistent Issue: If experiment in summer, heat may be a problem for sensors

99. 99 Potential Sites (continued) Need further investigation –Fort Bliss, TX Home of U.S. Army missile test sites Arid environment (heat in summer an issue) Not flat – has dunes –Nellis AFB (near Las Vegas, NV) Large Air Force test site Not sure of availability of large expanse of land –Bombing sites may be off-limits

100. 100 To Further Assess Sites Need –Experiment schedule Approximate times of technical tests and final experiment –Characteristics of sensors Heat tolerance Weather tolerance Capability of comms in flat to rugged/forested terrain –Assume 50 m transmission distance

101. 101 UNCLASSIFED – FOR OFFICIAL USE ONLY Preliminary Program Plan System Metrics Operational Metrics  Latency <10 seconds (90% of the time)  Probability of False Alarm (Pfa) <1 per day  Probability of Detection (Pd) 0.99 for vehicles 0.95 for dismounts  Classification TBD  XSM Node Lifetime TBD  Relay Node Lifetime TBD  Network Deployment time <1 hour  Network Uninstallation time < 1 hour  Cost Per node Per unit area covered Operational Metrics  Latency <10 seconds (90% of the time)  Probability of False Alarm (Pfa) <1 per day  Probability of Detection (Pd) 0.99 for vehicles 0.95 for dismounts  Classification TBD  XSM Node Lifetime TBD  Relay Node Lifetime TBD  Network Deployment time <1 hour  Network Uninstallation time < 1 hour  Cost Per node Per unit area covered Technical Metrics  Robustness Of node Of network  Sensor coverage  Sensor density  Effective bandwidth Of first-tier network Of second-tier network  Endurance Of node Of network Technical Metrics  Robustness Of node Of network  Sensor coverage  Sensor density  Effective bandwidth Of first-tier network Of second-tier network  Endurance Of node Of network

102. 102 UNCLASSIFED – FOR OFFICIAL USE ONLY Preliminary Program Plan Technology Transition  Transition Partners  Transition Process Joint DARPA/Partner identification of operational challenges and potential solutions DARPA development of technical solution to operational challenges Joint evaluation of technical solution Joint Memorandum of Agreeement (MOA) / Memorandum of Understanding (MOU)

103. 103 UNCLASSIFED – FOR OFFICIAL USE ONLY Extreme Scaling Planning Session  Technical challenges Hardware issues Technical shortfalls/challenges in the extreme scaling experiment Remote programming Software Tools  Process Tasks, Who does what Methodology, Tools for systems integration  Refine Program Plan and Schedule  Action Items  Other issues

104. 104 UNCLASSIFED – FOR OFFICIAL USE ONLY Hardware Issues  Current limitations  New features needed/desirable  Packaging considerations  Supermote and relay node requirements

105. 105 UNCLASSIFED – FOR OFFICIAL USE ONLY Current Limitations  Larger SRAM  Faster processor  More comms bandwidth  Robust antenna solution (pre-soldered or on-board)  ADC/radio modules freeze if a hardware interrupt is lost  Race conditions  Wider ADC  Set/reset for magnetometer  More magnetometer range  Software control for radio  Battery power indication  MIR interference at close range  Radio/MIR affected by ground absorption  Overheating effects on MIR and magnetometer  Solar Cell

106. 106 UNCLASSIFED – FOR OFFICIAL USE ONLY New Features  Sensor suite Passive IR, magnetometer, acoustic/seismic Performance characteristics (range)  Switchless turn-on, visual/acoustic aids for deployment/recovery  Supermote What kind of device (e.g., iPAQ, Stargate)  Relay nodes

107. 107 UNCLASSIFED – FOR OFFICIAL USE ONLY Packaging Considerations  Ruggedness Antenna Battery holder detaches from the board easily  Weather-proofing Light rain Heat  Deployment Stake or tripod to get stable connection to earth  Issues not considered Stealth, camouflaging Physical security

108. 108 UNCLASSIFED – FOR OFFICIAL USE ONLY Shortfalls and Challenges

109. 109 UNCLASSIFED – FOR OFFICIAL USE ONLY Remote Programming  What are the desirable features? Time to reprogram (order of a few minutes for a 100 nodes deployed in a 100m by 100m area) Area coverage Robustness Elementary security Issues – memory overrun, mote in infinite loop, … Multihop Ability to use the second-tier supermote (or not) for higher gain xmit

110. 110 UNCLASSIFED – FOR OFFICIAL USE ONLY Tools  Extend Tossim and Tinyviz  Scale-invariant Simulator What are the desirable features in this?  Tools for in-field deployment, testing, recovery  Loggers  Network monitoring  Node health monitoring, network health monitoring

111. 111 UNCLASSIFED – FOR OFFICIAL USE ONLY Process  Tasks Put time line on high-level tasks Assign POC for each TBD by OSU (Anish) in consultation with DARPA

112. 112 UNCLASSIFED – FOR OFFICIAL USE ONLY Preliminary Program Plan Roles and Responsibilities  Systems Integration Define system architecture and hardware/middleware services required Build and maintain integration testbed Integrate system components from Technology Developers Test, verify, and validate system components and overall system Manage field testing/evaluation  Technology Development Develop system components (both hardware and software) Fabricate system hardware components (including packaging) Develop and test algorithms and middleware services  Testing and Evaluation Plan field testing/evaluation Schedule testing facilities (e.g., MOUT site, other DoD facility) Support system deployment at testing facilities Conduct system training as necessary Define system metrics and document system performance  Transition Partners Support CONOP and TTP development Facilitate meetings between DARPA and User Communities Support technology transition activities (e.g., development of MOAs/MOUs)  Military Consultants Support CONOP and TTP development Interact with System Integrator, Technology Developers, and Transition Partners

113. 113 UNCLASSIFED – FOR OFFICIAL USE ONLY Refine Program Schedule  OSU (Anish) to refine program schedule in consultation with DARPA in time for the PI meeting

114. 114 UNCLASSIFED – FOR OFFICIAL USE ONLY Action Items  Application Definition OSU (Anish), Puritan Research (Norm), MITRE (Alex and Ken) are to create an application definition that will specify the extreme scaling plan. (Dec PI meeting) Anish will refine the schedule that Larry provides, update list of tasks, and provide metrics for dry runs as well as final field experiment (Dec PI meeting)  Power Management UCB is the lead on this component  Localization UIUC is the lead on this component; will provide a demonstration of localization in an 100m x 100m grid with 100 nodes spread about 10m apart supporting a nominal accuracy of 1ft by Jan 31, 2004.  Deployment CNS Technologies (Al Sciarretta) will make further plans after discussion with OSU and DARPA. (Dec PI meeting)  Clock Synchronization UCLA is the lead on this component; they will collaborate with other efforts (such as Iowa, UCB, Vanderbilt) to deliver a component that could possibly leverage the second-tier supermotes in the future, by Jan 31. Accuracy and further work to be determined following application definition.  Sensor Suite Study OSU is to report on the use of PIR, magnetometer, and acoustic/sensor nodes as the appropriate complement of sensors for the ESE; will interact with Crossbow; this report will influence the set of metrics, the deployment plan, localization requirements, and the tool development. (Dec PI meeting)  Remote Programming UCB is the lead on this; they will provide a first-class remote programming tool that can be used for in-situ reprogramming of mote fields (roughly a 100m x 100m grid of 100 nodes in a “few minutes”) by Jan 31, 2004.  Support Infrastructure OSU and Berkeley will prepare items for further discussion of the support infrastructure needed for the ESE; the support infrastructure will include tools for simulation, testing, evaluation, …; other instrumentation needed for debugging and ESE development etc. (Dec PI meeting)  Sensor Fusion OSU is the lead on this component; they will collaborate with UCB, UCLA, and other teams that can provide the expertise that’s needed