1. On Fault Tolerance in Wireless Ad Hoc Networks Seth Gilbert Nancy Lynch Celebration, 2008

2. Nancy Lynch 1994 Late 1980’s?? 1997 2002-2008 Through the years …

3. 1980 1984 1988 1992 1996 2000 2004 2008 FLP: Impossibility of distributed consensus with one faulty process DLS: Consensus in the Presence of Partial Synchrony LT: An Introduction to Input / Output Automata Fault tolerance Replication Consistency Formal Methods Simulation Relations, Invariant-based Arguments Timing Increasingly complex, increasingly dyamic: Group communication / membership Publish / Subscribe Peer-to-peer systems Wireless ad hoc networks

4. The Virtual Infrastructure Project

5.  Papers:  GeoQuorums: Implementing Atomic Memory in Mobile Ad Hoc Networks, DGLSW, DISC ’ 03, DC ’ 05  Virtual Mobile Nodes for Mobile Ad Hoc Networks, DGLSSW, DISC ’ 03  Consensus and Collision Detectors in Wireless Ad Hoc Networks, CDGNN, PODC ’ 05, DC ’ 08  Timed Virtual Stationary Automata for Mobile Networks, DGLLN, Allerton ’ 05, OPODIS ’ 05  Autonomous Virtual Mobile Nodes, DGSSW, DIALM-POMC ’ 05  A Middleware Framework for Robust Applications in Wireless Ad Hoc Networks, CDGN, Allerton ’ 05  Reconciling the theory and practice of unreliable wireless broadcast, CDGLNN, ADSN ’ 05  Self-Stabilizing Mobile Node Location Management and Message Routing, DLLN, SSS ’ 05  Motion Coordination Using Virtual Nodes, LMN, CDC ’ 05  The Virtual Node Layer: A Programming Abstraction for Wireless Sensor Networks, BGLNNS, WWWSNA ’ 07  A Virtual Node-Based Tracking Algorithm for Mobile Networks, NL, ICDCS ’ 07  Self-stabilization and Virtual Node Layer Emulations, NL, SSS ’ 07  Secret Swarm Unit: Reactive k-Secret Sharing, DLY, IndoCrypt ’ 07  Virtual Infrastructure for Collision-Prone Wireless Networks, CGL, PODC ’ 08  Theses:  Virtual Infrastructure for Wireless Ad Hoc Networks, G, PhD 2007  Air Traffic Control Using Virtual Stationary Automata, B, MEng 2007  Simulation and Evaluation of the Reactive Virtual Node Layer, S, MEng 2008  Virtual Stationary Timed Automata for Mobile Networks, N, PhD 2008  In Progress:  Self-Stabilizing Robot Formations over Unreliable Networks, GLMN  Using Virtual Infrastructure to Adapt Wireline Protocols to MANET, W  Virtual Infrastructure Routing for Mobile Ad Hoc Networks, DN

6. Scenarios: Sensor networks Social networks Coordination Wireless Ad Hoc Networks

7. Scenarios: Sensor networks Social networks Coordinated applications Wireless Ad Hoc Networks —environmental monitoring —intrusion detection —border monitoring —fire detection

8. Scenarios: Sensor networks Social networks Coordinated applications Wireless Ad Hoc Networks —messaging —conferences / events —HikingNet —TrafficNet

9. Scenarios: Sensor networks Social networks Coordination Wireless Ad Hoc Networks emergency response & military — firefighting — police response — terrorism

10. Scenarios: Sensor networks Social networks Coordination Wireless Ad Hoc Networks

11. Unreliable communication Unknown availability Wireless ad hoc networks are really hard to use. Noise Collisions Dynamic Unknown participants Unknown topology Fault prone Lost Messages

12. Fixed Infrastructure Deploy: —Base stations —Cell towers —Servers Problems: —Too expensive —Not feasible

13. Virtual Infrastructure Unreliable  ReliableAd hoc  Fixed net

14. Network Layers Service Middleware Wireless Ad Hoc Network Application

15. Network Layers Routing Tracking Virtual Infrastructure Wireless Ad Hoc Network Application

16. Building Virtual Infrastructure Basic idea: replicated state machine

17. Building Virtual Infrastructure Basic idea: replicated state machine 1. Each participant is a replica. 2. Replicas execute a consistency protocol 3. Leader / backup 4. Leader sends & receives messages for the virtual node

18. Today’s Questions 1. What is virtual infrastructure? 2. What can you do with it? — Dynamic distributed coordination. — Air traffic control 3. Does it really work? — Two simulation studies: routing and address allocation.

19. Dynamic Distributed Coordination Challenging problem: oHighly dynamic environment oUnreliable network oSafety-critical applications Ideal for Virtual Infrastructure solution: oStatic overlay oSimpler, verifiable algorithms oFate-sharing

20. ˇ

22. Dynamic Distributed Coordination Note: Number of (non-failed) robots unknown. Location of other robots unknown. Pattern may change over time.

23. Dynamic Distributed Coordination In each round: 1.All robots stop. 2.All robots send location info. 3.Coordinators exchange info. In each round: 4.Coordinators calculate. 5.Coordinators send out targets. 6.Robots move to target.

24. Dynamic Distributed Coordination Rule 1: If only 1 robot, keep it. Calculating new targets

25. Rule 2: If not on the curve and no neighbors on the curve: distribute evenly all but one. Dynamic Distributed Coordination Calculating new targets

26. Rule 3: If not on the curve: distribute among less populated neighbors on the curve. Dynamic Distributed Coordination Calculating new targets

27. Rule 4: If on the curve: distribute among less dense neighbors on the curve. Dynamic Distributed Coordination Calculating new targets

28. Rule 4: If on the curve: distribute among less dense neighbors on the curve. Dynamic Distributed Coordination Calculating new targets

29. Rule 5: Distribute robots evenly on the curve in each region. Dynamic Distributed Coordination Calculating new targets

30. Dynamic Distributed Coordination Step 1: Eventually, robots cease moving from regions “ off the curve ” to regions “ on the curve ”. Step 2: If neighbor g is the most dense neighbor of u after time t, then u is less dense than g after time t+1. Step 3: Eventually, robots remain always in the same region. Correctness

31. Dynamic Distributed Coordination Self-stabilization What happens when something goes wrong? Too many lost messages Too much churn INCONSISTENT REPLICAS Option 1: Design for the very, very worst case. Option 2: Design a system that can recover from faults.

32. Emulating Virtual Infrastructure Self-stabilization techniques Leader Election: oHeartbeats, timeouts oResolve leader competitions Replica Consistency: oLeader sends “checksums” of the state. oIf out-of-synch, then re-join.

33. Building Virtual Infrastructure Self-stabilization claims Assume that: oA is a self-stabilizing algorithm. oA is designed for the virtual infrastructure abstraction. oA is executed with the emulator. oThe system begins in an arbitrary (corrupt) state. Then if the system is eventually well-behaved: oFrom some point on, the state of A is as if it had really executed on a fixed infrastructure.

34. Dynamic Distributed Coordination Summary Coordination algorithm is self-stabilizing. oIn each round, all state is recalculated. oUnderlying virtual infrastructure emulation is self-stabilizing. Implications : oConverges to changing curve. oRecovers from network instability, lost messages, etc.

35. Dynamic Distributed Coordination Additional comments Tina Nolte Virtual Stationary Timed Automata for Mobile Networks PhD 2008

36. Dynamic Distributed Coordination Air traffic control Free Flight oNo flight plan, no control towers! oEach pilot chooses a route independently. oMore efficient: —Adapt to wind currents. —Avoid turbulence / bad weather.

37. Dynamic Distributed Coordination Air traffic control Goal: Free Flight oEach pilot chooses a route independently. oMore efficient: —Adapt to wind currents. —Avoid turbulence / bad weather. In the USA, minimum separation: 3 miles lateral distance OR 1000 feet altitude

38. Dynamic Distributed Coordination Additional comments Matthew D. Brown Air Traffic Control Using Virtual Stationary Automata MEng, 2008

39. Today’s Questions 1. What is virtual infrastructure? 2. What can you do with it? — Dynamic distributed coordination. 3. Does it really work? — Two simulation studies.

40. Simulating Virtual Infrastructure Study #1 —Routing / Geocast —Custom-built simulator (python) —Simple communication model Study #2 —Address allocation (i.e., DHCP) —ns2 simulator —802.11 MAC layer

41. GeoCast Location-based routing Source Destination

42. GeoCast Location-based routing Source Destination

43. Location Service Store current location at home Target geocast hash(id, 1) hash(id, 2)

44. Location Service Where are you? Target geocast hash(id, 2) Source hash(id, 1)

45. Routing Point-to-point communication Two step process: 1.Lookup destination location. 2.Geocast message to destination’s region.

46. 400 m 250 m Simulation Setup Number of devices: 25 / 50 / 100 Velocity: 0-20 meters / second Mobility model: Random waypoint Pause time: 100- 900s Simulation time: 1000 seconds Basic settings

47. 400 m 250 m Simulation Setup GeoCast: 10 send/receive pairs 1 msg every 5 secs Routing 10 send/receive pairs 1 msg every 0.5 secs 15 second simulation Application settings

48. Mobility and Density Percent of Time Non-Failed Pause Time 100% 80% 60% 20% 40% 200400600800 25 devices 100 devices 50 devices When density is sufficient, virtual nodes work.

49. Leadership Changes per Region 10 Pause Time 8 6 2 4 200400600800 100 devices There is continuous turn-over in the leader.

50. Message Overhead Messages per Region per second Pause Time 0.5 0.4 0.3 0.05 0.1 0.01 200400600800 Heartbeat Join Leader Most overhead is heartbeats. (Overhead is negligible.)

51. Geocast Latency Overhead VN-GeoCast is 2-3 times slower than simple GeoCast. Latency (in seconds) 0.5 Pause Time 0.4 0.3 0.1 0.2 200400600800 100 devices simple Geocast

52. Routing 79% 0.46 seconds 0.58 seconds Delivery Rate Median Latency Average Latency End-to-end performance Each message requires 3 GeoCast messages. ** devices=50, pausetime=400

53. Simulation Summary Virtual nodes are stable if: —sufficient density (e.g., 4/region), OR —low-enough churn Message overhead: negligible. GeoCast latency overhead: factor of 2. Routing: relatively slow.

54. Simulation Summary Additional comments Mike Spindel Simulation and Evaluation of the Reactive Virtual Node Layer MEng 2008

55. Simulating Virtual Infrastructure Study #1 —Routing / Geocast —Custom-built simulator (python) —Simple communication model Study #2 —Address allocation (i.e., DHCP) —ns2 simulator —802.11 MAC layer

56. —Mobile devices join and leave. —Each device needs an address. —Addresses should be assigned dynamically. —Addresses should be unique. Basic problem Address Allocation Challenges:  Highly dynamic.  No central authority.  Unreliable network.  Limited address pool.

57. Simple Scheme  Each region is allocated a cache of addresses.  Basic protocol:  Client send REQUEST  Server reply OFFER  Client send ACQUIRE  Server reply ACK  Renew protocol:  Client send RENEW  Server reply RACK  Message forwarding… REQUEST ACQUIRE RENEW OFFER ACK RACK Virtual Node Client

58. Number of devices: 160 MAC Layer: 802.11 Models collisions Mobility model: Random waypoint Simulation time: 40000 seconds 700 m 250 m Simulation Setup Basic settings

59. Number of addresses: 30 per region Lease time: 400 seconds Forwarding limit: 2 hop - REQUEST 2 hop - RACK Varying - RENEW 700 m 250 m Simulation Setup Application settings

60. Simulation Setup Simulation settings Very SlowSlow Medium Slow Medium Fast Fast Min. Speed (m/s)0.3650.731.462.927.3 Max. Speed (m/s)1.482.925.8411.6829.2 Average Pause Time (s)440022001100550220 Average Cross Time (s)82.2041.1020.5510.274.11

61. Message Overhead Messages per 400 secs Percent Heartbeats36076 Leader Request245 Leader Reply5011 Synch-Request204 Synch-Reply204 Total Message Overhead 474 Maximum observed: Less than 2-4.5kbps

62. Message Overhead Different speeds

63. Message Overhead Different densities

64. Protocol Performance Different speeds

65. Renewal cost Protocol Performance

66. Simulation Summary Message overhead: still negligible. —Even with collisions… —Backoff… —Bigger simulations… Si mple address allocation scheme: —Reasonably efficient… —Scales well…

67. Simulation Summary Additional comments Jiang Wu Using Virtual Infrastructure to Adapt Wirelines Protcols to MANET

68. Summary  What is virtual infrastructure?  Dynamic distributed coordination  Robotic motion coordination  Self-stabilization  (Preliminary) simulation results. The Virtual Infrastructure Project

69. Distributed Algorithms Focus on fault-tolerance —Replication —Consistency —Agreement Design principles —Abstraction / layered design —IOA / TIOA formalism Classical techniques, modern networks

70. Seth Gilbert George Varghese Boaz Patt-Shamir Jennifer Welch Brian Coan Kenneth Goldman Shinya Umeno Alex Cornejo Mark Tuttle Joshua Tauber Eugene Stark Rainer Gawlick Alan Fekete Victor Luchancgo Roberto Segala Rui Fan Tina Nolte Sayan Mitra Calvin Newport Carl Lividas Jim Burns Roger Khazan Roberto DePrisco Congratulations, Nancy, and thank you!!

71. The End