1. Outdoor Distributed Computing Liviu Iftode Department of Computer Science Rutgers University http://discolab.rutgers.edu Supported by NSF through ITR grant ANI-0121416 http://discolab.rutgers.edu

2. Traditional Programming Model zPrograms access data through variables zVariables mapped to physical memory locations zRuntime + OS guarantees reference consistency zAccess time has an (acceptable) upper bound Program Virtual Address Space Page Table + OS Physical Memory Variable access

3. “Indoor” Distributed Computing zComputing distributed for performance or fault tolerance zNodes are computationally equivalent zConfiguration is stable (failures are exceptions) zNetworking is robust and has acceptable delays zRelatively easy to program yMessage passing or shared memory yNaming easy to translate to network addresses yCorrect programs complete and return deterministic results

4. Distributed object tracking over a large geographical area Cars collaborating for a safer and more fluent traffic Computers “Move” Outdoors

5. Linux Car SensorsLinux Camera Linux Watch Outdoor Distributed Systems zNodes are volatile and/or mobile: dynamic configurations zNetworking is wireless and ad-hoc: unpredictable delays zLocation dictates the computation: naming must include location zHow to program and execute distributed applications outdoors? yTraditional distributed computing does not work

6. How to Program Outdoor Distributed Embedded Systems? zRecent research in networked embedded systems yHardware yOperating Systems yNetwork Protocols yData collection/dissemination in sensor networks zOur research focuses on programmability yHow to code distributed applications to execute over networks of embedded systems?

7. Traditional Distributed Computing Does Not Work Well over Ad-Hoc zFixed address naming and routing (e.g., IP) are too rigid zEnd-to-end data transfer may hardly complete zOutdoor distributed computing requires novel system architectures and programming models

8. Example  “Water the hottest spot on the Left Hill”  Number and location of mobile sprinklers is unknown  Configuration is not stable in time  sprinklers move  temperature changes  What is a good result? Mobile sprinkler with temperature sensor Left Hill Right Hill Hot spot

9. Quality of Result QoR Outdoors Adversity 0 100% ideal real

10. Talk at a Glance zOutdoor Programming Model zOutdoor Execution Model zApplications

11. Outdoor Programming Outdoor Programming – How to program an unknown number of mobile systems to execute an intrusion detection application in a certain physical space? - Programming model must be simple to use - Networking aspects should be hidden from programmer - Access to mobile systems as simple as access to variables - - A virtual address space for the physical space? Left HillRight Hill

12. Sequential Programming Example struct temp a[N]; for(i=0;i<N;i++) { if (a[i].temp> Max_temp) { Max_temp = a[i].temp; Max_id = i; } } a[Max_id].water = ON;

13. Distributed Programming using Message Passing Struct temp a[N/P], b[P]; for(i=0;i<N/P;i++) { if (a[i].temp> Max_temp) { Max_temp = a[i].temp; Max_id = i; } } send(0,Max_temp,sizeof(int)); If (my_node_id == 0) { for(i=0;j<P;i++) recv(i,&b[i],sizeof(int)); …

14. zApplications access distributed data through shared variables zRuntime system translates variable accesses into message passing (when necessary) Software Distributed Shared Memory Distributed Application Shared virtual address space Page Table + Message Passing Physical memories Variable accesses

15. Shared Memory Distributed Computing struct a[N], b[P]; Thread worker(k) { int Max_temp, Max_id,i; for(i=k*N/P; i< (k+1)*N/P; i++) { if (a[i].temp> Max_temp) { Max_temp = a[i].temp; Max_id = i; } } b[k]=Max_temp; barrier() if (k==0) {

16. From Indoor to Outdoor Computing Virtual Address SpaceSpace Region VariablesSpatial References Variables mapped to physical memory Spatial references mapped to systems embedded in the physical space Reference consistency Bounded access time ? ?

17. Spatial Programming (SP) at a Glance zOutdoor distributed applications are programmed using spatial references zA virtual address space hides networking zEmbedded systems/nodes named by their expected locations and properties

18. Space Region radius Hill = new Space({lat, long}, radius); {lat,long} Virtual representation of a physical space Virtual representation of a physical space Similar to a virtual address space in a conventional Similar to a virtual address space in a conventional computer system computer system

19. Spatial References – Defined as {space:property} pairs –Similar to variables in conventional programming –Used to name systems embedded in physical space –Indexes used to distinguish among similar systems in the same space region -Same index maps to the same system (reference consistency) -Distinct indexes map to distinct systems {Hill:robot[0]} {Hill:robot[1]} {Hill:truck} Hill

20. Reference Consistency zAt the time of the first access, a spatial reference is mapped to an unmapped embedded system located in the specified space zMappings maintained in per-application Mapping Tables (MT) zA spatial reference refers the same system as long as it is located in the same space, fails otherwise {space, property, index} {network_address, location}

21. Reference Consistency For (i=0;i<1000;i++) {Left_Hill:robot[i]}.stop =ON ; Left HillRight Hill {Hill:robot[0]} {Hill:robot[1]}

22. Reference Consistency (con’t) Left Hill Right Hill {Left_Hill:robot[0]}.move = ON; Left Hill Right Hill {Left_Hill:robot[0]}.move = OFF; Time

23. Space Casting Left HillRight Hill Left HillRight Hill {Right_Hill:robot[0]} {Left_Hill:{Right_Hill:robot[0]}} time

24. Relative Spaces Left Hill Right Hill {rangeOf({Left_Hill:robot[0]}, radius):robot[0]}

25. Space Composition Left Hill Right Hill {(Left_Hill + Right_Hill):robot[0]}

26. Bounding the Access Time zA spatial reference can take an infinite time yDiscover an unmapped system for a new spatial reference yMapped systems may move, go out of space, or disappear zHow to bound the time to access a spatial reference? y Associate an explicit timeout with each spatial reference access try{ {Hill:robot[0], timeout}.camera = ON; }catch(TimeoutException e){ // the programmer decides the next action }

27. Hills Configuration Changes During Execution Execution Time System State {Hills:robot[0]} {Hills:robot[1]} {Hills:robot[2]} Hills

28. for(i=0;i<1000;i++) try{ if ({Left_Hill:Hot[i], timeout}.temp > Max_temp) Max_temp = {Left_Hill:Hot[i], timeout}.temp; Max_id = i; }catch(TimeoutException e) break; {Left_Hill:Hot[Max_id]}.water = ON; Spatial Programming Example Application: Water the hottest spot on the Left Hill Mobile sprinkler with temperature sensor Left Hill Right Hill Hot spot

29. Talk at a Glance zOutdoor Programming Model zOutdoor Execution Model zApplications

30. Smart Message Architecture zSmart Message (SM) yUser-defined distributed application yExecutes on nodes of interest named by properties yMigrates between nodes of interest zSelf-Routing yApplication-controlled routing to perform migration yExecuted on every node on the path between nodes of interest zCooperative Nodes yExecution environment (Virtual Machine) yMemory addressable by names (Tag Space

31. Smart vs. “Dumb” Messages Data migration Mary’s lunch: Appetizer Entree Dessert ` Execution migration

32. Application Example n=0 while (n<NumTaxis) migrate(Taxi); migrate(Taxi); if (readTag(Available)) if (readTag(Available)) writeTag(Available, false); writeTag(Available, false); writeTag(Location, myLocation); writeTag(Location, myLocation); n++; n++; n=0 Taxi Taxi n=0n=0n=1n=1n=2 need 2 taxis data brick application code brick routing

33. Cooperative Node Architecture Network SMPlatform AdmissionManagerAdmissionManager Network SMSM Operating System & I/O CodeCacheCodeCache VirtualMachineVirtualMachine TagSpaceTagSpace SM Ready Queue

34. Smart Message Admission zPrevents SMs from migrating to nodes where they cannot execute zSMs specify resources requirements yLower bounds for virtual machine cycles, tag space memory, runtime memory, network bandwidth, etc. ySpecified by programmer or derived using compiler support zSMs accepted if the node can satisfy SM requirements zSMs transfer only the missing code bricks zMore resources can be granted according to admission policy yIf not granted, SM is allowed to migrate

35. Smart Message Execution zTakes place over a virtual machine zNon-preemptive but time bounded zEnds with migration or terminates zDuring execution, SMs can yCreate new SMs ySpawn new SMs yAccess the tag space yBlock on a tags to be updated

36. Tag Space zLocal memory that SMs can use ySM-created tags: limited lifetime, persistent across SM executions yI/O tags: maintained by the system zMultiple Roles yNode addressing migrate (tag,timeout) yI/O port access read (temperature) yStore data write (tag,value) yInter -SM communication ySynchronization on writeblock(tag,timeout) yRouting

37. Protection Domains for Tag Space zOwner: SM that creates the tag zFamily: all SMs having a common ancestor with the SM that creates the tag zOrigin: all SMs created on the same node as the family originator of the tag owner zCode: all SMs that carry the same code brick(s) zOthers: all SMs Family Code Origin Others Owner

38. Access Control Example (Code-based protection domain) Node5 Node3 Node4 SM2 C2 Cr Node1 Node2 Code Bricks Tag Owner = SM1 [Hash(Cr), RW] SM1 C1 Cr SM1 SM3 C3 C4 Access permission granted for SM2 Access permission denied for SM3 Cr Same routing used by SM1 and SM2

39. Application Example n=0 while (n<NumTaxis) migrate(Taxi); migrate(Taxi); if (readTag(Available)) if (readTag(Available)) writeTag(Available, false); writeTag(Available, false); writeTag(Location, myLocation); writeTag(Location, myLocation); n++; n++; n=0 Taxi Taxi n=0n=0n=1n=1n=2 need 2 taxis data brick application code brick routing

40. migrate() migrate() – implements routing algorithm – migrates application to next node of interest – node of interest named by tags – sys_migrate() – one hop migration – used in migrate to implement routing Migration migrate(Taxi) sys_migrate(2) sys_migrate(3) sys_migrate(4) TaxiTaxi 1 234

41. Network Routing Example 12i RouteToTaxi = 2 Taxi RouteToTaxi = ? migrate(Taxi){ while(!readTag(Taxi)) if (readTag(RouteToTaxi)) sys_migrate(readTag(RouteToTaxi)); else create_SM(DiscoverySM, Taxi); createTag(RouteToTaxi, lifetime, null); block_SM(RouteToTaxi, timeout); } RouteToTaxi = j

42. Self-Routing zSMs carry the routing brick and execute it at each node zSMs control their routing ySelect routing algorithm (migrate primitive) xMultiple library implementations xImplement a new one yChange routing algorithm during execution in response to xAdverse network conditions xApplication’s requirements

43. Dynamic Change of Routing SM starts with proactive routing migrate(Taxi, timeout1) Dense network Low mobility Proactive routing Sparse network High mobility On-demand routing migrate timeouts SM changes routing to on-demand migrate(Taxi, timeout2)

44. Prototype Implementation zModified version of Sun’s Java K Virtual Machine ySmall memory footprint (160KB) zSM and tag space primitives implemented inside virtual machine as native methods (efficiency) zImplemented I/O tags: GPS location, neighbor discovery, image capture, light sensor, system status Prototype Node with GPS receiver and video camera

45. Lightweight Migration zTraditional process migration difficult y Strong coupling between execution entity and host y OS state (e.g., open sockets, file descriptors) hard to transfer zTag space decouples the SM execution state from the OS state zSM migration transfers yData bricks explicitly specified by programmer as mobile data yMinimal execution control state required to resume the SM at the next node (e.g., instruction pointer, operand stack pointer)

46. Experimental Results for Simple Routing Algorithms Completion Time Routing algorithmCode not cached (ms)Code cached (ms) Geographic On-demand 415.6 126.6 506.6 314.7 user node node of interest intermediate node

47. Self-Routing Simulation starting node node of interest other node 3 nodes of interest located in the corners have to be visited in clockwise order vary the radius from 100m to 700m On-Demand Routing versus Geographic and On-Demand Routing

48. SP Implementation using SM zSP application -> SM zEmbedded system properties -> Tags zSM self-routing yContent-based routing using tags yGeographical routing zReference consistency yunique tag created on the node which a spatial reference is mapped to yMappingTable (MT) maps spatial reference to the unique tag and expected location yMT is carried by the SM that implements the SP application zSpatial reference access -> an SM migration to the node where the spatial reference is mapped to

49. Max_temp = {Left_Hill:Hot[1], timeout}.temp; SP using SMs: Example Mobile sprinkler with temperature sensors Left Hill Right Hill Hot spot Spatial Reference Access Smart Message ret = migrate_geo(location, timeout); if ret == LocationUnreachable ret = migrate_tag(yU78GH5, timeout); if (ret == OK) && (location == Left_Hill) return readTag(temp); else throw TimeoutException {Left_Hill,Hot,1}{yU78GH5,location} MT Code Brick

50. SP Application: Intrusion detection Code Size breakdown for SM (Application + SP Library) Code Size breakdown for SP library Testbed: 10 HP iPAQs with 802.11 cards and GPS devices user node light sensor camera node regular node monitoredspace

51. Execution Time Breakdown

52. Talk at a Glance zOutdoor Programming Model zOutdoor Execution Model zApplications

53. Intelligent Distributed Transportation Systems zTwo projects: yEZCab: An Automatic System for Booking Cabs yTraffic Viewer: A Scalable Traffic Monitoring System Infrastructure-free approach based on car-to-car communication short-range wireless communication location information from GPS receiver vehicle’s data from sensors through on-board diagnostic system (OBD-II)

54. EZCab Idea allow people to book cabs in dense cab area using ad-hoc networking allow people to book cabs in dense cab area using ad-hoc networking

55. Finding Free Cabs

56. Booking a cab

57. Phase 2 & 3: Report and Validation

58. TrafficView Enable drivers to monitor traffic in front of their cars, farther than they can seeEnable drivers to monitor traffic in front of their cars, farther than they can see Based exclusively on vehicle-to-vehicle ad hoc communicationBased exclusively on vehicle-to-vehicle ad hoc communication

59. How TrafficView Works Receive data from remote vehicle Non-validated dataset Validate Validated dataset Local data Display Broadcast data

60. Data Aggregation  How to aggregate data to collect monitor vehicles as far as possible with “ acceptable ” accuracy loss zNatural Solution yAggregate data for vehicles that are close to each other yPerform more aggregation as distance increases

61. Smart Phones as Universal Personal Servers zMobile phones and PDAs converge into Smart Phones yProgrammable yDual-connectivity (BT & GPRS) zHow to build pervasive applications on them?

62. Summary zSpatial Programming makes outdoor distributed computing simple zVolatility, mobility, configuration dynamics, ad-hoc networking are all hidden from programmer zSpatial Programming can be implemented using Smart Messages zOutdoor applications: intelligent transportation, pervasive computing using smart phones

63. Acknowledgements zFaculty yUli Kremer (check our PLDI’05 paper!) zGraduate Students yCristian Borcea (currently on the NJIT faculty) yPorlin Kang yNishkam Ravi yPravin Shankar yPeng Zhou yDeepa Iyer yAkhilesh Saxena zVisitors yChalermek Intanagonwiwat (post-doc, currently on the faculty of Chulalongkorn University, Thailand) yOriana Riva (grad student, University of Helsinki) yTamer Nadeem (grad student, UMD) yMarios Dikaiakos (faculty, Univ of Cyprus) zUndergraduate students: yNiket Desai, Peter Stern, Yan Nuriyev

64. Thank you! http://discolab.rutgers.edu