Outdoor Distributed Computing Cristian Borcea Department of Computer Science New Jersey Institute of Technology

2 Computers Embedded Everywhere

3 Distributed object tracking over a large geographical area Cars collaborating for a safer and more fluid traffic Outdoor Distributed Applications

4 Traditional (Indoor) Distributed Computing Commonly, computation is distributed for performance or fault tolerance Commonly, computation is distributed for performance or fault tolerance Relatively easy to program Relatively easy to program –Nodes are computationally equivalent –Networking is robust and has acceptable delays –Configuration is stable

5 Outdoor Distributed Computing Computation is physically distributed Computation is physically distributed –Nodes distributed across the physical space –Location-aware computing Hard to program Hard to program –Nodes are functionally heterogeneous and potentially mobile –Ad hoc wireless networks of embedded systems –Volatile configurations

6 How to Program Outdoor Distributed Applications? Recent research in networked embedded systems has focused on Recent research in networked embedded systems has focused on –Hardware (e.g., Berkeley’s motes, MIT’s crickets) –Operating Systems (e.g., TinyOS) –Network Protocols (e.g., Directed Diffusion, SPIN) –Data collection in sensor networks (e.g., TinyDB) My research focuses on programmability My research focuses on programmability –How to write distributed programs over outdoor networks of embedded systems?

7 Outdoor Programming Example Left Hill Right Hill Mobile robot with camera Motion Sensor Intrusion detection across the hills using motion sensors and mobile robots with cameras Intrusion detection across the hills using motion sensors and mobile robots with cameras Number and location of systems are unknown Number and location of systems are unknown Configuration is not stable over time Configuration is not stable over time –Intrusions can appear randomly –Robots can move

8 Traditional Distributed Computing Does Not Work Well Outdoors End-to-end data transfers may hardly complete End-to-end data transfers may hardly complete Fixed address naming and routing (e.g., IP) are too rigid Fixed address naming and routing (e.g., IP) are too rigid Difficult to deploy new applications in existing networks Difficult to deploy new applications in existing networks Outdoor distributed computing requires novel programming models and system architectures Outdoor distributed computing requires novel programming models and system architectures

9Outline Motivation Motivation Spatial Programming Model Spatial Programming Model Smart Messages System Architecture Smart Messages System Architecture Implementation Implementation Outdoor applications: EZCab and TrafficView Outdoor applications: EZCab and TrafficView Conclusions Conclusions

10 Traditional (Indoor) Programming Programs access data through variables Programs access data through variables Variables mapped to physical memory locations Variables mapped to physical memory locations Page Table and OS guarantee reference consistency Page Table and OS guarantee reference consistency Access time has an (acceptable) upper bound Access time has an (acceptable) upper bound Program Virtual Address Space Page Table + OS Physical Memory Variable access

11 From Indoor to Outdoor Computing Virtual Address Space Space Region Variables Spatial References Variables mapped to physical memory Spatial references mapped to systems embedded in the physical space Reference consistency Bounded access time ? ?

12 Spatial Programming (SP) at a Glance Provides a virtual name space over outdoor networks of embedded systems Provides a virtual name space over outdoor networks of embedded systems Embedded systems named by their locations and properties Embedded systems named by their locations and properties Runtime system takes care of name resolution, reference consistency, and networking aspects Runtime system takes care of name resolution, reference consistency, and networking aspects Implementation on top of Smart Messages: SP applications execute, sequentially, on each system referenced in their code Implementation on top of Smart Messages: SP applications execute, sequentially, on each system referenced in their code

13 radius Hill = new Space({lat, long}, radius); {lat,long} Space Regions Virtual representation of a physical space Virtual representation of a physical space Similar to a virtual address space in a conventional Similar to a virtual address space in a conventional computer system computer system Defined statically or dynamically Defined statically or dynamically

14 Spatial References Defined as {space:property} pairs Defined as {space:property} pairs Virtual names for embedded systems Virtual names for embedded systems Similar to variables in conventional programming Similar to variables in conventional programming Indexes used to distinguish among similar systems Indexes used to distinguish among similar systems in the same space region in the same space region {Hill:robot[0]} {Hill:robot[1]} {Hill:motion[0]} Hill r2 r7 m5

15 From Indoor to Outdoor Computing Virtual Address Space Space Region Variables Spatial References Variables mapped to physical memory Spatial references mapped to systems embedded in the physical space Reference consistency Bounded access time ? ?

16 Reference Consistency At first access, a spatial reference is mapped to an embedded system located in the specified space At first access, a spatial reference is mapped to an embedded system located in the specified space Mappings maintained in per-application Mapping Table (MT) – similar to a page table Mappings maintained in per-application Mapping Table (MT) – similar to a page table Subsequent accesses to the same spatial reference will reach the same system (using MT) as long as it is located in the same space region Subsequent accesses to the same spatial reference will reach the same system (using MT) as long as it is located in the same space region {space, property, index} {unique_address, location}

17 Reference Consistency Example Left Hill Right Hill {Left_Hill:robot[0]}.move = ON; r2 r7 r5 {Left_Hill:robot[0]}.move = OFF;

18 Space Casting {Left_Hill:robot[0]} Left Hill Right Hill {Right_Hill:(Left_Hill:robot[0])}r7

19 From Indoor to Outdoor Computing Virtual Address Space Space Region Variables Spatial References Variables mapped to physical memory Spatial references mapped to systems embedded in the physical space Reference consistency Bounded access time Mapping Table ?

20 Bounding the Access Time How to bound the time to access a spatial reference? How to bound the time to access a spatial reference? –Systems may move, go out of space, or disappear Solution: associate an explicit timeout with each spatial reference access Solution: associate an explicit timeout with each spatial reference access try{ {Hill:robot[0], timeout}.move = ON; {Hill:robot[0], timeout}.move = ON; }catch(TimeoutException e){ // the programmer decides the next action // the programmer decides the next action}

21 for(i=0; i<1000; i++) for(i=0; i<1000; i++) try{ try{ if ({Left_Hill:motion[i], timeout}.detect > Max_motion) if ({Left_Hill:motion[i], timeout}.detect > Max_motion) Max_motion = {Left_Hill:motion[i], timeout}.detect; Max_motion = {Left_Hill:motion[i], timeout}.detect; Max_id = i; Max_id = i; }catch(TimeoutException e) }catch(TimeoutException e) break; break; intrusionSpace = rangeOf({Left_Hill:motion[Max_id].location}, Range); intrusionSpace = rangeOf({Left_Hill:motion[Max_id].location}, Range); {intrusionSpace:robot[0]}.camera = ON; {intrusionSpace:robot[0]}.camera = ON; {intrusionSpace:robot[0]}.focus = {Left_Hill:motion[Max_id].location}; {intrusionSpace:robot[0]}.focus = {Left_Hill:motion[Max_id].location}; Spatial Programming Example - Find the sensor that detected the “strongest” motion on Left Hill - Turn a camera on in the proximity of this sensor Left Hill Right Hill

22Outline Motivation Motivation Spatial Programming Model Spatial Programming Model Smart Messages System Architecture Smart Messages System Architecture Implementation Implementation Outdoor applications: EZCab and TrafficView Outdoor applications: EZCab and TrafficView Conclusions Conclusions

23 “Dumb” Messages vs. “Smart” Messages Data migration Lunch: Lunch:AppetizerEntreeDessert Execution migration

24 Smart Messages at a Glance User-defined distributed applications User-defined distributed applications Composed of code bricks, data bricks, and execution control state Composed of code bricks, data bricks, and execution control state Execute on nodes of interest named by properties Execute on nodes of interest named by properties Migrate between nodes of interest Migrate between nodes of interest Self-route at every node in the path during migrations Self-route at every node in the path during migrations

25 Cooperative Node Architecture Network SMPlatform AdmissionManagerAdmissionManager Network SMSM Operating System & I/O CodeCacheCodeCache VirtualMachineVirtualMachine TagSpaceTagSpace SM Ready Queue

26 SM Execution at a Node Takes place over a virtual machine Takes place over a virtual machine Non-preemptive, but time bounded Non-preemptive, but time bounded Ends with a migration, or terminates Ends with a migration, or terminates During execution, SMs can During execution, SMs can –Spawn new SMs –Create smaller SMs out of their code and data bricks –Access the tag space –Block on a tag to be updated (update-based synchronization)

27 Tag Space Collection of application tags and I/O tags Collection of application tags and I/O tags Application tags: persistent memory across SM executions Application tags: persistent memory across SM executions I/O tags: access to operating system and I/O subsystem I/O tags: access to operating system and I/O subsystem Tags used for Tags used for –Content-based naming migrate(tag) –Inter-SM communication write(tag, data), read(tag) –Synchronization block(tag, timeout) –I/O access read(temperature) Protection domains for access control Protection domains for access control

28 SM Admission Ensures progress for all SMs in the network Ensures progress for all SMs in the network Prevents SMs from migrating to nodes that cannot provide enough resources Prevents SMs from migrating to nodes that cannot provide enough resources SMs specify lower bounds for resource requirements (e.g., memory, bandwidth) SMs specify lower bounds for resource requirements (e.g., memory, bandwidth) SMs accepted if the node can satisfy these requirements SMs accepted if the node can satisfy these requirements More resources can be granted according to admission policy More resources can be granted according to admission policy –If not granted, SMs are allowed to migrate

29 Application Example n=0 while (n<NumTaxis) migrate(Taxi); migrate(Taxi); if (readTag(Available)) if (readTag(Available)) writeTag(Available, false); writeTag(Available, false); writeTag(Location, myLocation); writeTag(Location, myLocation); n++; n++; n=0 Taxi Taxi n=0n=0n=1n=1n=2 need 2 taxis data brick application code brick routing SM

30 migrate() migrate() – migrates application to next node of interest – names nodes by tags – implements routing algorithm sys_migrate() sys_migrate() – one hop migration – used by migrate to implement routing SM Migration migrate(Taxi) sys_migrate(2) sys_migrate(3) sys_migrate(4) TaxiTaxi 1 234

31 Network Routing Example 12i RouteToTaxi = 2 Taxi RouteToTaxi = ?RouteToTaxi = j migrate(Taxi){ while(readTag(Taxi) == null) if (readTag(RouteToTaxi)) sys_migrate(readTag(RouteToTaxi)); else create_SM(DiscoverySM, Taxi); createTag(RouteToTaxi, lifetime, null); block_SM(RouteToTaxi, timeout); }

32 SM Self-Routing SMs carry the routing and execute it at each node SMs carry the routing and execute it at each node Routing information stored in tag space Routing information stored in tag space SMs control their routing SMs control their routing –Select routing algorithm (migrate primitive) Multiple library implementations Multiple library implementations Implement a new one Implement a new one –Change routing algorithm during execution in response to Adverse network conditions Adverse network conditions Application’s requirements Application’s requirements

33Outline Motivation Motivation Spatial Programming Model Spatial Programming Model Smart Messages System Architecture Smart Messages System Architecture Implementation Implementation Outdoor applications: EZCab and TrafficView Outdoor applications: EZCab and TrafficView Conclusions Conclusions

34 Spatial Programming Implementation Using Smart Messages SP application translates into an SM SP application translates into an SM –Spatial reference access translates into an SM migration to the mapped node –Embedded system properties: Tags SM self-routes using geographical routing and content-based routing SM self-routes using geographical routing and content-based routing Reference consistency Reference consistency − Unique addresses (stored in mapping table) are unique tags created at nodes − SM carries the mapping table

35 SM Prototype Implementation Modified version of Sun’s Java K Virtual Machine Modified version of Sun’s Java K Virtual Machine –Small memory footprint (160KB) SM and tag space primitives implemented inside virtual machine as native methods (efficiency) SM and tag space primitives implemented inside virtual machine as native methods (efficiency) I/O tags: GPS location, neighbor discovery, image capture, light sensor, system status I/O tags: GPS location, neighbor discovery, image capture, light sensor, system status Ad hoc networks of HP iPAQ PDAs running Linux

36 Smart Phones: The Mobile Computers of the Future Accessing our mobile data anytime/anywhere Proximity-aware interactions with embedded systems and services Developed SM implementation that works over unmodified Java Virtual Machines using bytecode instrumentation Developed SM implementation that works over unmodified Java Virtual Machines using bytecode instrumentation

37Outline Motivation Motivation Spatial Programming Model Spatial Programming Model Smart Messages System Architecture Smart Messages System Architecture Implementation Implementation Outdoor applications: EZCab and TrafficView Outdoor applications: EZCab and TrafficView Conclusions Conclusions

38 Booking Cabs on Broadway Phase1: Fight with other people for a cab The nightmare! Phase2: Call a dispatching center … and wait, and wait

39 EZCab System Overview Need a cab Use mobile ad hoc networks of cabs to book a free cab Use mobile ad hoc networks of cabs to book a free cab Each cab has short-range wireless interface and GPS Each cab has short-range wireless interface and GPS

40 EZCab Implementation Prototype over Smart Messages – –Evaluated over small scale networks (8 nodes) Developed and simulated probabilistic routing algorithms to find free cabs – –Each routing table entry corresponds to a neighbor address and the estimated probability to find a free cab through it – –Probabilities account for the distance and number of free cabs reachable through a neighbor – –Proactive probabilistic routing works best

41 How to Provide Dynamic, Real-Time View of the Road Traffic? What’s in front of that bus? What’s behind the bend? On rainy days On foggy days

42 TrafficView System Overview 3 cars on Route 1 Traffic Information Displayed in the Last Car Uses vehicle-to-vehicle ad hoc networks Uses vehicle-to-vehicle ad hoc networks Vehicles have embedded computer, GPS receiver, and short-range wireless interface (IEEE ) Vehicles have embedded computer, GPS receiver, and short-range wireless interface (IEEE )

43 TrafficView Implementation Implemented in C over Linux Implemented in C over Linux –In the process of porting it over Smart Messages Developed accurate road identification software Developed accurate road identification software Designed and evaluated scalable data aggregation algorithms Designed and evaluated scalable data aggregation algorithms –Use aggregation to put as much data as possible in one packet –Aggregate data for vehicles that are close to each other –Perform more aggregation as distance increases –Maintain “ acceptable ” accuracy loss

44Conclusions Spatial Programming makes outdoor distributed computing simple Spatial Programming makes outdoor distributed computing simple –Volatility, mobility, configuration dynamics, ad hoc networking are hidden from programmer Implementation on top of Smart Messages Implementation on top of Smart Messages –Easy to deploy new applications in the network –Quick adaptation to highly dynamic network configurations Demonstrated real-life outdoor distributed applications Demonstrated real-life outdoor distributed applications

45 What should we expect in the future? Pervasive computing applications will span non- traditional computing domains – –Healthcare, transportation, homeland security, etc. To make the pervasive computing a reality, we need a significant amount of system research – –Crossroad between networking, operating systems, embedded systems, human-computer interaction, computer vision, etc. Inter-disciplinary research is the key to success in pervasive computing – –Collaborations with civil engineering, cognitive sciences, biology, etc.

46Acknowledgments Liviu Iftode, Ulrich Kremer, Chalermek Intanagonwiwat Liviu Iftode, Ulrich Kremer, Chalermek Intanagonwiwat Porlin Kang, Nishkam Ravi, Deepa Iyer, Akhilesh Saxena, Peng Zhou, Tamer Nadeem, Bogdan Dorohonceanu Porlin Kang, Nishkam Ravi, Deepa Iyer, Akhilesh Saxena, Peng Zhou, Tamer Nadeem, Bogdan Dorohonceanu

47 Thank you!