A Lightweight Platform for Integration of Resource Limited Devices into Pervasive Grids Stavros Isaiadis and Vladimir Getov University of Westminster

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 2 Current Grid Systems Motivation Project Overview Research Issues Operational Overview Conclusion Outline

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 3 Current Grid Systems Grids were initially developed to support large (mainly) scientific data-centric applications Resources were in abundance and Grids orchestrated sharing in large scale and/or harnessing of idle resources Current Grid systems assume: –Plenty of resources –Relatively reliable and static networks Traditionally, Grid systems are not concerned with the platform’s footprint: platform is “heavy” and cannot be deployed in resource limited devices

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 4 Grids and “Small” Devices But networks have evolved. The “edge” of networks increasingly consists of resource limited and/or mobile devices The Grid, however, hasn’t yet allowed the integration of these devices While access to the Grid and job submission is generally supported (e.g. through portals), contribution of resources is very difficult Resource constraints make installation of Grid middleware prohibiting

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 5 Current Grid Systems Motivation Project Overview Research Issues Operational Overview Conclusion Outline

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 6 But Why Integrate Resource Limited Devices? Future Grid systems should be truly pervasive and ubiquitous Mobile and pervasive computing attract a big part of current research and industry funds and hold a big share of the market pie Small and/or mobile devices increasingly offer functionality not available in traditional Grid nodes –multimedia equipment, sensors, global positioning systems etc.

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 7 Current Grid Systems Motivation Project Overview Research Issues Operational Overview Conclusion Outline

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 8 A Lightweight Grid Platform A lightweight platform will provide the foundations for our project: Should be based on the CoreGRID component model Will consist of a small well defined set of core components Extra functionality through pluggable components Small footprint and communication overhead so that it won’t drain resources

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 9 Current Grid Systems Motivation Project Overview Research Issues Operational Overview Conclusion Outline

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 10 Dealing with Device Dynamicity Increased dynamicity is not acceptable in complex and heavily loaded Grid systems Solution: –Hide devices behind a proxy and delegate finer control (scheduling, monitoring, recovery etc) to a local community system –Devices now form a virtual “cluster” and are presented as a single entity to the Grid.

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 11 Virtual Aggregator Services Similar services (that implement the same interface) are aggregated and published through a single interface at the proxy The aggregator service implementation orchestrates access to the services it represents (by coordinating the various components like schedulers, indexes etc.) A single consistent interface to all aggregated resources is presented regardless of the internal state of the “cluster”

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 12 Making an Unstable Environment Virtually Stable When aggregating services that provide controlled access to raw resources (like cpu cycles, memory, storage etc.), we don’t present the total aggregate but only a fraction This way we can mask internal failures by reallocating resources from within the cluster without notifying (and thus placing extra burden on) higher level Grid components

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 13 Dealing with Failures Even behind a cluster, small devices still suffer from a high number of failures and we must provide the best possible quality of service Solution: –A monitoring and failure recovery framework to predict, detect, diagnose and recover from failures –Forecasting components to support the community scheduling system providing recommendations on the resource usage plans

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 14 The Monitoring Framework A set of small monitoring agents installed in the devices. They collect state information and forward them to the proxy A set of monitoring components at the proxy that collect and analyze the information –A monitoring “gateway” makes all decisions and orchestrates the other components –A forecasting component provides predictions on the best possible usage plan (according to usage and failure history) –Failure recovery components decide on the recovery plans based on predefined scenarios

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 15 Current Grid Systems Motivation Project Overview Research Issues Operational Overview Conclusion Outline

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 16 Component Prototypes Currently using Globus Toolkit 3.2 middleware Will port to GT4 in the very near future Prototyping components using Java Prototypes are quite restrictive at this stage but they will: Help us get a better insight Help us identify possible obstacles Provide the skeleton for the functional final platform

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 17 Operational Overview: Registration of Services AGGREGATED SERVICE INTERFACES RESOURCE META-DATA REPOSITORY Add/Update Interface Update Index Discovery and Interface Identification Registration LIVE RESOURCE INDEX

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 18 Operational Overview: Job Submission Job Submission Forward Consult Application Tasks App/ Code Job Requirements Distribution AGGREGATED SERVICE INTERFACES SERVICE IMPLEMENTATION LIVE RESOURCE INDEX TASK EXTRACTION ENGINE COMMUNITY SCHEDULING SYSTEM

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 19 Operational Overview: Failure Recovery JOB META-DATA REPOSITORY Event notification + status info diagnosis forward update Event notification + status info FORECASTING COMPONENT LIVE RESOURCE INDEX MONITORING GATEWAY FAILURE DIAGNOSIS FAILURE RECOVERY plan PLAN DEPLOYMENT

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 20 Current Grid Systems Motivation Project Overview Research Issues Operational Overview Conclusion Outline

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 21 Conclusion The lightweight platform could provide the foundations for truly ubiquitous Grids Coupled with a lightweight monitoring framework, it is a big first step towards full and efficient integration of resource limited devices into the Grid The early prototyping phase will help us gain a better insight on the issues that might arise and will provide the skeleton for the final deliverable platform

Questions? …or comments, suggestions, ideas… Contact: