Presentation is loading. Please wait.

Presentation is loading. Please wait.

Tools and Services for Data Intensive Research Roger Barga, Architect eXtreme Computing Group, Microsoft Research An Elephant Through the Eye of a Needle.

Similar presentations


Presentation on theme: "Tools and Services for Data Intensive Research Roger Barga, Architect eXtreme Computing Group, Microsoft Research An Elephant Through the Eye of a Needle."— Presentation transcript:

1 Tools and Services for Data Intensive Research Roger Barga, Architect eXtreme Computing Group, Microsoft Research An Elephant Through the Eye of a Needle

2 Select eXtreme Computing Group (XCG) Initiatives Cloud Computing Futures – ab initio R&D on cloud hardware/software infrastructure Multicore academic engagement – Universal Parallel Computing Research Centers (UPCRCs) Software incubations – Multicore applications, power management, scheduling Quantum computing – Topological quantum computing investigations Security and cryptography – Theoretical explorations and software tools Research cloud engagement – Worldwide government and academic research partnerships – Inform next generation cloud computing infrastructure

3 Data Intensive Research The nature of scientific computing is changing – It is about the data… Hypothesis-driven research – I have an idea, let me verify it... Exploratory – What correlations can I glean from everyones data? Requires different tools and techniques – Exploratory analysis relies on data mining, viz analytics – grep is not a data mining tool, and neither is a DBMS… Massive, multidisciplinary data – Rising rapidly and at unprecedented scale

4 Research 1.Have good idea 2.Write proposal 3.Wait 6 months 4.If successful, wait 3 months 5.Install Computers 6.Start Work Science Start-ups 1.Have good idea 2. Write Business Plan 3.Ask VCs to fund 4.If successful.. 5.Install Computers 6.Start Work Cloud Computing Model 1.Have good idea 2.Grab nodes from Cloud provider 3.Start Work 4.Pay for what you used also scalability, cost, sustainability Why Commercial Clouds are Important * * Slide used with permission of Paul Watson, University of Newcastle (UK)

5 Moores Law favored consumer commodities Economics drove enormous improvements Specialized processors and mainframes faltered The commodity software industry was born The Pull of Economics (follow the money) This will drive changes in research computing and cloud infrastructure Just as did killer micros and inexpensive clusters

6 Drinking from the Twitter Fire Hose Assume the order of magnitude of the twitter user base is in the MM range, lets crank this up to the 500M range. The average Twitter user is generating a relatively low incoming message rate right now, assume that a users devices (phone, car, PC) are enhanced to begin auto-generating periodic Twitter messages on their behalf, e.g. with location pings and solving other problems that twitterbots are emerging to address. So lets say the input rate grows again to 10x-100x what it was in the previous step.twitterbots On the input end Start with the twitter fire hose, messages flowing inbound at specific rate. Enrich each element with significantly more metadata, e.g. geolocation.

7 On the input end On the output end: three different usage modalities Each user has one or more agents they run on their behalf, monitoring this input stream. This might just be a client that displays a stream that is incoming from or #topics or the (user standing queries). A user can do more general queries from a search page. This query may have more unstructured search terms than the above, and it is expected not just to be going against incoming stream but against much larger corpus of messages from the entire input stream that has been persisted for days, weeks, months, years… Finally, analytical tools or bots whose purpose is to do trend analysis on the knowledge popping out of the stream, in real-time. Whether seeded with an interest (let me know when a problem pops up with that will damage my companys reputation) or just discovering a topic from the noise (let me know when a new hot news item emerges), both must be possible. Drinking from the Twitter Fire Hose

8 Pause for Moment… Defining representative challenges or quests to focus group attention is an excellent way to proceed as a community Publishing a whitepaper articulating these challenges is a great way to allow others to contribute to a shared research agenda Make simulated and reference data sets available to ground such a distributed research effort

9 On the input end On the output end: three different usage modalities A combination of live data, including streaming, and historical data Lots of necessary technology, but no single technology is sufficient If this is going to be successful it must be accessible to the masses Simple to use and highly scalable, which is extremely difficult because in actuality it is not simple… Drinking from the Twitter Fire Hose

10 This Talk is About Intersection of four fundamental strategies 1.Distribute Data and perform Parallel Processing 2.Parallel operations to take advantage of multiple cores ; 3.Reduce the size of the data accessed – Data compression – Data structures that limit the amount of data required for queries; 4.Stream data processing to extract information before storage Effort to build & port tools for data intensive research in the cloud – None have run in the cloud to date or at scale we are targeting… Able to handle torrential streams of live and historical data – Goal is simplicity and ease-of-use combined with scalability

11 Microsofts Dryad Continuously deployed since 2006 Running on >> 10 4 machines Sifting through > 10Pb data daily Runs on clusters > 3000 machines Handles jobs with > 10 5 processes each Used by >> 100 developers Rich platform for data analysis Microsoft Research, Silicon Valley Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, Dennis Fetterly

12 Pause for Moment… Data-Intensive Computing Symposium, 2007 Dryad is now freely available Thanks to Geoffrey Fox (Indiana) and Magda Balazinska (UW) as early adopters Commitment by External Research (MSR) to support research community use

13 Simple Programming Model Terasort, well known benchmark, time to sort time 1 TB data [J. Gray 1985] Sequential scan/disk = 4.6 hours DryadLINQ provides simple but powerful programming model Only few lines of code needed to implement Terasort, benchmark May 2008 DryadLINQ result: 349 seconds (5.8 min) Cluster of 240 AMD64 (quad) machines, 920 disks Code: 17 lines of LINQ DryadDataContext ddc = new DryadDataContext(fileDir); DryadTable records = ddc.GetPartitionedTable (file); var q = records.OrderBy(x => x); q.ToDryadPartitionedTable(output);

14 LINQ Microsofts Language INtegrated Query – Available in Visual Studio 2008 A set of operators to manipulate datasets in.NET – Support traditional relational operators Select, Join, GroupBy, Aggregate, etc. Data model – Data elements are strongly typed.NET objects – Much more expressive than SQL tables Extremely extensible – Add new custom operators – Add new execution providers

15 Dryad Generalizes Unix Pipes Unix Pipes: 1-D grep | sed | sort | awk | perl Dryad: 2-D, multi-machine, virtualized grep 1000 | sed 500 | sort 1000 | awk 500 | perl 50

16 Dryad Job Structure grep sed sort awk perl grep sed sort awk Input files Vertices (processes) Output files Channels Stage Channel is a finite streams of items NTFS files (temporary) TCP pipes (inter-machine) Memory FIFOs (intra-machine) Channel is a finite streams of items NTFS files (temporary) TCP pipes (inter-machine) Memory FIFOs (intra-machine)

17 Dryad System Architecture Files, TCP, FIFO, Network job schedule data plane control plane NSPD V VV Job managercluster

18 JM code Vertex Code Dryad Job Staging 1. Build 2. Send.exe 3. Start JM 5. Generate graph 7. Serialize vertices 8. Monitor vertex execution 4. Query cluster resources Cluster services 6. Initialize vertices

19 Dryad Scheduler is a State Machine Static optimizer builds execution graph – Vertex can run anywhere once all its inputs are ready. Dynamic optimizer mutates running graph – Distributes code, routes data; – Schedules processes on machines near data; – Adjusts available compute resources at each stage; – Automatically recovers computation, adjusts for overload o If A fails, run it again; o If As inputs are gone, run upstream vertices again (recursively); o If A is slow, run a copy elsewhere and use output from one that finishes first. – Masks failures in cluster and network;

20 PLINQ Local Machine.Net program (C#, VB, F#, etc) Execution Engines Query Objects LINQ-to-IMDB DryadLINQ LINQ-to-CEP LINQ provider interface Scalability Single-core Multi-core Cluster

21 LINQ == Tree of Operators A query is comprised of a tree of operators As with a program AST, these trees can be analyzed, rewritten This is why PLINQ can safely introduce parallelism q = from x in A where p(x) select x 3 ; Intra-operator: Inter-operator: Both composed: Nesting queries inside of others is common PLINQ can fuse partitions var q1 = from x in A select x*2; var q2 = q1.Sum();

22 Combining with PLINQ Query DryadLINQ PLINQ subquery

23 Combining with LINQ-to-IMDB DryadLINQ Subquery Query LINQ-to-IMDB Historical Reference Data

24 Combining with LINQ-to-CEP DryadLINQ Subquery Query LINQ-to-IMDB Subquery LINQ-to-CEP Live Streaming Data

25 Cost of storing data – few cents/month/MB Cost of acquiring data – negligible Extracting insight while acquiring data - priceless Mining historical data for ways to extract insight – precious CEDR CEP – the engine that makes it possible Consistent Streaming Through Time: A Vision for Event Stream Processing Roger S. Barga, Jonathan Goldstein, Mohamed H. Ali, Mingsheng Hong In the proceedings of CIDR 2007 Consistent Streaming Through Time: A Vision for Event Stream Processing Roger S. Barga, Jonathan Goldstein, Mohamed H. Ali, Mingsheng Hong In the proceedings of CIDR 2007

26 Complex Event Processing Complex Event Processing (CEP) is the continuous and incremental processing of event (data) streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

27 The CEDR (Orinoco) Algebra Leverages existing SQL understanding – Streaming extensions to relational algebra – Query integration with host languages (LINQ) S emantics are independent of order of arrival – Specify a standing event query – Separately specify desired disorder handling strategy – Many interesting repercussions Consistent Streaming Through Time: A Vision for Event Stream Processing Roger S. Barga, Jonathan Goldstein, Mohamed H. Ali, Mingsheng Hong In the proceedings of CIDR 2007 Consistent Streaming Through Time: A Vision for Event Stream Processing Roger S. Barga, Jonathan Goldstein, Mohamed H. Ali, Mingsheng Hong In the proceedings of CIDR 2007

28 CEDR (Orinoco) Overview Currently processing over 400M events per day for internal application (5000 events/sec)

29 Reference Data on Azure Ocean Science data on Azure SDS-relational Two terabytes of coastal and model data Collaboration with Bill Howe (Univ of Washington) Computational finance data on Azure SDS-relational BATS, daily tick data for stocks (10 years) XBRL call report for banks (10,000 banks) Working with IRIS to store select seismic data on Azure. IRIS consortium based in Seattle (NSF) collects and distributes global seismological data. Data sets requested by researchers worldwide Includes HD videos, seismograms, images, data from major seismic events.

30 Data growing exponentially: big data, with big implications… Implications for research environments and cloud infrastructure Building cloud analysis & storage tools for data intensive research – Implementing key services for science (PhyloD for HIV researchers) – Host select data sets for multidisciplinary data analysis Ongoing discussions for research access to Azure – Many PB of storage and hundreds of thousands of core-hours – Internet2/ESnet connections, w/ service peering at high bandwidth – Drive negotiations with ISVs for pay-as-you-go licensing (MATLAB) Academic access to Azure through our MSDN program Technical engagement team to onboard research groups – Tools for data analysis, data storage services, and visual analytics Summary

31 Questions


Download ppt "Tools and Services for Data Intensive Research Roger Barga, Architect eXtreme Computing Group, Microsoft Research An Elephant Through the Eye of a Needle."

Similar presentations


Ads by Google