Download presentation
Presentation is loading. Please wait.
Published byHelen Porter Modified over 9 years ago
1
© 2012 IBM Corporation Platform Computing 1 IBM ® Platform™ Symphony A Smarter Platform for Distributed Computing and Big Data Analytics Updated for Platform Symphony version 6.1.1 Gord Sissons Platform Symphony Product Marketing Scott Campbell Platform Symphony Product Manager
2
© 2012 IBM Corporation Platform Computing 2 Product Positioning Platform Symphony
3
© 2012 IBM Corporation Platform Computing 3 Platform LSF Platform Symphony Command line driven (bsub) API driven (C++, C#, Java) Each job discrete Many tasks per session Medium / long job execution time Task computation time may be very short - milliseconds Overhead to schedule a job relatively high - seconds Very fast scheduling, ultra-low latency < 1 millisecond The compute resources allocated to a job generally “static” Service instances can flex and be re- allocated very rapidly Generally relies on a shared file system to move data Fast “in-band” communications to move data Data requirements met by shared FS or parallel file systems Customer requires MapReduce: Hadoop, Pig, Hbase etc.. What to position when?
4
© 2012 IBM Corporation Platform Computing 4 IBM Platform LSF Family IBM Platform Symphony Family Batch, command line oriented, MPI Batch scheduling policies Sophisticated scheduling policies Portals, process automation Service-oriented, API driven Extreme throughput / low latency Agile fine-grained resource sharing Big Data, Hadoop requirements Complementary solutions IBM Platform Cluster Manager Advanced Edition Platform Cluster Manager AE an provision both Platform LSF and Symphony clusters IBM Platform Symphony and IBM Platform LSF can share the same infrastructure Resource Orchestrator (EGO)
5
© 2012 IBM Corporation Platform Computing 5 Customer challenges Need for deeper more thorough analysis Results increasingly time critical Insatiable appetite for analytic capacity Exploding data volumes IT budgets not keeping pace with demand Ever increasing expectations
6
© 2012 IBM Corporation Platform Computing 6 Parallelizable problems demand fresh approaches Calculate this now! – over 500,000 scenarios, 500 instruments, 200 time steps. Financial Market & Credit Risk, Insurance Mine 24 months of credit card purchases for 30,000,000 cardholders to identify credit-worthy customers by geography Credit-scoring, ETL, Fraud Detection Contrails – perform assembly and mapping of large genomes in hours rather than weeks using MapReduce programming model. Life Sciences – Genome Mapping Perform designs of experiment and parametric sweeps for a variety of computer-aided design applications to find optimal designs without physical prototyping. CAE – Parametric sweeps, DOE
7
© 2012 IBM Corporation Platform Computing 7 Product Overview IBM Platform Symphony
8
© 2012 IBM Corporation Platform Computing 8 IBM Platform Symphony is the most powerful management software for running distributed applications and big data analytics on a scalable, multi-tenant, heterogeneous grid. It accelerates a wide variety of parallel applications, quickly computing results while making optimal use of available infrastructure. About IBM Platform Symphony
9
© 2012 IBM Corporation Platform Computing 9 IBM Platform Symphony A heterogeneous grid management platform A high-performance SOA middleware environment Supports diverse compute & data intensive applications ISV applications In-house developed applications (C/C++, C#/.NET, Java, Excel, R etc) Optimized low-latency Hadoop compatible run-time Can be used to launch, persist and manage non-grid aware application services React instantly to time critical-requirements A multi-tenant shared services platform with unique resource sharing capabilities A limited-use run-time for Platform Symphony (called Adaptive MapReduce) included in IBM InfoSphere ® BigInsights ® 2.1
10
© 2012 IBM Corporation Platform Computing 10 IBM Platform Symphony – Production Proven “IBM Platform Symphony provided us with a 100-fold capacity increase in support of Dodd-Frank compliance” Head of Risk, Major North American Exchange “Along with enabling us to maximize the use of all our available computing resources, the Platform Symphony grid is enabling our IT users to be more creative in how they share computing resources to do their development and testing work, thus helping us achieve our expensive efficiency targets.” SVP, Wall Street Investment Bank http://www.youtube.com/watch?v=MMsYacyNLo4 “Platform Symphony and our compute efficiency program has shown that we can massively reduce the number of cores used while simultaneously providing improved service levels” Executive Director, Chief Business Technologist, Major Global Investment Bank “IBM then proved its approach, taking an existing job that required two hours on 20,000 cores and running it in one hour on 10,000 cores. Achieving twice the performance on half the infrastructure was absolutely compelling for us.” CIO Major Wall Street Investment Bank “We realized an Immediate 40% cost deferral on H/W purchase based on 30,000 cores to start, resulting in an initial savings of $4.7M per year.” European Investment Bank “Exact risk analysis is a prerequisite for any transaction. After deploying the grid computing solution, we were suddenly able to compute tasks in real-time that used to run only once a day on one specific machine.” General Manager, European Commodities Exchange "Platform Computing and its enterprise grid solution enable us to share a formerly heterogeneous and distributed hardware infrastructure across applications regardless of their location, operating system and application logic, therefore helping us to achieve our internal efficiency targets while at the same time improving our performance and service quality." Head of Global Markets and Treasury Infrastructure, Major European Investment Bank
11
© 2012 IBM Corporation Platform Computing 11 About IBM Platform Symphony Resource Orchestration Workload Manager C C C C C C C C C C C C D D D D D D D D D D D D C C C C C C A A A A A A A A A A A A A A A A B B B B B B B B B B B B B B B B B B Third Party Trading & Risk Platforms In-house Applications InfoSphere BigInsights A B C D Multiple users, applications and lines of business on a shared, heterogeneous, multi-tenant grid
12
© 2012 IBM Corporation Platform Computing 12 Platform Symphony Infrastructure App 1 App 2 App 3 ClientPlatform Symphony Management Hosts Compute Nodes Master Candidates Platform Enterprise Reporting Framework SHARED Storage to persist Transactions Dedicated Hardware Harvested Hardware
13
© 2012 IBM Corporation Platform Computing 13 IBM Platform Symphony – Data Intensive Applications Platform Management Console Platform Enterprise Reporting Framework Resource Orchestrator Low-latency Service-oriented Application Middleware Service Instance Manager (SIM) Enhanced MapReduce Processing Framework DATA INTENSIVE Platform Symphony Core COMPUTE INTENSIVE
14
© 2012 IBM Corporation Platform Computing 14 Enables sharing while preserving ownership Near 100% sustained resource utilization Allocations flex quickly to reflect business priorities Support new applications with existing infrastructure Platform Symphony improves on application SLAs while using resources more efficiently than competing grid managers IBM Platform Symphony – Sophisticated Resource Sharing
15
© 2012 IBM Corporation Platform Computing 15 Ressourcen-Sharing-Policies : Ownership policy Sharing policy Borrowing/lending policy Reclamation policy Rank IBM Platform Symphony – Sophisticated Resource Sharing
16
© 2012 IBM Corporation Platform Computing 16 EGO Concepts EGO Platform LSF® Batch Processing Accounting System Company Intranet Web-based Storefront (Marketing gets 20% of resource from 9-5 If response on storefront > 7 sec, then…) 1 – Consumers (requests) 2 - Resources 3 – Policy 4 – Activities
17
© 2012 IBM Corporation Platform Computing 17 IBM Platform Symphony
18
© 2012 IBM Corporation Platform Computing 18 Platform Symphony – Compute Intensive Applications Service Instance Manager (SIM) Platform Management Console Platform Enterprise Reporting Framework Low-latency Service-oriented Application Middleware Enhanced MapReduce Processing Framework DATA INTENSIVE COMPUTE INTENSIVE Resource Orchestrator Platform Symphony Core COMPUTE INTENSIVE
19
© 2012 IBM Corporation Platform Computing 19 Platform Symphony – Compute Intensive Applications Low latency / Hi-throughput Sub-millisecond response >17,000 tasks per second throughput* Large scale 10,000 cores per application 40,000 cores per grid Cost efficient, shared services Multi-tenant grid solution Guarantees SLAs while encouraging resource sharing Easy to on-board new grid applications Maximizes use of grid resources Heterogeneous & open Linux, Windows / Windows HPC, AIX, Solaris C/C++, C#, Java, Excel, Python, R* Smart data handling, Data Affinity * Benchmark refers to task sending throughput. Revolution R is available from third party.
20
© 2012 IBM Corporation Platform Computing 20 SD Client SD Client SSM Heterogeneous Applications Available off-grid servers & virtualization platforms Harvestable desktops SD SSM SIM Service SIM Service SIM Service SD SSM SIM Service SIM Service SIM Service SIM Service SIM Service SIM Service SIM Service SIM Service SIM Service SIM Service SIM Service C++ SIM Service SIM Service SIM Service SIM Service SIM Service SIM Service SIM Service SIM Service C++ SIM Service C++ SIM Service C++ Grid Orchestration Services Management Hosts Compute Hosts Platform Symphony provides fast, reliable services for compute and data intensive applications IBM Platform Symphony – Simplified Architecture
21
© 2012 IBM Corporation Platform Computing 21 Other Grid Server Broker Engines Each engine polls broker ~5 times per second (configurable) Send work when engine ready Client Serialize input data Network transport (client to broker) Wait for engine to poll broker Network transport (broker to engine) De-serialize Input data Compute Result Serialize result Post result back to broker Time … Broker Compute time Platform Symphony is (much) faster because: Efficient C language routines use CDR (common data representation) and IOCP rather than slow, heavy-weight XML data encoding) Network transit time is reduced by avoiding text based HTTP protocol and encoding data in more compact CDR binary format Processing time for all Symphony services is reduced by using a native HPC C/C++ implementation for system services rather than Java Platform Symphony has a more efficient “push model” that avoids entirely the architectural problems with polling Platform Symphony Serialize input Network transport SSM Compute time & logging Time Network transport (SSM to engine) De-serialize … Serialize Network transport (engine to SSM) Compute result No wait time due to polling, faster serialization/de-serialization, More network efficient protocol Why IBM Platform Symphony is faster and more scalable
22
© 2012 IBM Corporation Platform Computing 22 Latency Scale Inefficient scheduling, polling model & heavy-weight transport protocols limit scalability. Other Grid Servers Symphony With a zero-wait time “push model” and efficient binary protocols, Symphony scales until the “wire” is saturated Why IBM Platform Symphony is faster and more scalable
23
© 2012 IBM Corporation Platform Computing 23 Client-Server Model, no peer-to-peer like MPI Simple interface o Programmer minimally aware of scheduling, preemption, provisioning, etc. Ideally suited for “embarrassingly parallel” work o Tasks are atomic and in general small o Little or no communication between concurrently executing tasks o No communication between task and client while task computes Also other paradigms are supported like parent / child IBM Platform Symphony Programming concept
24
© 2012 IBM Corporation Platform Computing 24 Service Oriented Approach Symphony Client Input Service Output o Client: Client: Initiates work, controls sessions (i.e. jobs), sends messages & receives replies (jobs or error messages) o Service: Receives messages, processes the jobs and sends replies
25
© 2012 IBM Corporation Platform Computing 25 IBM Platform Symphony API Initialize and Connect Initialize and Connect Create Session ( valuation for Instrument X) Create Session ( valuation for Instrument X) Send inputs (instrument info, Market info) Send inputs (instrument info, Market info) Retrieve outputs (valuation answer) Retrieve outputs (valuation answer) Client Service Implement Symphony Service Interface -onInvoke(taskPointer) Implement Symphony Service Interface -onInvoke(taskPointer) Preparation Distribution Aggregation Quick and simple application integration through Symphony API
26
© 2012 IBM Corporation Platform Computing 26 IBM Platform Symphony API The targeted API can be extended as needed without affecting the core Supports all most programming languages through layered architecture
27
© 2012 IBM Corporation Platform Computing 27 IBM Platform Symphony serialization Input Message Output Message Platform Symphony Input Message Output Message Client ApplicationService Serialize Deserialize Compute task IBM Platform Symphony serialization supports mixing of programming languages between client and service for maximum reusability
28
© 2012 IBM Corporation Platform Computing 28 IBM Plafrom Symphony Deployment Symphony automatically deploys and updates packages (binaries) needed to run certain workload By seperation of packages and application definition, service packages can be reused for several applications Supports seperation of concerns by package dependencies (seperate package for business and technical layer) when different deployment cycles are needed Applications can be associated with different packages so that you can run different application versions at the same time Automated Package Deployment
29
© 2012 IBM Corporation Platform Computing 29 IBM Platform Symphony – Data Intensive Applications Platform Management Console Platform Enterprise Reporting Framework Resource Orchestrator Low-latency Service-oriented Application Middleware Service Instance Manager (SIM) Enhanced MapReduce Processing Framework DATA INTENSIVE COMPUTE INTENSIVE Platform Symphony Core COMPUTE INTENSIVE
30
© 2012 IBM Corporation Platform Computing 30 Best-in class Hadoop MapReduce implementation Software framework supporting distributed computing on large data sets Key advantages Higher performance – ~7x faster than Hadoop for short-run jobs Reliable, highly available, rolling upgrades Resources managed dynamically Fully compatible: Java MR, PIG, HIVE, Oozie, Hbase Open data architecture: File systems & databases IBM Platform Symphony Advanced Edition Enhanced MapReduce processing framework
31
© 2012 IBM Corporation Platform Computing 31 Resource Orchestrator Job Controller & Scheduler Job Controller & Scheduler Map Task Local Storage Input Folder Output folder Split data and allocate resources for applications Map Task Map Task(s) Reduce Task(s) Platform Symphony Hadoop MapReduce API Pluggable Distributed File System / Storage Input Folder Output folder Fully Hadoop compatible MapReduce implementation. Platform MapReduce Includes HDFS and supports GPFS and other system & database technologies. IBM Platform Symphony Advanced Edition Enhanced MapReduce processing framework
32
© 2012 IBM Corporation Platform Computing 32 Symphony versus Hadoop – HiBench KMeans Symphony provides 3X improvement in iterative workload
33
© 2012 IBM Corporation Platform Computing 33 Hadoop “state of the art” Latest scheduler performance Platform Symphony 5.2 MapReduce engine performance 66X Improvement Hadoop “sleep test” – scheduling performance Symphony vs. commercial and open source Hadoop distros
34
© 2012 IBM Corporation Platform Computing 34 About the Berkeley Facebook SWIM benchmark “Real-world” MapReduce benchmark – essentially “replays” production Facebook workload traces and workloads from other production environments Developed by Yanpei Chen (now at Cloudera) and others at @ UCB - https://github.com/SWIMProjectUCB/SWIM/wiki https://github.com/SWIMProjectUCB/SWIM/wiki Viewed as an advance over existing synthetic MapReduce benchmarks including GridMix2, PigMix, Hive BM etc. Designed to scale to different sized clusters Represents workloads comprised of short, large and huge jobs stressing disk, network IO, CPU and memory exhibiting the characteristics as the production workload Advantages of SWIM over other benchmarks promoted at Hadoop World 2011 - http://www.slideshare.net/cloudera/hadoop-world-2011-hadoop-and-performance-todd- lipcon-yanpei-chen-cloudera http://www.slideshare.net/cloudera/hadoop-world-2011-hadoop-and-performance-todd- lipcon-yanpei-chen-cloudera Significant because: Using a real-world workload, IBM BigInsights on a low-latency Platform Computing scheduler (Adaptive MapReduce) delivers better performance, and supports equivalent workloads with much less infrastructure
35
© 2012 IBM Corporation Platform Computing 35 SWIM Benchmark results Hadoop 1.1.1 vs. IBM BigInsights 2.1 with Adaptive MapReduce Total run-time lower is better 2.6 x faster!
36
© 2012 IBM Corporation Platform Computing 36 About the SLEEP benchmark “Sleep” is one of the standard tests included in the Hadoop distribution. As the name implies, tasks simply sleep for a specified duration for each map and reduce task dispatched to the cluster. Recognized as an effective way to measure the scheduling efficiency of Hadoop distributions - http://www.slideshare.net/cloudera/hadoop-world-2011- hadoop-and-performance-todd-lipcon-yanpei-chen-cloudera http://www.slideshare.net/cloudera/hadoop-world-2011- hadoop-and-performance-todd-lipcon-yanpei-chen-cloudera A standard test is to schedule 5,000 map tasks and a single reduce task all having a run-time of just 1 millisecond. This stresses the scheduling capacity of a Hadoop cluster by essentially running “no-op” workloads. $ hadoop jar examples.jar sleep –mt1 –rt1 –m5000 –r1 Significant because: –IBM BigInsights demonstrates a ~10 times improvement in raw scheduling throughput with dramatically lower per-task latency even when Hadoop is tuned to use a fast 300 msec heartbeat interval
37
© 2012 IBM Corporation Platform Computing 37 SLEEP Test Benchmark results Hadoop 1.1.1 vs. IBM BigInsights 2.1 with Adaptive MapReduce Task throughput Higher is better 10 x faster!
38
© 2012 IBM Corporation Platform Computing 38 IBM Platform Symphony – Data Intensive Applications Platform Management Console Platform Enterprise Reporting Framework Resource Orchestrator Low-latency Service-oriented Application Middleware Service Instance Manager (SIM) Enhanced MapReduce Processing Framework DATA INTENSIVE Platform Symphony Core COMPUTE INTENSIVE
39
© 2012 IBM Corporation Platform Computing 39 IBM Platform Symphony Mangement Console Provides centralized log retrieval, resource and workload monitoring
40
© 2012 IBM Corporation Platform Computing 40 IBM Platform Symphony – Data Intensive Applications Platform Management Console Platform Enterprise Reporting Framework Resource Orchestrator Low-latency Service-oriented Application Middleware Service Instance Manager (SIM) Enhanced MapReduce Processing Framework DATA INTENSIVE Platform Symphony Core COMPUTE INTENSIVE
41
© 2012 IBM Corporation Platform Computing 41 IBM Platform Symphony Reporting
42
© 2012 IBM Corporation Platform Computing 42 IBM Platform Symphony Reporting SELECT TIME_STAMP, RESOURCE_NAME, ATTRIBUTE_NAME, ATTRIBUTE_VALUE_NUM FROM RESOURCE_METRICS WHERERESOURCE_NAME IN ('elecerf1.euro.corp.platform.com') AND ATTRIBUTE_NAME = 'ut' AND (TIME_STAMP >= TIMESTAMP(DATE({fn TIMESTAMPADD( SQL_TSI_DAY, - 1,CURRENT_TIMESTAMP)}), SUBSTR(CHAR(TIME({fn TIMESTAMPADD( SQL_TSI_DAY, - 1,CURRENT_TIMESTAMP)})), 1, 5) || ':00') AND TIME_STAMP < TIMESTAMP(DATE(CURRENT_TIMESTAMP), SUBSTR(CHAR(TIME(CURRENT_TIMESTAMP)), 1, 5) || ':00')) ORDER BY TIME_STAMP, RESOURCE_NAME Standard and Custom reports for workload and resources
43
© 2012 IBM Corporation Platform Computing 43 What makes Symphony unique? Higher quality results faster Starts & runs jobs the fastest Scales the highest Lower cost Uses infrastructure more efficiently Easier to manage Simplifies application integration Better resource sharing Integrated compute + data services Sophisticated hierarchical sharing model Harvesting & multi-site sharing options Smarter data handling Optimized, low-latency MapReduce implementation Considers data locality when scheduling tasks Adapts to multiple data sources World-wide support & services Consulting, Customer Education, Comprehensive support services
44
© 2012 IBM Corporation Platform Computing 44 IBM Platform Symphony 6.1.x.x Editions DEVELOPDEPLOYSCALECONVERGE IBM BigInsights Enterprise Edition (OEM license) Platform Symphony Developer Platform Symphony Express Platform Symphony Standard Platform Symphony Advanced Low Latency HPC SOA Agile Service & Task Scheduling - Dynamic Resource Orchestration -- Standard & Custom Reporting --- Server, VM, Desktop Harvesting Capability --- Data Affinity ---- MapReduce Framework -- Multi-Cluster Management ---- Max Hosts / CoresPer BI licensing2 Hosts240 Cores5K Hosts, 40K Cores Max Applications1 – BI only - 5300 Desktop Harvesting --- Server & VM Harvesting --- GPU --- Platform Analytics --- GPFSFPO in BI --
45
© 2012 IBM Corporation Platform Computing 45 IBM Platform Symphony – add-on products IBM Platform Symphony Desktop Harvesting IBM Platform Symphony GPU Harvesting IBM Platform Symphony Server/VM Harvesting Platform Management Console Platform Enterprise Reporting Framework Resource Orchestrator Low-latency Service- oriented Application Middleware Service Instance Manager (SIM) Enhanced MapReduce Processing Framework DATA INTENSIVE COMPUTE INTENSIVE Platform Symphony Core COMPUTE INTENSIVE IBM Platform Analytics IBM Platform Symphony Coprocessor Harvesting IBM Platform LSF add-on for Symphony
46
© 2012 IBM Corporation Platform Computing 46 IBM Platform Symphony Services World-class support organization Comprehensive support offerings Consistently high customer quality scores Tailored consulting offerings Self-serve support options Mission Critical Premium Standard Technical Account Manager, Quarterly reports Technical Critical Care, Migration Planning Assistance Proactively investigate customer-specific Issues Direct Access to R&D & Product Management Assigned Support Engineer (ASE) Faster SLA, Regular updates, fast ticket review Multi-site Co-ordination, Remote Health Monitoring Developer Support, Maintain customer profile After Hours Critical Changeover Coverage Local Business Hour Support Severity One 24x7 Hotline Software Q&A and Usage Assistance Software Upgrade and Patches Knowledge Base Articles, unlimited tickets
47
© 2012 IBM Corporation Platform Computing 47 IBM Platform Symphony Education Multiple training alternatives: Public or private classes On-premise or remote education options Customized curriculums Tailored to administrators or developers Top-quality instructors Platform Symphony Administration Platform Symphony Application Developers Platform MapReduce for IT Managers, Developers, System and Storage Administrators Platform MapReduce Training for System Administrators
48
© 2012 IBM Corporation Platform Computing 48 Purpose-built for both compute and data intensive applications Supports diverse, heterogeneous analytic workloads Better performance – more throughput with less hardware Sophisticated multi-tenancy – guarantee service levels, avoid cost Production proven at scale Complements and extends IBM’s Big Data portfolio IBM Platform Symphony – key messages
49
© 2012 IBM Corporation Platform Computing 49 IBM Platform Symphony New Features in 6.1.1
50
© 2012 IBM Corporation Platform Computing 50 New add-ons Platform Symphony co-processor harvesting for better integration and resource management of applications built for Intel ® Phi ® Support for IBM Platform LSF Standard Edition as a guest scheduler on an IBM Platform Symphony cluster MapReduce performance enhancements Terasort (1TB) – 18.8% performance improvement over Platform Symphony 6.1.0.1 Sleep job performance (50K mapper, 1 reducer) – 6.1% performance improvement over Platform Symphony 6.1.0.1 24.4% efficiency gain in runtime of SWIM benchmark with Facebook workload (no change in elapsed completion time) over 6.1.0.1 IBM Platform Symphony What’s new in IBM Platform Symphony 6.1.1
51
© 2012 IBM Corporation Platform Computing 51 New platform support Platform Symphony MultiCluster support for Linux ® & PowerLinux ® Symphony Developer Edition supports latest ECLIPSE IDE IBM DB2 support for Symphony reporting database Platform Analytics add-on now supporting PowerLinux GPU Harvesting support on Microsoft ® Windows ® Platform Analytics data loaders now available on Microsoft Windows Management enhancements Re-design of main GUI dashboard view Direct upgrade to Symphony 6.1.1 from Symphony 5.1, 6.1 and 6.1.0.1 EGO CLI enhanced to include auditing Security enhancements Microsoft Active Directory integration for single sign-on Improved Kerberos support Kerberos support for MapReduce workloads Role-based access for starting and stopping EGO services IBM Platform Symphony What’s new in IBM Platform Symphony 6.1.1
52
© 2012 IBM Corporation Platform Computing 52 WHAT’S NEW: Native support for Xeon Phi Tools for grid-enabling co-processor based applications Easily manage large Xeon Phi compute clusters Harvest available co-processor resources at run-time to maximize utilization and sharing CLIENT BENEFITS: Improve application performance Minimize application development & deployment time Ensure application reliability Simplify management and administration Reduce costs by maximizing the use of co-processors and sharing them efficiently among applications and users Learn More: http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/symphony/ Build, deploy and manage scaled-out high-performance applications using Intel ® Xeon Phi TM. Harvest co-processor resources and share them among users, applications and departments. IBM Platform Symphony Co-Processor Harvesting
53
© 2012 IBM Corporation Platform Computing 53 Share cluster resources between Platform Symphony workloads, Big Data workloads and traditional Platform LSF workloads for maximum resource utilization IBM Platform LSF add-on for Platform Symphony Resource Orchestration Workload Manager C C C C C C C C C C C C D D D D D D D D D D D D C C C C C C A A A A A A A A A A A A A A A A B B B B B B B B B B B B B B B B B B Third Party ISV + Proprietary Trading & Risk Platforms Batch workloads (SPSS, IBM DataStage, MATLAB, SAS etc.) InfoSphere BigInsights & 3 rd Party Hadoop A B C D IBM Platform LSF – CLI / APISymphony SOA / SymExecSymphony MapReduce
54
© 2012 IBM Corporation Platform Computing 54 IBM Platform Symphony Summary Faster & more scalable Heterogeneous & open Share infrastructure more efficiently Support compute & data intensive workloads Deploy applications faster Simplify management Top quality service and support
55
© 2012 IBM Corporation Platform Computing 55 Target Markets for Platform Symphony
56
© 2012 IBM Corporation Platform Computing 56 Financial Services Manufacturing Health & Life Sciences Government – Intelligence Oil & Gas Media & Entertainment E-Gaming Telco Retail Utilities … Compute IntensiveData Intensive Financial Services Health & Life Sciences Government – Intelligence Telco Retail Utilities Social Networks Internet Service Providers E-Gaming Media & Entertainment … Degree of Fit Target Markets
57
© 2012 IBM Corporation Platform Computing 57 12 of top 20 banks including 3 of the top 5 rely on Platform Symphony Production proven
58
© 2012 IBM Corporation Platform Computing 58 Key Partners: Financial Services & Insurance
59
© 2012 IBM Corporation Platform Computing 59 IBM Algorithmics IBM Infosphere BigInsights Microsoft Excel Calypso GGY AXIS SunGard Front Arena, Adaptiv Sophis Risque Mathworks MatLab R Statistical Language Any Hadoop / BigInsights Application Pig Hive Oozie Datameer GPFS Think Platform Symphony for these applications! Integrated Applications These are examples of applications that have been integrated with IBM Platform Symphony. The level of effort required to effect the integration will depend on details such as software versions and hardware and operating system platforms deployed.
60
© 2012 IBM Corporation Platform Computing 60 State of the Art Grid Integration Accelerate a variety of analytic applications on a shared grid Grid provides needed capacity to meet new regulatory requirements Superior utilization keeps infrastructure costs to a minimum Agile scheduling enables time-critical analysis to immediately pre- empt longer running simulations enabling faster decisions Why Platform Symphony for Algorithmics? Superior performance and scale per unit infrastructure cost Unique dynamic lending and borrowing model maximizes efficiency and enables time-critical tasks to run faster “Push model” schedules time critical simulations instantly Common job nomenclature between Algo Batch Environment and Platform Symphony simplifies management Share the grid between Algorithmics and third-party analytic applications Unify management of heterogeneous grid applications Algorithmics, an IBM Company Full-featured Enterprise Risk Management Solution Enables counter-party credit risk (CCR) modeling with incremental CVA Provides needed capacity to meet new regulatory requirements Good solution for both large and mid-sized banks Enables banks to move from “measuring” to “managing” risk Key Partners: Within the IBM Family
61
© 2012 IBM Corporation Platform Computing 61 Trends Lower Latency (µ secs) Trade Volumes (10x in 5 years) Massive Data Real-time, Intraday vs. Overnight Regulation Dodd-Frank– “too big to fail” Reduce systemic risk & increase transparency Short Selling, High Leverage, Credit Crunch RegNMS, MiFID, Basel III, Stress Testing, etc Standardized, exchange traded derivates Technology Faster server hardware, co-location, networks Larger and faster data stores and caches Increasing reliance on electronic decision making Platform Symphony is uniquely positioned to support a growing number of compute and data intensive problems Financial Services Momentum
62
© 2012 IBM Corporation Platform Computing 62 Hadoop MapReduce increasingly common in Financial Services Batch applications Real Time SOA Services MapReduce Non- MapReduce Non- MapReduce CVA VaR ALM AML Sensitivity analysis Credit Scoring Mortgage Analytics Variable Annuity Backtesting ETL Trade Surveillance Strategy Mining ETL Datameer Talend FS Private Analytic Cloud Public Cloud (Share Results) Non-MapReduce Applications MapReduce Applications Financial Services Momentum
63
© 2012 IBM Corporation Platform Computing 63 Market Risk (VaR) calculations Credit risk including counterparty risk (CCR) Credit Value Adjustments (CVA) Equity derivatives trading Stochastic volatility modeling Back testing, stress testing Actuarial analysis & modeling ETL process acceleration Fraud Detection Mining of unstructured data Financial Services Use-cases
64
© 2012 IBM Corporation Platform Computing 64 Government Huge amounts of unstructured data ingested 24x7. Requires filter logic to store only information viewed as valuable. Already starting to implement MapReduce logic. Mission critical – No single points of failure allowed. Systems ‘always on’. Need to analyze data in native formats. Life Sciences ‘Big Data Biology’ emerging as a MapReduce workload. Early adopter stage of the industry. Opportunity to engage and provide leadership with enterprise ready solutions. Largely University and grant driven. High availability and SLA management is important. Other Use-cases
65
© 2012 IBM Corporation Platform Computing 65 Telco Already dealing with the ‘big data’ problem. Mission Critical to Business Information in the form of text messages, telephone call logs, e-mail, system log data, etc Requires enterprise level technology and service agreements (SLA driven) Retail Customer behavior & trend analysis driving huge and complex analytics. Already dealing with ‘big data’ problem. Global retail companies and marketing research providers Traditionally dominated by Database Warehouse models. Expect enterprise class infrastructure and tools. Other Use-cases
66
© 2012 IBM Corporation Platform Computing 66 Customer Use Cases for Platform Symphony
67
© 2012 IBM Corporation Platform Computing 67 Société Générale "Because of Platform’s extensive experience and the track record of its technology in the industrial market, we felt that Platform Symphony was mature enough for us to adopt it for our own competitive advantage and enhanced business performance." Alain Benoist Debt & Finance CIO, Corporate & Investment Banking Customer Business Problem Increasing competitive pressure Need to provide faster advice to clients Meet stringent regulatory requirements Require 24x7 reliability Solution Symphony selected after grid “bake-off” Growing Grid, thousands of cores Business Results 80% reduction in job run-times Reduced a 2.5 hour job to 10 minutes Reduced administrator workload Process reliability improvements
68
© 2012 IBM Corporation Platform Computing 68 UniCredit Customer Business Problem Multiple siloed grid systems Under-utilized infrastructures Cost pressures on IT infrastructure Need to deploy new application services Solution Platform Symphony - Murex, Sophis, Full re-valuation engine (FRE) Platform Symphony Virtual Server Harvesting & Platform ISF Business Results Reduced energy & data center cost Reduced hardware footprint Improved performance & service quality Dynamic provisioning of servers "Platform Computing and its enterprise grid solution enable us to share a formerly heterogeneous and distributed hardware infrastructure across applications regardless of their location, operating system and application logic, therefore helping us to achieve our internal efficiency targets while at the same time improving our performance and service quality" Lorenzo Cervellin Head of Global Markets and Treasury Infrastructure and Support at UniCredit Global Information Services
69
© 2012 IBM Corporation Platform Computing 69 Platform Symphony Selling Platform Symphony
70
© 2012 IBM Corporation Platform Computing 70 Open & heterogeneous: multi-OS, multi-framework Superior performance and scalability More capable integrated MapReduce framework Lowest latency (for time-critical applications) Flexible dynamic resource sharing Most compelling roadmap, vendor commitment Strongest consulting & support organization Platform Symphony Why we win The only HPC SOA grid manager & middleware with big data built-in
71
© 2012 IBM Corporation Platform Computing 71 Customer is deploying or has deployed a grid environment Considering solutions like Tibco GridServer, Windows HPC Big Data solutions like Cloudera, MAPR or HortonWorks Any compute intensive, parallel workload (e.g. monte-carlo) Customer is talking about Hadoop or SOA on a grid Financial risk applications, Excel services Customer describes problems managing unstructured data Discussions about MapReduce, Pig, Hive or HBASE Identifying sales opportunities Things to look for: Don’t get left behind ! There are lots of Hadoop MapReduce projects. Customers will see value in Platform Symphony
72
© 2012 IBM Corporation Platform Computing 72 Current IBM or Platform Customers running grid environments Customers running Tibco GridServer environments Existing Algorithmics customers Chief Architects, Enterprise Architects In Finance, CRO, key risk application owners Senior Data Analysts Hadoop MapReduce project owners Data Warehouse owners VP Data Mining VP Information Management Who to call on: Identifying sales opportunities
73
© 2012 IBM Corporation Platform Computing 73 Several types of competiion ISV applications with native distributed computing capability Commercial HPC SOA solutions (Tibco, Microsoft) Open source & commercial MapReduce implementations Batch workload management solutions (for some use cases) In-house developed solutions Winning against the competition Competitive strategies will vary by competitor
74
© 2012 IBM Corporation Platform Computing 74 Positioning with InfoSphere BigInsights IBM Platform Symphony Augments Big Insights 66 times “raw” scheduling performance Enables sharing, multi-tenancy Share among multiple heterogeneous grid applications Better performance with less hardware IBM Platform Symphony dramatically accelerates Big Data workloads
75
© 2012 IBM Corporation Platform Computing 75 Symphony versus Hadoop – Validated with specific customer use case Life Sciences – Platform Symphony MapReduce Framework
76
© 2012 IBM Corporation Platform Computing 76 Platform Symphony vs. Grid Middleware vendors
77
© 2012 IBM Corporation Platform Computing 77 Platform Symphony vs. Hadoop solution vendors
78
© 2012 IBM Corporation Platform Computing 78 Symphony is better because: Performance: Low latency architecture - short-run job performance boosted 10X. Dynamic Resource Management: Slot allocation changes dynamically based on job priority and server thresholds. Loaning and borrowing Application Life Cycle: Support for rolling upgrades of Symphony software. Support for multiple versions of Hadoop co-existing on the same cluster. Reliability: Symphony makes all MapReduce and HDFS related services highly available (name nodes, job trackers, task trackers etc.) and is a proven technology. Sophisticated Scheduling Engine: Fair share scheduling with 10,000 levels of prioritization. Pre-emptive and resource threshold based scheduling with run time change management. Open: Support for multiple APIs and languages. It is fully compatible with Java, Pig, Hive and other MR apps. Symphony also supports multiple data sources, including HDFS, GPFS and many others. Management tools: Platform Symphony provides a comprehensive management capability for troubleshooting alerting and tracking jobs as well as rich reporting capabilities Winning against the competition Apache Hadoop (open-source)
79
© 2012 IBM Corporation Platform Computing 79 Platform Symphony Product Promotion & Sales Tools
80
© 2012 IBM Corporation Platform Computing 80 Product Promotion & Sales tools Brochure IBM Platform Symphony Product Family Datasheets IBM Platform Symphony Developer Edition IBM Platform Symphony for GPUs IBM Platform Symphony Desktop Harvesting IBM Platform Symphony VM & Server Harvesting IBM Platform Symphony MapReduce Framework
81
© 2012 IBM Corporation Platform Computing 81 Product Promotion & Sales tools White Papers Platform Symphony Performance Update Accelerating Excel Applications with Symphony Developer’s Guide to SOA Apps in Symphony Grid enabling Algorithmics with Platform Symphony Improving the efficiency of GPU clusters Top 5 challenges for Hadoop MapReduce in Enterprise Architecture of an enterprise Class Distributed Runtime Engine Reference architecture for Bio-informative MapReduce applications
82
© 2012 IBM Corporation Platform Computing 82
83
© 2012 IBM Corporation Platform Computing 83 IBM Platform Symphony Extra Slides
84
© 2012 IBM Corporation Platform Computing 84 File System / Data Store Connectors (Distributed parallel fault-tolerant file systems / Relational & MPP Databases ) Distributed, Highly Available Runtime Infrastructure Optimized Communication / Object Serialization / Session Management Commercial ISVs In-house Applications Application workflows Data Intensive Apps Hadoop open- source projects Policy Based Scheduling: Service orchestration, Fine-grained task scheduling, SLA mgmt. Enhanced MapReduce Framework HPC SOA Framework MapReduce Apps DDT Data Affinity MR API (Java/C++) VB6 / COM Platform Resource Orchestrator MPP Database Relational Database Scale-out File Systems GPFS HDFS Integrations / Solutions Platform ISF (Private Cloud Mgmt) Distributed Cache (Gemstone) External Public Cloud Adapters Low-latency client & service APIs Management & reporting APIs Optimized data handling APIs IBM Platform Symphony – Simplified Architecture
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.