Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Data Center and Hadoop Jacob Rapp, Cisco

Similar presentations


Presentation on theme: "The Data Center and Hadoop Jacob Rapp, Cisco"— Presentation transcript:

1 The Data Center and Hadoop Jacob Rapp, Cisco

2 Hadoop Considerations Traffic Types, Job Patterns, Network Considerations, Compute Network Integration Co-exist with current Data Center infrastructure Open, Programmable and Application-Aware Networks Multi-tenancy Remove the “Silo clusters” 2

3 3

4 4 Analyze Extract Transform Load (ETL) Explode Reduce Ingress vs. Egress Data Set 1:0.3 Ingress vs. Egress Data Set 1:1 Ingress vs. Egress Data Set 1:2 The Time the reducers start is dependent on: mapred.reduce.slowstart.co mpleted.maps It doesn’t change the amount of data sent to Reducers, but may change the timing to send that data

5 5 Small Flows/Messaging (Admin Related, Heart-beats, Keep-alive, delay sensitive application messaging) Small – Medium Incast (Hadoop Shuffle) Large Flows (HDFS Ingest) Large Incast (Hadoop Replication)

6 6 Many-to-Many Traffic Pattern Map 1Map 2Map NMap 3 Reducer 1Reducer 2Reducer 3Reducer N HDFS Shuffle Output Replication NameNode JobTracker ZooKeeper

7 Analyze Simulated with Shakespeare Wordcount Extract Transform Load (ETL) Simulated with Yahoo TeraSort Extract Transform Load (ETL) Simulated with Yahoo TeraSort with output replication Job Patterns have varying impact on network utilization

8 8

9 9  Network Attributes  Architecture  Availability  Capacity, Scale & Oversubscription  Flexibility  Management & Visibility Integration Considerations

10 10 Single 1GE 100% Utilized Dual 1GE 75% Utilized 10GE 40% Utilized Generally 1G is being used largely due to the cost/performance trade-offs. Though 10GE can provide benefits depending on workload

11 No single point of failure from network view point. No impact on job completion time NIC bonding configured at Linux – with LACP mode of bonding Effective load-sharing of traffic flow on two NICs. Recommended to change the hashing to src-dst-ip-port (both network and NIC bonding in Linux) for optimal load-sharing 11

12 1GE vs. 10GE Buffer Usage 12 Moving from 1GE to 10GE actually lowers the buffer requirement at the switching layer. By moving to 10GE, the data node has a wider pipe to receive data lessening the need for buffers on the network as the total aggregate transfer rate and amount of data does not increase substantially. This is due, in part, to limits of I/O and Compute capabilities

13 Goals Extensive Validation of Hadoop Workload Reference Architecture Make it easy for Enterprise Demystify Network for Hadoop Deployment Integration with Enterprise with efficient choices of network topology/devices Findings 10G and/or Dual attached server provides consistent job completion time & better buffer utilization 10G provide reduce burst at the access layer Dual Attached Sever is recommended design – 1G or 10G. 10G for future proofing Rack failure has the biggest impact on job completion time Does not require non-blocking network Latency does not matter much in Hadoop workloads 13 More Details From Hadoop Summit 2012 at:

14 14

15 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15 n # show interface brief Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # Eth1/1 1 eth access up none 10G(D) -- Eth1/2 1 eth access up none 10G(D) -- Eth1/3 1 eth access up none 10G(D) -- Eth1/4 1 eth access up none 10G(D) -- Eth1/5 1 eth access up none 10G(D) –-. Eth1/33 1 eth access up none 10G(D) -- Eth1/34 1 eth access up none 10G(D) -- Eth1/35 1 eth access down SFP not inserted 10G(D) -- Eth1/36 1 eth access down SFP not inserted 10G(D) -- Eth1/37 1 eth access down Administratively down 10G(D) –.

16 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16 n # show mac address-table dynamic Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since first seen,+ - primary entry using vPC Peer- Link VLAN MAC Address Type age Secure NTFY Ports * 1 e8b7.484d.a208 dynamic F F Eth1/31 * 1 e8b7.484d.a20a dynamic F F Eth1/31 * 1 e8b7.484d.a73e dynamic F F Eth1/34 * 1 e8b7.484d.a740 dynamic F F Eth1/34 * 1 e8b7.484d.ad15 dynamic F F Eth1/28 * 1 e8b7.484d.ad17 dynamic F F Eth1/28 * 1 e8b7.484d.b3e9 dynamic F F Eth1/25 * 1 e8b7.484d.b3eb dynamic F F Eth1/25. MAC Addresses of the connected devices … and the port they are on…

17 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17 n # portServerMap ======================================= Port Server FQDN Eth1/1 c200-m2-10g2-001.cluster10g.com Eth1/2 c200-m2-10g2-002.cluster10g.com Eth1/3 c200-m2-10g2-003.cluster10g.com Eth1/4 c200-m2-10g2-004.cluster10g.com Eth1/5 c200-m2-10g2-005.cluster10g.com Eth1/6 c200-m2-10g2-006.cluster10g.com Eth1/7 c200-m2-10g2-031.cluster10g.com Eth1/8 c200-m2-10g2-008.cluster10g.com Eth1/9 c200-m2-10g2-009.cluster10g.com Eth1/11 c200-m2-10g2-011.cluster10g.com.

18 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18 n # trackerList =========================================== Port Server Server Port Eth1/2 c200-m2-10g Eth1/3 c200-m2-10g Eth1/4 c200-m2-10g Eth1/5 c200-m2-10g Eth1/6 c200-m2-10g Eth1/7 c200-m2-10g Eth1/8 c200-m2-10g Eth1/9 c200-m2-10g Eth1/11 c200-m2-10g Eth1/12 c200-m2-10g

19 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19 n # bufferServerMap =================================================================== Port Server 1sec 5sec 60sec 5min 1hr Eth1/1 c200-m2-10g KB 0KB 0KB 0KB 0KB Eth1/2 c200-m2-10g KB 384KB 1536KB 2304KB 2304KB Eth1/3 c200-m2-10g KB 384KB 1152KB 1536KB 1536KB Eth1/4 c200-m2-10g KB 384KB 2304KB 2304KB 2304KB Eth1/5 c200-m2-10g KB 384KB 768KB 1536KB 1536KB Eth1/6 c200-m2-10g KB 2304KB 2304KB 2304KB 2304KB Eth1/7 c200-m2-10g KB 384KB 3456KB 3840KB 3840KB Eth1/8 c200-m2-10g KB 768KB 2688KB 2688KB 2688KB Eth1/9 c200-m2-10g KB 384KB 2304KB 2304KB 2304KB Eth1/11 c200-m2-10g KB 384KB 1920KB 1920KB 1920KB. Eth1/1(c200-m2-10g2-001) has 0 buffer usage because it’s the name node

20 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20 n # jobsBuffer Hadoop Job Info... =================================================================== 1 jobs currently running JobId RunTime(secs) User Priority job_ _ hadoop NORMAL =================================================================== Buffer Info - Per Port Port Server 1sec 5sec 60sec 5min 1hr Eth1/1 c200-m2-10g KB 0KB 0KB 0KB 0KB Eth1/2 c200-m2-10g KB 384KB 768KB 768KB 768KB Eth1/3 c200-m2-10g KB 384KB 1152KB 1152KB 1152KB Eth1/4 c200-m2-10g KB 1536KB 1536KB 1536KB 1536KB Eth1/5 c200-m2-10g KB 768KB 1152KB 1152KB 1152KB. What jobs were running during peak buffer usage … and for how long were they running

21 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21 n (config)# jobsBuffer Hadoop Job Info... =================================================================== 0 jobs currently running JobId RunTime(secs) User Priority =================================================================== Buffer Info - Per Port Port Server 1sec 5sec 60sec 5min 1hr Eth1/1 c200-m2-10g KB 0KB 0KB 0KB 0KB Eth1/2 c200-m2-10g KB 0KB 0KB 1920KB 1920KB Eth1/3 c200-m2-10g KB 0KB 0KB 2304KB 2304KB Eth1/4 c200-m2-10g KB 0KB 0KB 2688KB 2688KB Eth1/5 c200-m2-10g KB 0KB 0KB 2304KB 2304KB Eth1/6 c200-m2-10g KB 0KB 0KB 2304KB 2304KB Eth1/7 c200-m2-10g KB 0KB 0KB 1920KB 2688KB. Historic look at the buffer usage …

22 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22

23 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23

24 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24

25 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25 Buffer Usage Shuffle Replication Reduce Map

26 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26 (Python Socket) Push Data Application Logs PTP Grandmaster (OPTIONAL) Analyze API to Application Info Synchronize Time github.com/datacenter

27 27

28 28  Hadoop + HBASE  Job Based  Department Based Various Multitenant Environments Need to understand Traffic Patterns Scheduling Dependent Permissions and Scheduling Dependent

29 29 Map 1Map 2Map NMap 3 Reducer 1 Reducer 2 Reducer 3 Reducer N HDFS Shuffle Output Replication Region Server Client Major Compaction Read Update Read Major Compaction

30 30 Hbase During Major Compaction Read/Update Latency Comparison of Non- QoS vs. QoS Policy ~45% for Read Improvement Switch Buffer Usage With Network QoS Policy to prioritize Hbase Update/Read Operations

31 Switch Buffer Usage With Network QoS Policy to prioritize Hbase Update/Read Operations Hbase + Hadoop Map Reduce Read/Update Latency Comparison of Non- QoS vs. QoS Policy ~60% for Read Improvement

32 Cisco Unified Data Center UNIFIED FABRIC UNIFIED COMPUTING Highly Scalable, Secure Network Fabric Modular Stateless Computing Elements UNIFIED MANAGEMENT Automated Management THANK YOU FOR LISTENING kloadautomation Manages Enterprise Workloads Cisco.com Big Data Data Center Script Examples from Presentation: github.com/datacenter


Download ppt "The Data Center and Hadoop Jacob Rapp, Cisco"

Similar presentations


Ads by Google