Presentation is loading. Please wait.

Presentation is loading. Please wait.

Hadoop Operations – Best Practices from the Field October 17, 2014 Chris Nauroth Suresh Srinivas

Similar presentations


Presentation on theme: "Hadoop Operations – Best Practices from the Field October 17, 2014 Chris Nauroth Suresh Srinivas"— Presentation transcript:

1 Hadoop Operations – Best Practices from the Field October 17, 2014 Chris Nauroth Suresh Srinivas

2 © Hortonworks Inc About Us Chris Nauroth Member of Technical Staff, Hortonworks –Apache Hadoop committer and PMC member –Major contributor to HDFS ACLs, Windows compatibility, and operability improvements Hadoop user since 2010 –Prior employment experience deploying, maintaining and using Hadoop clusters Suresh Srinivas Architect & Founder at Hortonworks –Long time Apache Hadoop committer and PMC member –Designed and developed many key Hadoop features Experience from supporting many clusters –Including some of the world’s largest Hadoop clusters Page 2 Architecting the Future of Big Data

3 © Hortonworks Inc Agenda Analysis of Hadoop Support Cases –Support case trends –Configuration –Documentation –Software Improvements Key Learnings and Best Practices –HDFS ACLs –HDFS Snapshots –YARN Application Timeline Server Page 3 Architecting the Future of Big Data

4 © Hortonworks Inc Support Cases: Setting the Context Hortonworks Support –Multiple tiers of support contacts –Support engineers trained and knowledgeable across the entire Hadoop ecosystem –Cases may escalate to subject matter experts for depth in one particular area –Challenging cases may escalate to Apache committers at Hortonworks if additional expertise is required Apache Community Support for user questions and support –https://issues.apache.org/jira for reporting confirmed bugs –Apache Hadoop users, contributors, committers and PMC members all participate actively in these forums to help resolve issues Page 4 Architecting the Future of Big Data

5 © Hortonworks Inc Support Case Analysis Methodology Inspected over 2 years of support case history across hundreds of customers Broad inclusion of 29 Hadoop ecosystem and related projects Multiple versions of Hadoop in deployments –2 major versions: Hadoop 1.x and 2.x –~3 minor versions within each major version –~3 patch releases per minor version –~15 total releases and updates Distinct deployment environments –Cluster sizes ranging from 10s to 1000s of nodes –Different management environments and operational practices –Various deployment techniques: Ambari, Chef, RPMs, etc. Page 5 Architecting the Future of Big Data

6 © Hortonworks Inc Support Case Trends – Cases per Month Page 6 Architecting the Future of Big Data

7 © Hortonworks Inc Support Case Trends – Cases per Month What is the spike in May 2014? –More users –More total users means more total support cases –More features –Many upgrades of existing clusters from Hadoop 1 to Hadoop 2 –Many conversions to HA deployments –Many conversions to secured deployments –More integration –Many sites running separate Hadoop 1 and Hadoop 2 clusters simultaneously –Questions around migrating data between clusters at 2 different versions (DistCp) Page 7 Architecting the Future of Big Data

8 © Hortonworks Inc Support Case Trends – Proportional Cases per Month Page 8 Architecting the Future of Big Data

9 © Hortonworks Inc Support Case Trends – Root Cause Page 9 Architecting the Future of Big Data

10 © Hortonworks Inc Support Case Trends Highlights –Core Hadoop components (HDFS, YARN and MapReduce) are used across all deployments, and therefore receive proportionally more support cases than other ecosystem components. –Misconfiguration is the dominant root cause. –Documentation is a close second. –We are constantly improving the code to eliminate operational issues, help with diagnosis and provide increased visibility. Page 10 Architecting the Future of Big Data

11 Configuration

12 © Hortonworks Inc Hardware and Cluster Sizing Considerations –Larger clusters heal faster on nodes or disk failure –Machines with huge storage take longer to recover –More racks give more failure domains Recommendations –Get good-quality commodity hardware –Buy the sweet-spot in pricing: 3TB disk, 96GB, 8-12 cores –More memory is better – real time is memory hungry! –Before considering fatter machines (1U 6 disks vs. 2U 12 disks) –Get to machines or 3-4 racks –Use pilot cluster to learn about load patterns –Balanced hardware for I/O, compute or memory bound –More details - Page 12

13 © Hortonworks Inc Configuration Avoid JVM issues –Use 64 bit JVM for all daemons –Compressed OOPS enabled by default (6 u23 and later) –Java heap size –Set same max and starting heapsize, Xmx == Xms –Avoid java defaults – configure NewSize and MaxNewSize –Use 1/8 to 1/6 of max size for JVMs larger than 4G –Configure –XX:PermSize=128 MB, -XX:MaxPermSize=256 MB –Use low-latency GC collector –-XX:+UseConcMarkSweepGC, -XX:ParallelGCThreads= –High on Namenode and JobTracker or ResourceManager –Important JVM configs to help debugging –-verbose:gc -Xloggc: -XX:+PrintGCDetails –-XX:ErrorFile= –-XX:+HeapDumpOnOutOfMemoryError Page 13

14 © Hortonworks Inc Configuration Multiple redundant dirs for namenode metadata –One of dfs.namenode.name.dir should be on NFS –NFS softmount - tcp,soft,intr,timeo=20,retrans=5 Configure open fd ulimit –Default 1024 is too low –16K for datanodes, 64K for Master nodes Use version control for configuration! Page 14

15 © Hortonworks Inc Configuration Use disk fail in place for datanodes: dfs.datanode.failed.volumes.tolerated –Disk failure is no longer datanode failure –Especially important for large density nodes Set dfs.namenode.name.dir.restore to true –Restores NN storage directory during checkpointing Take periodic backups of namenode metadata –Make copies of the entire storage directory Set aside a lot of disk space for NN logs –It is verbose – set aside multiple GBs –Many installs configure this too small –NN logs roll with in minutes – hard to debug issues Page 15

16 © Hortonworks Inc Monitor Usage Cluster storage, nodes, files, blocks grows –Update NN heap, handler count, number of DN xceivers –Tweak other related config periodically Monitor the hardware usage for your work load –Disk I/O, network I/O, CPU and memory usage –Use this information when expanding cluster capacity Monitor the usage with HADOOP metrics –JVM metrics – GC times, Memory used, Thread Status –RPC metrics – especially latency to track slowdowns –HDFS metrics –Used storage, # of files and blocks, total load on the cluster –File System operations –MapReduce Metrics –Slot utilization and Job status Tweak configurations during upgrades/maintenance on an ongoing basis Page 16

17 Documentation

18 © Hortonworks Inc Documentation Continual Investment in Documentation –Hortonworks Data Platform Documentation –http://docs.hortonworks.com/ –Apache Hadoop Documentation –http://hadoop.apache.org/docs/current/ Apache Hadoop Documentation –We welcome your requests in Apache jira for documentation improvements. –Create issues with the “documentation” label. –Getting the end user perspective is extremely valuable. –We would be grateful to receive documentation patches. –It’s a great way to get started in the Apache Hadoop open source process. –Search for unresolved issues with the “documentation” label. –https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUC E)%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20documentation Page 18 Architecting the Future of Big Data

19 Software Improvements Real Incidents and Software Improvements to Address Them

20 © Hortonworks Inc Don’t edit the metadata files! Editing can corrupt the cluster state –Might result in loss of data Real incident –NN misconfigured to point to another NN’s metadata –DNs can’t register due to namespace ID mismatch –System detected the problem correctly –Safety net ignored by the admin! –Admin edits the namenode VERSION file to match ids What Happens Next? Page 20

21 © Hortonworks Inc Improvement Pause deletion of blocks when the namenode starts up –https://issues.apache.org/jira/browse/HDFS-6186 –Supports configurable delay of block deletions after NameNode startup –Gives an admin extra time to diagnose before deletions begin Show when block deletion will start after NameNode startup in WebUI –https://issues.apache.org/jira/browse/HDFS-6385 –The web UI already displays the number of pending block deletions –This will enhance the display to indicate when actual deletion will begin Page 21 Architecting the Future of Big Data

22 © Hortonworks Inc Guard Against Accidental Deletion rm –r deletes the data at the speed of Hadoop! –ctrl-c of the command does not stop deletion! –Undeleting files on datanodes is hard & time consuming –Immediately shutdown NN, unmount disks on datanodes –Recover deleted files –Start namenode without the delete operation in edits Enable Trash Real Incident –Customer is running a distro of Hadoop with trash not enabled –Deletes a large dir (100 TB) and shuts down NN immediately –Support person asks NN to be restarted to see if trash is enabled! What happens next? Now HDFS has Snapshots! Page 22

23 © Hortonworks Inc Improvement HDFS Snapshots –https://issues.apache.org/jira/browse/HDFS-2802 –A snapshot is a read-only point-in-time image of part of the file system –A snapshot created before a deletion can be used to restore deleted data –More coverage of snapshots later in the presentation HDFS ACLs –https://issues.apache.org/jira/browse/HDFS-4685 –Finer-grained control of file permissions can help prevent an accidental deletion –More coverage of ACLs later in the presentation Page 23 Architecting the Future of Big Data

24 © Hortonworks Inc Unexpected error during HA HDFS upgrade Background: HDFS HA Architecture –http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html Real Incident –During upgrade, NameNode calls every JournalNode to request backup of metadata directory, which renames “current” directory to “previous.tmp”. –Permissions incorrect on metadata directory for 1 out of 3 JournalNodes. –The hdfs user is not authorized to rename. Backup fails for that JournalNode, so upgrade process aborts with error. What happens next? Page 24 Architecting the Future of Big Data

25 © Hortonworks Inc Improvement Improve diagnostics on storage directory rename operations by using native code. –https://issues.apache.org/jira/browse/HDFS-7118 –Logs additional root cause information for rename failure. For example, EACCES Split error checks in into separate conditions to improve diagnostics. –https://issues.apache.org/jira/browse/HDFS-7119 –Splits a log message about failure to delete or rename into separate log messages to clarify which specific action failed When aborting NameNode or JournalNode, write the contents of the metadata directories and permissions to logs. –https://issues.apache.org/jira/browse/HDFS-7120 –Usually the first information asked of the user, so we can automate this For JournalNode operations that must succeed on all nodes, execute a pre-check to verify that the operation can succeed. –https://issues.apache.org/jira/browse/HDFS-7121 –Prevents need for manual cleanup on 2 out of 3 JournalNodes where backup succeeded Page 25 Architecting the Future of Big Data

26 © Hortonworks Inc Support Case Trends Highlights Revisited –Core Hadoop components (HDFS, YARN and MapReduce) are used across almost all deployments, and therefore receive proportionally more support cases than other ecosystem components. –Action: Focus efforts on core Hadoop first to improve operability of the platform. –Misconfiguration is the dominant root cause. –Action: Publish configuration best practices and advise on the need for ongoing review of configuration as cluster usage patterns change over time. –Documentation is a close second. –Action: Contribute frequently to product documentation, both in open source Apache Hadoop and in the distro. End user documentation is a gating factor for launching new features. We welcome your requests in Apache jira for documentation improvements, and we welcome your patches! –Code changes often can be implemented to eliminate an operational issue, help with diagnosis or provide increased visibility. –Action: After resolution of each support case, consider potential product improvements. For example, can logging be improved? Small code changes can have a big impact. Page 26 Architecting the Future of Big Data

27 Key Learnings and Best Practices Features that Help Improve Production Operations

28 © Hortonworks Inc HDFS ACLs Existing HDFS POSIX permissions good, but not flexible enough –Permission requirements may differ from the natural organizational hierarchy of users and groups. HDFS ACLs augment the existing HDFS POSIX permissions model by implementing the POSIX ACL model. –An ACL (Access Control List) provides a way to set different permissions for specific named users or named groups, not only the file’s owner and file’s group. Page 28 Architecting the Future of Big Data

29 © Hortonworks Inc HDFS File Permissions Example Authorization requirements: –In a sales department, they would like a single user Maya (Department Manager) to control all modifications to sales data –Other members of sales department need to view the data, but can’t modify it. –Everyone else in the company must not be allowed to view the data. Can be implemented via the following: Read/Write perm for user maya User Group Read perm for group sales File with sales data

30 © Hortonworks Inc HDFS ACLs Problem –No longer feasible for Maya to control all modifications to the file –New Requirement: Maya, Diane and Clark are allowed to make modifications –New Requirement: New group called executives should be able to read the sales data –Current permissions model only allows permissions at 1 group and 1 user Solution: HDFS ACLs –Now assign different permissions to different users and groups Owner Group Others HDFS Directory … rwx Group D … rwx Group F … rwx User Y … rwx

31 © Hortonworks Inc HDFS ACLs New Tools for ACL Management (setfacl, getfacl) –hdfs dfs -setfacl -m group:execs:r-- /sales-data –hdfs dfs -getfacl /sales-data # file: /sales-data # owner: maya # group: sales user::rw- group::r-- group:execs:r-- mask::r-- other::-- –How do you know if a directory has ACLs set? –hdfs dfs -ls /sales-data Found 1 items -rw-r maya sales :31 /sales- data

32 © Hortonworks Inc HDFS ACLs Default ACLs –hdfs dfs -setfacl -m default:group:execs:r-x /monthly-sales-data –hdfs dfs -mkdir /monthly-sales-data/JAN –hdfs dfs –getfacl /monthly-sales-data/JAN –# file: /monthly-sales-data/JAN # owner: maya # group: sales user::rwx group::r-x group:execs:r-x mask::r- x other::--- default:user::rwx default:group::r-x default:group:execs:r-x default:mask::r- x default:other::---

33 © Hortonworks Inc HDFS ACLs Best Practices Start with traditional HDFS permissions to implement most permission requirements. Define a smaller number of ACLs to handle exceptional cases. A file with an ACL incurs an additional cost in memory in the NameNode compared to a file that has only traditional permissions. Page 33 Architecting the Future of Big Data

34 © Hortonworks Inc HDFS Snapshots –A snapshot is a read-only point-in-time image of part of the file system –Performance: snapshot creation is instantaneous, regardless of data size or subtree depth –Reliability: snapshot creation is atomic –Scalability: snapshots do not create extra copies of data blocks –Useful for protecting against accidental deletion of data Example: Daily Feeds hdfs dfs -ls /daily-feeds Found 5 items drwxr-xr-x - chris supergroup :36 /daily-feeds/ drwxr-xr-x - chris supergroup :36 /daily-feeds/ drwxr-xr-x - chris supergroup :37 /daily-feeds/ drwxr-xr-x - chris supergroup :37 /daily-feeds/ drwxr-xr-x - chris supergroup :37 /daily-feeds/ Page 34 Architecting the Future of Big Data

35 © Hortonworks Inc HDFS Snapshots Create a snapshot after each daily load hdfs dfsadmin -allowSnapshot /daily-feeds Allowing snaphot on /daily-feeds succeeded hdfs dfs -createSnapshot /daily-feeds snapshot-to Created snapshot /daily-feeds/.snapshot/snapshot-to User accidentally deletes data for hdfs dfs -ls /daily-feeds Found 4 items drwxr-xr-x - chris supergroup :36 /daily-feeds/ drwxr-xr-x - chris supergroup :36 /daily-feeds/ drwxr-xr-x - chris supergroup :37 /daily-feeds/ drwxr-xr-x - chris supergroup :37 /daily-feeds/ Page 35 Architecting the Future of Big Data

36 © Hortonworks Inc HDFS Snapshots Snapshots to the rescue: the data is still in the snapshot hdfs dfs -ls /daily-feeds/.snapshot/snapshot-to Found 5 items drwxr-xr-x - chris supergroup :36 /daily- feeds/.snapshot/snapshot-to / drwxr-xr-x - chris supergroup :36 /daily- feeds/.snapshot/snapshot-to / drwxr-xr-x - chris supergroup :37 /daily- feeds/.snapshot/snapshot-to / drwxr-xr-x - chris supergroup :37 /daily- feeds/.snapshot/snapshot-to / drwxr-xr-x - chris supergroup :37 /daily- feeds/.snapshot/snapshot-to / Restore data from hdfs dfs -cp /daily-feeds/.snapshot/snapshot-to / /daily-feeds Page 36 Architecting the Future of Big Data

37 © Hortonworks Inc YARN Application Timeline Server Stores data about YARN application execution –Generic data –YARN container utilization –Metrics related to containers –Application-specific data –MapReduce jobs and their tasks –Tez DAG execution Provides CLI for accessing data –Useful for ad-hoc queries or scripted analysis Provides REST API for accessing data –Consumed by UI front-ends such as Apache Ambari Page 37 Architecting the Future of Big Data

38 © Hortonworks Inc Querying a Map Reduce Job Entity curl { "entity": "job_ _0001", "entitytype": "MAPREDUCE_JOB", "events": [ { "eventinfo": { "FINISHED_MAPS": 2, "FINISHED_REDUCES": 1, "FINISH_TIME": , "JOB_STATUS": "SUCCEEDED" }, "eventtype": "JOB_FINISHED", "timestamp": } ], "relatedentities": { "MAPREDUCE_TASK": [ "task_ _0001_m_000000" ] }, "starttime": } Page 38 Architecting the Future of Big Data

39 © Hortonworks Inc Querying a Map Task Entity curl { "entity": "task_ _0001_m_000000", "entitytype": "MAPREDUCE_TASK", "events": [ { "eventtype": "TASK_FINISHED", "timestamp": }, { "eventinfo": { "SPLIT_LOCATIONS": "localhost", "START_TIME": , "TASK_TYPE": "MAP" }, "eventtype": "TASK_STARTED", "timestamp": } ], } Page 39 Architecting the Future of Big Data

40 © Hortonworks Inc Summary Configuration –Prevent garbage collection issues –Configure for redundancy –Retune configuration in response to metrics Documentation –End user perspective is crucial –Please consider contributing to Apache Hadoop documentation HDFS ACLs –Implement fine-grained authorization rules on files –Can protect against accidental file manipulations HDFS Snapshots –Point-in-time image of part of the filesystem –Useful for restoring to a prior state after accidental file manipulation YARN Application Timeline Server –Provides generic and application-specific data about YARN application execution –Useful for analyzing cluster usage patterns Page 40 Architecting the Future of Big Data

41 © Hortonworks Inc Thank you, Q&A Page 41 ResourceLocation Hardware Recommendations for Apache Hadoop guide/content/ch_hardware-recommendations.html Hadoop Documentation Issues https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20HA DOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20resolution%20%3 D%20Unresolved%20AND%20labels%20%3D%20documentation HDFS operational and debuggability improvements https://issues.apache.org/jira/browse/HDFS-6185 HDFS ACLs Blog Posthttp://hortonworks.com/blog/hdfs-acls-fine-grained-permissions-hdfs-files-hadoop/ HDFS Snapshots Blog Posthttp://hortonworks.com/blog/protecting-your-enterprise-data-with-hdfs-snapshots/ YARN Timeline Server Documentation site/TimelineServer.html Learn more


Download ppt "Hadoop Operations – Best Practices from the Field October 17, 2014 Chris Nauroth Suresh Srinivas"

Similar presentations


Ads by Google