Presentation is loading. Please wait.

Presentation is loading. Please wait.

Tang Zhenkun

Similar presentations


Presentation on theme: "Tang Zhenkun"— Presentation transcript:

1 Tang Zhenkun Email: tangzk2011@163.comtangzk2011@163.com

2  Mapreduce Basics  Hadoop Counters  Hadoop Log Info(slf4j)  Unit Test(JUnit, MRUnit)  Guava(Google Core Libraries for Java 1.6+)  Others  References

3  Hadoop job submit flow

4  Hadoop Web GUI

5  Hadoop job submit flow 1. Invisible to details 2. None step-through Debug If errors? Not just pray!

6  Command Errors, Grammar Errors  Check, and check, and check again…  Logic Errors  That is the point that we need to deal with.

7

8  Hadoop Standard Counters  Map output records  Reduce output records  Custom Counters

9 context.getCounter(counterName); context.getCounter(groupName, counterName); 输入文件

10

11

12  Stdout does not work.  System.out.println() X  Use Logger.  Eg: log4j, slf4j

13  SLF4j – Simple Logging Façade for Java.  Simple, easy to use.

14

15

16  TDD, Test-Driven Development,

17  JUnit(Unit Test for Java)  #Unit(for C#)  XUnit

18  小孩分油问题:两个小孩去打油,一人带 了一个一斤的空瓶,另一个带了一个七两、 一个三两的空瓶。原计划各打一斤油,可 是由于所带的钱不够,只好两人合打了一 斤 (10 两 ) 油,在回家的路上,二人想平分这 一斤油,可是又没有其它工具。试仅用三 个瓶子 ( 一斤、七两、三两 ) 精确地分出两个 半斤油来。

19  Define a state:  Each represents the 10 ounces, 7ounces, and 3 ounces bottle.  Define the Operation : multiAndPlus(X, b)  Eg: pour 10 ounces from the first(10o) bottle to the third one.

20 MatTest.java Mat.java

21  @Test  @Before, @After  Assert*  And last, RUN in Java Normal Application.

22  MRUnit, Unit Test for Hadoop Mapreduce

23  MapDriver  ReduceDriver  MapReduceDriver  withInput(key, value)  withOutput(key, value)  runTest()  And last, RUN in Java Normal Application.

24

25  The Art of Assertion in CH5 of Programming Pearls, Second Edition.  Assert in Java  assert  assert :  But, you must run the application with enabling assertions implicitly.(java -ea )  Precondition in Guava

26  Guava, Google Core Libraries for Java 1.6+  Preconditions checkArgument(i >= 0, "Argument was %s but expected nonnegative", i); checkArgument(i %s", i, j);

27  Other useful libraries. http://code.google.com/p/guava-libraries/

28 自定义数据类型 CustomType 自定义 Partitioner

29

30 Partitioner: return Key % 3 When change to: (return key / 3), and change the number of reduce tasks to 4 Totally ordering.

31  Maven  Auto endependency management  Hadoop Remote Debug  JDWP, Java Debug Wire Protocol  HPROF  Analysis tools in JDK

32  Hadoop, the Definitive Guide, Second Edition.  http://www.junit.org/ http://www.junit.org/  http://incubator.apache.org/mrunit/ http://incubator.apache.org/mrunit/  http://code.google.com/p/guava-libraries/  http://insightfullogic.com/blog/2011/oct/21/5- reasons-use-guava/ http://insightfullogic.com/blog/2011/oct/21/5- reasons-use-guava/

33


Download ppt "Tang Zhenkun"

Similar presentations


Ads by Google