Presentation is loading. Please wait.

Presentation is loading. Please wait.

Apache Bigtop Working Group Cluster stuff. Cloud computing.

Similar presentations

Presentation on theme: "Apache Bigtop Working Group Cluster stuff. Cloud computing."— Presentation transcript:

1 Apache Bigtop Working Group Cluster stuff

2 Cloud computing

3 Bigtop Administration Make sure you are signed up on the bigtop- dev mailing list. Lots of info which will never get repeated if you miss it Bigtop-user, bigtop-dev

4 Bigtop Administration Sign up for jira

5 Bigtop Administration – Registration, Join Biocurious. Pays for space nobody takes a cut of this Registration – Free drinks – Registration = AWS Credits. Cancelling IntelliJ. Expires end of April. –

6 Newbie Slide Structure: – Do labs Lab 1 Modified to take 1-2 weeks. Update the wiki with your findings Lab 2 Build Bigtop 0.3.0; Can start projects here, do Jira tickets Lab 3 map reduce program Lab 4 Run the unit tests under the component downloads Lab 5 Run the integration tests Lab 6 Puppet, deploy and run Lab 7 Port a module – Labs are changing; not a class. Requires time commitment – Demo, doesnt need to be working; for your benefit not ours

7 Lab 1 Install bigtop. Web search for apache bigtop, go to wiki link IGTOP/Index IGTOP/Index IGTOP/How+to+install+Hadoop+distribution+f rom+Bigtop IGTOP/How+to+install+Hadoop+distribution+f rom+Bigtop

8 Lab 1 Install bigtop, run all the components, Hive/Hbase/Pig/Hadoop/Mahout/Oozie There are bugs, document them Add the sample programs in quickstart to the wiki. Not all are included yet

9 Lab 1 Update the wiki Sqoop open (User group meeting next week) Flume/Flume NG (open/nothing) Zookeeper(open/nothing)

10 Hadoop Components Old: Dont stop at running Pi as test of HDFS Still missing: Run Terasort in Hadoop, need cluster IGTOP/How+to+install+Hadoop+distribution+f rom+Bigtop IGTOP/How+to+install+Hadoop+distribution+f rom+Bigtop Whirr may need patch depending on where you run it from

11 Mahout Dont run jar like in Hadoop Scripts handle downloading and clustering, demo, etc.. Under /examples/bin. Bigtop puts example/bin under /usr/share/doc/mahout. Is this correct? Not documentation Add documentation to wiki Ticket filed

12 Oozie Oozie runs, forget the error message, set to highest version

13 Oozie

14 Flume/Flume NG New patch checkin for Flume NG Testing

15 Whirr sudo apt-get install whirr Run as: whirr launch-cluster --config /udt/lib/whirr/recipes/ If successful will see directory under ~/.whirr whirr.log mvn clean install

16 Puppet sudo apt-get install puppet facter fails

17 Ticket Questions/Demo Bigtop install should include stable for ubuntu? Diff between stable and bigtop incubating. There used to be a diff. Monitoring, ->metrics2 Ganglia or JMX? All components w/daemon; Bruno has Ganglia recipes to monitor status of cluster. Hadoop monitoring: performance and functionality. Hooked up to kerberos/ commercial version is Cloudera manager. Networking, i/o, block sizes, swap space, disk space. Stable vs. incubating? Anwar: LogMining (M/R, clickstream and FE log data, exception on day to day basis);

Download ppt "Apache Bigtop Working Group Cluster stuff. Cloud computing."

Similar presentations

Ads by Google