1. Hadoop 2.2.0 Installation and Setup on Ubuntu 12.04.3
CT Yang
Department of Computer Science
Tunghai University

2. Hadoop Document, http://hadoop.apache.org/docs/r2.2.0/
Hadoop Common: The common utilities that support the other Hadoop modules.
Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.
Hadoop YARN: A framework for job scheduling and cluster resource management.
Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.
2019/1/13

3. Other Hadoop-related projects at Apache
Ambari™: A web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters which includes support for Hadoop HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides a dashboard for viewing cluster health such as heatmaps and ability to view MapReduce, Pig and Hive applications visually along with features to diagnose their performance characteristics in a user-friendly manner.
Avro™: A data serialization system.
Cassandra™: A scalable multi-master database with no single points of failure.
Chukwa™: A data collection system for managing large distributed systems.
HBase™: A scalable, distributed database that supports structured data storage for large tables.
Hive™: A data warehouse infrastructure that provides data summarization and ad hoc querying.
Mahout™: A Scalable machine learning and data mining library.
Pig™: A high-level data-flow language and execution framework for parallel computation.
ZooKeeper™: A high-performance coordination service for distributed applications.
2019/1/13

4. OS: Ubnutu LTS
MyHadoop-master
MyHadoop-node
MyHadoop-node

5. 修改hosts
sudo vim /etc/hosts

6. 修改hostname
sudo vim /etc/hostname
sudo service hostname start
重新登入

7. 安裝Java JDK sudo apt-get -y install openjdk-7-jdk
sudo ln -s /usr/lib/jvm/java-7-openjdk-amd64 /usr/l ib/jvm/jdk

8. 新增hadoop使用者 sudo addgroup hadoop sudo adduser --ingroup hadoop hduser
sudo adduser hduser sudo

9. 建立SSH免密碼登入 ssh-keygen -t rsa -f ~/.ssh/id_rsa -P ""
cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
scp –r ~/.ssh MyHadoop-node01:~/

10. 下載hadoop
cd ~
wget ommon/hadoop-2.2.0/hadoop tar.gz
tar zxf hadoop tar.gz
mv hadoop tar.gz hadoop

11. 新增環境變數 vim .bashrc export JAVA_HOME=/usr/lib/jvm/jdk/
export HADOOP_INSTALL=/home/hduser/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL

12. 設定hadoop config cd hadoop/etc/hadoop vim hadoop-env.sh
將export JAVA_HOME這一行做修改

13. 設定hadoop config(cont.)
vim core-site.xml
<property>
<name>fs.default.name</name>
<value>hdfs://MyHadoop-master:9000</value>
</property>

14. 設定hadoop config(cont.)
vim yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<name>yarn.resourcemanager.hostname</name>
<value>MyHadoop-master</value>

15. 設定hadoop config(cont.)
cp mapred-site.xml.template mapred-site.xml
vim mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

16. 設定hadoop config(cont.)
mkdir -p ~/mydata/hdfs/namenode
mkdir -p ~/mydata/hdfs/datanode
vim hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<name>dfs.namenode.name.dir</name>
<value>/home/hduser/mydata/hdfs/namenode</value>
<name>dfs.datanode.data.dir</name>
<value>/home/hduser/mydata/hdfs/datanode</value>

17. 設定hadoop config(cont.)
vim slaves
MyHadoop-node01
MyHadoop-node02

18. 複製hadoop給所有node
scp -r /home/hduser/hadoop MyHadoop-node01:/home/hd user

19. 格式化HDFS
hdfs namenode -format

20. 啟動Hadoop
start-all.sh

21. 使用jps查看java正在運行的程式
jps

22. Hadoop監控網頁
MyHadoop-master:8088

23. 範例程式 cd /home/hduser/hadoop
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce- examples jar pi 2 5

24. 停止hadoop 服務
stop-all.sh

25. XML預設資料
dist/hadoop-common/core-default.xml
client/hadoop-mapreduce-client-core/mapred-default.xml
common/yarn-default.xml
dist/hadoop-hdfs/hdfs-default.xml