CSE 548 Advanced Computer Network Security Trust in MobiCloud using Hadoop Framework Updates Sayan Kole Jaya Chakladar Group No: 1

Overview Installation of Hadoop –Single node –Distributed cluster Understanding the existing trust system and its suitability as a MapReduce application

Project Tasks (updated) Tasks Responsible Status Learn MapReduce and HadoopJaya & Sayan100 % Install and configure Hadoop in MobiCloud Jaya & Sayan60 % Develop UI web applicationJaya15 % Search mapper algorithmSayan25 % Search reduction algorithmJaya25 % HDFS data store creation and updates Jaya & Sayan60 % Testing and problem resolutionJaya & SayanNot started Delivery and demoJaya & SayanNot started

Software and Hardware Requirements Hadoop Java 6 SSH Database software e.g. MySQL or Apache HDFS 4 virtual machines

MapReduce EMT Application Tier 1 is not suitable as Map Reduce application – onetime job Tier 2 Trust Unknown trust evaluation Propagate trust amongst a web of users. Directed graph

MapReduce EMT Application Find paths between s and d Use path weights as trust values Incorporate number of hops

Hadoop Single Cluster Installation Prerequisites Java 6 –Add the canonical partner repository to the apt repository –Update the source list –Install JDK –Select Sun’s Java as the default on the machine Add a dedicated Hadoop system user Configure SSH –Configure SSH access for Hadoop system user –Generate an SSH key for Hadoop user –Enable SSH access to local mahine with the new key created Disable IPv6

Hadoop Single Cluster Installation Download Hadoop from Apache mirror sites and extract Set JAVA_HOME in /conf/hadoop-env.sh Configure core-site.xml –Set path for hadoop.tmp.dir to local directory –Set the HDFS variable Configure mapred-site.xml to set the host and port of mapReduce job tracker. Configure hdfs-site.xml to specify the number of replications for each file in the system.

Hadoop Single Cluster Installation Format the Hadoop HDFS name node – make sure data is backed up Start a single node cluster, this starts the name node, data node, job tracker & task tracker.

Hadoop Multiple Cluster Installation Setup two single node clusters Designate one as master and the other one as slave Shutdown clusters in both machines Update /etc/hosts on both machines with appropriate names (master and slave) and addresses SSH configuration between master and slave –Hadoop user must connect to users master and slave –Password less connection

Hadoop Multiple Cluster Installation Master node runs master daemons like name node for HDFS and job tracker Both nodes run slave daemons like data node for HDFS and task tracker

Hadoop Multiple Cluster Installation Master vs. Slave configuration –On master /conf/master lists the master –On slaves, /conf/slaves lists two entries master and slave Update core-site.xml on all machines to setup fs.default.name as hdfs://master: Update mapred-site.xml on all sites to fix mapread.job.tracker as master: Change dfs.replication variable in hdfs-site.xml to the number of sites avaiable, 4 in our case. Format the name node

Hadoop Multiple Cluster Installation Start up the multi-node cluster –Start HDFS daemons like name node and data node daemons in master and slaves respectively –Start Map reduce daemons like job tracker on master and task tracker in slaves

Word Count Application – Single Node Hadoop

Word Count Output

Challenges faced so far Multi node setup errors Setup SSH without password requests Synchronization between VMs Big assumption – Tier 2 Trust is a good MapReduce application

Project Time Line Week 1Week 2Week 3Week 4Week 5Week 6Week 7Week 8Week 9 Week 10 Week 11 Week 12 Study and understand Mapreduce and Hadoop Install and configure Hadoop Run simple application and demonstrate correctness of implementation Create Mapreduce algorithm particular to specific problem/application Develop the user interface/frontend Installation on Mobicloud Stress checking and testing Analyze and interpret the results Present the application

Q & A