Installation using brew
$brew install hadoop ==> Downloading http://www.apache.org/dyn/closer.cgi?path=hadoop/core/hadoop-1.0.3/hadoop-1.0.3.tar.gz ==> Best Mirror http://apache.petsads.us/hadoop/core/hadoop-1.0.3/hadoop-1.0.3.tar.gz ######################################################################## 100.0% ==> Caveats In Hadoop's config file: /usr/local/Cellar/hadoop/1.0.3/libexec/conf/hadoop-env.sh $JAVA_HOME has been set to be the output of: /usr/libexec/java_home ==> Summary /usr/local/Cellar/hadoop/1.0.3: 267 files, 64M, built in 2.6 minutes $
Conf files to edit
vi /usr/local/Cellar/hadoop/1.0.3/libexec/conf/hadoop-env.sh vi /usr/local/Cellar/hadoop/1.0.3/libexec/conf/core-site.xml vi /usr/local/Cellar/hadoop/1.0.3/libexec/conf/mapred-site.xml vi /usr/local/Cellar/hadoop/1.0.3/libexec/conf/hdfs-site.xml
Format HDFS
hadoop namenode -format
Hadoop start
$/usr/local/Cellar/hadoop/1.0.3/bin/start-all.sh starting namenode, logging to /usr/local/Cellar/hadoop/1.0.3/libexec/bin/../logs/hadoop-aaa-namenode-aaa.local.out Password: localhost: starting datanode, logging to /usr/local/Cellar/hadoop/1.0.3/libexec/bin/../logs/hadoop-aaa-datanode-aaa.local.out Password: localhost: starting secondarynamenode, logging to /usr/local/Cellar/hadoop/1.0.3/libexec/bin/../logs/hadoop-aaa-secondarynamenode-aaa.local.out starting jobtracker, logging to /usr/local/Cellar/hadoop/1.0.3/libexec/bin/../logs/hadoop-aaa-jobtracker-aaa.local.out Password: localhost: starting tasktracker, logging to /usr/local/Cellar/hadoop/1.0.3/libexec/bin/../logs/hadoop-aaa-tasktracker-aaa.local.out $
Quick Verification
When you run "ps -aef | grep hadoop" the following processes should be listed
* JobTracker
* TaskTracker
* NameNode
* DataNode
* SecondaryNameNode
Hadoop sample program
$hadoop jar /usr/local/Cellar/hadoop/1.0.3/libexec/hadoop-examples-1.0.3.jar pi 10 100 Number of Maps = 10 Samples per Map = 100 Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 Wrote input for Map #3 Wrote input for Map #4 Wrote input for Map #5 Wrote input for Map #6 Wrote input for Map #7 Wrote input for Map #8 Wrote input for Map #9 Starting Job 12/10/13 00:41:45 INFO mapred.FileInputFormat: Total input paths to process : 10 12/10/13 00:41:45 INFO mapred.JobClient: Running job: job_201210130034_0001 12/10/13 00:41:46 INFO mapred.JobClient: map 0% reduce 0% 12/10/13 00:42:01 INFO mapred.JobClient: map 20% reduce 0% 12/10/13 00:42:07 INFO mapred.JobClient: map 40% reduce 0% 12/10/13 00:42:10 INFO mapred.JobClient: map 40% reduce 6% 12/10/13 00:42:13 INFO mapred.JobClient: map 60% reduce 13% 12/10/13 00:42:19 INFO mapred.JobClient: map 80% reduce 13% 12/10/13 00:42:25 INFO mapred.JobClient: map 100% reduce 20% 12/10/13 00:42:31 INFO mapred.JobClient: map 100% reduce 30% 12/10/13 00:42:37 INFO mapred.JobClient: map 100% reduce 100% 12/10/13 00:42:42 INFO mapred.JobClient: Job complete: job_201210130034_0001 12/10/13 00:42:42 INFO mapred.JobClient: Counters: 27 12/10/13 00:42:42 INFO mapred.JobClient: Job Counters 12/10/13 00:42:42 INFO mapred.JobClient: Launched reduce tasks=1 12/10/13 00:42:42 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=60557 12/10/13 00:42:42 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 12/10/13 00:42:42 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 12/10/13 00:42:42 INFO mapred.JobClient: Launched map tasks=10 12/10/13 00:42:42 INFO mapred.JobClient: Data-local map tasks=10 12/10/13 00:42:42 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=35812 12/10/13 00:42:42 INFO mapred.JobClient: File Input Format Counters 12/10/13 00:42:42 INFO mapred.JobClient: Bytes Read=1180 12/10/13 00:42:42 INFO mapred.JobClient: File Output Format Counters 12/10/13 00:42:42 INFO mapred.JobClient: Bytes Written=97 12/10/13 00:42:42 INFO mapred.JobClient: FileSystemCounters 12/10/13 00:42:42 INFO mapred.JobClient: FILE_BYTES_READ=226 12/10/13 00:42:42 INFO mapred.JobClient: HDFS_BYTES_READ=2380 12/10/13 00:42:42 INFO mapred.JobClient: FILE_BYTES_WRITTEN=239494 12/10/13 00:42:42 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=215 12/10/13 00:42:42 INFO mapred.JobClient: Map-Reduce Framework 12/10/13 00:42:42 INFO mapred.JobClient: Map output materialized bytes=280 12/10/13 00:42:42 INFO mapred.JobClient: Map input records=10 12/10/13 00:42:42 INFO mapred.JobClient: Reduce shuffle bytes=252 12/10/13 00:42:42 INFO mapred.JobClient: Spilled Records=40 12/10/13 00:42:42 INFO mapred.JobClient: Map output bytes=180 12/10/13 00:42:42 INFO mapred.JobClient: Total committed heap usage (bytes)=1931190272 12/10/13 00:42:42 INFO mapred.JobClient: Map input bytes=240 12/10/13 00:42:42 INFO mapred.JobClient: Combine input records=0 12/10/13 00:42:42 INFO mapred.JobClient: SPLIT_RAW_BYTES=1200 12/10/13 00:42:42 INFO mapred.JobClient: Reduce input records=20 12/10/13 00:42:42 INFO mapred.JobClient: Reduce input groups=20 12/10/13 00:42:42 INFO mapred.JobClient: Combine output records=0 12/10/13 00:42:42 INFO mapred.JobClient: Reduce output records=0 12/10/13 00:42:42 INFO mapred.JobClient: Map output records=20 Job Finished in 56.845 seconds Estimated value of Pi is 3.14800000000000000000 $
Hadoop stop
/usr/local/Cellar/hadoop/1.0.3/bin/stop-all.sh
