Thursday, 14 February 2013

Writing MapReduce Program on Hadoop - Context

Map and Reduce

Map Reduce works by breaking the processing into two phases: Map phase and Reduce phase. Each phase has the key-value pair as input and type of key and value can be chosen by the programmer.

Data flow in the Map and Reduce:
Input ==> Map ==> Mapper Output ==> Sort and shuffle ==> Reduce ==> Final Output

Steps to Write the Hadoop Map Reduce in Java

Map Reduce program need three things: Map, Reduce and Some code to run job(Here we will call it as Invoker)

1). Create the Map(Any Name) class and map function was represented by org.apache.hadoop.mapreduce.Mapper.class which declares an abstract map() method.
[code lang="java"]import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class Map extends Mapper<;LongWritable, Text, Text,IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key,Text value,Context context) throws IOException, InterruptedException {
word.set(value.toString());
context.write(word, one);
}
}

[/code]
Explanation:
The Mapper class is the generic class with four formal parameters(input key, input value, output key and output value). Here input key is LongWritable(Long representation by hadoop), input value is the Text(String representation by hadoop), output key is text(keyword) and output value is Intwritable(int representation by hadoop). All above hadoop datatypes are same as java datatype expect that are optimized for network serialization.

2). Create the Reducer(Any Name) class and reduce function was represented by org.apache.hadoop.mapreduce.Reducer.class which declares an abstract reduce() method.
[code lang="java"]import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;
import java.util.Iterator;

public class Reduce extends Reducer<Text, IntWritable, Text,IntWritable> {
@Override
protected void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException {
int sum = 0;
for(IntWritable intWritable : values){
sum += intWritable.get();
}
context.write(key, new IntWritable(sum));
}
}
[/code]
Explanation:
The Reducer class is the generic class with four formal parameters(input key, input value, output key and output value). Here input key and input value type must match with Mapper output, output key is text(keyword) and output value is Intwritable(number of occurence).

3) We are ready with Map and Reduce implementation, then we need to have the invoker for confguring the Hadoop job and invoke the Map Reduce program.
[code lang="java"]import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount{
public static void main(String[] args) throws Exception {
Configuration configuration = new Configuration();
configuration.set("fs.default.name", "hdfs://localhost:10011");
configuration.set("mapred.job.tracker","localhost:10012");

Job job = new Job(configuration, "Word Count");

job.setJarByClass(WordCount.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

//Submit the job to the cluster and wait for it to finish.
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
[/code]
4). Compile code by using following command
mkdir WordCount
javac -classpath ${HADOOP_HOME}/hadoop-0.20.2+228-core.jar -d WordCount path/*.java

5). Create the jar by using command
jar -cvf ~/WordCount.jar -C WordCount/ .

6). Create the input file in the local file system

Eg : mkdir /home/user1/wordcount/input
cd /wordcount/input
gedit file01
gedit file02

and so on..

7). Copy the input file in local file system to HDFS
$HADOOP_HOME/bin/hadoop fs -cp ~/wordcount/input/file01 /home/user1/dfs/input/file01
$HADOOP_HOME/bin/hadoop fs -cp ~/wordcount/input/file02 /home/user1/dfs/input/file02

8). Execute the jar as follows:
$HADOOP_HOME/bin/hadoop jar WordCount.jar WordCount /home/user1/dfs/input /home/user1/dfs/output

9). After execution get completed, below set of commands is used to view the reduce file that are generated
$HADOOP_HOME/bin/hadoop fs -ls /home/user1/dfs/output/

10). To view the output, use this below given command
$HADOOP_HOME/bin/hadoop fs -cat hdfs:///home/user1/dfs/output/part-00000
$HADOOP_HOME/bin/hadoop fs -cat hdfs:///home/user1/dfs/output/part-00001
$HADOOP_HOME/bin/hadoop fs -cat hdfs:///home/user1/dfs/output/part-00002

and so on...

Upcoming Post: Using Distributed Cache in Java Hadoop MapReduce.

Thursday, 24 January 2013

Recovering corrupted files in Linux

There are many compression technique that you use and compress a file in Linux.

Few techniques are gzip, tar and lot more.

At times when you decompress/uncompress a file after a long time you may get few errors like.

"Unexpected EOF." "Files not in gzip format" or "Files not in gz format" something like this.

This is because the files may be corrupted.

There is a recovery tool kit and it is mainly for gzip but this can also be used for tar.gz too.

STEPS FOR RECOVERY :

Go to the link http://www.urbanophile.com/arenn/coding/gzrt/

And click on the gzrt-0.6.tar.gz link, then gzrt-0.6.tar.gz file will be downloaded to your machine.

Go to the specified location and do the following.

$ tar -xzvf gzrt-0.6.tar.gz
$ cd gzrt-0.6
$ make

Then copy the corrupted file to this folder and execute the following command.

$./gzrecover ABC.gz

Then the recovered file will be in the name of ABC.recovered

Hence there is no problem in dealing with corrupted files in Linux any more. :)

Tuesday, 22 January 2013

Downloading videos in Linux

There is a simple software (clip grab) that allows you to download online videos in Linux.

Execute the following commands in your terminal.

############################

sudo add-apt-repository ppa:clipgrab-team/ppa

sudo apt-get update

sudo apt-get install clipgrab

##########################

The software is now installed , to run the software execute the following command

#############

clipgrab

################

Steps to download :

1. Go to the "SETTINGS" tab in the GUI of Clip grab and mention the destination path where you need to save the video.

2. Then under downloads tab, Paste the web-link from which u want to download the video.

3. Click the "grab this clip" button.

The video will start getting downloaded in the specified destination.

Thursday, 17 January 2013

We are what we think...

"We are what we repeatedly do. Excellence, then, is not an act but a habit"

At the End of the day, we are what we think and do the entire day.

Our thoughts becomes things. So control your thoughts and think only what you want, then its yours at the some point of time.

Have good thoughts, So that only good things happens to you.

Keep smiling and spread the happiness.

Tuesday, 8 May 2012

My Own Line

Don’t act to satisfy others…Be true to your beloved ones…

Monday, 7 May 2012

How to mount pendrive in Linux/Unix/Centos

In shell, Enter as superuser(root) mode by using command: su -
Then
  • type command: fdisk -l
This will list the memory device that are connect to system as follows: /dev/hda1,/dev/hda2 and pendrive as follows: /dev/sda1 or /dev/sdb1
  • then create directory /mnt by using command
  • mkdir urfolder
Before that move to /mnt folder by issuing this command:
cd /mnt
  • Then use command: mount /dev/sda1 /mnt/urfolder
/dev/sda1 that you got by issuing the command fdisk -l
  • Now you can access your pendrive in /mnt/urfolder
To unmount the pendrive:
  • Give umount /dev/sda1

Tuesday, 21 February 2012

Make the Technology help you Go Green

There are many things in our day to day life through which we can contribute to Go green.

ATM has become a part of our routine. For sure we will be using it at least one time a day.

There is a option that the robotic money lender tells us  " Do you want a receipt for your transaction ? "

Without even reading this statement completely ....We will give "yes" for whatever question ATM throws to us.

After taking the money and  card out of the machine , we will get a slip.

The slip will find itself in the trash can , the very next moment it comes out.

Please Don't take the slip unless it is very necessary.

This is a simple good deed, which quite a few of us will do that.

It doesn't mean that the others don't know it actually.  It means that the others don't care for it.

The following Induced me to write this post.

when we cleaned our house , the majority of the waste  that we found is this ATM slips.

Thus we decided to create an awareness about it and spread the same.

Follow such simple things that can have a great Impact on the environment.

" Together we can make a difference..." ...Go Green.....