Light smart by going green

I was in dilemma, when I started to write this blog. Because the content that am going to narrate here is a well known fact.  But still, a recent incident has urged me post this blog.

How many of us have cursed the Tamil Nadu or Indian government for inconsistent power supply. We blame political parties, politicians and even God at times for lack of rain which has a direct influence in power production.  But we fail to think that there is only few that can be done by the government and we as a citizen of this country need to take responsibility and do some possible deed.

There is a simple step that you can contribute to nature in two ways, one by consuming less energy and another by polluting the environment less.

Its nothing but replacing incandescent bulbs with CFL bulbs. How this will help us ???.



A CFL consumes 25% less energy when compared to an incandescent bulb. This saves electricity by 70%. An incandescent bulb glowing for 1 hr consumes 60 watts, whereas a CFL glowing for 1 hour consumes 10 watts only. You will get the same glow from both these bulbs. We save 50 Watts of energy per hour, if we do a simple change.  This is how we consume less energy.

"We never know the worth of water till the well is dry".  ~Thomas Fuller,Gnomologia. Same is the case for power we waste most of the time without knowing its value. 

On the other side of the story, If we light an incandescent lamp for one hour it produces 5 kg of co2, which is an endanger for the environment. This is equal to riding a bike for 8 km. So the first immediate step we need to do is to change all the incandescent bulb with CFL wherever  we could.  So that you can contribute something valuable to the environment and to the nation by saving a decent amount of electricity.

You may have lot of questions that, who is buying incandescent these days and where will be these kind of bulbs found now a days. Yes, we have many people around who are unaware of it. The reason why am writing this here is, one of my well educated friend has recently bought a incandescent bulb to our home.  That's the time I have decided to post this blog and create awareness about the same.  You can find these kind of bulbs in rural areas mostly than the urban. It is not uncommon to find these kind of bulb in rural areas.

If we change every such Bulb in our state(Tamil Nadu), we can get an extra 15 mins of power for each day in our state. We can do this instead blaming our government or politicians. If we could make it happen and if we do the same in all states of our country, we could save a lot of power and that can be used for many other industrialization purposes.

I thought of this blog a long time back and drafted this, But I need to be a "Doer before a Sayer".  It took a months time for me to change all incandescent lamps wherever I could (In my home, My Grandma's home, Our current rented home in Chennai). Totally 5 is what I have changed so far.  Even its a little am happy that I have contributed something to the environment and did a good deed as a citizen to my country.

Little progress together makes a brighter and greener future. I want my friends to do and Share the same.

Lets create a better future together.....

GenericOptionsParser, Tool, and ToolRunner for running Hadoop Job

Hadoop comes with a few helper classes for making it easier to run jobs from the command line. GenericOptionsParser is a class that interprets common Hadoop command-line options and sets them on a Configuration object for your application to use as desired. You don’t usually use GenericOptionsParser directly, as it’s more convenient to implement the Tool interface and run your application with the ToolRunner, which uses GenericOptionsParser internally:

public interface Tool extends Configurable {
        int run(String [] args) throws Exception;
    }

Below example shows a very simple implementation of Tool, for running the Hadoop Map Reduce Job.

public class WordCountConfigured extends Configured implements Tool {
        @Override
        public int run(String[] args) throws Exception {
        Configuration conf = getConf();

        return 0;
        }        
    }
    public static void main(String[] args) throws Exception {
        int exitCode = ToolRunner.run(new WordCountConfigured(), args);
        System.exit(exitCode);
    }

We make WordCountConfigured a subclass of Configured, which is an implementation of the Configurable interface. All implementations of Tool need to implement Configurable (since Tool extends it), and subclassing Configured is often the easiest way to achieve this. The run() method obtains the Configuration using Configurable’s getConf() method, and then iterates over it, printing each property to standard output.

WordCountConfigured’s main() method does not invoke its own run() method directly. Instead, we call ToolRunner’s static run() method, which takes care of creating a Configuration object for the Tool, before calling its run() method. ToolRunner also uses a GenericOptionsParser to pick up any standard options specified on the command line, and set them on the Configuration instance. We can see the effect of picking up the properties specified in conf/hadoop-localhost.xml by running the following command:

hadoop WordCountConfigured -conf conf/hadoop-localhost.xml -D mapred.job.tracker=localhost:10011 -D mapred.reduce.tasks=n

Options specified with -D take priority over properties from the configuration files. This is very useful: you can put defaults into configuration files, and then override them with the -D option as needed. A common example of this is setting the number of reducers for a MapReduce job via -D mapred.reduce.tasks=n. This will override the number of reducers set on the cluster, or if set in any client-side configuration files. The other options that GenericOptionsParser and ToolRunner support are listed in Table.

GenericOptionsParser and ToolRunner option Description

Property>	Description
-D property=value	Sets the given Hadoop configuration property to the given value. Overrides any default or site properties in the configuration, and any properties set via the -conf option.
-conf filename ...	Adds the given files to the list of resources in the configuration. This is a convenient way to set site properties, or to set a number of properties at once.
-fs uri	Sets the default filesystem to the given URI. Shortcut for -D fs.default.name=uri
-jt host:port	Sets the jobtracker to the given host and port. Shortcut for -D mapred.job.tracker=host:port
-files file1,file2,...	Copies the specified files from the local filesystem (or any filesystem if a scheme is specified) to the shared filesystem used by the jobtracker (usually HDFS) and makes them available to MapReduce programs in the task’s working directory.
-archives archive1,archive2,...	Copies the specified archives from the local filesystem (or any filesystem if a scheme is specified) to the shared filesystem used by the jobtracker (usually HDFS), unarchives them, and makes them available to MapReduce programs in the task’s working directory.
-libjars jar1,jar2,...	Copies the specified JAR files from the local filesystem (or any filesystem if a scheme is specified) to the shared filesystem used by the jobtracker (usually HDFS), and adds them to the MapReduce task’s classpath. This option is a useful way of shipping JAR files that a job is dependent on

Apache Ant - Tutorial

1. Build Tools
Build tools are used to automate the repetitive task like compiling source code, generating documentation, running tests, uilding the jar and so on. Some of the well known build tools are Apache Ant, Maven.

2. Overview of Ant

Ant(Another Neat Tool) is the java library and mostly used to building and deploying the java application. Ant provides the built-in tools to compile, build, test and packing the java application. Ant builds are based on three blocks.
Tasks: Task is the unit of work. For example compile, packing.
Targets: Targets can be invoked via Ant.
Extension Points: Extension points are same as targets and it won't any operation and just coordinate the targets.

3. Building the Java Application: Using Apache Ant

Create the build.xml(not as same) in your project root directory. Given below is the sample build xml(self explanatory - comments inline)

<?xml version="1.0" ?>
<project name="nRelate Analytics" >
    <!-- Sets variables which can later be used. -->
    <!-- The value of a property is accessed via ${} -->
    <property name="src.dir" location="src" />
    <property name="build.dir" location="build" />
    <property name="dist.dir" location="dist" />
    <property name="lib.dir" location="lib" />

    <!--
      Create a classpath container which can be later used in the ant task
    -->
    <path id="build.classpath">
        <fileset dir="${lib.dir}">
            <include name="**/*.jar" />
        </fileset>
    </path>

    <!-- Deletes the existing build, dist directory-->
    <target name="clean">
        <delete dir="${build.dir}" />
        <delete dir="${dist.dir}" />
    </target>

    <!-- Creates the  build, dist directory-->
    <target name="makedir">
        <mkdir dir="${build.dir}" />
        <mkdir dir="${dist.dir}" />
    </target>

    <!-- Compiles the java code (including the usage of library -->
    <target name="compile" depends="clean, makedir">
        <javac srcdir="${src.dir}" destdir="${build.dir}" classpathref="build.classpath">
        </javac>
    </target>

    <!--Creates the deployable jar file  -->
    <target name="createjar" depends="compile">
        <jar destfile="${dist.dir}LogNormalizer.jar" basedir="${build.dir}">
        <!-- Adding to Manifest file, from which class to start exceution -->
        <manifest>
            <attribute name="Main-Class" value="test.MainClass" />
        </manifest>
        </jar>
    </target>
</project>

4) Run your Ant build from the command line

Open a command line and switch to your project directory. Type in the following commands.

# Run the build
ant -f build.xml
# build.xml is default you can also use
ant
Specify the target as below.
#Run the build.xml by specifying the target.
ant -f build.xml createjar

The build should finish successfully and generate the build artifacts under the dist directory.

Subscribe to get updates on this article.

My Personal collections

This is my personal collection of books that I wish to read and also books that I have accomplished so far. I firmly believe in  Thomas Jefferson's quote - "A life without a book is meaningless". I want my life to be meaningful and so here comes this blog. I will keep on updating this list and also welcome people who suggest me some good books and give me some feedback.

Drowned ...

Two States - Chetan bhagat

This is beautiful love story between an IIM guy from Delhi and a girl from Tamil nadu. This is how they fall in love and carrying it till their marriage. Over coming their problems of being from different caste, culture, language and family back ground. This is the author's own story. A good one to read.

One night at the call center - Chetan bhagat

 Another drama from Chetan, where the entire book is about the story that happens over a night. This explains the life at call center and along with a love drama in between. A kind of interesting, though it doesn't sound as two states.

Who will cry when you die - Robin Sharma

Author Robin speaks about life's simple things which will make life happier. This will be a collection of different lessons where he touch base all areas from being kind to others, mental and physical fitness, going green, being positive and lot more.

The Greatness guide - Robin Sharma

Robin explains how one should lead his life both as a leader and a follower. He has embedded great sayings of Legends and  proverbs that will sting to your mind. He simply states that , "the first point of how a world sees you, should begin from you.

Five point someone - Chetan bhagat

 This another real life incident happened to author in IIT. A mix of fun and love in his college days. Well written in simple language as like his other books. Once you start, you can never close without finishing it.

What young India Wants - Chetan Bhagat

Its the collection of essays and articles written by the author in various newspapers in the north. He has also some good letters written by him to Political leaders. Most of the incidents in this book speaks about things that cling around northern part of India.

The Alchemist - Paul Coelho

This is a story where the author speaks about treasure, desert, travel and love. Its about a young boy who risks his life in fulfilling his destiny. He insists that "“If you really desire to achieve something, the whole world conspires in helping you to achieve it”.

Revolution 2020 - Chetan Bhagat

A triangle love story between three. One guy who failed to IIT but becoming a director of an Engineering college. Another guy who cracked the exams to get in to the best college and at last ended up in running his own news paper publication who is willing to correct the corrupted nation with his thoughts and broad mind. The final touch of whom the girl will be married to is an interesting ending.

Kane and Abel - Jeffrey Archer

This is one of the best books that I have ever read. It explains life of 2 persons from their birth to death. The journey they went through, difficulties in bringing themselves up their life. I highly recommend this book for anyone. The author has kept his nerve through out the story. I have decided to read the few other books of this author. He deserves the title to be one of the best authors in the world.

To be drowned....

A Thousand Splendid Suns -  Khaled Hosseini

Google Launches Realtime API for Google Drive

Google has launched a Realtime API for Google Drive, enabling developers to create apps that can tap into Drive's real-time editing capabilities.

The API provides developers with collaborative versions of data objects such as maps, lists, strings, and JSON values, which are automatically synchronized, and all modifications to them are stored. Developers can create apps that read from and write to these objects like any local object.

If the basic set of objects isn't enough, developers can also create custom objects and references, which includes trees and arbitrary graph structures.

interested developers can check out the Google Drive Realtime API technical documentation here.

 

Source: Mashable.com

Excellence is a drive from inside

Shared in the Cognizant Internal Mail Thread.

Excellence is a drive from inside

A tourist once visited a temple under construction where he saw a sculptor making an idol of God. Suddenly he noticed a similar idol lying nearby.

Surprised, he asked the sculptor, "Do you need two statues of the same idol?"

"No," said the sculptor without looking up, "We need only one, but the first one got damaged at the last stage."

The gentleman examined the idol and found no apparent damage.

"Where is the damage?" he asked.

"There is a scratch on the nose of the idol." said the sculptor, still busy with his work.

"Where are you going to install the idol?"

The sculptor replied that it would be installed on a pillar twenty feet high.

"If the idol is that far, who is going to know that there is a scratch on the nose?" the gentleman asked...

The sculptor stopped his work, looked up at the gentleman, smiled and said, "I will know it."

Moral :

Excellence is not for someone else to notice but for your own satisfaction and efficiency.

The desire to excel is exclusive of the fact whether someone else appreciates.

Writing MapReduce Program on Hadoop - Context

Map and Reduce

Map Reduce works by breaking the processing into two phases: Map phase and Reduce phase. Each phase has the key-value pair as input and type of key and value can be chosen by the programmer.

Data flow in the Map and Reduce:

Input ==> Map ==> Mapper Output ==> Sort and shuffle ==> Reduce ==> Final Output

Steps to Write the Hadoop Map Reduce in Java

Map Reduce program need three things: Map, Reduce and Some code to run job(Here we will call it as Invoker)

1). Create the Map(Any Name) class and map function was represented by org.apache.hadoop.mapreduce.Mapper.class which declares an abstract map() method.
[code lang="java"]import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class Map extends Mapper<;LongWritable, Text, Text,IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key,Text value,Context context) throws IOException, InterruptedException {
word.set(value.toString());
context.write(word, one);
}
}

[/code]
Explanation:
The Mapper class is the generic class with four formal parameters(input key, input value, output key and output value). Here input key is LongWritable(Long representation by hadoop), input value is the Text(String representation by hadoop), output key is text(keyword) and output value is Intwritable(int representation by hadoop). All above hadoop datatypes are same as java datatype expect that are optimized for network serialization.

2). Create the Reducer(Any Name) class and reduce function was represented by org.apache.hadoop.mapreduce.Reducer.class which declares an abstract reduce() method.
[code lang="java"]import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;
import java.util.Iterator;

public class Reduce extends Reducer<Text, IntWritable, Text,IntWritable> {
@Override
protected void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException {
int sum = 0;
for(IntWritable intWritable : values){
sum += intWritable.get();
}
context.write(key, new IntWritable(sum));
}
}
[/code]
Explanation:
The Reducer class is the generic class with four formal parameters(input key, input value, output key and output value). Here input key and input value type must match with Mapper output, output key is text(keyword) and output value is Intwritable(number of occurence).

3) We are ready with Map and Reduce implementation, then we need to have the invoker for confguring the Hadoop job and invoke the Map Reduce program.
[code lang="java"]import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount{
public static void main(String[] args) throws Exception {
Configuration configuration = new Configuration();
configuration.set("fs.default.name", "hdfs://localhost:10011");
configuration.set("mapred.job.tracker","localhost:10012");

Job job = new Job(configuration, "Word Count");

job.setJarByClass(WordCount.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

//Submit the job to the cluster and wait for it to finish.
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
[/code]
4). Compile code by using following command

mkdir WordCount
javac -classpath ${HADOOP_HOME}/hadoop-0.20.2+228-core.jar -d WordCount path/*.java

5). Create the jar by using command

jar -cvf ~/WordCount.jar -C WordCount/ .

6). Create the input file in the local file system

Eg : mkdir /home/user1/wordcount/input

cd /wordcount/input
gedit file01
gedit file02

and so on..

7). Copy the input file in local file system to HDFS

$HADOOP_HOME/bin/hadoop fs -cp ~/wordcount/input/file01 /home/user1/dfs/input/file01
$HADOOP_HOME/bin/hadoop fs -cp ~/wordcount/input/file02 /home/user1/dfs/input/file02

8). Execute the jar as follows:

$HADOOP_HOME/bin/hadoop jar WordCount.jar WordCount /home/user1/dfs/input /home/user1/dfs/output

9). After execution get completed, below set of commands is used to view the reduce file that are generated

$HADOOP_HOME/bin/hadoop fs -ls /home/user1/dfs/output/

10). To view the output, use this below given command

$HADOOP_HOME/bin/hadoop fs -cat hdfs:///home/user1/dfs/output/part-00000
$HADOOP_HOME/bin/hadoop fs -cat hdfs:///home/user1/dfs/output/part-00001
$HADOOP_HOME/bin/hadoop fs -cat hdfs:///home/user1/dfs/output/part-00002

and so on...

Upcoming Post: Using Distributed Cache in Java Hadoop MapReduce.