Logo F2FInterview

Hadoop Interview Questions

Q   |   QA

50070

To have hadoop use a custom partitioner you will have to do minimum the following three

  • Create a new class that extends Partitioner class
  • Override method getPartition
  • In the wrapper that runs the Map Reducer, either

    •  add the custom partitioner to the job programtically using method setPartitionerClass or
    •  add the custom partitioner to the job as a config file (if your wrapper reads from config file or oozie)

There can be several ways of doing this but most common ways are

  • By using counters
  • The web interface provided by Hadoop framework

Its an open ended question but most candidates, if they have written a production job, should talk about some type of alert mechanisn like email is sent or there monitoring system sends an alert. Since Hadoop works on unstructured data, its very important to have a good alerting system for errors since unexpected data can very easily break the job.

This is an open ended question but a candidate who claims to be an intermediate developer and has worked on large data set (10-20GB min) should have run into this problem. There can be many ways to handle this problem but most common way is to alter your algorithm and break down the job into more map reduce phase or use a combiner if possible.

In order to link this F2FInterview's page as Reference on your website or Blog, click on below text area and pres (CTRL-C) to copy the code in clipboard or right click then copy the following lines after that paste into your website or Blog.

Get Reference Link To This Page: (copy below code by (CTRL-C) and paste into your website or Blog)
HTML Rendering of above code: