hadoop - Write Reducer output of a Mapreduce job to a single File -

-

i have written map-reduce job data in hbase. contains multiple mappers , single reducer. reducer method takes in data supplied mapper , analytic on it. after processing complete data in hbase wanted write data file in hdfs through single reducer. presently able write data hdfs every time new 1 unable figure how write final conclusion hdfs @ last.

so, if trying write final result single reducer hdfs, can try 1 of approaches below -

  1. use hadoop api filesystem's create() function write hdfs reducer.
  2. emit single key , value reducer after final calculation
  3. override reducers cleanup() function , point (1) there.

details on 3:

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/reducer.html#cleanup-org.apache.hadoop.mapreduce.reducer.context-

hope helps.


Comments

Post a Comment

Popular posts from this blog

javascript - Count length of each class -

-
i have below code in this fiddle . html <span class="myname">john smith</span> <span class="myname">john smith</span> <span class="myname">john smith</span> <span class="myname">john smith</span> <span class="myname">john smith</span> <span class="myname">john smith</span> javascript $('.myname').each(function(){ var text = $(this).html().split(' '), len = text.length, test = $(this).val().length, result = []; console.log('outside: ' + test); for( var = 0; < len; i++ ) { result[i] = '<span class="name-'+[i]+'" >' + text[i] + '</span>'; console.log('inside: ' + test); } $(this).html(result.join(' ')); console.log('after
Read more

What design pattern is this code in Javascript? -

-
this question has answer here: why write “.call(this)” @ end of javascript anonymous function? [duplicate] 4 answers what kind of design pattern , significance of using closure? (function(){ // code here }).call(this); edit then difference between above code , following this keyword still refer same object in both ways. (function(){ // code here })(); thats invoked function expression. more info here: http://benalman.com/news/2010/11/immediately-invoked-function-expression/ the purpose run code while protecting scope (so variables declared within don't leak out global scope. update call sets value of this function being applied on. without it, value set window object, there set outer scope.
Read more

hadoop - Restrict secondarynamenode to be installed and run on any other node in the cluster -

-
i have have cloudera cdh3u4 cluster setup. have secondarynamenode daemon running on 1 of nodes. i have ensure user has root access node on cluster (for example, datanode not have other daemons running) should not able start instance of secondarynamenode daemon on node (thereby having 2 secondarynamenodes on cluster) how can achieve this? thanks in advance. add property hdfs-site.xml namedfs.secondary.http.address valuesecondarynamenode-host address or ip:50090
Read more