hadoop - How to process large file with one record dependent on another in MapReduce -

-

i have scenario there large file , line 1 record might have dependency on 1000th line data , line 1 , 1000 can part of separate spilts. understanding of framework record reader going return 1 key, value pair mapper , each k,v pair independent of another. since file has been divided splits , want (i.e. splittable false no option), can handle anyhow may writing own record reader, mapper or reducer?

dependency -

row1: a,b,c,d,e,f

row2: x,y,z,p,q,r

now x in row2 need used d in row1 desired output.

thanks.

i think need implement reducer side join. here can see better explanation of it: http://hadooped.blogspot.mx/2013/09/reduce-side-joins-in-java-map-reduce.html.

both related values have end in same reducer (defined key , partitioner) , should grouped (groupingcomparator) , may use secondsort order grouped values.


Comments

Post a Comment

Popular posts from this blog

java - Date formats difference between yyyy-MM-dd'T'HH:mm:ss and yyyy-MM-dd'T'HH:mm:ssXXX -

-
i trying parse date 2014-12-03t10:05:59.5646+08:00 using these 2 formats: yyyy-mm-dd't'hh:mm:ss yyyy-mm-dd't'hh:mm:ssxxx when parse using yyyy-mm-dd't'hh:mm:ss works fine, when parse yyyy-mm-dd't'hh:mm:ssxxx parseexception thrown. which correct format parse date , difference between these 2 formats? note : cannot use joda :( those valid formats: yyyy-mm-dd't'hh:mm:ss.sssz >>> e.g.: 2001-07-04t12:08:56.235-0700 yyyy-mm-dd't'hh:mm:ss.sssxxx >>> e.g.: 2001-07-04t12:08:56.235-07:00 edit: btw, "x" refer (iso 8601 time zone)
Read more

c# - Get rid of xmlns attribute when adding node to existing xml -

-
i have xml-file: <ns2:root xmlns:ns2="namespace"> <ns2:a> <ns2:b>some content</b> <ns2:c>some content</c> <ns2:d>some content</d> </a> </root> i need add new node in specific place, code is: var doc = xdocument.load(file); xnamespace ns2 = "namespace"; doc.element(ns2 + "root").element(ns2 + "a").element(ns2 + "c").addafterself( new xelement(ns2+"new", new xelement("new1", new xelement("new2","some content"), new xelement("new3", "some content")))); the output is: <ns2:root xmlns:ns2="namespace"> <ns2:a> <ns2:b>some content</b> <ns2:c>some content</c> <ns2:new> <new1 xmlns=""> <new2>some content</new2> ...
Read more