File readers inside of Mappers ad Reducers

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

File readers inside of Mappers ad Reducers

Dennis Kubes
How would someone go about reading Map and Sequence file contents in
Mappers and Reducers?  Is it best to only use the addInputDirectory
method and find a way to get all of the data one needs by key or is
there a good way to read file contents inside of map and reduce calls.

I am not asking about what classes (i.e. MapFile.Reader, etc.)  I want
to know a consistent way to do it.  Right now I am using a static
reference in the main class, called by an inner class.  This works well
on one machine but I don't think it is going to work across nodes.  Here
is how I have it now:

class x {

  private static MapFile.Reader reader = null;

  public x() {
    reader = new MapFile.Reader...
    create mapreduce job
       setMapperClass(InnerX.class);
    submit job
    close reader.
  }

  class InnerX {
 
    map(WritableComparable key, Writable value, OutputCollector output,
Reporter reporter) {
      do something with reader...
    }
  }

}

Is there a better way to do this?

Dennis