Quantcast

ArrayWritable usage

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

ArrayWritable usage

Dhruv Kumar
I'm having some difficulty with using ArrayWritable in the following test
code:

    ArrayWritable array = new ArrayWritable(IntWritable.class);

      IntWritable[] ints = new IntWritable[4];

      for (int i =0 ; i < 4; i++) {
        ints[i] = new IntWritable(i);

      }

      array.set(ints);

      writer.append(new LongWritable(1), array);

      writer.close();

After starting the map reduce application's test, I'm getting the following
errors which I suspect are due to improper use of ArrayWritable. Can someone
tell me if the code given above is correct?

java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.io.ArrayWritable.<init>()
    at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
    at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62)
    at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
    at
org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817)
    at
org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790)
    at
org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74)
    at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531)
    at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
    at org.apache.hadoop.mapred.Child.main(Child.java:253)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.io.ArrayWritable.<init>()
    at java.lang.Class.getConstructor0(Class.java:2706)
    at java.lang.Class.getDeclaredConstructor(Class.java:1985)
    at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
    ... 15 more
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ArrayWritable usage

Joey Echeverria
ArrayWritable doesn't serialize type information. You need to subclass it
(e.g. IntArrayWritable) and create a no arg constructor which calls
super(IntWritable.class).

Use this instead of ArrayWritable directly. If you want to store more than
one type, look at the source for MapWritable to see how it generates type
codes.

-Joey
On Jul 4, 2011 2:55 PM, "Dhruv Kumar" <[hidden email]> wrote:

> I'm having some difficulty with using ArrayWritable in the following test
> code:
>
> ArrayWritable array = new ArrayWritable(IntWritable.class);
>
> IntWritable[] ints = new IntWritable[4];
>
> for (int i =0 ; i < 4; i++) {
> ints[i] = new IntWritable(i);
>
> }
>
> array.set(ints);
>
> writer.append(new LongWritable(1), array);
>
> writer.close();
>
> After starting the map reduce application's test, I'm getting the
following
> errors which I suspect are due to improper use of ArrayWritable. Can
someone
> tell me if the code given above is correct?
>
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> org.apache.hadoop.io.ArrayWritable.<init>()
> at
>
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> at
>
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62)
> at
>
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
> at
>
org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817)
> at
>
org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790)
> at
>
org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74)
> at
>
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531)

> at
> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
>
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
> Caused by: java.lang.NoSuchMethodException:
> org.apache.hadoop.io.ArrayWritable.<init>()
> at java.lang.Class.getConstructor0(Class.java:2706)
> at java.lang.Class.getDeclaredConstructor(Class.java:1985)
> at
>
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
> ... 15 more
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ArrayWritable usage

Dhruv Kumar
Thanks,  your answer and the documentation at
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/ArrayWritable.htmlfixed
the issue.

For others benefit, I have reproduced the solution below.

Modified test code:

   SequenceFile.Writer writer = SequenceFile.createWriter(fs, conf, new
Path("/home/input/in"), LongWritable.class, IntArrayWritable.class);

      IntArrayWritable array = new IntArrayWritable();

      IntWritable[] ints = new IntWritable[4];

      for (int i =0 ; i < 4; i++) {
        ints[i] = new IntWritable(i);

      }

      array.set(ints);

      writer.append(new LongWritable(1), array);

      writer.close();



New IntArrayWritableClass:

public class IntArrayWritable extends ArrayWritable {
  public IntArrayWritable() {
    super(IntWritable.class);
  }
}


On Mon, Jul 4, 2011 at 3:00 PM, Joey Echeverria <[hidden email]> wrote:

> ArrayWritable doesn't serialize type information. You need to subclass it
> (e.g. IntArrayWritable) and create a no arg constructor which calls
> super(IntWritable.class).
>
> Use this instead of ArrayWritable directly. If you want to store more than
> one type, look at the source for MapWritable to see how it generates type
> codes.
>
> -Joey
> On Jul 4, 2011 2:55 PM, "Dhruv Kumar" <[hidden email]> wrote:
> > I'm having some difficulty with using ArrayWritable in the following test
> > code:
> >
> > ArrayWritable array = new ArrayWritable(IntWritable.class);
> >
> > IntWritable[] ints = new IntWritable[4];
> >
> > for (int i =0 ; i < 4; i++) {
> > ints[i] = new IntWritable(i);
> >
> > }
> >
> > array.set(ints);
> >
> > writer.append(new LongWritable(1), array);
> >
> > writer.close();
> >
> > After starting the map reduce application's test, I'm getting the
> following
> > errors which I suspect are due to improper use of ArrayWritable. Can
> someone
> > tell me if the code given above is correct?
> >
> > java.lang.RuntimeException: java.lang.NoSuchMethodException:
> > org.apache.hadoop.io.ArrayWritable.<init>()
> > at
> >
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> > at
> >
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62)
> > at
> >
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
> > at
> >
>
> org.apache.hadoop.io.SequenceFile$Reader.deserializeValue(SequenceFile.java:1817)
> > at
> >
>
> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1790)
> > at
> >
>
> org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74)
> > at
> >
>
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531)
> > at
> > org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> > at org.apache.hadoop.mapred.Child.main(Child.java:253)
> > Caused by: java.lang.NoSuchMethodException:
> > org.apache.hadoop.io.ArrayWritable.<init>()
> > at java.lang.Class.getConstructor0(Class.java:2706)
> > at java.lang.Class.getDeclaredConstructor(Class.java:1985)
> > at
> >
>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
> > ... 15 more
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ArrayWritable usage

zanurag
Hi Dhruv,
Is this working well for you ?? Are you able to do IntWritable [] abc = array.get();

I am trying similar thing for IntTwoDArrayWritable.
The array.set works but array.get returns Writable[][] and I am not able
to cast it to IntWritable[][].
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ArrayWritable usage

Brock Noland
Hi,

ArrayWritable is a touch hard to use. Say you have an array of
IntWritable[]. The get() method or ArrayWritable, after
serializations/deserialization, does in fact return an array of type
Writable. As such you cannot cast it directly to IntWritable[]. Individual
elements are of type IntWritable and can be cast as such.

Will not work:

IntWritable[] array = (IntWritable[]) writable.get();

Will work:

for(Writable element : writable.get()) {
  IntWritable intWritable = (IntWritable)element;
}

Brock

On Sat, Dec 10, 2011 at 3:58 PM, zanurag <[hidden email]> wrote:

> Hi Dhruv,
> Is this working well for you ?? Are you able to do IntWritable [] abc =
> array.get();
>
> I am trying similar thing for IntTwoDArrayWritable.
> The array.set works but array.get returns Writable[][] and I am not able
> to cast it to IntWritable[][].
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/ArrayWritable-usage-tp3138520p3576386.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
Loading...