> Hrm.... no. I've lowered it to -1, but I can try 1 again.
>
> On Sun, Sep 26, 2010 at 6:47 AM, Ted Yu <
[hidden email]> wrote:
> > Have you tried lowering mapred.job.reuse.jvm.num.tasks ?
> >
> > On Sun, Sep 26, 2010 at 3:30 AM, Bradford Stephens <
> >
[hidden email]> wrote:
> >
> >> Nope, that didn't seem to help.
> >>
> >> On Sun, Sep 26, 2010 at 1:00 AM, Bradford Stephens
> >> <
[hidden email]> wrote:
> >> > I'm going to try running it on high-RAM boxes with -Xmx4096m or so,
> >> > see if that helps.
> >> >
> >> > On Sun, Sep 26, 2010 at 12:55 AM, Bradford Stephens
> >> > <
[hidden email]> wrote:
> >> >> Greetings,
> >> >>
> >> >> I'm running into a brain-numbing problem on Elastic MapReduce. I'm
> >> >> running a decent-size task (22,000 mappers, a ton of GZipped input
> >> >> blocks, ~1TB of data) on 40 c1.xlarge nodes (7 gb RAM, ~8 "cores").
> >> >>
> >> >> I get failures randomly --- sometimes at the end of my 6-step
> process,
> >> >> sometimes at the first reducer phase, sometimes in the mapper. It
> >> >> seems to fail in multiple areas. Mostly in the reducers. Any ideas?
> >> >>
> >> >> Here's the settings I've changed:
> >> >> -Xmx400m
> >> >> 6 max mappers
> >> >> 1 max reducer
> >> >> 1GB swap partition
> >> >> mapred.job.reuse.jvm.num.tasks=50
> >> >> mapred.reduce.parallel.copies=3
> >> >>
> >> >>
> >> >> java.lang.OutOfMemoryError: GC overhead limit exceeded
> >> >> at java.nio.CharBuffer.wrap(CharBuffer.java:350)
> >> >> at java.nio.CharBuffer.wrap(CharBuffer.java:373)
> >> >> at
> >> java.lang.StringCoding$StringDecoder.decode(StringCoding.java:138)
> >> >> at java.lang.StringCoding.decode(StringCoding.java:173)
> >> >> at java.lang.String.(String.java:443)
> >> >> at java.lang.String.(String.java:515)
> >> >> at
> >> org.apache.hadoop.io.WritableUtils.readString(WritableUtils.java:116)
> >> >> at
> >> cascading.tuple.TupleInputStream.readString(TupleInputStream.java:144)
> >> >> at
> >> cascading.tuple.TupleInputStream.readType(TupleInputStream.java:154)
> >> >> at
> >>
> cascading.tuple.TupleInputStream.getNextElement(TupleInputStream.java:101)
> >> >> at
> >>
> cascading.tuple.hadoop.TupleElementComparator.compare(TupleElementComparator.java:75)
> >> >> at
> >>
> cascading.tuple.hadoop.TupleElementComparator.compare(TupleElementComparator.java:33)
> >> >> at
> >>
> cascading.tuple.hadoop.DelegatingTupleElementComparator.compare(DelegatingTupleElementComparator.java:74)
> >> >> at
> >>
> cascading.tuple.hadoop.DelegatingTupleElementComparator.compare(DelegatingTupleElementComparator.java:34)
> >> >> at
> >>
> cascading.tuple.hadoop.DeserializerComparator.compareTuples(DeserializerComparator.java:142)
> >> >> at
> >>
> cascading.tuple.hadoop.GroupingSortingComparator.compare(GroupingSortingComparator.java:55)
> >> >> at
> >> org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373)
> >> >> at
> >> org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:136)
> >> >> at
> >> org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103)
> >> >> at
> >>
> org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:335)
> >> >> at
> >> org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350)
> >> >> at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156)
> >> >> at
> >>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2645)
> >> >> at
> >>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2586)
> >> >>
> >> >> --
> >> >> Bradford Stephens,
> >> >> Founder, Drawn to Scale
> >> >> drawntoscalehq.com
> >> >> 727.697.7528
> >> >>
> >> >>
http://www.drawntoscalehq.com -- The intuitive, cloud-scale data
> >> >> solution. Process, store, query, search, and serve all your data.
> >> >>
> >> >>
http://www.roadtofailure.com -- The Fringes of Scalability, Social
> >> >> Media, and Computer Science
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Bradford Stephens,
> >> > Founder, Drawn to Scale
> >> > drawntoscalehq.com
> >> > 727.697.7528
> >> >
> >> >
http://www.drawntoscalehq.com -- The intuitive, cloud-scale data
> >> > solution. Process, store, query, search, and serve all your data.
> >> >
> >> >
http://www.roadtofailure.com -- The Fringes of Scalability, Social
> >> > Media, and Computer Science
> >> >
> >>
> >>
> >>
> >> --
> >> Bradford Stephens,
> >> Founder, Drawn to Scale
> >> drawntoscalehq.com
> >> 727.697.7528
> >>
> >>
http://www.drawntoscalehq.com -- The intuitive, cloud-scale data
> >> solution. Process, store, query, search, and serve all your data.
> >>
> >>
http://www.roadtofailure.com -- The Fringes of Scalability, Social
> >> Media, and Computer Science
> >>
> >
>
>
>
> --
> Bradford Stephens,
> Founder, Drawn to Scale
> drawntoscalehq.com
> 727.697.7528
>
>
http://www.drawntoscalehq.com -- The intuitive, cloud-scale data
> solution. Process, store, query, search, and serve all your data.
>
>
http://www.roadtofailure.com -- The Fringes of Scalability, Social
> Media, and Computer Science
>
Locked
