Quantcast

LZO exception decompressing (returned -8)

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

LZO exception decompressing (returned -8)

Marc Sturlese
Hey there,
I've been running a cluster for over a year and was getting a lzo decompressing exception less than once a month. Suddenly it happens almost once per day. Any ideas what could be causing it? I'm with hadoop 0.20.2
I've thought in moving to snappy but would like to know why this happens more often now

The exception happens always when the reducer gets data from the map and looks like:

Error: java.lang.InternalError: lzo1x_decompress returned: -8
        at com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect(Native Method)
        at com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:305)
        at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:76)
        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1553)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1432)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1285)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1216)

Thanks in advance.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LZO exception decompressing (returned -8)

Joey Echeverria
Which version of the Hadoop LZO library are you using? It looks like something I'm pretty sure was fixed in a newer version.

-Joey



On Feb 28, 2012, at 4:58, Marc Sturlese <[hidden email]> wrote:

> Hey there,
> I've been running a cluster for over a year and was getting a lzo
> decompressing exception less than once a month. Suddenly it happens almost
> once per day. Any ideas what could be causing it? I'm with hadoop 0.20.2
> I've thought in moving to snappy but would like to know why this happens
> more often now
>
> The exception happens always when the reducer gets data from the map and
> looks like:
>
> Error: java.lang.InternalError: lzo1x_decompress returned: -8
>    at com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect(Native
> Method)
>    at
> com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:305)
>    at
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:76)
>    at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
>    at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1553)
>    at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1432)
>    at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1285)
>    at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1216)
>
> Thanks in advance.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3783652.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LZO exception decompressing (returned -8)

Marc Sturlese
I'm with 0.4.9 (think is the latest)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LZO exception decompressing (returned -8)

Joey Echeverria
Try 0.4.15. You can get it from here:

https://github.com/toddlipcon/hadoop-lzo

Sent from my iPhone

On Feb 28, 2012, at 6:49, Marc Sturlese <[hidden email]> wrote:

> I'm with 0.4.9 (think is the latest)
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3783927.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LZO exception decompressing (returned -8)

Marc Sturlese
This post was updated on .
Tried but still getting the error with the 0.4.15 lzo version. Really lost with this.
My hadoop release is 0.20.2 from more than a year ago. Could this be related to the problem?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LZO exception decompressing (returned -8)

Harsh J-2
Marc,

Was the lzo libs on your server upgraded to a higher version recently?

Also, when you deployed a built copy of 0.4.15, did you ensure you
replaced the older native libs for hadoop-lzo as well?

On Fri, Mar 2, 2012 at 9:05 AM, Marc Sturlese <[hidden email]> wrote:
> Tried but still getting the error 0.4.15. Really lost with this.
> My hadoop release is 0.20.2 from more than a year ago. Could this be related
> to the problem?
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3792484.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.



--
Harsh J
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LZO exception decompressing (returned -8)

Marc Sturlese
Yes, The steps I followed where:
1-Intall lzo 2.06 in a machine with the same kernel as my nodes.
2-Compile there lzo 0.4.15 (in /lib replaced cdh3u3 per my hadoop 0.20.2 release)
3-Replace hadoop-lzo-0.4.9.jar for the now compiled hadoop-lzo-0.4.15.jar in the hadoop lib directory of all my nodes and master
4-Put de generated native files in the native lib directory of all the nodes and master
5-In my jar job, replaced the jar library hadoop-lzo-0.4.9.jar for hadoop-lzo-0.4.15.jar

And sometimes when a job is running I get (4 times so the job gets killed):

...org.apache.hadoop.mapred.ReduceTask: Shuffling 3188320 bytes (1025174 raw bytes) into RAM from attempt_201202291221_1501_m_000480_0
2012-03-02 02:32:55,496 INFO org.apache.hadoop.mapred.ReduceTask: Task attempt_201202291221_1501_r_000105_0: Failed fetch #1 from attempt_201202291221_1501_m_000046_0
2012-03-02 02:32:55,496 WARN org.apache.hadoop.mapred.ReduceTask: attempt_201202291221_1501_r_000105_0 adding host hadoop-01.backend to penalty box, next contact in 4 seconds
2012-03-02 02:32:55,496 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201202291221_1501_r_000105_0: Got 1 map-outputs from previous failures
2012-03-02 02:32:55,497 FATAL org.apache.hadoop.mapred.TaskRunner: attempt_201202291221_1501_r_000105_0 : Map output copy failure : java.lang.InternalError: lzo1x_decompress returned: -8
        at com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect(Native Method)
        at com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:305)
        at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:76)
        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1553)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1432)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1285)
        at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1216)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LZO exception decompressing (returned -8)

Marc Sturlese
I use to have 2.05 but now as I said I installed 2.06
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LZO exception decompressing (returned -8)

Joey Echeverria
I know this doesn't fix lzo, but have you considered Snappy for the
intermediate output compression? It gets similar compression ratios
and compress/decompress speed, but arguably has better Hadoop
integration.

-Joey

On Thu, Mar 1, 2012 at 10:01 PM, Marc Sturlese <[hidden email]> wrote:
> I use to have 2.05 but now as I said I installed 2.06
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3792511.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.



--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LZO exception decompressing (returned -8)

Marc Sturlese
Absolutely. In case I don't find the root of the problem soon I'll definitely try it.
Loading...