[jira] Created: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space'

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space'

JIRA jira@apache.org
randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space'
---------------------------------------------------------------------

                 Key: HADOOP-849
                 URL: http://issues.apache.org/jira/browse/HADOOP-849
             Project: Hadoop
          Issue Type: Bug
            Reporter: Arun C Murthy
             Fix For: 0.9.2


Reproducible, tried to increase the child jvm's heapsize via
<property>
  <name>mapred.child.java.opts</name>
  <value>-Xmx512m</value>
</property>

without any difference, still fails.

Need to investigate further.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-849?page=all ]

Arun C Murthy updated HADOOP-849:
---------------------------------

    Summary: randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task  (was: randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space')

> randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-849
>                 URL: http://issues.apache.org/jira/browse/HADOOP-849
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>             Fix For: 0.9.2
>
>
> Reproducible, tried to increase the child jvm's heapsize via
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx512m</value>
> </property>
> without any difference, still fails.
> Need to investigate further.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Assigned: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-849?page=all ]

Devaraj Das reassigned HADOOP-849:
----------------------------------

    Assignee: Devaraj Das

> randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-849
>                 URL: http://issues.apache.org/jira/browse/HADOOP-849
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>         Assigned To: Devaraj Das
>             Fix For: 0.9.2
>
>
> Reproducible, tried to increase the child jvm's heapsize via
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx512m</value>
> </property>
> without any difference, still fails.
> Need to investigate further.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-849?page=all ]

Devaraj Das updated HADOOP-849:
-------------------------------

    Attachment: 849.patch

This patch fixes the problem. The merge code that opens map output files for reading used to never close the empty map output files (empty means that the sequence files contains just the sequence file header without any key/val data). For the RandomWriter case, this problem shows up as OutOfMemory, which basically means that the program is out of file descriptors after merging 100s of empty map output files (in the case of RandomWriter, all map outputs are empty; the data is written directly to the dfs).

> randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-849
>                 URL: http://issues.apache.org/jira/browse/HADOOP-849
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>         Assigned To: Devaraj Das
>             Fix For: 0.9.2
>
>         Attachments: 849.patch
>
>
> Reproducible, tried to increase the child jvm's heapsize via
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx512m</value>
> </property>
> without any difference, still fails.
> Need to investigate further.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ http://issues.apache.org/jira/browse/HADOOP-849?page=all ]

Devaraj Das updated HADOOP-849:
-------------------------------

    Status: Patch Available  (was: Open)

> randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-849
>                 URL: http://issues.apache.org/jira/browse/HADOOP-849
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>         Assigned To: Devaraj Das
>             Fix For: 0.9.2
>
>         Attachments: 849.patch
>
>
> Reproducible, tried to increase the child jvm's heapsize via
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx512m</value>
> </property>
> without any difference, still fails.
> Need to investigate further.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ http://issues.apache.org/jira/browse/HADOOP-849?page=comments#action_12461176 ]
           
Hadoop QA commented on HADOOP-849:
----------------------------------

+1, because http://issues.apache.org/jira/secure/attachment/12347969/849.patch applied and successfully tested against trunk revision r489707.

> randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-849
>                 URL: http://issues.apache.org/jira/browse/HADOOP-849
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>         Assigned To: Devaraj Das
>             Fix For: 0.9.2
>
>         Attachments: 849.patch
>
>
> Reproducible, tried to increase the child jvm's heapsize via
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx512m</value>
> </property>
> without any difference, still fails.
> Need to investigate further.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ http://issues.apache.org/jira/browse/HADOOP-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12461825 ]

Owen O'Malley commented on HADOOP-849:
--------------------------------------

+1

> randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-849
>                 URL: http://issues.apache.org/jira/browse/HADOOP-849
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>         Assigned To: Devaraj Das
>             Fix For: 0.9.2
>
>         Attachments: 849.patch
>
>
> Reproducible, tried to increase the child jvm's heapsize via
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx512m</value>
> </property>
> without any difference, still fails.
> Need to investigate further.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ http://issues.apache.org/jira/browse/HADOOP-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12461842 ]

Owen O'Malley commented on HADOOP-849:
--------------------------------------

How critical is this bug fix? It doesn't really require another 9.x bug fix release does it?

> randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-849
>                 URL: http://issues.apache.org/jira/browse/HADOOP-849
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>         Assigned To: Devaraj Das
>             Fix For: 0.9.2
>
>         Attachments: 849.patch
>
>
> Reproducible, tried to increase the child jvm's heapsize via
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx512m</value>
> </property>
> without any difference, still fails.
> Need to investigate further.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12461916 ]

Devaraj Das commented on HADOOP-849:
------------------------------------

The bug fix will be critical for cases where empty map outputs are generated. RandomWriter is one instance of that but, in general, I think that's not the usual case.

> randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-849
>                 URL: https://issues.apache.org/jira/browse/HADOOP-849
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>         Assigned To: Devaraj Das
>             Fix For: 0.9.2
>
>         Attachments: 849.patch
>
>
> Reproducible, tried to increase the child jvm's heapsize via
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx512m</value>
> </property>
> without any difference, still fails.
> Need to investigate further.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/HADOOP-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12461956 ]

Andrzej Bialecki  commented on HADOOP-849:
------------------------------------------

I have experienced this problem when trying to debug a mapred application - it was very difficult to figure out what was wrong, because this error was obscuring the real reason (which was a bug in my Mapper class).

Also, speaking with my Nutch hat on, if there are plans for substantial API changes in trunk/ it would be good to have a bugfix release, which is still API compatible, and which Nutch could use - there have been tons of fixes since 0.9.2 ...

> randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-849
>                 URL: https://issues.apache.org/jira/browse/HADOOP-849
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>         Assigned To: Devaraj Das
>             Fix For: 0.9.2
>
>         Attachments: 849.patch
>
>
> Reproducible, tried to increase the child jvm's heapsize via
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx512m</value>
> </property>
> without any difference, still fails.
> Need to investigate further.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/HADOOP-849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-849:
--------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 0.9.2)
                   0.10.0
           Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, Devaraj!

> randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-849
>                 URL: https://issues.apache.org/jira/browse/HADOOP-849
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>         Assigned To: Devaraj Das
>             Fix For: 0.10.0
>
>         Attachments: 849.patch
>
>
> Reproducible, tried to increase the child jvm's heapsize via
> <property>
>   <name>mapred.child.java.opts</name>
>   <value>-Xmx512m</value>
> </property>
> without any difference, still fails.
> Need to investigate further.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

Doug Cutting
In reply to this post by JIRA jira@apache.org
Andrzej Bialecki (JIRA) wrote:
> Also, speaking with my Nutch hat on, if there are plans for substantial API changes in trunk/ it would be good to have a bugfix release, which is still API compatible, and which Nutch could use - there have been tons of fixes since 0.9.2 ...

The current plan is to make the Hadoop 0.10.0 release this Friday,
barring objections.

There is one significant incompatible change in this release:

https://issues.apache.org/jira/browse/HADOOP-451

We could make a Hadoop 0.9.3 release containing the patch for HADOOP-849
(more work for me).  Instead, Nutch could simply build & commit a
patched version of 0.9.2, or Nutch could upgrade to Hadoop 0.10.0 (less
work for me).  Thoughts?

Doug
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

Andrzej Białecki-2
Doug Cutting wrote:

> Andrzej Bialecki (JIRA) wrote:
>> Also, speaking with my Nutch hat on, if there are plans for
>> substantial API changes in trunk/ it would be good to have a bugfix
>> release, which is still API compatible, and which Nutch could use -
>> there have been tons of fixes since 0.9.2 ...
>
> The current plan is to make the Hadoop 0.10.0 release this Friday,
> barring objections.
>
> There is one significant incompatible change in this release:
>
> https://issues.apache.org/jira/browse/HADOOP-451

Do you think this causes compatibility problems when reading/writing
existing Nutch data? I.e. if we upgrade Nutch to 0.10, is there an issue
here(apart from API changes) that could cause older data to become
unreadable?

>
> We could make a Hadoop 0.9.3 release containing the patch for
> HADOOP-849 (more work for me).  Instead, Nutch could simply build &
> commit a patched version of 0.9.2, or Nutch could upgrade to Hadoop
> 0.10.0 (less work for me).  Thoughts?

Hmm. Let me check how much work is involved in upgrading Nutch to 0.10
on the API level ... sooner or later Nutch will have to follow these
changes anyway, the question is if we have enough resources to do it now.

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

Nigel Daley
In reply to this post by Doug Cutting

On Jan 3, 2007, at 10:40 AM, Doug Cutting wrote:

> Andrzej Bialecki (JIRA) wrote:
>> Also, speaking with my Nutch hat on, if there are plans for  
>> substantial API changes in trunk/ it would be good to have a  
>> bugfix release, which is still API compatible, and which Nutch  
>> could use - there have been tons of fixes since 0.9.2 ...
>
> The current plan is to make the Hadoop 0.10.0 release this Friday,  
> barring objections.

I'd like to propose these get committed for 0.10.0.  I've been  
testing and benchmarking with them since Dec 22 and they greatly help  
overall stability and performance:

https://issues.apache.org/jira/browse/HADOOP-840
https://issues.apache.org/jira/browse/HADOOP-846

I realize they need a final +1 from a code reviewer, but I'm hoping  
that will happen today.

Cheers,
Nige
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

Doug Cutting
In reply to this post by Andrzej Białecki-2
Andrzej Bialecki wrote:
>> There is one significant incompatible change in this release:
>>
>> https://issues.apache.org/jira/browse/HADOOP-451
>
> Do you think this causes compatibility problems when reading/writing
> existing Nutch data? I.e. if we upgrade Nutch to 0.10, is there an issue
> here(apart from API changes) that could cause older data to become
> unreadable?

The incompatibility is code-level, not file-level.  Existing data on
disk should be fine, only APIs have changed.

Doug
Reply | Threaded
Open this post in threaded view
|

Re: [jira] Commented: (HADOOP-849) randomwriter fails with 'java.lang.OutOfMemoryError: Java heap space' in the 'reduce' task

Andrzej Białecki-2
In reply to this post by Andrzej Białecki-2
Andrzej Bialecki wrote:
>>
>> We could make a Hadoop 0.9.3 release containing the patch for
>> HADOOP-849 (more work for me).  Instead, Nutch could simply build &
>> commit a patched version of 0.9.2, or Nutch could upgrade to Hadoop
>> 0.10.0 (less work for me).  Thoughts?
>
> Hmm. Let me check how much work is involved in upgrading Nutch to 0.10
> on the API level ... sooner or later Nutch will have to follow these
> changes anyway, the question is if we have enough resources to do it now.

I just upgraded Nutch to use Hadoop 0.10.1, so 0.9.3 is not needed from
my POV. Thanks!

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com