Too many open files

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Too many open files

Howie Wang
Hi,

I think I remember seeing some messages about "Too many open files"
when merging a while ago. I recently started getting this on Nutch 0.7
using JDK 1.4.2 on WinXP while I was trying to reparse a segment.
I looked around and I found this Java bug:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4189011

It looks like on Windows, you can work around this by switching to
JDK 1.5. I didn't recompile since I thought 0.7 didn't compile against
JDK 1.5. It seemed to run fine for me by just using the 1.4 compiled
jar file and using the 1.5 Java executable.

It still seems that this will be an issue for other platforms, which have
hard limits on the number of file descriptors. Is there a file descriptor
leak in the merge or parsing code?

Howie


Reply | Threaded
Open this post in threaded view
|

Re: Too many open files

Thomas Delnoij-3
> t still seems that this will be an issue for other platforms, which have
hard limits on the number of file descriptors.

What platforms?

You can use ulimit -n to increase the limit on Unix/Linux systems.

Rgrds, Thomas



On 6/13/06, Howie Wang <[hidden email]> wrote:

> Hi,
>
> I think I remember seeing some messages about "Too many open files"
> when merging a while ago. I recently started getting this on Nutch 0.7
> using JDK 1.4.2 on WinXP while I was trying to reparse a segment.
> I looked around and I found this Java bug:
>
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4189011
>
> It looks like on Windows, you can work around this by switching to
> JDK 1.5. I didn't recompile since I thought 0.7 didn't compile against
> JDK 1.5. It seemed to run fine for me by just using the 1.4 compiled
> jar file and using the 1.5 Java executable.
>
> It still seems that this will be an issue for other platforms, which have
> hard limits on the number of file descriptors. Is there a file descriptor
> leak in the merge or parsing code?
>
> Howie
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Too many open files

Howie Wang
You're right. I guess I misunderstood the term hard limit
when talking about file descriptor limits.

Still, why is Nutch opening so many file descriptors during
merge or reparse? 2000+ open file descriptors doesn't seem
intentional. Plus, my DB is not that big (~1M pages).

Has anyone seen this with 0.8?

>You can use ulimit -n to increase the limit on Unix/Linux systems.
>
>Rgrds, Thomas
>
>
>
>On 6/13/06, Howie Wang <[hidden email]> wrote:
>>Hi,
>>
>>I think I remember seeing some messages about "Too many open files"
>>when merging a while ago. I recently started getting this on Nutch 0.7
>>using JDK 1.4.2 on WinXP while I was trying to reparse a segment.
>>I looked around and I found this Java bug:
>>
>>http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4189011
>>
>>It looks like on Windows, you can work around this by switching to
>>JDK 1.5. I didn't recompile since I thought 0.7 didn't compile against
>>JDK 1.5. It seemed to run fine for me by just using the 1.4 compiled
>>jar file and using the 1.5 Java executable.
>>
>>It still seems that this will be an issue for other platforms, which have
>>hard limits on the number of file descriptors. Is there a file descriptor
>>leak in the merge or parsing code?
>>
>>Howie
>>
>>
>>


Reply | Threaded
Open this post in threaded view
|

Re: Too many open files

Thomas Delnoij-3
> Has anyone seen this with 0.8?

I think everybody has seen this :)

It *is* intentional and part of how Nutch and MapReduce/Hadoop works, I believe.

Rgrds, Thomas

On 6/16/06, Howie Wang <[hidden email]> wrote:

> You're right. I guess I misunderstood the term hard limit
> when talking about file descriptor limits.
>
> Still, why is Nutch opening so many file descriptors during
> merge or reparse? 2000+ open file descriptors doesn't seem
> intentional. Plus, my DB is not that big (~1M pages).
>
>
> >You can use ulimit -n to increase the limit on Unix/Linux systems.
> >
> >Rgrds, Thomas
> >
> >
> >
> >On 6/13/06, Howie Wang <[hidden email]> wrote:
> >>Hi,
> >>
> >>I think I remember seeing some messages about "Too many open files"
> >>when merging a while ago. I recently started getting this on Nutch 0.7
> >>using JDK 1.4.2 on WinXP while I was trying to reparse a segment.
> >>I looked around and I found this Java bug:
> >>
> >>http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4189011
> >>
> >>It looks like on Windows, you can work around this by switching to
> >>JDK 1.5. I didn't recompile since I thought 0.7 didn't compile against
> >>JDK 1.5. It seemed to run fine for me by just using the 1.4 compiled
> >>jar file and using the 1.5 Java executable.
> >>
> >>It still seems that this will be an issue for other platforms, which have
> >>hard limits on the number of file descriptors. Is there a file descriptor
> >>leak in the merge or parsing code?
> >>
> >>Howie
> >>
> >>
> >>
>
>
>