Created: (SOLR-173) "too many open files" with posting to update handler

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Created: (SOLR-173) "too many open files" with posting to update handler

Sebastian Nagel (Jira)
"too many open files" with posting to update handler
----------------------------------------------------

                 Key: SOLR-173
                 URL: https://issues.apache.org/jira/browse/SOLR-173
             Project: Solr
          Issue Type: Bug
          Components: update
    Affects Versions: 1.2
            Reporter: Ryan McKinley


From brian:

1) Download trunk/nightly
2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
3) java -jar start.jar...
3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
4) After a few seconds/minutes jetty will crash with "too many open files"

- - - - -

all you've got to do is

apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done

with the request handler pointing to /update. Use

# lsof | grep solr | wc -l

to watch the fdescs fly.



--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

Updated: (SOLR-173) "too many open files" with posting to update handler

Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ryan McKinley updated SOLR-173:
-------------------------------

    Attachment: SOLR-173-open-files-bug.patch

This is not a real 'fix' but it does make the problem go away.  I don't understand how or why.  Essentially if you use the QueryResponseWriter it runs out of open files - if you write the output directly, it is fine.

It does not seem to make any difference what is in the response.

Anyone have any idea what could be going on?

- - - -

to make post.sh work with the update handler stuff, you will need to add the content type to <commit/>

curl $URL --data-binary '<commit/>' -H 'Content-type:text/xml; charset=utf-8'


> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Attachments: SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

Commented: (SOLR-173) "too many open files" with posting to update handler

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475664 ]

Bertrand Delacretaz commented on SOLR-173:
------------------------------------------

After making the change shown above in solrconfig.xml, a commit is enough to make the number of open file handles grow:

  curl http://localhost:8983/solr/update --data-binary '<commit/>'  -H 'Content-type:text/xml; charset=utf-8'

lsof shows that many of these file handles point to files in data/index, which have been deleted during the commit:

  lsof -p 9563 | grep data/index | wc -l           (where 9563 is my solr's process ID)

shows 398 file handles after a few commits, although my data/index dir contains only 47 files.

So it looks like something is keeping useless open handles to "old" index files after a commit.



> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Attachments: SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

Commented: (SOLR-173) "too many open files" with posting to update handler

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

    [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475668 ]

Yonik Seeley commented on SOLR-173:
-----------------------------------

The problem is the use of the response writers for something they weren't originally indended for.

Requests are pretty much always associated with an index searcher.  The searcher is grabbed on-demand and used for the remainder of the request's lifetime.  This is important (lucene docids can change, etc).  At the end of the lifetime of a request, close() *must* be called to decrement the reference count and close the searcher if necessary.  An open searcher holds the files of the index open (so it's view of the index never changes).

Problem #1: it doesn't look like doFilter() closes the request object.
Problem #2: update handlers should not request a searcher in the first place... the response writers need to be modified to call getSearcher() on-demand.

> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Attachments: SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

Updated: (SOLR-173) "too many open files" with posting to update handler

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ryan McKinley updated SOLR-173:
-------------------------------

    Attachment: SOLR-173-open-files-bug.patch

this patch takes care of Problem #1, it does not touch #2

closing the request object keeps the open files from stacking up.

thanks Yonik

> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Attachments: SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

Updated: (SOLR-173) "too many open files" with posting to update handler

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ryan McKinley updated SOLR-173:
-------------------------------

    Attachment: SOLR-173-open-files-bug.patch

modifying for problem #2 was easier then i expected, it only touches
 JSONResponseWriter.java
 TextResponseWriter.java
 XMLWriter.java

rather then initalize the searcher in the constructor, they load a local SolrIndexSearcher inside the writeDocList() function

> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Attachments: SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

RE: Updated: (SOLR-173) "too many open files" with posting toupdate handler

Jeryl
I remember when using Lucene directly in a project, I noticed this error
only comes up if the system is deployed on UNIX boxes..in our case
SOLARIS.

You can increase the number of allowed "open files" on unix ..i forget
the command but the default is 256.   We increased it, and the problem
went away.(but the patch looks to address it programmatically as well.)

Jeryl Cook

-----Original Message-----
From: Ryan McKinley (JIRA) [mailto:[hidden email]]
Sent: Saturday, February 24, 2007 7:43 PM
To: [hidden email]
Subject: [jira] Updated: (SOLR-173) "too many open files" with posting
toupdate handler


     [
https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.p
lugin.system.issuetabpanels:all-tabpanel ]

Ryan McKinley updated SOLR-173:
-------------------------------

    Attachment: SOLR-173-open-files-bug.patch

modifying for problem #2 was easier then i expected, it only touches
 JSONResponseWriter.java
 TextResponseWriter.java
 XMLWriter.java

rather then initalize the searcher in the constructor, they load a local
SolrIndexSearcher inside the writeDocList() function

> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Attachments: SOLR-173-open-files-bug.patch,
SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to
<requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell
script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open
files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do
./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

Re: Updated: (SOLR-173) "too many open files" with posting toupdate handler

Ryan McKinley
This problem is different then the normal situation where you can
increase it with "ulimit -n XXX".

This is a real bug introduced by SOLR-104.  As Yonik pointed out, the
filter does not close its request and when the searcher is opened for
an update (unnecessary) it hangs on to closed files indefinitely.

I think the patch on SOLR-173 is good to go - if anyone has a chance
to look at it, that would be great.

thanks
ryan


On 2/26/07, Cook, Jeryl <[hidden email]> wrote:

> I remember when using Lucene directly in a project, I noticed this error
> only comes up if the system is deployed on UNIX boxes..in our case
> SOLARIS.
>
> You can increase the number of allowed "open files" on unix ..i forget
> the command but the default is 256.   We increased it, and the problem
> went away.(but the patch looks to address it programmatically as well.)
>
> Jeryl Cook
>
> -----Original Message-----
> From: Ryan McKinley (JIRA) [mailto:[hidden email]]
> Sent: Saturday, February 24, 2007 7:43 PM
> To: [hidden email]
> Subject: [jira] Updated: (SOLR-173) "too many open files" with posting
> toupdate handler
>
>
>      [
> https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.p
> lugin.system.issuetabpanels:all-tabpanel ]
>
> Ryan McKinley updated SOLR-173:
> -------------------------------
>
>     Attachment: SOLR-173-open-files-bug.patch
>
> modifying for problem #2 was easier then i expected, it only touches
>  JSONResponseWriter.java
>  TextResponseWriter.java
>  XMLWriter.java
>
> rather then initalize the searcher in the constructor, they load a local
> SolrIndexSearcher inside the writeDocList() function
>
> > "too many open files" with posting to update handler
> > ----------------------------------------------------
> >
> >                 Key: SOLR-173
> >                 URL: https://issues.apache.org/jira/browse/SOLR-173
> >             Project: Solr
> >          Issue Type: Bug
> >          Components: update
> >    Affects Versions: 1.2
> >            Reporter: Ryan McKinley
> >         Attachments: SOLR-173-open-files-bug.patch,
> SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch
> >
> >
> > From brian:
> > 1) Download trunk/nightly
> > 2) Change line 347 of example/solr/conf/solrconfig.xml to
> <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> > 3) java -jar start.jar...
> > 3) Run post.sh a bunch of times on the same xml file... (in a shell
> script or whatever)
> > 4) After a few seconds/minutes jetty will crash with "too many open
> files"
> > - - - - -
> > all you've got to do is
> > apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do
> ./post.sh hd.xml; done
> > with the request handler pointing to /update. Use
> > # lsof | grep solr | wc -l
> > to watch the fdescs fly.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
Reply | Threaded
Open this post in threaded view
|

Assigned: (SOLR-173) "too many open files" with posting to update handler

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man reassigned SOLR-173:
-----------------------------

    Assignee: Hoss Man

this patch changes the public API of XMLWriter in a way that isn't backwards compatible if anyone was using it in their own plugin ... but i'm guessing the number of people doing that is fairly low (if any) and the added safety it provides seems worth while ... doing some sanity testing and then i'll commit.

> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Assigned To: Hoss Man
>         Attachments: SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply | Threaded
Open this post in threaded view
|

Resolved: (SOLR-173) "too many open files" with posting to update handler

Sebastian Nagel (Jira)
In reply to this post by Sebastian Nagel (Jira)

     [ https://issues.apache.org/jira/browse/SOLR-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man resolved SOLR-173.
---------------------------

    Resolution: Fixed

commited

> "too many open files" with posting to update handler
> ----------------------------------------------------
>
>                 Key: SOLR-173
>                 URL: https://issues.apache.org/jira/browse/SOLR-173
>             Project: Solr
>          Issue Type: Bug
>          Components: update
>    Affects Versions: 1.2
>            Reporter: Ryan McKinley
>         Assigned To: Hoss Man
>         Attachments: SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch, SOLR-173-open-files-bug.patch
>
>
> From brian:
> 1) Download trunk/nightly
> 2) Change line 347 of example/solr/conf/solrconfig.xml to   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
> 3) java -jar start.jar...
> 3) Run post.sh a bunch of times on the same xml file... (in a shell script or whatever)
> 4) After a few seconds/minutes jetty will crash with "too many open files"
> - - - - -
> all you've got to do is
> apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./post.sh hd.xml; done
> with the request handler pointing to /update. Use
> # lsof | grep solr | wc -l
> to watch the fdescs fly.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.