# open files with SolrCloud

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

# open files with SolrCloud

Sami Siren-2
I have a simple solrcloud setup from trunk with default configs; 1
shard with one replica. As few other people have reported there seems
to be some kind of leak somewhere that causes the number of open files
to grow over time when doing indexing.

One thing that correlates with the open file count that the jvm
reports is the count of deleted files that solr still keeps open (not
sure if the problem is this or something else). The deleted but not
closed files are all ending with "nrm.cfs", for example

<solr_home>/solr/data/index/_jwk_nrm.cfs (deleted)

Any ideas about what could be the cause for this? I don't even know
where to start looking...

--
 Sami Siren
Reply | Threaded
Open this post in threaded view
|

Re: # open files with SolrCloud

Sami Siren-2
On Thu, Apr 19, 2012 at 3:12 PM, Sami Siren <[hidden email]> wrote:

> I have a simple solrcloud setup from trunk with default configs; 1
> shard with one replica. As few other people have reported there seems
> to be some kind of leak somewhere that causes the number of open files
> to grow over time when doing indexing.
>
> One thing that correlates with the open file count that the jvm
> reports is the count of deleted files that solr still keeps open (not
> sure if the problem is this or something else). The deleted but not
> closed files are all ending with "nrm.cfs", for example
>
> <solr_home>/solr/data/index/_jwk_nrm.cfs (deleted)
>
> Any ideas about what could be the cause for this? I don't even know
> where to start looking...

I think the problem is not just the (deleted) files that counts
towards the max open files the user is allowed to have. There's also
the mmapped files that are kept open and count towards the limit
configured in /proc/sys/vm/max_map_count and eventually lead to
exceptions like java.io.IOException: Map failed

there's plenty of those too visible to lsof (all kinds of lucene index
files), for example:

java    32624  sam  DEL    REG                8,3            6425004
/home/sam/example/solr/data/index/_4.fdx
java    32624  sam  DEL    REG                8,3            6424981
/home/sam/example/solr/data/index/_4.fdt
java    32624  sam  DEL    REG                8,3            6425019
/home/sam/example/solr/data/index/_4_nrm.cfs
java    32624  sam  DEL    REG                8,3            6425016
/home/sam/example/solr/data/index/_4_0.tim
java    32624  sam  DEL    REG                8,3            6425015
/home/sam/example/solr/data/index/_4_0.prx
java    32624  sam  DEL    REG                8,3            6425014
/home/sam/example/solr/data/index/_4_0.frq

Why are all these files, regular and mmapped kept open even if merging
has occurred in the background and the files are deleted?

-
 Sami Siren
Reply | Threaded
Open this post in threaded view
|

Re: # open files with SolrCloud

Yonik Seeley-2-2
I can reproduce some kind of searcher leak issue here, even w/o
SolrCloud, and I've opened
https://issues.apache.org/jira/browse/SOLR-3392

-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10
Reply | Threaded
Open this post in threaded view
|

Re: # open files with SolrCloud

Gopal Patwa
Yonik, This same issue we have on our production with Solr 4 Trunk build
running on Cent OS, JDK 6 64-bit

I have reported "java.io.IOException: Map failed" and "Too many open files"
issue, i seems their is a search leak in Solr which is not closing them and
file being kept open.

It would be great help if we can resolve this issue, I was going to try
latest build but it seems this issue is not resolved yet

http://lucene.472066.n3.nabble.com/Large-Index-and-OutOfMemoryError-Map-failed-td3872891.html


On Sat, Apr 21, 2012 at 11:57 AM, Yonik Seeley
<[hidden email]>wrote:

> I can reproduce some kind of searcher leak issue here, even w/o
> SolrCloud, and I've opened
> https://issues.apache.org/jira/browse/SOLR-3392
>
> -Yonik
> lucenerevolution.com - Lucene/Solr Open Source Search Conference.
> Boston May 7-10
>
Reply | Threaded
Open this post in threaded view
|

Re: # open files with SolrCloud

Gopal Patwa
forgot to mention we are not using Solr Cloud yet but we use Lucene NRT
feature,  This issue is happening WITHOUT Solr Cloud

On Sat, Apr 21, 2012 at 8:14 PM, Gopal Patwa <[hidden email]> wrote:

> Yonik, This same issue we have on our production with Solr 4 Trunk build
> running on Cent OS, JDK 6 64-bit
>
> I have reported "java.io.IOException: Map failed" and "Too many open
> files" issue, i seems their is a search leak in Solr which is not closing
> them and file being kept open.
>
> It would be great help if we can resolve this issue, I was going to try
> latest build but it seems this issue is not resolved yet
>
>
> http://lucene.472066.n3.nabble.com/Large-Index-and-OutOfMemoryError-Map-failed-td3872891.html
>
>
> On Sat, Apr 21, 2012 at 11:57 AM, Yonik Seeley <[hidden email]
> > wrote:
>
>> I can reproduce some kind of searcher leak issue here, even w/o
>> SolrCloud, and I've opened
>> https://issues.apache.org/jira/browse/SOLR-3392
>>
>> -Yonik
>> lucenerevolution.com - Lucene/Solr Open Source Search Conference.
>> Boston May 7-10
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: # open files with SolrCloud

Sami Siren-2
In reply to this post by Yonik Seeley-2-2
On Sat, Apr 21, 2012 at 9:57 PM, Yonik Seeley
<[hidden email]> wrote:
> I can reproduce some kind of searcher leak issue here, even w/o
> SolrCloud, and I've opened
> https://issues.apache.org/jira/browse/SOLR-3392

With the fix integrated. I do not see the leaking problem anymore with
my setup so it seems to be working now.

--
 Sami Siren
Reply | Threaded
Open this post in threaded view
|

Re: # open files with SolrCloud

Gopal Patwa
Great! I am going to try new Solr 4 build from April 23rd

On Sun, Apr 22, 2012 at 11:35 PM, Sami Siren <[hidden email]> wrote:

> On Sat, Apr 21, 2012 at 9:57 PM, Yonik Seeley
> <[hidden email]> wrote:
> > I can reproduce some kind of searcher leak issue here, even w/o
> > SolrCloud, and I've opened
> > https://issues.apache.org/jira/browse/SOLR-3392
>
> With the fix integrated. I do not see the leaking problem anymore with
> my setup so it seems to be working now.
>
> --
>  Sami Siren
>