Issues with stale searchers.

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Issues with stale searchers.

Jeremy Carroll-3
I have Solr 1.3 running on Apache Tomcat 5.5.27. I'm running into an issue where searchers are opened up right away when tomcat starts, and never goes away. This is causing read locks on the Lucene index holding open deleted files during merges. This causes our server to run out of disk space in our index. Wondering what is causing this issue as I have been searching for two days without any real answers.

Thanks,

LSOF Output
java       7322   tomcat   70r      REG              253,0      2569538    2883610 /opt/solr/data/index/_m5n.cfs (deleted)
java       7322   tomcat   71r      REG              253,0      2338291    2883609 /opt/solr/data/index/_m5m.cfs (deleted)
java       7322   tomcat   72r      REG              253,0     13398930    2883608 /opt/solr/data/index/_m5l.cfs (deleted)
java       7322   tomcat   73r      REG              253,0      2692917    2883598 /opt/solr/data/index/_m5k.cfs (deleted)
java       7322   tomcat   74r      REG              253,0     32324600    2883592 /opt/solr/data/index/_m5j.cfx (deleted)
java       7322   tomcat   75r      REG              253,0      6767344    2883603 /opt/solr/data/index/_m5j.cfs (deleted)
java       7322   tomcat   76r      REG              253,0     32324600    2883592 /opt/solr/data/index/_m5j.cfx (deleted)
java       7322   tomcat   77r      REG              253,0     15937346    2883600 /opt/solr/data/index/_m5i.cfs (deleted)

Stats page on Solr Admin


                 Searcher@66952905 main
class:     org.apache.solr.search.SolrIndexSearcher
version:                1.0
description:        index searcher
stats:     searcherName : Searcher@66952905 main
caching : true
numDocs : 187169908
maxDoc : 187169908
readerImpl : ReadOnlyMultiSegmentReader
readerDir : org.apache.lucene.store.FSDirectory@/opt/solr/data/index
indexVersion : 1224609883675
openedAt : Thu Mar 12 17:13:15 CDT 2009
registeredAt : Thu Mar 12 17:13:23 CDT 2009
warmupTime : 0

name:   core
class:
version:                1.0
description:        SolrCore
stats:     coreName :
startTime : Thu Mar 12 17:13:15 CDT 2009
refCount : 2
aliases : []

name:   searcher
class:     org.apache.solr.search.SolrIndexSearcher
version:                1.0
description:        index searcher
stats:     searcherName : Searcher@66952905 main
caching : true
numDocs : 187169908
maxDoc : 187169908
readerImpl : ReadOnlyMultiSegmentReader
readerDir : org.apache.lucene.store.FSDirectory@/opt/solr/data/index
indexVersion : 1224609883675
openedAt : Thu Mar 12 17:13:15 CDT 2009
registeredAt : Thu Mar 12 17:13:23 CDT 2009
warmupTime : 0

Jeremy Carroll
Sr. Network Engineer
Networked Insights


Reply | Threaded
Open this post in threaded view
|

Re: Issues with stale searchers.

Yonik Seeley-2-2
On Thu, Mar 12, 2009 at 6:29 PM, Jeremy Carroll
<[hidden email]> wrote:
> I have Solr 1.3 running on Apache Tomcat 5.5.27. I'm running into an issue where searchers are opened up right away when tomcat starts, and never goes away. This is causing read locks on the Lucene index holding open deleted files during merges.


Deleted files being held open can be normal - that's the current
IndexSearcher serving requests (even though those files may have been
deleted by the IndexWriter already).

Looking at your Stats, I only see one  Searcher, so things look fine there too.

-Yonik
http://www.lucidimagination.com


 This causes our server to run out of disk space in our index.
Wondering what is causing this issue as I have been searching for two
days without any real answers.

>
> Thanks,
>
> LSOF Output
> java       7322   tomcat   70r      REG              253,0      2569538    2883610 /opt/solr/data/index/_m5n.cfs (deleted)
> java       7322   tomcat   71r      REG              253,0      2338291    2883609 /opt/solr/data/index/_m5m.cfs (deleted)
> java       7322   tomcat   72r      REG              253,0     13398930    2883608 /opt/solr/data/index/_m5l.cfs (deleted)
> java       7322   tomcat   73r      REG              253,0      2692917    2883598 /opt/solr/data/index/_m5k.cfs (deleted)
> java       7322   tomcat   74r      REG              253,0     32324600    2883592 /opt/solr/data/index/_m5j.cfx (deleted)
> java       7322   tomcat   75r      REG              253,0      6767344    2883603 /opt/solr/data/index/_m5j.cfs (deleted)
> java       7322   tomcat   76r      REG              253,0     32324600    2883592 /opt/solr/data/index/_m5j.cfx (deleted)
> java       7322   tomcat   77r      REG              253,0     15937346    2883600 /opt/solr/data/index/_m5i.cfs (deleted)
>
> Stats page on Solr Admin
>
>
>                 Searcher@66952905 main
> class:     org.apache.solr.search.SolrIndexSearcher
> version:                1.0
> description:        index searcher
> stats:     searcherName : Searcher@66952905 main
> caching : true
> numDocs : 187169908
> maxDoc : 187169908
> readerImpl : ReadOnlyMultiSegmentReader
> readerDir : org.apache.lucene.store.FSDirectory@/opt/solr/data/index
> indexVersion : 1224609883675
> openedAt : Thu Mar 12 17:13:15 CDT 2009
> registeredAt : Thu Mar 12 17:13:23 CDT 2009
> warmupTime : 0
>
> name:   core
> class:
> version:                1.0
> description:        SolrCore
> stats:     coreName :
> startTime : Thu Mar 12 17:13:15 CDT 2009
> refCount : 2
> aliases : []
>
> name:   searcher
> class:     org.apache.solr.search.SolrIndexSearcher
> version:                1.0
> description:        index searcher
> stats:     searcherName : Searcher@66952905 main
> caching : true
> numDocs : 187169908
> maxDoc : 187169908
> readerImpl : ReadOnlyMultiSegmentReader
> readerDir : org.apache.lucene.store.FSDirectory@/opt/solr/data/index
> indexVersion : 1224609883675
> openedAt : Thu Mar 12 17:13:15 CDT 2009
> registeredAt : Thu Mar 12 17:13:23 CDT 2009
> warmupTime : 0
>
> Jeremy Carroll
> Sr. Network Engineer
> Networked Insights
>
>
Reply | Threaded
Open this post in threaded view
|

RE: Issues with stale searchers.

Jeremy Carroll-3
If that's the case it is causing out of disk issues with Solr. We have a 187m document count index which is about ~200Gb in size. Over a period of about a week after optimizations, etc... the open file but deleted count grows very large. Causing the system to not be able to optimize due to lack of disk space. Also new documents that are indexed are not showing up in search results.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
Sent: Thursday, March 12, 2009 7:43 PM
To: [hidden email]
Subject: Re: Issues with stale searchers.

On Thu, Mar 12, 2009 at 6:29 PM, Jeremy Carroll
<[hidden email]> wrote:
> I have Solr 1.3 running on Apache Tomcat 5.5.27. I'm running into an issue where searchers are opened up right away when tomcat starts, and never goes away. This is causing read locks on the Lucene index holding open deleted files during merges.


Deleted files being held open can be normal - that's the current
IndexSearcher serving requests (even though those files may have been
deleted by the IndexWriter already).

Looking at your Stats, I only see one  Searcher, so things look fine there too.

-Yonik
http://www.lucidimagination.com


 This causes our server to run out of disk space in our index.
Wondering what is causing this issue as I have been searching for two
days without any real answers.

>
> Thanks,
>
> LSOF Output
> java       7322   tomcat   70r      REG              253,0      2569538    2883610 /opt/solr/data/index/_m5n.cfs (deleted)
> java       7322   tomcat   71r      REG              253,0      2338291    2883609 /opt/solr/data/index/_m5m.cfs (deleted)
> java       7322   tomcat   72r      REG              253,0     13398930    2883608 /opt/solr/data/index/_m5l.cfs (deleted)
> java       7322   tomcat   73r      REG              253,0      2692917    2883598 /opt/solr/data/index/_m5k.cfs (deleted)
> java       7322   tomcat   74r      REG              253,0     32324600    2883592 /opt/solr/data/index/_m5j.cfx (deleted)
> java       7322   tomcat   75r      REG              253,0      6767344    2883603 /opt/solr/data/index/_m5j.cfs (deleted)
> java       7322   tomcat   76r      REG              253,0     32324600    2883592 /opt/solr/data/index/_m5j.cfx (deleted)
> java       7322   tomcat   77r      REG              253,0     15937346    2883600 /opt/solr/data/index/_m5i.cfs (deleted)
>
> Stats page on Solr Admin
>
>
>                 Searcher@66952905 main
> class:     org.apache.solr.search.SolrIndexSearcher
> version:                1.0
> description:        index searcher
> stats:     searcherName : Searcher@66952905 main
> caching : true
> numDocs : 187169908
> maxDoc : 187169908
> readerImpl : ReadOnlyMultiSegmentReader
> readerDir : org.apache.lucene.store.FSDirectory@/opt/solr/data/index
> indexVersion : 1224609883675
> openedAt : Thu Mar 12 17:13:15 CDT 2009
> registeredAt : Thu Mar 12 17:13:23 CDT 2009
> warmupTime : 0
>
> name:   core
> class:
> version:                1.0
> description:        SolrCore
> stats:     coreName :
> startTime : Thu Mar 12 17:13:15 CDT 2009
> refCount : 2
> aliases : []
>
> name:   searcher
> class:     org.apache.solr.search.SolrIndexSearcher
> version:                1.0
> description:        index searcher
> stats:     searcherName : Searcher@66952905 main
> caching : true
> numDocs : 187169908
> maxDoc : 187169908
> readerImpl : ReadOnlyMultiSegmentReader
> readerDir : org.apache.lucene.store.FSDirectory@/opt/solr/data/index
> indexVersion : 1224609883675
> openedAt : Thu Mar 12 17:13:15 CDT 2009
> registeredAt : Thu Mar 12 17:13:23 CDT 2009
> warmupTime : 0
>
> Jeremy Carroll
> Sr. Network Engineer
> Networked Insights
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Issues with stale searchers.

Yonik Seeley-2-2
On Thu, Mar 12, 2009 at 9:38 PM, Jeremy Carroll
<[hidden email]> wrote:
> If that's the case it is causing out of disk issues with Solr. We have a 187m document count index which is about ~200Gb in size. Over a period of about a week after optimizations, etc... the open file but deleted count grows very large. Causing the system to not be able to optimize due to lack of disk space. Also new documents that are indexed are not showing up in search results.

Multiply the index size by 3 to get the max disk space.
- 1 for the index currently open for searching
- up to 1 for new segments written by the index writer (including merges)
- up to 1 when the index writer does major merges or optimizes (the
index writer can't delete the old segment files until it's sure that
the new index has been written successfully).

That said, what you are seeing could be normal, or could be a bug.

-Yonik