Tomcat holding deleted snapshots until it's restarted - SOLVED!!!

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Tomcat holding deleted snapshots until it's restarted - SOLVED!!!

Marc Sturlese
This post was updated on .
Hey Yonik,
I tested the last nightly build and still happens... but I have solved it! I tell you my solution, it seems to be working well but just want to be sure that it doesn't have any bad effects as for me this is one of the most complicated parts of the Solr source (the fact of dealing with multiple indexsearchers in a syncronized way).
I noticed that in the SolrCore.java, there's a part in the function getSearcher where there is a comment saying:

// we are all done with the old searcher we used
// for warming...

And after that the code is:
if (currSearcherHolderF!=null) currSearcherHolderF.decref();

The problem here is that this old SolrIndexSearcher is never closed and never removed from _searchers
What I have done:

if (currSearcherHolderF!=null){

        currSearcherHolderF.get().close(); //close SolrIndexSearcher proper
        currSearcherHolderF.decref();
        _searchers.remove(); //remove the
}

Doing that... if I do a "lsof | grep tomcat" will see that tomcat is not holding deleted files anymore (as indexsearcher was proper close) and the _searchers var will not accumulate infinite references...
It sorts the problem in the stats screen aswell... after 5 full-imports it just shows one IndexSearcher
What do you think?








Hey there,
I have noticed that once snapshots are deleted, tomcat keeps holding references to them.Due to this disk space will never set free until Tomcat is restarted, I have realized that doing a lsof | grep 'tomcat', the result look like:
...

java 22015 tomcat 614r REG 253,0 1149569723 1093456605 /var/local/solr/data/index/_1fb.fdt (deleted)
java 22015 tomcat 615r REG 253,0 12724500 1093456606 /var/local/solr/data/index/_1fb.fdx (deleted)
java 22015 tomcat 616r REG 253,0 175953343 1093456607 /var/local/solr/data/index/_1fb.tis (deleted)
java 22015 tomcat 617r REG 253,0 1989522 1102344114 /var/local/solr/sdata/index/_1fb.tii (deleted)
java 22015 tomcat 618r REG 253,0 178646437 1102344140 /var/local/solr/data/index/_1fb.frq (deleted)
java 22015 tomcat 619r REG 253,0 108460405 1102344154 /var/local/solr/sdata/index/_1fb.prx (deleted)

...

How can I make tomcat free the snapshots. Or even better... why is it happening?
Thanks in advance
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

Shalin Shekhar Mangar
On Wed, Mar 11, 2009 at 2:47 PM, Marc Sturlese <[hidden email]>wrote:

>
> How can I make tomcat free the snapshots. Or even better... why is it
> happening?
>

Is this on the rsync replication or the new java replication in trunk?

--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

Marc Sturlese
I am using the scripts of Collection Distribution.
The problem is just happening in the master, not in the slaves.
Do you have any clue? I am fighting against this since a few days ago...
Thanks in advance

Shalin Shekhar Mangar wrote
On Wed, Mar 11, 2009 at 2:47 PM, Marc Sturlese <marc.sturlese@gmail.com>wrote:

>
> How can I make tomcat free the snapshots. Or even better... why is it
> happening?
>

Is this on the rsync replication or the new java replication in trunk?

--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

Shalin Shekhar Mangar
On Wed, Mar 11, 2009 at 3:07 PM, Marc Sturlese <[hidden email]>wrote:

>
> I am using the scripts of Collection Distribution.
> The problem is just happening in the master, not in the slaves.
> Do you have any clue? I am fighting against this since a few days ago...
> Thanks in advance
>
> No Marc, I'm clueless too. I haven't seen this behavior before. Which OS
and filesystem are you using? Perhaps someone else may have more insight.

--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

Marc Sturlese
Hey Shalin,
I am using XFS file system with Debian Linux version 2.6.26-1-amd64.
Tomcat 5.5 server and java 1.6

As I optimize the index always before doing an snapshot (so the hard links will have the size of the whole index every time they are shot) I will try to modify snapshooter to create the snapshots doing a normal copy instead of using hard links.

One question, are you using jetty or tomcat? I have thought that maybe this just happens with tomcat...
If something comes to your mind please let me know. I have to be restarting my production tomcat every time the disk is consumed and it's being a bit messy... hope the modification in snapshooter works...

Thanks in advance


Shalin Shekhar Mangar wrote
On Wed, Mar 11, 2009 at 3:07 PM, Marc Sturlese <marc.sturlese@gmail.com>wrote:

>
> I am using the scripts of Collection Distribution.
> The problem is just happening in the master, not in the slaves.
> Do you have any clue? I am fighting against this since a few days ago...
> Thanks in advance
>
> No Marc, I'm clueless too. I haven't seen this behavior before. Which OS
and filesystem are you using? Perhaps someone else may have more insight.

--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

Yonik Seeley-2-2
In reply to this post by Marc Sturlese
On Wed, Mar 11, 2009 at 5:17 AM, Marc Sturlese <[hidden email]> wrote:
> I have noticed that once snapshots are deleted, tomcat keeps holding
> references to them.Due to this disk space will never set free until Tomcat
> is restarted, I have realized that doing a lsof | grep 'tomcat', the result
> look like:

That suggests that the index files aren't being closed for some reason.
After the new index searcher is registered, do you see the close of
the old index reader in the logs?
Do you have any Solr modifications or plugins?

-Yonik
http://www.lucidimagination.com


> java 22015 tomcat 614r REG 253,0 1149569723 1093456605
> /var/local/solr/data/index/_1fb.fdt (deleted)
> java 22015 tomcat 615r REG 253,0 12724500 1093456606
> /var/local/solr/data/index/_1fb.fdx (deleted)
> java 22015 tomcat 616r REG 253,0 175953343 1093456607
> /var/local/solr/data/index/_1fb.tis (deleted)
> java 22015 tomcat 617r REG 253,0 1989522 1102344114
> /var/local/solr/sdata/index/_1fb.tii (deleted)
> java 22015 tomcat 618r REG 253,0 178646437 1102344140
> /var/local/solr/data/index/_1fb.frq (deleted)
> java 22015 tomcat 619r REG 253,0 108460405 1102344154
> /var/local/solr/sdata/index/_1fb.prx (deleted)
>
> ...
>
> How can I make tomcat free the snapshots. Or even better... why is it
> happening?
> Thanks in advance
> --
> View this message in context: http://www.nabble.com/Tomcat-holding-deleted-snapshots-until-it%27s-restarted-tp22451252p22451252.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

Marc Sturlese
Hey Yonik,
I have realized that the problem is happening not because of the snapshots. I mean, I do an index and optimize it. Modifiy the database and to delta-import and optimize again... After that I do an lsof of the tomcat user and I see the old index still holded by tomcat...

...
ava      12814      tomcat   88r      REG        8,2 149279785  2902977 /home/smack/solr/data/index/_56n.frq (deleted)
java      12814      tomcat   89r      REG        8,2  91797827  2902978 /home/smack/solr/data/index/_56n.prx (deleted)
java      12814      tomcat   90r      REG        8,2   4196563  2902979 /home/smack/solr/data/index/_56n.nrm (deleted)
...

I am using not a plugin, just implemented a transformer to modify the data but this should'nt afect.
At least I see that actually the problem seems to be that an indexSearcher or indexWriter is not beig closed...
I have checked the log and it is closing an indexWriter and registering a new searcher but can't see that the older one is closed:

Mar 11 17:02:37 127.0.0.1 solr: 3926476 [Thread-47] DEBUG org.apache.solr.update.SolrIndexWriter - Closing Writer DirectUpdateHandler2
Mar 11 17:02:38 127.0.0.1 solr: 3927659 [Thread-47] INFO  org.apache.solr.core.SolrCore - TrovitDeletionPolicy.onCommit: commits:
Mar 11 17:02:38 127.0.0.1 solr: 3927660 [Thread-47] INFO  org.apache.solr.core.SolrCore - last commit = 1234970421620
Mar 11 17:02:38 127.0.0.1 solr: 3927924 [Thread-47] INFO  org.apache.solr.search.SolrIndexSearcher - Opening Searcher@a9f827 main
Mar 11 17:02:38 127.0.0.1 solr: 3927924 [Thread-47] INFO  org.apache.solr.update.UpdateHandler - end_commit_flush
Mar 11 17:02:38 127.0.0.1 solr: 3927925 [pool-11-thread-1] INFO  org.apache.solr.search.SolrIndexSearcher - autowarming Searcher@a9f827 main from Searcher@1e0bf98 main         fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=3,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0,item_fa_region={field=fa_region,memSize=5599982,tindexSize=56,time=48,phase1=44,nTerms=50,bigTerms=3,termInstances=488689,uses=3},item_fa_city_area={field=fa_city_area,memSize=5599654,tindexSize=50,time=469,phase1=465,nTerms=18,bigTerms=0,termInstances=352,uses=3},item_fa_city={field=fa_city,memSize=5633866,tindexSize=1494,time=94,phase1=91,nTerms=5767,bigTerms=0,termInstances=685467,uses=3}}
Mar 11 17:02:38 127.0.0.1 solr: 3927925 [pool-11-thread-1] INFO  org.apache.solr.search.SolrIndexSearcher - autowarming result for Searcher@a9f827 main         fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 11 17:02:38 127.0.0.1 solr: 3927925 [pool-11-thread-1] INFO  org.apache.solr.search.SolrIndexSearcher - autowarming Searcher@a9f827 main from Searcher@1e0bf98 main         filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=20,warmupTime=279,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 11 17:02:38 127.0.0.1 solr: 3928161 [pool-11-thread-1] INFO  org.apache.solr.search.SolrIndexSearcher - autowarming result for Searcher@a9f827 main         filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=20,warmupTime=236,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 11 17:02:38 127.0.0.1 solr: 3928161 [pool-11-thread-1] INFO  org.apache.solr.search.SolrIndexSearcher - autowarming Searcher@a9f827 main from Searcher@1e0bf98 main         queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=10,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 11 17:02:38 127.0.0.1 solr: 3928171 [pool-11-thread-1] INFO  org.apache.solr.search.SolrIndexSearcher - autowarming result for Searcher@a9f827 main         queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=10,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 11 17:02:38 127.0.0.1 solr: 3928171 [pool-11-thread-1] INFO  org.apache.solr.search.SolrIndexSearcher - autowarming Searcher@a9f827 main from Searcher@1e0bf98 main         documentCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 11 17:02:38 127.0.0.1 solr: 3928171 [pool-11-thread-1] INFO  org.apache.solr.search.SolrIndexSearcher - autowarming result for Searcher@a9f827 main         documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 11 17:02:38 127.0.0.1 solr: 3928171 [pool-11-thread-1] INFO  org.apache.solr.core.SolrCore - QuerySenderListener sending requests to Searcher@a9f827 main
Mar 11 17:02:38 127.0.0.1 solr: 3928260 [pool-11-thread-1] INFO  org.apache.solr.core.SolrCore - UnInverted multi-valued field {field=fa_city,memSize=5633866,tindexSize=1494,time=85,phase1=81,nTerms=5767,bigTerms=0,termInstances=685467,uses=0}
Mar 11 17:02:39 127.0.0.1 solr: 3928330 [pool-11-thread-1] INFO  org.apache.solr.core.SolrCore - UnInverted multi-valued field {field=fa_city_area,memSize=5599654,tindexSize=50,time=70,phase1=66,nTerms=18,bigTerms=0,termInstances=352,uses=0}
Mar 11 17:02:39 127.0.0.1 solr: 3928377 [pool-11-thread-1] INFO  org.apache.solr.core.SolrCore - UnInverted multi-valued field {field=fa_region,memSize=5599982,tindexSize=56,time=47,phase1=44,nTerms=50,bigTerms=3,termInstances=488689,uses=0}
Mar 11 17:02:39 127.0.0.1 solr: 3928378 [pool-11-thread-1] INFO  org.apache.solr.core.SolrCore - [search_homes_es] webapp=null path=null params={start=0&q=solr&rows=10} hits=0 status=0 QTime=207
Mar 11 17:02:39 127.0.0.1 solr: 3928426 [pool-11-thread-1] INFO  org.apache.solr.core.SolrCore - [search_homes_es] webapp=null path=null params={start=0&q=rocks&rows=10} hits=5 status=0 QTime=48
Mar 11 17:02:39 127.0.0.1 solr: 3928437 [pool-11-thread-1] INFO  org.apache.solr.core.SolrCore - [search_homes_es] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml} hits=0 status=0 QTime=10
Mar 11 17:02:39 127.0.0.1 solr: 3928437 [pool-11-thread-1] INFO  org.apache.solr.core.SolrCore - QuerySenderListener done.
Mar 11 17:02:39 127.0.0.1 solr: 3928439 [pool-11-thread-1] INFO  org.apache.solr.core.SolrCore - [search_homes_es] Registered new searcher Searcher@a9f827 main

Definitely looks like the problem is around here... Do you know where in the source code should I try to close that indexSeracher?

Thanks for answering as I am really stuck with this problem...



Yonik Seeley-2 wrote
On Wed, Mar 11, 2009 at 5:17 AM, Marc Sturlese <marc.sturlese@gmail.com> wrote:
> I have noticed that once snapshots are deleted, tomcat keeps holding
> references to them.Due to this disk space will never set free until Tomcat
> is restarted, I have realized that doing a lsof | grep 'tomcat', the result
> look like:

That suggests that the index files aren't being closed for some reason.
After the new index searcher is registered, do you see the close of
the old index reader in the logs?
Do you have any Solr modifications or plugins?

-Yonik
http://www.lucidimagination.com


> java 22015 tomcat 614r REG 253,0 1149569723 1093456605
> /var/local/solr/data/index/_1fb.fdt (deleted)
> java 22015 tomcat 615r REG 253,0 12724500 1093456606
> /var/local/solr/data/index/_1fb.fdx (deleted)
> java 22015 tomcat 616r REG 253,0 175953343 1093456607
> /var/local/solr/data/index/_1fb.tis (deleted)
> java 22015 tomcat 617r REG 253,0 1989522 1102344114
> /var/local/solr/sdata/index/_1fb.tii (deleted)
> java 22015 tomcat 618r REG 253,0 178646437 1102344140
> /var/local/solr/data/index/_1fb.frq (deleted)
> java 22015 tomcat 619r REG 253,0 108460405 1102344154
> /var/local/solr/sdata/index/_1fb.prx (deleted)
>
> ...
>
> How can I make tomcat free the snapshots. Or even better... why is it
> happening?
> Thanks in advance
> --
> View this message in context: http://www.nabble.com/Tomcat-holding-deleted-snapshots-until-it%27s-restarted-tp22451252p22451252.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

Yonik Seeley-2-2
On Wed, Mar 11, 2009 at 12:23 PM, Marc Sturlese <[hidden email]> wrote:
> I have checked the log and it is closing an indexWriter and registering a
> new searcher but can't see that the older one is closed:

On a quiet system, you should see the original searcher closed right
after the new searcher is registered.

Example:
Mar 11, 2009 2:22:25 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [] Registered new searcher Searcher@1f1cbf6 main
Mar 11, 2009 2:22:25 PM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@acdd02 main

> I am using not a plugin, just implemented a transformer to modify the data but this should'nt afect.

Transformer?  from Data Import Handler?
What version of Solr are you using?

-Yonik
http://www.lucidimagination.com
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

Marc Sturlese
Yes, I coded a transformer to deal with the data from a mysql table before index it with dataimporthandler.
I am actually using a nightly build from middle january with all the concurrency bugs of dataimporthandler fixed.
After lots of tracing I think the problem could be in the commit void of the DirectUpdateHandler2.class but still not able to hit where.
If the problem is not there the other thing that comes to my mind is lucene2.9-dev... maybe there's a problem closing indexWriter?... opiously it's just a thought.


Yonik Seeley-2 wrote
On Wed, Mar 11, 2009 at 12:23 PM, Marc Sturlese <marc.sturlese@gmail.com> wrote:
> I have checked the log and it is closing an indexWriter and registering a
> new searcher but can't see that the older one is closed:

On a quiet system, you should see the original searcher closed right
after the new searcher is registered.

Example:
Mar 11, 2009 2:22:25 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [] Registered new searcher Searcher@1f1cbf6 main
Mar 11, 2009 2:22:25 PM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@acdd02 main

> I am using not a plugin, just implemented a transformer to modify the data but this should'nt afect.

Transformer?  from Data Import Handler?
What version of Solr are you using?

-Yonik
http://www.lucidimagination.com
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

hossman

: If the problem is not there the other thing that comes to my mind is
: lucene2.9-dev... maybe there's a problem closing indexWriter?... opiously
: it's just a thought.

you never answered yoniks question about wether you see any "Closing
Searcher" messagges in your log, also it's useful to know what you see in
the CORE section when you look at stats.jsp ... typically the "main"
searcher is listed there twice, but during warming you'll see the old
searcher as well ... if older searchers aren't getting closed for some
reason, they should be listed there.

i'd start by confirming/ruling out hte old searchers before speculating
about the indexwriter or other problems.

: > On a quiet system, you should see the original searcher closed right
: > after the new searcher is registered.
: >
: > Example:
: > Mar 11, 2009 2:22:25 PM org.apache.solr.core.SolrCore registerSearcher
: > INFO: [] Registered new searcher Searcher@1f1cbf6 main
: > Mar 11, 2009 2:22:25 PM org.apache.solr.search.SolrIndexSearcher close
: > INFO: Closing Searcher@acdd02 main



-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

Marc Sturlese
The old IndexSearcher is beeing closed correctly:

2009-03-12 13:05:06,200 [pool-7-thread-1] INFO  org.apache.solr.core.SolrCore - [core_01] Registered new searcher Searcher@c6692 main
2009-03-12 13:05:06,200 [pool-7-thread-1] INFO  org.apache.solr.search.SolrIndexSearcher - Closing Searcher@1c5cd7

main
hossman wrote
: If the problem is not there the other thing that comes to my mind is
: lucene2.9-dev... maybe there's a problem closing indexWriter?... opiously
: it's just a thought.

you never answered yoniks question about wether you see any "Closing
Searcher" messagges in your log, also it's useful to know what you see in
the CORE section when you look at stats.jsp ... typically the "main"
searcher is listed there twice, but during warming you'll see the old
searcher as well ... if older searchers aren't getting closed for some
reason, they should be listed there.

i'd start by confirming/ruling out hte old searchers before speculating
about the indexwriter or other problems.

: > On a quiet system, you should see the original searcher closed right
: > after the new searcher is registered.
: >
: > Example:
: > Mar 11, 2009 2:22:25 PM org.apache.solr.core.SolrCore registerSearcher
: > INFO: [] Registered new searcher Searcher@1f1cbf6 main
: > Mar 11, 2009 2:22:25 PM org.apache.solr.search.SolrIndexSearcher close
: > INFO: Closing Searcher@acdd02 main



-Hoss
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

Marc Sturlese
I have noticed that the first time I execute full import (having an old index in the index folder) once it is done, the old indexsearcher will be closed:

2009-03-12 13:05:06,200 [pool-7-thread-1] INFO  org.apache.solr.core.SolrCore - [core_01] Registered new searcher Searcher@c6692 main
2009-03-12 13:05:06,200 [pool-7-thread-1] INFO  org.apache.solr.search.SolrIndexSearcher - Closing Searcher@1c5cd7

The problem is that if I do another full-import... the old searcher will not be closed, there will just appear the line:
2009-03-12 13:05:06,200 [pool-7-thread-1] INFO  org.apache.solr.core.SolrCore - [core_01] Registered new searcher Searcher@c6692 main

If I keep doing full-imports the ols searchers will never be closed. seems that they are just closed in the first full import...
Does it mean something to anyone?

Marc Sturlese wrote
The old IndexSearcher is beeing closed correctly:

2009-03-12 13:05:06,200 [pool-7-thread-1] INFO  org.apache.solr.core.SolrCore - [core_01] Registered new searcher Searcher@c6692 main
2009-03-12 13:05:06,200 [pool-7-thread-1] INFO  org.apache.solr.search.SolrIndexSearcher - Closing Searcher@1c5cd7

main
hossman wrote
: If the problem is not there the other thing that comes to my mind is
: lucene2.9-dev... maybe there's a problem closing indexWriter?... opiously
: it's just a thought.

you never answered yoniks question about wether you see any "Closing
Searcher" messagges in your log, also it's useful to know what you see in
the CORE section when you look at stats.jsp ... typically the "main"
searcher is listed there twice, but during warming you'll see the old
searcher as well ... if older searchers aren't getting closed for some
reason, they should be listed there.

i'd start by confirming/ruling out hte old searchers before speculating
about the indexwriter or other problems.

: > On a quiet system, you should see the original searcher closed right
: > after the new searcher is registered.
: >
: > Example:
: > Mar 11, 2009 2:22:25 PM org.apache.solr.core.SolrCore registerSearcher
: > INFO: [] Registered new searcher Searcher@1f1cbf6 main
: > Mar 11, 2009 2:22:25 PM org.apache.solr.search.SolrIndexSearcher close
: > INFO: Closing Searcher@acdd02 main



-Hoss
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

hossman
: I have noticed that the first time I execute full import (having an old index
: in the index folder) once it is done, the old indexsearcher will be closed:
        ...
: The problem is that if I do another full-import... the old searcher will not
: be closed, there will just appear the line:
        ...
: If I keep doing full-imports the ols searchers will never be closed. seems
: that they are just closed in the first full import...
: Does it mean something to anyone?

Hmmm... sounds like maybe DIH is triggering something weird.  

Just to clarify:
a) what does the stats page show (in terms of the number of
Searchers listed in the CORE section) after a couple of full imports?  
b) can you reproduce this doing full builds even with replication
disabled?
c) can you reproduce this using the example DIH configs?




-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

Marc Sturlese
: Just to clarify:
: a) what does the stats page show (in terms of the number of
: Searchers listed in the CORE section) after a couple of full imports?  

After 4 full-imports it will show 3 indexsearchers. I have also printed the var "_searchers" from SolrCore.java and it shows me 3 indexsearchers.

: b) can you reproduce this doing full builds even with replication
: disabled?

I have replication disabled. I use solr collection distribution but for all this tests I am not even using that. I just use one machine and index just in there
 
: c) can you reproduce this using the example DIH configs?
My configs look really similar than defaults. I get data from mysql database in data-config.xml. Solrconfig.xml has the caches and warmings same as defaults.
I have disabled solrdeletionpolicystuff (and replication aswell).

I have checked the oficial 1.3 release and I hace seen DirectUpdateHandler2.java is quite different that the one in the nightlys. In the commit void... 1.3 is calling a closeSearcher function :

public void commit(CommitUpdateCommand cmd) throws IOException {

    if (cmd.optimize) {
      optimizeCommands.incrementAndGet();
    } else {
      commitCommands.incrementAndGet();
    }

    Future[] waitSearcher = null;
    if (cmd.waitSearcher) {
      waitSearcher = new Future[1];
    }

    boolean error=true;
    iwCommit.lock();
    try {
      log.info("start "+cmd);

      if (cmd.optimize) {
        closeSearcher();
        openWriter();
        writer.optimize(cmd.maxOptimizeSegments);
      }

      closeSearcher();
      closeWriter();

These closeSearcher function doesn't exist in the nightly (I supose all the proces works in a different way now).
It seems that once DataImportHandler does the first import touches something that makes IndexSearchers to not set free never again.




hossman wrote
: I have noticed that the first time I execute full import (having an old index
: in the index folder) once it is done, the old indexsearcher will be closed:
        ...
: The problem is that if I do another full-import... the old searcher will not
: be closed, there will just appear the line:
        ...
: If I keep doing full-imports the ols searchers will never be closed. seems
: that they are just closed in the first full import...
: Does it mean something to anyone?

Hmmm... sounds like maybe DIH is triggering something weird.  

Just to clarify:
a) what does the stats page show (in terms of the number of
Searchers listed in the CORE section) after a couple of full imports?  
b) can you reproduce this doing full builds even with replication
disabled?
c) can you reproduce this using the example DIH configs?




-Hoss
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted

Yonik Seeley-2-2
On Thu, Mar 12, 2009 at 1:34 PM, Marc Sturlese <[hidden email]> wrote:
> : Just to clarify:
> : a) what does the stats page show (in terms of the number of
> : Searchers listed in the CORE section) after a couple of full imports?
>
> After 4 full-imports it will show 3 indexsearchers. I have also printed the
> var "_searchers" from SolrCore.java and it shows me 3 indexsearchers.

Definitely seems like a bug somewhere...
Could you try a recent nightly build to see if it's fixed or not?

-Yonik
http://www.lucidimagination.com
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted - SOLVED!!!

Marc Sturlese
In reply to this post by Marc Sturlese
Hey Yonik,
I tested the last nightly build and still happens... but I have solved it! I tell you my solution, it seems to be working well but just want to be sure that it doesn't have any bad effects as for me this is one of the most complicated parts of the Solr source (the fact of dealing with multiple indexsearchers in a syncronized way).
I noticed that in the SolrCore.java, there's a part in the function getSearcher where there is a comment saying:

// we are all done with the old searcher we used
// for warming...

And after that the code is:
if (currSearcherHolderF!=null) currSearcherHolderF.decref();

The problem here is that this old SolrIndexSearcher is never closed and never removed from _searchers
What I have done:

if (currSearcherHolderF!=null){

        currSearcherHolderF.get().close(); //close SolrIndexSearcher proper
        currSearcherHolderF.decref();
        _searchers.remove(); //remove the
}

Doing that... if I do a "lsof | grep tomcat" will see that tomcat is not holding deleted files anymore (as indexsearcher was proper close) and the _searchers var will not accumulate infinite references...
It sorts the problem in the stats screen aswell... after 5 full-imports it just shows one IndexSearcher
What do you think?
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted - SOLVED!!!

Yonik Seeley-2-2
decref() decrements the reference count and closes the searcher when
it reaches 0 (no more users).
Forcing it to close at the point you did is unsafe since other threads
may still be using that searcher.
The real issue lies somewhere else - either a stuck thread, or some
code that is not decrementing the reference when it's done.  It's most
likely the latter.

We need to get to the root cause.  Can you open a JIRA bug for this?

-Yonik
http://www.lucidimagination.com

On Fri, Mar 13, 2009 at 12:39 PM, Marc Sturlese <[hidden email]> wrote:

>
> Hey Yonik,
> I tested the last nightly build and still happens... but I have solved it! I
> tell you my solution, it seems to be working well but just want to be sure
> that it doesn't have any bad effects as for me this is one of the most
> complicated parts of the Solr source (the fact of dealing with multiple
> indexsearchers in a syncronized way).
> I noticed that in the SolrCore.java, there's a part in the function
> getSearcher where there is a comment saying:
>
> // we are all done with the old searcher we used
> // for warming...
>
> And after that the code is:
> if (currSearcherHolderF!=null) currSearcherHolderF.decref();
>
> The problem here is that this old SolrIndexSearcher is never closed and
> never removed from _searchers
> What I have done:
>
> if (currSearcherHolderF!=null){
>
>        currSearcherHolderF.get().close(); //close SolrIndexSearcher proper
>        currSearcherHolderF.decref();
>        _searchers.remove(); //remove the
> }
>
> Doing that... if I do a "lsof | grep tomcat" will see that tomcat is not
> holding deleted files anymore (as indexsearcher was proper close) and the
> _searchers var will not accumulate infinite references...
> It sorts the problem in the stats screen aswell... after 5 full-imports it
> just shows one IndexSearcher
> What do you think?
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted - SOLVED!!!

Marc Sturlese
Ok, I will open a bug issue now.

> Forcing it to close at the point you did is unsafe since other threads
> may still be using that searcher.

Can you give me an example where other threads would be using that searcher? (As I said I find this part of the source dufficult to understand and imagined was missing something...)



Yonik Seeley-2 wrote
decref() decrements the reference count and closes the searcher when
it reaches 0 (no more users).
Forcing it to close at the point you did is unsafe since other threads
may still be using that searcher.
The real issue lies somewhere else - either a stuck thread, or some
code that is not decrementing the reference when it's done.  It's most
likely the latter.

We need to get to the root cause.  Can you open a JIRA bug for this?

-Yonik
http://www.lucidimagination.com

On Fri, Mar 13, 2009 at 12:39 PM, Marc Sturlese <marc.sturlese@gmail.com> wrote:
>
> Hey Yonik,
> I tested the last nightly build and still happens... but I have solved it! I
> tell you my solution, it seems to be working well but just want to be sure
> that it doesn't have any bad effects as for me this is one of the most
> complicated parts of the Solr source (the fact of dealing with multiple
> indexsearchers in a syncronized way).
> I noticed that in the SolrCore.java, there's a part in the function
> getSearcher where there is a comment saying:
>
> // we are all done with the old searcher we used
> // for warming...
>
> And after that the code is:
> if (currSearcherHolderF!=null) currSearcherHolderF.decref();
>
> The problem here is that this old SolrIndexSearcher is never closed and
> never removed from _searchers
> What I have done:
>
> if (currSearcherHolderF!=null){
>
>        currSearcherHolderF.get().close(); //close SolrIndexSearcher proper
>        currSearcherHolderF.decref();
>        _searchers.remove(); //remove the
> }
>
> Doing that... if I do a "lsof | grep tomcat" will see that tomcat is not
> holding deleted files anymore (as indexsearcher was proper close) and the
> _searchers var will not accumulate infinite references...
> It sorts the problem in the stats screen aswell... after 5 full-imports it
> just shows one IndexSearcher
> What do you think?
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat holding deleted snapshots until it's restarted - SOLVED!!!

Yonik Seeley-2-2
On Fri, Mar 13, 2009 at 1:00 PM, Marc Sturlese <[hidden email]> wrote:
>
> Ok, I will open a bug issue now.
>
>> Forcing it to close at the point you did is unsafe since other threads
>> may still be using that searcher.
>
> Can you give me an example where other threads would be using that searcher?

Any searches that started before the new searcher was registered will
still be using the old searcher.
  Thread A starts executing a search request with Searcher1
  Thread B issues a "commit"
      - close the writer
      - open Searcher2
      - register Searcher2 (and decrement Searcher1 ref count)
  Thread B finishes
  Thread A finishes (decrement Searcher1 ref count)

-Yonik
http://www.lucidimagination.com