Fwd: Re: suggester/autocomplete locks file preventing replication

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: Re: suggester/autocomplete locks file preventing replication

tmenzel
cross posting this issue to the dev list in the hope to get a response here...


-------- Original Message --------
Subject: Re: suggester/autocomplete locks file preventing replication
Date: Thu, 21 Jun 2012 17:11:40 +0200
From: tom [hidden email]
Reply-To: [hidden email]
To: [hidden email]


pocking into the code i think the FileDictionary class is the culprit:
It takes an InputStream as a ctor argument but never releases the 
stream. what puzzles me is that the class seems to allow a one-time 
iteration and then the stream is useless, unless i'm missing smth. here.

is there a good reason for this or rather a bug?
should i move the topic to the dev list?


On 21.06.2012 14:49, tom wrote:
> BTW: a core unload doesnt release the lock either ;(
>
>
> On 21.06.2012 14:39, tom wrote:
>> hi,
>>
>> i'm using the suggester with a file like so:
>>
>>   <searchComponent class="solr.SpellCheckComponent" name="suggest">
>>     <lst name="spellchecker">
>>       <str name="name">suggest</str>
>>       <str 
>> name="classname">org.apache.solr.spelling.suggest.Suggester</str>
>>       <str 
>> name="lookupImpl">org.apache.solr.spelling.suggest.fst.FSTLookup</str>
>>       <!-- Alternatives to lookupImpl:
>>         org.apache.solr.spelling.suggest.fst.FSTLookup [finite state 
>> automaton]
>>         org.apache.solr.spelling.suggest.jaspell.JaspellLookup 
>> [default, jaspell-based]
>>         org.apache.solr.spelling.suggest.tst.TSTLookup [ternary trees]
>>       -->
>>       <!-- the indexed field to derive suggestions from -->
>>       <!-- TODO must change this to spell or smth alike later -->
>>       <str name="field">content</str>
>>       <float name="threshold">0.05</float>
>>       <str name="buildOnCommit">true</str>
>>       <str name="weightBuckets">100</str>
>>       <str name="sourceLocation">autocomplete.dictionary</str>
>>     </lst>
>>   </searchComponent>
>>
>> when trying to replicate i get the following error message on the 
>> slave side:
>>
>>  2012-06-21 14:34:50,781 ERROR 
>> [pool-3-thread-1                              ] 
>> handler.ReplicationHandler                    - SnapPull failed
>> org.apache.solr.common.SolrException: Unable to rename: <path>  
>> autocomplete.dictionary.20120620120611
>>     at 
>> org.apache.solr.handler.SnapPuller.copyTmpConfFiles2Conf(SnapPuller.java:642)
>>     at 
>> org.apache.solr.handler.SnapPuller.downloadConfFiles(SnapPuller.java:526)
>>     at 
>> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:299)
>>     at 
>> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268)
>>     at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
>>     at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>     at 
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>>     at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>     at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>     at java.lang.Thread.run(Thread.java:619)
>>
>> so i dug around it and found out that the solr's java process holds a 
>> lock on the autocomplete.dictionary file. any reason why this is so?
>>
>> thx,
>>
>> running:
>> solr 3.5
>> win7
>>
>
>
>






Reply | Threaded
Open this post in threaded view
|

Re: Re: suggester/autocomplete locks file preventing replication

Simon Willnauer


On Fri, Jun 22, 2012 at 10:37 AM, tom <[hidden email]> wrote:
cross posting this issue to the dev list in the hope to get a response here...

I think you are right. Closing the Stream / Reader is the responsibility of the caller not the FileDictionary IMO but solr doesn't close it so that might cause your problems. Are you running on windows by any chance?
I will create an issue and fix it.

simon


-------- Original Message --------
Subject: Re: suggester/autocomplete locks file preventing replication
Date: Thu, 21 Jun 2012 17:11:40 +0200
From: tom [hidden email]
Reply-To: [hidden email]
To: [hidden email]


pocking into the code i think the FileDictionary class is the culprit:
It takes an InputStream as a ctor argument but never releases the 
stream. what puzzles me is that the class seems to allow a one-time 
iteration and then the stream is useless, unless i'm missing smth. here.

is there a good reason for this or rather a bug?
should i move the topic to the dev list?


On <a href="tel:21.06.2012%2014" value="+12106201214" target="_blank">21.06.2012 14:49, tom wrote:
> BTW: a core unload doesnt release the lock either ;(
>
>
> On <a href="tel:21.06.2012%2014" value="+12106201214" target="_blank">21.06.2012 14:39, tom wrote:
>> hi,
>>
>> i'm using the suggester with a file like so:
>>
>>   <searchComponent class="solr.SpellCheckComponent" name="suggest">
>>     <lst name="spellchecker">
>>       <str name="name">suggest</str>
>>       <str 
>> name="classname">org.apache.solr.spelling.suggest.Suggester</str>
>>       <str 
>> name="lookupImpl">org.apache.solr.spelling.suggest.fst.FSTLookup</str>
>>       <!-- Alternatives to lookupImpl:
>>         org.apache.solr.spelling.suggest.fst.FSTLookup [finite state 
>> automaton]
>>         org.apache.solr.spelling.suggest.jaspell.JaspellLookup 
>> [default, jaspell-based]
>>         org.apache.solr.spelling.suggest.tst.TSTLookup [ternary trees]
>>       -->
>>       <!-- the indexed field to derive suggestions from -->
>>       <!-- TODO must change this to spell or smth alike later -->
>>       <str name="field">content</str>
>>       <float name="threshold">0.05</float>
>>       <str name="buildOnCommit">true</str>
>>       <str name="weightBuckets">100</str>
>>       <str name="sourceLocation">autocomplete.dictionary</str>
>>     </lst>
>>   </searchComponent>
>>
>> when trying to replicate i get the following error message on the 
>> slave side:
>>
>>  2012-06-21 14:34:50,781 ERROR 
>> [pool-3-thread-1                              ] 
>> handler.ReplicationHandler                    - SnapPull failed
>> org.apache.solr.common.SolrException: Unable to rename: <path>  
>> autocomplete.dictionary.20120620120611
>>     at 
>> org.apache.solr.handler.SnapPuller.copyTmpConfFiles2Conf(SnapPuller.java:642)
>>     at 
>> org.apache.solr.handler.SnapPuller.downloadConfFiles(SnapPuller.java:526)
>>     at 
>> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:299)
>>     at 
>> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268)
>>     at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
>>     at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>     at 
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>>     at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>     at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>     at java.lang.Thread.run(Thread.java:619)
>>
>> so i dug around it and found out that the solr's java process holds a 
>> lock on the autocomplete.dictionary file. any reason why this is so?
>>
>> thx,
>>
>> running:
>> solr 3.5
>> win7
>>
>
>
>







Reply | Threaded
Open this post in threaded view
|

Re: Re: suggester/autocomplete locks file preventing replication

Simon Willnauer


On Fri, Jun 22, 2012 at 11:47 AM, Simon Willnauer <[hidden email]> wrote:


On Fri, Jun 22, 2012 at 10:37 AM, tom <[hidden email]> wrote:
cross posting this issue to the dev list in the hope to get a response here...

I think you are right. Closing the Stream / Reader is the responsibility of the caller not the FileDictionary IMO but solr doesn't close it so that might cause your problems. Are you running on windows by any chance?
I will create an issue and fix it.

hmm I just looked at it and I see a IOUtils.close call in FileDictionary


are you using solr 3.6?  

simon


-------- Original Message --------
Subject: Re: suggester/autocomplete locks file preventing replication
Date: Thu, 21 Jun 2012 17:11:40 +0200
From: tom [hidden email]
Reply-To: [hidden email]
To: [hidden email]


pocking into the code i think the FileDictionary class is the culprit:
It takes an InputStream as a ctor argument but never releases the 
stream. what puzzles me is that the class seems to allow a one-time 
iteration and then the stream is useless, unless i'm missing smth. here.

is there a good reason for this or rather a bug?
should i move the topic to the dev list?


On <a href="tel:21.06.2012%2014" value="+12106201214" target="_blank">21.06.2012 14:49, tom wrote:
> BTW: a core unload doesnt release the lock either ;(
>
>
> On <a href="tel:21.06.2012%2014" value="+12106201214" target="_blank">21.06.2012 14:39, tom wrote:
>> hi,
>>
>> i'm using the suggester with a file like so:
>>
>>   <searchComponent class="solr.SpellCheckComponent" name="suggest">
>>     <lst name="spellchecker">
>>       <str name="name">suggest</str>
>>       <str 
>> name="classname">org.apache.solr.spelling.suggest.Suggester</str>
>>       <str 
>> name="lookupImpl">org.apache.solr.spelling.suggest.fst.FSTLookup</str>
>>       <!-- Alternatives to lookupImpl:
>>         org.apache.solr.spelling.suggest.fst.FSTLookup [finite state 
>> automaton]
>>         org.apache.solr.spelling.suggest.jaspell.JaspellLookup 
>> [default, jaspell-based]
>>         org.apache.solr.spelling.suggest.tst.TSTLookup [ternary trees]
>>       -->
>>       <!-- the indexed field to derive suggestions from -->
>>       <!-- TODO must change this to spell or smth alike later -->
>>       <str name="field">content</str>
>>       <float name="threshold">0.05</float>
>>       <str name="buildOnCommit">true</str>
>>       <str name="weightBuckets">100</str>
>>       <str name="sourceLocation">autocomplete.dictionary</str>
>>     </lst>
>>   </searchComponent>
>>
>> when trying to replicate i get the following error message on the 
>> slave side:
>>
>>  2012-06-21 14:34:50,781 ERROR 
>> [pool-3-thread-1                              ] 
>> handler.ReplicationHandler                    - SnapPull failed
>> org.apache.solr.common.SolrException: Unable to rename: <path>  
>> autocomplete.dictionary.20120620120611
>>     at 
>> org.apache.solr.handler.SnapPuller.copyTmpConfFiles2Conf(SnapPuller.java:642)
>>     at 
>> org.apache.solr.handler.SnapPuller.downloadConfFiles(SnapPuller.java:526)
>>     at 
>> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:299)
>>     at 
>> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268)
>>     at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
>>     at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>     at 
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>>     at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>     at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>     at java.lang.Thread.run(Thread.java:619)
>>
>> so i dug around it and found out that the solr's java process holds a 
>> lock on the autocomplete.dictionary file. any reason why this is so?
>>
>> thx,
>>
>> running:
>> solr 3.5
>> win7
>>
>
>
>








Reply | Threaded
Open this post in threaded view
|

Re: Re: suggester/autocomplete locks file preventing replication

Simon Willnauer
here is the issue https://issues.apache.org/jira/browse/SOLR-3570

On Fri, Jun 22, 2012 at 11:55 AM, Simon Willnauer <[hidden email]> wrote:


On Fri, Jun 22, 2012 at 11:47 AM, Simon Willnauer <[hidden email]> wrote:


On Fri, Jun 22, 2012 at 10:37 AM, tom <[hidden email]> wrote:
cross posting this issue to the dev list in the hope to get a response here...

I think you are right. Closing the Stream / Reader is the responsibility of the caller not the FileDictionary IMO but solr doesn't close it so that might cause your problems. Are you running on windows by any chance?
I will create an issue and fix it.

hmm I just looked at it and I see a IOUtils.close call in FileDictionary


are you using solr 3.6?  

simon


-------- Original Message --------
Subject: Re: suggester/autocomplete locks file preventing replication
Date: Thu, 21 Jun 2012 17:11:40 +0200
From: tom [hidden email]
Reply-To: [hidden email]
To: [hidden email]


pocking into the code i think the FileDictionary class is the culprit:
It takes an InputStream as a ctor argument but never releases the 
stream. what puzzles me is that the class seems to allow a one-time 
iteration and then the stream is useless, unless i'm missing smth. here.

is there a good reason for this or rather a bug?
should i move the topic to the dev list?


On <a href="tel:21.06.2012%2014" value="+12106201214" target="_blank">21.06.2012 14:49, tom wrote:
> BTW: a core unload doesnt release the lock either ;(
>
>
> On <a href="tel:21.06.2012%2014" value="+12106201214" target="_blank">21.06.2012 14:39, tom wrote:
>> hi,
>>
>> i'm using the suggester with a file like so:
>>
>>   <searchComponent class="solr.SpellCheckComponent" name="suggest">
>>     <lst name="spellchecker">
>>       <str name="name">suggest</str>
>>       <str 
>> name="classname">org.apache.solr.spelling.suggest.Suggester</str>
>>       <str 
>> name="lookupImpl">org.apache.solr.spelling.suggest.fst.FSTLookup</str>
>>       <!-- Alternatives to lookupImpl:
>>         org.apache.solr.spelling.suggest.fst.FSTLookup [finite state 
>> automaton]
>>         org.apache.solr.spelling.suggest.jaspell.JaspellLookup 
>> [default, jaspell-based]
>>         org.apache.solr.spelling.suggest.tst.TSTLookup [ternary trees]
>>       -->
>>       <!-- the indexed field to derive suggestions from -->
>>       <!-- TODO must change this to spell or smth alike later -->
>>       <str name="field">content</str>
>>       <float name="threshold">0.05</float>
>>       <str name="buildOnCommit">true</str>
>>       <str name="weightBuckets">100</str>
>>       <str name="sourceLocation">autocomplete.dictionary</str>
>>     </lst>
>>   </searchComponent>
>>
>> when trying to replicate i get the following error message on the 
>> slave side:
>>
>>  2012-06-21 14:34:50,781 ERROR 
>> [pool-3-thread-1                              ] 
>> handler.ReplicationHandler                    - SnapPull failed
>> org.apache.solr.common.SolrException: Unable to rename: <path>  
>> autocomplete.dictionary.20120620120611
>>     at 
>> org.apache.solr.handler.SnapPuller.copyTmpConfFiles2Conf(SnapPuller.java:642)
>>     at 
>> org.apache.solr.handler.SnapPuller.downloadConfFiles(SnapPuller.java:526)
>>     at 
>> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:299)
>>     at 
>> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268)
>>     at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
>>     at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>     at 
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>>     at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>     at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>     at java.lang.Thread.run(Thread.java:619)
>>
>> so i dug around it and found out that the solr's java process holds a 
>> lock on the autocomplete.dictionary file. any reason why this is so?
>>
>> thx,
>>
>> running:
>> solr 3.5
>> win7
>>
>
>
>









Reply | Threaded
Open this post in threaded view
|

Re: suggester/autocomplete locks file preventing replication

tmenzel
ah thx and to answer ur question:

no we are still on 3.5 and i had tested it on win7 (see the very end of this email thread)
and after having briefly looked @ the 3.6 code the class has changed quite a bit since 3.5...

On 22.06.2012 12:16, Simon Willnauer wrote:
here is the issue https://issues.apache.org/jira/browse/SOLR-3570

On Fri, Jun 22, 2012 at 11:55 AM, Simon Willnauer <[hidden email]> wrote:


On Fri, Jun 22, 2012 at 11:47 AM, Simon Willnauer <[hidden email]> wrote:


On Fri, Jun 22, 2012 at 10:37 AM, tom <[hidden email]> wrote:
cross posting this issue to the dev list in the hope to get a response here...

I think you are right. Closing the Stream / Reader is the responsibility of the caller not the FileDictionary IMO but solr doesn't close it so that might cause your problems. Are you running on windows by any chance?
I will create an issue and fix it.

hmm I just looked at it and I see a IOUtils.close call in FileDictionary


are you using solr 3.6?  

simon


-------- Original Message --------
Subject: Re: suggester/autocomplete locks file preventing replication
Date: Thu, 21 Jun 2012 17:11:40 +0200
From: tom [hidden email]
Reply-To: [hidden email]
To: [hidden email]


pocking into the code i think the FileDictionary class is the culprit:
It takes an InputStream as a ctor argument but never releases the 
stream. what puzzles me is that the class seems to allow a one-time 
iteration and then the stream is useless, unless i'm missing smth. here.

is there a good reason for this or rather a bug?
should i move the topic to the dev list?


On <a moz-do-not-send="true" href="tel:21.06.2012%2014" value="+12106201214" target="_blank">21.06.2012 14:49, tom wrote:
> BTW: a core unload doesnt release the lock either ;(
>
>
> On <a moz-do-not-send="true" href="tel:21.06.2012%2014" value="+12106201214" target="_blank">21.06.2012 14:39, tom wrote:
>> hi,
>>
>> i'm using the suggester with a file like so:
>>
>>   <searchComponent class="solr.SpellCheckComponent" name="suggest">
>>     <lst name="spellchecker">
>>       <str name="name">suggest</str>
>>       <str 
>> name="classname">org.apache.solr.spelling.suggest.Suggester</str>
>>       <str 
>> name="lookupImpl">org.apache.solr.spelling.suggest.fst.FSTLookup</str>
>>       <!-- Alternatives to lookupImpl:
>>         org.apache.solr.spelling.suggest.fst.FSTLookup [finite state 
>> automaton]
>>         org.apache.solr.spelling.suggest.jaspell.JaspellLookup 
>> [default, jaspell-based]
>>         org.apache.solr.spelling.suggest.tst.TSTLookup [ternary trees]
>>       -->
>>       <!-- the indexed field to derive suggestions from -->
>>       <!-- TODO must change this to spell or smth alike later -->
>>       <str name="field">content</str>
>>       <float name="threshold">0.05</float>
>>       <str name="buildOnCommit">true</str>
>>       <str name="weightBuckets">100</str>
>>       <str name="sourceLocation">autocomplete.dictionary</str>
>>     </lst>
>>   </searchComponent>
>>
>> when trying to replicate i get the following error message on the 
>> slave side:
>>
>>  2012-06-21 14:34:50,781 ERROR 
>> [pool-3-thread-1                              ] 
>> handler.ReplicationHandler                    - SnapPull failed
>> org.apache.solr.common.SolrException: Unable to rename: <path>  
>> autocomplete.dictionary.20120620120611
>>     at 
>> org.apache.solr.handler.SnapPuller.copyTmpConfFiles2Conf(SnapPuller.java:642)
>>     at 
>> org.apache.solr.handler.SnapPuller.downloadConfFiles(SnapPuller.java:526)
>>     at 
>> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:299)
>>     at 
>> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268)
>>     at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
>>     at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>     at 
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>>     at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>     at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>     at java.lang.Thread.run(Thread.java:619)
>>
>> so i dug around it and found out that the solr's java process holds a 
>> lock on the autocomplete.dictionary file. any reason why this is so?
>>
>> thx,
>>
>> running:
>> solr 3.5
>> win7
>>
>
>
>











Reply | Threaded
Open this post in threaded view
|

Re: suggester/autocomplete locks file preventing replication

Simon Willnauer


On Fri, Jun 22, 2012 at 1:06 PM, tom <[hidden email]> wrote:
ah thx and to answer ur question:

no we are still on 3.5 and i had tested it on win7 (see the very end of this email thread)
and after having briefly looked @ the 3.6 code the class has changed quite a bit since 3.5...

yeah that is right! I rewrote most of the parts in the suggest code I think this should be fine now. So you might want to upgrade to 3.6 to get this fixed.

simon 


On 22.06.2012 12:16, Simon Willnauer wrote:
here is the issue https://issues.apache.org/jira/browse/SOLR-3570

On Fri, Jun 22, 2012 at 11:55 AM, Simon Willnauer <[hidden email]> wrote:


On Fri, Jun 22, 2012 at 11:47 AM, Simon Willnauer <[hidden email]> wrote:


On Fri, Jun 22, 2012 at 10:37 AM, tom <[hidden email]> wrote:
cross posting this issue to the dev list in the hope to get a response here...

I think you are right. Closing the Stream / Reader is the responsibility of the caller not the FileDictionary IMO but solr doesn't close it so that might cause your problems. Are you running on windows by any chance?
I will create an issue and fix it.

hmm I just looked at it and I see a IOUtils.close call in FileDictionary


are you using solr 3.6?  

simon


-------- Original Message --------
Subject: Re: suggester/autocomplete locks file preventing replication
Date: Thu, 21 Jun 2012 17:11:40 +0200
From: tom [hidden email]
Reply-To: [hidden email]
To: [hidden email]


pocking into the code i think the FileDictionary class is the culprit:
It takes an InputStream as a ctor argument but never releases the 
stream. what puzzles me is that the class seems to allow a one-time 
iteration and then the stream is useless, unless i'm missing smth. here.

is there a good reason for this or rather a bug?
should i move the topic to the dev list?


On <a href="tel:21.06.2012%2014" value="+12106201214" target="_blank">21.06.2012 14:49, tom wrote:
> BTW: a core unload doesnt release the lock either ;(
>
>
> On <a href="tel:21.06.2012%2014" value="+12106201214" target="_blank">21.06.2012 14:39, tom wrote:
>> hi,
>>
>> i'm using the suggester with a file like so:
>>
>>   <searchComponent class="solr.SpellCheckComponent" name="suggest">
>>     <lst name="spellchecker">
>>       <str name="name">suggest</str>
>>       <str 
>> name="classname">org.apache.solr.spelling.suggest.Suggester</str>
>>       <str 
>> name="lookupImpl">org.apache.solr.spelling.suggest.fst.FSTLookup</str>
>>       <!-- Alternatives to lookupImpl:
>>         org.apache.solr.spelling.suggest.fst.FSTLookup [finite state 
>> automaton]
>>         org.apache.solr.spelling.suggest.jaspell.JaspellLookup 
>> [default, jaspell-based]
>>         org.apache.solr.spelling.suggest.tst.TSTLookup [ternary trees]
>>       -->
>>       <!-- the indexed field to derive suggestions from -->
>>       <!-- TODO must change this to spell or smth alike later -->
>>       <str name="field">content</str>
>>       <float name="threshold">0.05</float>
>>       <str name="buildOnCommit">true</str>
>>       <str name="weightBuckets">100</str>
>>       <str name="sourceLocation">autocomplete.dictionary</str>
>>     </lst>
>>   </searchComponent>
>>
>> when trying to replicate i get the following error message on the 
>> slave side:
>>
>>  2012-06-21 14:34:50,781 ERROR 
>> [pool-3-thread-1                              ] 
>> handler.ReplicationHandler                    - SnapPull failed
>> org.apache.solr.common.SolrException: Unable to rename: <path>  
>> autocomplete.dictionary.20120620120611
>>     at 
>> org.apache.solr.handler.SnapPuller.copyTmpConfFiles2Conf(SnapPuller.java:642)
>>     at 
>> org.apache.solr.handler.SnapPuller.downloadConfFiles(SnapPuller.java:526)
>>     at 
>> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:299)
>>     at 
>> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268)
>>     at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
>>     at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>     at 
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>>     at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>>     at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>     at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>     at java.lang.Thread.run(Thread.java:619)
>>
>> so i dug around it and found out that the solr's java process holds a 
>> lock on the autocomplete.dictionary file. any reason why this is so?
>>
>> thx,
>>
>> running:
>> solr 3.5
>> win7
>>
>
>
>