solr 4.2.1 still has problems with index version and index generation

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

solr 4.2.1 still has problems with index version and index generation

Bernd Fehling
I know there was some effort to fix this but I must report
that solr 4.2.1 has still problems with index version and
index generation numbering in master/slave mode with replication.

Test was:
1. installed solr 4.2.1 on master and build index from scratch
2. installed solr 4.2.1 on slave with empty index
3. replicated master to slave, everything was fine, both in sync
4. deleted index on master with *:* and build index from scratch
5. replicated index from master to slave

RESULT: slave has different (higher) version number and is with generation 1 ahead :-(

Regards
Bernd
Reply | Threaded
Open this post in threaded view
|

Re: solr 4.2.1 still has problems with index version and index generation

Tom Gullo
I'm on 4.1 and I have a similar problem.  Except for the version number everything else seems to be fine.  Is that what other people are seeing?
Reply | Threaded
Open this post in threaded view
|

Re: solr 4.2.1 still has problems with index version and index generation

Joel Bernstein
This is the jira issue that addresses this:

https://issues.apache.org/jira/browse/SOLR-4661

I'll try to find some time today and test out the patch and see how things
look.


On Mon, Apr 8, 2013 at 7:18 AM, Tom Gullo <[hidden email]> wrote:

> I'm on 4.1 and I have a similar problem.  Except for the version number
> everything else seems to be fine.  Is that what other people are seeing?
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/solr-4-2-1-still-has-problems-with-index-version-and-index-generation-tp4054444p4054463.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



--
Joel Bernstein
Professional Services LucidWorks
Reply | Threaded
Open this post in threaded view
|

Re: solr 4.2.1 still has problems with index version and index generation

Chris Hostetter-3
In reply to this post by Bernd Fehling
: I know there was some effort to fix this but I must report
: that solr 4.2.1 has still problems with index version and
: index generation numbering in master/slave mode with replication.
        ...
: RESULT: slave has different (higher) version number and is with generation 1 ahead :-(

Can you please provide more details...

* are you using autocommit? with what settings?
* are you using openSearcher=false in any of your commits?
* where exactly are you looking that you see the master/slave out of sync?
* are you observing any actual problems, or just seeing that the
gen/version are reported as different?

As Joel mentioned, there is an open Jira related purely to the *display*
of information about gen/version between master & slave, because in many
cases the "searcher" in use on the master may refer to an older commit
point, but it doesn't mean there is any actual problem in replication --
the slave is still fetching & seraching the latest commit from the master
as intended....

https://issues.apache.org/jira/browse/SOLR-4661


-Hoss
Reply | Threaded
Open this post in threaded view
|

Re: solr 4.2.1 still has problems with index version and index generation

Bernd Fehling
Hi Hoss,

we don't use autoCommit and autoSoftCommit.
We don't use openSearcher.
We don't use transaction log.

I can see it in the AdminGUI and with
<a href="http://master_host:port/solr/replication?command=indexversion">http://master_host:port/solr/replication?command=indexversion

All files are replicated from master to slave, nothing lost.
It is just that the gen/version differs and breaks our cronjobs which
worked since solr 2.x.
As you mentioned, it seams that the latest commit is fetched.

Strange thing is,
we start with a clean, empty index on master. With all commands we send
a commit=true and, where applicable, an optimze=true.
The master is always in state "optimzed and current" when replicating.

How can it be that the searcher on master is referring to an older commit point
if there is no such point. The logs show _AFTER_ the last optimize has finished
a new searcher is started and the old one is closed.

Also, we have replicateAfter startup, commit and optimize set but the AdminGUI
and replication details only report replicateAfter commit and startup.
Not really an error, but not that what is really set in the config!

Very strange, I will try the patch.

Regards
Bernd


Am 08.04.2013 20:12, schrieb Chris Hostetter:

> : I know there was some effort to fix this but I must report
> : that solr 4.2.1 has still problems with index version and
> : index generation numbering in master/slave mode with replication.
> ...
> : RESULT: slave has different (higher) version number and is with generation 1 ahead :-(
>
> Can you please provide more details...
>
> * are you using autocommit? with what settings?
> * are you using openSearcher=false in any of your commits?
> * where exactly are you looking that you see the master/slave out of sync?
> * are you observing any actual problems, or just seeing that the
> gen/version are reported as different?
>
> As Joel mentioned, there is an open Jira related purely to the *display*
> of information about gen/version between master & slave, because in many
> cases the "searcher" in use on the master may refer to an older commit
> point, but it doesn't mean there is any actual problem in replication --
> the slave is still fetching & seraching the latest commit from the master
> as intended....
>
> https://issues.apache.org/jira/browse/SOLR-4661
>
>
> -Hoss
>
Reply | Threaded
Open this post in threaded view
|

Re: solr 4.2.1 still has problems with index version and index generation

Bernd Fehling
Looking a bit deeper showed that replication?command=commit reports the
right indexversion, generation and filelist.
<arr name="commits">
 <lst>
  <long name="indexVersion">1365357951589</long>
  <long name="generation">198</long>
   <arr name="filelist">
...

And with replication?command=details I also see the correct commit part as
above, BUT where the hell are the wrong info below the commit array are
coming from?
<str name="isMaster">true</str>
<str name="isSlave">false</str>
<long name="indexVersion">1365357774190</long>
<long name="generation">197</long>

The command replication?command=filelist&generation=197
replies with
<str name="status">invalid index generation</str>

Have a look into the sources:
Ahh, it is build in getReplicationDetails with:
details.add("isMaster", String.valueOf(isMaster));
details.add("isSlave", String.valueOf(isSlave));
long[] versionAndGeneration = getIndexVersion();
details.add("indexVersion", versionAndGeneration[0]);
details.add(GENERATION, versionAndGeneration[1]);

So getIndexVersion() gets a wrong version and generation, but why?
It first gets the searcher from the core and then tries to get
via the IndexReader the IndexCommit and then the commitData.

I think I should use remote debugging on master.
At least I now know that it is the master.

Regards
Bernd


Am 09.04.2013 08:35, schrieb Bernd Fehling:

> Hi Hoss,
>
> we don't use autoCommit and autoSoftCommit.
> We don't use openSearcher.
> We don't use transaction log.
>
> I can see it in the AdminGUI and with
> <a href="http://master_host:port/solr/replication?command=indexversion">http://master_host:port/solr/replication?command=indexversion
>
> All files are replicated from master to slave, nothing lost.
> It is just that the gen/version differs and breaks our cronjobs which
> worked since solr 2.x.
> As you mentioned, it seams that the latest commit is fetched.
>
> Strange thing is,
> we start with a clean, empty index on master. With all commands we send
> a commit=true and, where applicable, an optimze=true.
> The master is always in state "optimzed and current" when replicating.
>
> How can it be that the searcher on master is referring to an older commit point
> if there is no such point. The logs show _AFTER_ the last optimize has finished
> a new searcher is started and the old one is closed.
>
> Also, we have replicateAfter startup, commit and optimize set but the AdminGUI
> and replication details only report replicateAfter commit and startup.
> Not really an error, but not that what is really set in the config!
>
> Very strange, I will try the patch.
>
> Regards
> Bernd
>
>
> Am 08.04.2013 20:12, schrieb Chris Hostetter:
>> : I know there was some effort to fix this but I must report
>> : that solr 4.2.1 has still problems with index version and
>> : index generation numbering in master/slave mode with replication.
>> ...
>> : RESULT: slave has different (higher) version number and is with generation 1 ahead :-(
>>
>> Can you please provide more details...
>>
>> * are you using autocommit? with what settings?
>> * are you using openSearcher=false in any of your commits?
>> * where exactly are you looking that you see the master/slave out of sync?
>> * are you observing any actual problems, or just seeing that the
>> gen/version are reported as different?
>>
>> As Joel mentioned, there is an open Jira related purely to the *display*
>> of information about gen/version between master & slave, because in many
>> cases the "searcher" in use on the master may refer to an older commit
>> point, but it doesn't mean there is any actual problem in replication --
>> the slave is still fetching & seraching the latest commit from the master
>> as intended....
>>
>> https://issues.apache.org/jira/browse/SOLR-4661
>>
>>
>> -Hoss
>>

--
*************************************************************
Bernd Fehling                    Bielefeld University Library
Dipl.-Inform. (FH)                LibTec - Library Technology
Universit√§tsstr. 25                  and Knowledge Management
33615 Bielefeld
Tel. +49 521 106-4060       bernd.fehling(at)uni-bielefeld.de

BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: solr 4.2.1 still has problems with index version and index generation

Chris Hostetter-3

: And with replication?command=details I also see the correct commit part as
: above, BUT where the hell are the wrong info below the commit array are
: coming from?

Please read the details in the previously mentioned Jira issue...

https://issues.apache.org/jira/browse/SOLR-4661

The indexVersion and generation you are looking at refer to the speciics
of the IndexReader as used by the *seracher* on the master server -- but
in addition to situations like openSearcher=false, there are some
optimizations in place such that Solr/Lucene is smart enough to realize
when an "empty commit" doesn't change the IndexReader it continues to use
the previous commit point...

https://issues.apache.org/jira/browse/SOLR-4661?focusedCommentId=13620195&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13620195

...but from the perspective of the slave, this is still a commit that
needs replicated and loaded.

Hence the current objective of the patch in SOLR-3855: add more details to
the command=details response (as well as the Admin UI) to clearly
distinguish between the gen/ver of the currently replicatable commit and
the gen/ver of the currently open searcher.

All available information suggests that this is purely a problem of
conveying information to users via command=details -- Replication is
behaving as designed using the correct information about hte commit
points.



-Hoss