solr 4.3, autocommit, maxdocs

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

solr 4.3, autocommit, maxdocs

Jonathan Rochkind
I have a solr 4.3 instance I am in the process of standing up. It
started out with an empty index.

I have in it's solrconfig.xml,

   <updateHandler class="solr.DirectUpdateHandler2">
     <autoCommit>
       <maxDocs>100000</maxDocs>
       <openSearcher>false</openSearcher>
     </autoCommit>
   <updateHandler>

I have an index process running, that has currently added around 400k
documents to Solr.

I had expected that a 'commit' would be run every 100k documents, from
the above configuration, so 4 commits would have been run by now, and
I'd see documents in the index.

However, when I look in the Solr admin interface, at my core's
'overview' page, it still says num docs 0, segment count 0.  When I
expected num docs 400k at this point.

Is there something I'm misunderstanding about the configuration or the
admin interface? Or am I right in my expectations, but something else
must be going wrong?

Thanks for any advice,

Jonathan
Reply | Threaded
Open this post in threaded view
|

Re: solr 4.3, autocommit, maxdocs

Jason Hellman
Jonathan,

Please note the openSearcher=false part of your configuration.  This is why you don't see documents.  The commits are occurring, and being written to segments on disk, but they are not visible to the search engine because a Solr searcher class has not opened them for visibility.

You can either change the value to true, or alternatively call a deterministic commit call at the end of your load (a solr/update?commit=true will default to openSearcher=true).  

Hope that's of use!

Jason


On Jul 15, 2013, at 9:52 AM, Jonathan Rochkind <[hidden email]> wrote:

> I have a solr 4.3 instance I am in the process of standing up. It started out with an empty index.
>
> I have in it's solrconfig.xml,
>
>  <updateHandler class="solr.DirectUpdateHandler2">
>    <autoCommit>
>      <maxDocs>100000</maxDocs>
>      <openSearcher>false</openSearcher>
>    </autoCommit>
>  <updateHandler>
>
> I have an index process running, that has currently added around 400k documents to Solr.
>
> I had expected that a 'commit' would be run every 100k documents, from the above configuration, so 4 commits would have been run by now, and I'd see documents in the index.
>
> However, when I look in the Solr admin interface, at my core's 'overview' page, it still says num docs 0, segment count 0.  When I expected num docs 400k at this point.
>
> Is there something I'm misunderstanding about the configuration or the admin interface? Or am I right in my expectations, but something else must be going wrong?
>
> Thanks for any advice,
>
> Jonathan

Reply | Threaded
Open this post in threaded view
|

Re: solr 4.3, autocommit, maxdocs

Jonathan Rochkind
Ah, thanks for this explanation. Although I don't entirely understand
it, I am glad there is an expected explanation!

This Solr instance is actually set up to be a replication master. It
never gets searched itself, it just replicates to slaves that get searched.

Perhaps some time in the past (I am migrating from an already set up
Solr 1.4 instance), I set this value to false, figuring it was not
neccesary to actually open a searcher, since the master does not get
searched itself ordinarily.

Despite the opensearcher=false... once committed, are the committed docs
still going to be sent via replication to a slave, is the index used for
replication actually changed, even though a searcher hasn't been opened
to take account of it?  Or will the opensearcher=false keep the commits
from being seen by replication slaves too?

Thanks for any tips,

Jonathan

On 7/15/13 12:57 PM, Jason Hellman wrote:

> Jonathan,
>
> Please note the openSearcher=false part of your configuration.  This is why you don't see documents.  The commits are occurring, and being written to segments on disk, but they are not visible to the search engine because a Solr searcher class has not opened them for visibility.
>
> You can either change the value to true, or alternatively call a deterministic commit call at the end of your load (a solr/update?commit=true will default to openSearcher=true).
>
> Hope that's of use!
>
> Jason
>
>
> On Jul 15, 2013, at 9:52 AM, Jonathan Rochkind <[hidden email]> wrote:
>
>> I have a solr 4.3 instance I am in the process of standing up. It started out with an empty index.
>>
>> I have in it's solrconfig.xml,
>>
>>   <updateHandler class="solr.DirectUpdateHandler2">
>>     <autoCommit>
>>       <maxDocs>100000</maxDocs>
>>       <openSearcher>false</openSearcher>
>>     </autoCommit>
>>   <updateHandler>
>>
>> I have an index process running, that has currently added around 400k documents to Solr.
>>
>> I had expected that a 'commit' would be run every 100k documents, from the above configuration, so 4 commits would have been run by now, and I'd see documents in the index.
>>
>> However, when I look in the Solr admin interface, at my core's 'overview' page, it still says num docs 0, segment count 0.  When I expected num docs 400k at this point.
>>
>> Is there something I'm misunderstanding about the configuration or the admin interface? Or am I right in my expectations, but something else must be going wrong?
>>
>> Thanks for any advice,
>>
>> Jonathan
>