Threaded updating?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Threaded updating?

Scott Leonard-2
Can I speedup a solr add-post by threading an add process and asynchronously
updating records? Or does solr do some locking that would make this
irrelevant.

.scott

Reply | Threaded
Open this post in threaded view
|

Re: Threaded updating?

Yonik Seeley-2
On 1/24/07, Scott Leonard <[hidden email]> wrote:
> Can I speedup a solr add-post by threading an add process and asynchronously
> updating records? Or does solr do some locking that would make this
> irrelevant.

Yes.  Parsing of the XML and text analysis of the documents would all
proceed in parallel (more or less, there are some small
synchronization point to keep track of what docs were added, and
Lucene currently blocks new documents from being added while a segment
merge is in progress).

Hopefully we can have a component that does this on the server side
for large numbers of documents at some point (if a bunch of documents
are added in a single request).

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Threaded updating?

Mike Klaas
In reply to this post by Scott Leonard-2
On 1/24/07, Scott Leonard <[hidden email]> wrote:
> Can I speedup a solr add-post by threading an add process and asynchronously
> updating records? Or does solr do some locking that would make this
> irrelevant.

Some update tasks can be parallelized and some cannot.  You will see a
speedup by adding docs on multiple threads, but it will not be a
linear increase unless your application's analysis requirements are
rather cpu-intensive.

-Mike