Parallel access to TermPositions API

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Parallel access to TermPositions API

Chris Bamford
Hi,

I need to load a huge amount of TermPositions in a short space of time
(millions of Documents, sub-second).
Does the IndexReader's API support multiple accesses to allow several
parallel threads to consume a chunk each?

Thanks for any ideas / pointers.

- Chris



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Parallel access to TermPositions API

Federico Fissore
Chris Bamford, il 14/04/2011 20:11, ha scritto:
> Hi,
>
> I need to load a huge amount of TermPositions in a short space of time
> (millions of Documents, sub-second).
> Does the IndexReader's API support multiple accesses to allow several
> parallel threads to consume a chunk each?
>

AFAIK, you cannot share one instance of TermPositions, but each of your
thread may have its own dedicated instance, so each will start with a
different call to skipTo or seek before consuming

Anyway the I/O will be a problem: you can have a 16 way server, but if
your index is on one harddrive, that will be your bottleneck

HTH

Federico

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]