the impact of thousands of field in a single document

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

the impact of thousands of field in a single document

Fang_Li
Hi,

Due to requirement, we need to construct a Lucene document with tens of
thousands of Field. Did anyone try this? What's the performance penalty
comparing with one single field to store all tokens for both indexing
and searching?

Thanks,
Li

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: the impact of thousands of field in a single document

Yonik Seeley
On Wed, Feb 18, 2009 at 3:26 AM,  <[hidden email]> wrote:
> Due to requirement, we need to construct a Lucene document with tens of
> thousands of Field. Did anyone try this? What's the performance penalty
> comparing with one single field to store all tokens for both indexing
> and searching?

It's doable.
Searching performance won't suffer at all.
Retrieving stored fields (after a search) will suffer somewhat.
Indexing performance will suffer somewhat.

The main problem will be norms, which aren't sparse and take up one
byte per document per field in the index, unless you omit the norms.

-Yonik
Lucene/Solr? http://www.lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]