StoredFields directly from a file

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

StoredFields directly from a file

John Wang-9
Hi folks:

A question on indexing StoredFields (using the compressioncodec):

I have a flat file storing in sequential order contents I intend to add to stored fields. I keep a list of lengths in memory:

(byte,byte),(byte,byte,byte)...  <--- file
[2,3,...] <-- memory

where each element correspond to the storedField for a doc.
doc1: 2 bytes
doc2: 3 bytes

Currently, I would need to read them into a BytesRef allocating either bunch of small byte[] or load this entire file into a rather large byte[].

Do you think it makes sense to change the StoredField api to read from a more generic version of BytesRef, that it can point either to file or memory?