Problem with tokenStreamValue() method in Field class

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with tokenStreamValue() method in Field class

mika03

Hi,

I am slightly confused.
The following code generates the output below.

Document doc = hits.doc(i);
System.out.println(doc.getFields());
Field f = doc.getField("Text");
System.out.println(f);
TokenStream ts = f.tokenStreamValue();
System.out.println(ts);
while (true){
        Token token = ts.next();
        if (token==null) break;
        System.out.print(token+" ");
}
System.out.println();


[stored/uncompressed,indexed,tokenized<Title:Infrared Data Association>, stored/uncompressed,indexed,tokenized<Type:Text>, stored/uncompressed,indexed,tokenized<Text:This is a test>]
stored/uncompressed,indexed,tokenized<Text:This is a test>
null
java.lang.NullPointerException
at ... (IndexQuerier.java:86)
...

So I am accessing a field which is displayed as "tokenized", but I still can't access its "tokenStreamValue()"? Why is that? Is there some way to get to tokens? Or did I do something wrong during indexing?


Regards,
Mika
Reply | Threaded
Open this post in threaded view
|

Re: Problem with tokenStreamValue() method in Field class

hossman

you are running into one of hte problems relating to "Field" being reused
by both the indexing code and the searching code.

things like the tokenStreamValue() and readerValue() only have meaning on
Fields that are about to be indexed ... Field objects returned from
searches will never return data from those methods.

Field objects returned from a search will only ever have the "stored"
value of a Field, which is available from stringValue()

: I am slightly confused.
: The following code generates the output below.
:
: Document doc = hits.doc(i);
: System.out.println(doc.getFields());
: Field f = doc.getField("Text");
: System.out.println(f);
: TokenStream ts = f.tokenStreamValue();
: System.out.println(ts);
: while (true){
: Token token = ts.next();
: if (token==null) break;
: System.out.print(token+" ");
: }
: System.out.println();
:
:
: [stored/uncompressed,indexed,tokenized<Title:Infrared Data Association>,
: stored/uncompressed,indexed,tokenized<Type:Text>,
: stored/uncompressed,indexed,tokenized<Text:This is a test>]
: stored/uncompressed,indexed,tokenized<Text:This is a test>
: null
: java.lang.NullPointerException
: at ... (IndexQuerier.java:86)
: ...
:
: So I am accessing a field which is displayed as "tokenized", but I still
: can't access its "tokenStreamValue()"? Why is that? Is there some way to get
: to tokens? Or did I do something wrong during indexing?
:
:
: Regards,
: Mika
: --
: View this message in context: http://www.nabble.com/Problem-with-tokenStreamValue%28%29-method-in-Field-class-tp15048591p15048591.html
: Sent from the Lucene - Java Users mailing list archive at Nabble.com.
:
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: [hidden email]
: For additional commands, e-mail: [hidden email]
:



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]