Foot, Inch: Stripping Out Special Characters: DisMax: WhitespaceTokenizer vs. Keyword Tokenizer

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Foot, Inch: Stripping Out Special Characters: DisMax: WhitespaceTokenizer vs. Keyword Tokenizer

Fuad Efendi
Hello,


I finally got it work: search for 5’ 3” (5 feet 3 inches)

It is strange for me that if I use WhitespaceTokenizer for field query-type analyzer then it will receive only 5 and 3 with special characters removed.

It is also strange that EDisMax does not strips out odd number of quotes.

But it works fine with KeywordTokenizer.

Any idea why? Thanks,


-- 
Fuad Efendi
http://www.tokenizer.ca
Data Mining, Vertical Search