tokenizer to strip a set of characters

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

tokenizer to strip a set of characters

Stephane Nicoll
Hi,

I am using lucene 3.6 and I am looking to a tokenized that would remove
certain characters when they are present at the beginning or at the end of
a token.

I initially used the StandardAnalyzer and switched to the
WhitespaceAnalyser because it was too agressive for my use case.

A few examples:

   - foo, -> foo (comma at the end)
   - foo. -> foo (period at the end)
   - foo!!!! -> foo
   - foo?! -> foo
   - ,foo -> foo (comma at the beginning of a word is a typo mistake but
   should be handled-

Is there a configurable tokenizer I could use for this?

Thanks,
S.
Reply | Threaded
Open this post in threaded view
|

Re: tokenizer to strip a set of characters

Jack Krupansky-2
The word delimiter filter has the ability to pass a table which specifies
the type for a character:

http://lucene.apache.org/core/4_5_1/analyzers-common/org/apache/lucene/analysis/miscellaneous/WordDelimiterFilter.html

http://lucene.apache.org/core/4_5_1/analyzers-common/org/apache/lucene/analysis/miscellaneous/WordDelimiterFilter.html#WordDelimiterFilter(org.apache.lucene.analysis.TokenStream,
byte[], int, org.apache.lucene.analysis.util.CharArraySet)

There is also a regex token filter that you could use to make fine
adjustments, like character allowed within tokens but ignored at the start
or end.

-- Jack Krupansky

-----Original Message-----
From: Stephane Nicoll
Sent: Thursday, November 21, 2013 9:42 AM
To: [hidden email]
Subject: tokenizer to strip a set of characters

Hi,

I am using lucene 3.6 and I am looking to a tokenized that would remove
certain characters when they are present at the beginning or at the end of
a token.

I initially used the StandardAnalyzer and switched to the
WhitespaceAnalyser because it was too agressive for my use case.

A few examples:

   - foo, -> foo (comma at the end)
   - foo. -> foo (period at the end)
   - foo!!!! -> foo
   - foo?! -> foo
   - ,foo -> foo (comma at the beginning of a word is a typo mistake but
   should be handled-

Is there a configurable tokenizer I could use for this?

Thanks,
S.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]