Twitter analyser

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Twitter analyser

Stephane Nicoll
Hi,

I am building an application that indexes tweet and offer some basic
search facilities on them.

I am trying to find a combination where the following would work:

* foo matches the foo word, a mention (@foo) or the hashtag (#foo)
* @foo only matches the mention
* #foo matches only the hashtag

It should matches complete word so I used the WhiteSpaceAnalyzer for indexing.

Any recommendation for this use case?

Thanks !
S.

Sent from my iPhone

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Twitter analyser

Erick Erickson
If your universe of items you want to match this way is small,
consider something akin to synonyms. Your indexing process
emits two tokens, with and without the @ or # which should
cover your situation.

FWIW,
Erick


On Tue, Nov 5, 2013 at 2:40 AM, Stéphane Nicoll
<[hidden email]>wrote:

> Hi,
>
> I am building an application that indexes tweet and offer some basic
> search facilities on them.
>
> I am trying to find a combination where the following would work:
>
> * foo matches the foo word, a mention (@foo) or the hashtag (#foo)
> * @foo only matches the mention
> * #foo matches only the hashtag
>
> It should matches complete word so I used the WhiteSpaceAnalyzer for
> indexing.
>
> Any recommendation for this use case?
>
> Thanks !
> S.
>
> Sent from my iPhone
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Twitter analyser

Stephane Nicoll
Hi,

Thanks for the reply. It's an index with tweets so any word really is a
target for this. This would mean a significant increase of the index. My
volumes are really small so that shouldn't be a problem (but
performance/scalability is a concern).

I have the control over the query. Another solution would be to translate a
query on "foo" to "foo or #foo or @foo"

WDYT?

Thanks!
S.




On Tue, Nov 5, 2013 at 2:17 PM, Erick Erickson <[hidden email]>wrote:

> If your universe of items you want to match this way is small,
> consider something akin to synonyms. Your indexing process
> emits two tokens, with and without the @ or # which should
> cover your situation.
>
> FWIW,
> Erick
>
>
> On Tue, Nov 5, 2013 at 2:40 AM, Stéphane Nicoll
> <[hidden email]>wrote:
>
> > Hi,
> >
> > I am building an application that indexes tweet and offer some basic
> > search facilities on them.
> >
> > I am trying to find a combination where the following would work:
> >
> > * foo matches the foo word, a mention (@foo) or the hashtag (#foo)
> > * @foo only matches the mention
> > * #foo matches only the hashtag
> >
> > It should matches complete word so I used the WhiteSpaceAnalyzer for
> > indexing.
> >
> > Any recommendation for this use case?
> >
> > Thanks !
> > S.
> >
> > Sent from my iPhone
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Twitter analyser

Erick Erickson
You have to get the values _into_ the index with the special characters,
that's where the issue is. Depending on your analysis chain special
characters may or may not be even in your index to search in the first
place.

So it's not how many different words are after the special characters as
much as how many special characters there are. So what I'm thinking is
that as you index documents, you detect #foo, #blah, #whatever and
index #foo, foo, #blah, blah etc. If all you have to do is specially handle
tokens that start with just a few different chars it's not very hard.

FWIW,
Erick


On Tue, Nov 5, 2013 at 8:33 AM, Stephane Nicoll
<[hidden email]>wrote:

> Hi,
>
> Thanks for the reply. It's an index with tweets so any word really is a
> target for this. This would mean a significant increase of the index. My
> volumes are really small so that shouldn't be a problem (but
> performance/scalability is a concern).
>
> I have the control over the query. Another solution would be to translate a
> query on "foo" to "foo or #foo or @foo"
>
> WDYT?
>
> Thanks!
> S.
>
>
>
>
> On Tue, Nov 5, 2013 at 2:17 PM, Erick Erickson <[hidden email]
> >wrote:
>
> > If your universe of items you want to match this way is small,
> > consider something akin to synonyms. Your indexing process
> > emits two tokens, with and without the @ or # which should
> > cover your situation.
> >
> > FWIW,
> > Erick
> >
> >
> > On Tue, Nov 5, 2013 at 2:40 AM, Stéphane Nicoll
> > <[hidden email]>wrote:
> >
> > > Hi,
> > >
> > > I am building an application that indexes tweet and offer some basic
> > > search facilities on them.
> > >
> > > I am trying to find a combination where the following would work:
> > >
> > > * foo matches the foo word, a mention (@foo) or the hashtag (#foo)
> > > * @foo only matches the mention
> > > * #foo matches only the hashtag
> > >
> > > It should matches complete word so I used the WhiteSpaceAnalyzer for
> > > indexing.
> > >
> > > Any recommendation for this use case?
> > >
> > > Thanks !
> > > S.
> > >
> > > Sent from my iPhone
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Twitter analyser

Jack Krupansky-2
In reply to this post by Stephane Nicoll
You can specify custom character types with the word delimiter filter, so
you could define "@" and "#" as "digit" and set SPLIT_ON_NUMERICS. This
would cause "@foo" to tokenize as two adjacent terms, ditto for "#foo".
Unfortunately, A user name or tag that starts with a digit would not
tokenize as desired, but that seems uncommon. "foo" would match all three
since the "@" or "#" would tokenize as a separate term.

Use:

public WordDelimiterFilter(TokenStream in,
                           byte[] charTypeTable,
                           int configurationFlags,
                           CharArraySet protWords)

See:
http://lucene.apache.org/core/4_5_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/WordDelimiterFilter.html

-- Jack Krupansky
-----Original Message-----
From: Stéphane Nicoll
Sent: Tuesday, November 05, 2013 2:40 AM
To: [hidden email]
Subject: Twitter analyser

Hi,

I am building an application that indexes tweet and offer some basic
search facilities on them.

I am trying to find a combination where the following would work:

* foo matches the foo word, a mention (@foo) or the hashtag (#foo)
* @foo only matches the mention
* #foo matches only the hashtag

It should matches complete word so I used the WhiteSpaceAnalyzer for
indexing.

Any recommendation for this use case?

Thanks !
S.

Sent from my iPhone

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Twitter analyser

Lance Norskog-2
In reply to this post by Stephane Nicoll
This is a parts-of-speech analyzer for tweets. It would make your index
far more useful.

http://www.ark.cs.cmu.edu/TweetNLP/

On 11/04/2013 11:40 PM, Stéphane Nicoll wrote:

> Hi,
>
> I am building an application that indexes tweet and offer some basic
> search facilities on them.
>
> I am trying to find a combination where the following would work:
>
> * foo matches the foo word, a mention (@foo) or the hashtag (#foo)
> * @foo only matches the mention
> * #foo matches only the hashtag
>
> It should matches complete word so I used the WhiteSpaceAnalyzer for indexing.
>
> Any recommendation for this use case?
>
> Thanks !
> S.
>
> Sent from my iPhone
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

Reply | Threaded
Open this post in threaded view
|

Re: Twitter analyser

Stephane Nicoll
In reply to this post by Erick Erickson
Hi,

This is what I've tried:
https://gist.github.com/anonymous/7383104

So far so good except that something is definitely wrong in my code as the
synonym is not emitted as a valid token it seems. This is how my indexing
analyzer is built:

 private static final class MyIndexAnalyzer extends ReusableAnalyzerBase {
        @Override
        protected TokenStreamComponents createComponents(String fieldName,
Reader reader) {
            final Tokenizer tokenizer = new
WhitespaceTokenizer(Version.LUCENE_36, reader);
            final TwitterFilter twitterFilter = new
TwitterFilter(tokenizer);
            final LowerCaseFilter filter = new
LowerCaseFilter(Version.LUCENE_36, twitterFilter);
            return new TokenStreamComponents(tokenizer, filter);
        }
    }

I am expecting the lower filter to pick up the synonym exactly the same way
as the original token but it does not. If I have the following tweet "Bla
Bla #SomeTAG", "#sometag" matches but "sometag" does not. All other use
cases not involving a case mismatch work as expected.

Does anyone knows what's wrong in my code?

Thanks for the support!

S.



On Tue, Nov 5, 2013 at 2:17 PM, Erick Erickson <[hidden email]>wrote:

> If your universe of items you want to match this way is small,
> consider something akin to synonyms. Your indexing process
> emits two tokens, with and without the @ or # which should
> cover your situation.
>
> FWIW,
> Erick
>
>
> On Tue, Nov 5, 2013 at 2:40 AM, Stéphane Nicoll
> <[hidden email]>wrote:
>
> > Hi,
> >
> > I am building an application that indexes tweet and offer some basic
> > search facilities on them.
> >
> > I am trying to find a combination where the following would work:
> >
> > * foo matches the foo word, a mention (@foo) or the hashtag (#foo)
> > * @foo only matches the mention
> > * #foo matches only the hashtag
> >
> > It should matches complete word so I used the WhiteSpaceAnalyzer for
> > indexing.
> >
> > Any recommendation for this use case?
> >
> > Thanks !
> > S.
> >
> > Sent from my iPhone
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Twitter analyser

Stephane Nicoll
Replying to self: silly me. I am obviously creating the array with the
wrong length.
final String term = new String(buffer, 1, length);

should be replaced by
final String term = new String(buffer, 1, length -1);

and the silly trim can go away. I guess I need more coffee.

S.




On Sat, Nov 9, 2013 at 9:45 AM, Stephane Nicoll
<[hidden email]>wrote:

> Hi,
>
> This is what I've tried:
> https://gist.github.com/anonymous/7383104
>
> So far so good except that something is definitely wrong in my code as the
> synonym is not emitted as a valid token it seems. This is how my indexing
> analyzer is built:
>
>  private static final class MyIndexAnalyzer extends ReusableAnalyzerBase {
>         @Override
>         protected TokenStreamComponents createComponents(String fieldName,
> Reader reader) {
>             final Tokenizer tokenizer = new
> WhitespaceTokenizer(Version.LUCENE_36, reader);
>             final TwitterFilter twitterFilter = new
> TwitterFilter(tokenizer);
>             final LowerCaseFilter filter = new
> LowerCaseFilter(Version.LUCENE_36, twitterFilter);
>             return new TokenStreamComponents(tokenizer, filter);
>         }
>     }
>
> I am expecting the lower filter to pick up the synonym exactly the same
> way as the original token but it does not. If I have the following tweet
> "Bla Bla #SomeTAG", "#sometag" matches but "sometag" does not. All other
> use cases not involving a case mismatch work as expected.
>
> Does anyone knows what's wrong in my code?
>
> Thanks for the support!
>
> S.
>
>
>
> On Tue, Nov 5, 2013 at 2:17 PM, Erick Erickson <[hidden email]>wrote:
>
>> If your universe of items you want to match this way is small,
>> consider something akin to synonyms. Your indexing process
>> emits two tokens, with and without the @ or # which should
>> cover your situation.
>>
>> FWIW,
>> Erick
>>
>>
>> On Tue, Nov 5, 2013 at 2:40 AM, Stéphane Nicoll
>> <[hidden email]>wrote:
>>
>> > Hi,
>> >
>> > I am building an application that indexes tweet and offer some basic
>> > search facilities on them.
>> >
>> > I am trying to find a combination where the following would work:
>> >
>> > * foo matches the foo word, a mention (@foo) or the hashtag (#foo)
>> > * @foo only matches the mention
>> > * #foo matches only the hashtag
>> >
>> > It should matches complete word so I used the WhiteSpaceAnalyzer for
>> > indexing.
>> >
>> > Any recommendation for this use case?
>> >
>> > Thanks !
>> > S.
>> >
>> > Sent from my iPhone
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: [hidden email]
>> > For additional commands, e-mail: [hidden email]
>> >
>> >
>>
>
>