[jira] [Created] (LUCENE-3880) UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Created] (LUCENE-3880) UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended

JIRA jira@apache.org
UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended
---------------------------------------------------------------------------------------------

                 Key: LUCENE-3880
                 URL: https://issues.apache.org/jira/browse/LUCENE-3880
             Project: Lucene - Java
          Issue Type: Bug
    Affects Versions: 3.5, 4.0
            Reporter: Steven Rowe
            Assignee: Steven Rowe
            Priority: Minor


As [reported by Kai Gülzau on solr-user|http://markmail.org/message/n32kji3okqm2c5qn]:

UAX29URLEmailTokenizer seems to split at the wrong place:

mailto:[hidden email] ->
mailto:test
example.org

As a workaround I use

<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="mailto:" replacement="mailto: "/>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (LUCENE-3880) UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steven Rowe updated LUCENE-3880:
--------------------------------

    Description:
As [reported by Kai Gülzau on solr-user|http://markmail.org/message/n32kji3okqm2c5qn]:

UAX29URLEmailTokenizer seems to split at the wrong place:

{noformat}mailto:[hidden email]{noformat} ->
{noformat}mailto:test{noformat}
{noformat}example.org{noformat}

As a workaround I use

{code:xml}
<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="mailto:" replacement="mailto: "/>
{code}

  was:
As [reported by Kai Gülzau on solr-user|http://markmail.org/message/n32kji3okqm2c5qn]:

UAX29URLEmailTokenizer seems to split at the wrong place:

mailto:[hidden email] ->
mailto:test
example.org

As a workaround I use

<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="mailto:" replacement="mailto: "/>


   

> UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended
> ---------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3880
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3880
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 3.5, 4.0
>            Reporter: Steven Rowe
>            Assignee: Steven Rowe
>            Priority: Minor
>
> As [reported by Kai Gülzau on solr-user|http://markmail.org/message/n32kji3okqm2c5qn]:
> UAX29URLEmailTokenizer seems to split at the wrong place:
> {noformat}mailto:[hidden email]{noformat} ->
> {noformat}mailto:test{noformat}
> {noformat}example.org{noformat}
> As a workaround I use
> {code:xml}
> <charFilter class="solr.PatternReplaceCharFilterFactory" pattern="mailto:" replacement="mailto: "/>
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (LUCENE-3880) UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232284#comment-13232284 ]

Steven Rowe commented on LUCENE-3880:
-------------------------------------

[RFC 2368|http://tools.ietf.org/html/rfc2368] describes URLs employing the {{mailto:}} scheme; this RFC has been obsoleted by [RFC 6068|http://tools.ietf.org/html/rfc6068], which describes the {{mailto:}} *URI* scheme.

{{mailto:}} URIs can contain multiple email addresses, and fielded information including CC, BCC, Subject, and Body - in short, the entire contents of an email message.

However, a significant proportion of (probably most) {{mailto:}} URIs in the wild contain just a single email address.  Short of handling all aspects of the {{mailto:}} scheme (out of scope for this issue), I think it would be useful to employ a trick similar to the charFilter hack described by Kai Gülzau: explicitly split "mailto:" off from a following email address, allowing the email address to be recognized as such.

               

> UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended
> ---------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3880
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3880
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 3.5, 4.0
>            Reporter: Steven Rowe
>            Assignee: Steven Rowe
>            Priority: Minor
>
> As [reported by Kai Gülzau on solr-user|http://markmail.org/message/n32kji3okqm2c5qn]:
> UAX29URLEmailTokenizer seems to split at the wrong place:
> {noformat}mailto:[hidden email]{noformat} ->
> {noformat}mailto:test{noformat}
> {noformat}example.org{noformat}
> As a workaround I use
> {code:xml}
> <charFilter class="solr.PatternReplaceCharFilterFactory" pattern="mailto:" replacement="mailto: "/>
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (LUCENE-3880) UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steven Rowe updated LUCENE-3880:
--------------------------------

    Attachment: LUCENE-3880.patch

Patch, adding a test for the triggering example, and another test illustrating some of the challenges of handling full mailto: syntax.

This change triggers a new version for UAX29URLEmailTokenizer, and I've taken advantage of that to update to the most recent top level domain definitions.

I think this is ready to commit.
               

> UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended
> ---------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3880
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3880
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 3.5, 4.0
>            Reporter: Steven Rowe
>            Assignee: Steven Rowe
>            Priority: Minor
>         Attachments: LUCENE-3880.patch
>
>
> As [reported by Kai Gülzau on solr-user|http://markmail.org/message/n32kji3okqm2c5qn]:
> UAX29URLEmailTokenizer seems to split at the wrong place:
> {noformat}mailto:[hidden email]{noformat} ->
> {noformat}mailto:test{noformat}
> {noformat}example.org{noformat}
> As a workaround I use
> {code:xml}
> <charFilter class="solr.PatternReplaceCharFilterFactory" pattern="mailto:" replacement="mailto: "/>
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (LUCENE-3880) UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232305#comment-13232305 ]

Uwe Schindler commented on LUCENE-3880:
---------------------------------------

Can we maybe (as discussed before) also add a corresponding Analyzer (clone of StandardAna)?
               

> UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended
> ---------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3880
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3880
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 3.5, 4.0
>            Reporter: Steven Rowe
>            Assignee: Steven Rowe
>            Priority: Minor
>         Attachments: LUCENE-3880.patch
>
>
> As [reported by Kai Gülzau on solr-user|http://markmail.org/message/n32kji3okqm2c5qn]:
> UAX29URLEmailTokenizer seems to split at the wrong place:
> {noformat}mailto:[hidden email]{noformat} ->
> {noformat}mailto:test{noformat}
> {noformat}example.org{noformat}
> As a workaround I use
> {code:xml}
> <charFilter class="solr.PatternReplaceCharFilterFactory" pattern="mailto:" replacement="mailto: "/>
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (LUCENE-3880) UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232308#comment-13232308 ]

Steven Rowe commented on LUCENE-3880:
-------------------------------------

bq. Can we maybe (as discussed before) also add a corresponding Analyzer (clone of StandardAna)?

+1
               

> UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended
> ---------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3880
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3880
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 3.5, 4.0
>            Reporter: Steven Rowe
>            Assignee: Steven Rowe
>            Priority: Minor
>         Attachments: LUCENE-3880.patch
>
>
> As [reported by Kai Gülzau on solr-user|http://markmail.org/message/n32kji3okqm2c5qn]:
> UAX29URLEmailTokenizer seems to split at the wrong place:
> {noformat}mailto:[hidden email]{noformat} ->
> {noformat}mailto:test{noformat}
> {noformat}example.org{noformat}
> As a workaround I use
> {code:xml}
> <charFilter class="solr.PatternReplaceCharFilterFactory" pattern="mailto:" replacement="mailto: "/>
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Resolved] (LUCENE-3880) UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/LUCENE-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steven Rowe resolved LUCENE-3880.
---------------------------------

       Resolution: Fixed
    Fix Version/s: 4.0
                   3.6
    Lucene Fields: New,Patch Available  (was: New)

Committed to trunk and branch_3x.

Thanks Kai!
               

> UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended
> ---------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3880
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3880
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 3.5, 4.0
>            Reporter: Steven Rowe
>            Assignee: Steven Rowe
>            Priority: Minor
>             Fix For: 3.6, 4.0
>
>         Attachments: LUCENE-3880.patch
>
>
> As [reported by Kai Gülzau on solr-user|http://markmail.org/message/n32kji3okqm2c5qn]:
> UAX29URLEmailTokenizer seems to split at the wrong place:
> {noformat}mailto:[hidden email]{noformat} ->
> {noformat}mailto:test{noformat}
> {noformat}example.org{noformat}
> As a workaround I use
> {code:xml}
> <charFilter class="solr.PatternReplaceCharFilterFactory" pattern="mailto:" replacement="mailto: "/>
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (LUCENE-3880) UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234341#comment-13234341 ]

Kai Gülzau commented on LUCENE-3880:
------------------------------------

That was fast! Thank _you_ :-)
               

> UAX29URLEmailTokenizer fails to recognize emails as such when the mailto: scheme is prepended
> ---------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3880
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3880
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 3.5, 4.0
>            Reporter: Steven Rowe
>            Assignee: Steven Rowe
>            Priority: Minor
>             Fix For: 3.6, 4.0
>
>         Attachments: LUCENE-3880.patch
>
>
> As [reported by Kai Gülzau on solr-user|http://markmail.org/message/n32kji3okqm2c5qn]:
> UAX29URLEmailTokenizer seems to split at the wrong place:
> {noformat}mailto:[hidden email]{noformat} ->
> {noformat}mailto:test{noformat}
> {noformat}example.org{noformat}
> As a workaround I use
> {code:xml}
> <charFilter class="solr.PatternReplaceCharFilterFactory" pattern="mailto:" replacement="mailto: "/>
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]