[jira] Created: (NUTCH-237) Carrot2 clustering plugin upgrade.

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (NUTCH-237) Carrot2 clustering plugin upgrade.

Michael Gibney (Jira)
Carrot2 clustering plugin upgrade.
----------------------------------

         Key: NUTCH-237
         URL: http://issues.apache.org/jira/browse/NUTCH-237
     Project: Nutch
        Type: Improvement
    Reporter: Dawid Weiss
    Priority: Trivial


This is an upgrade of the clustering plugin to the newest release (1.0.2).

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-237) Carrot2 clustering plugin upgrade.

Michael Gibney (Jira)
     [ http://issues.apache.org/jira/browse/NUTCH-237?page=all ]

Dawid Weiss updated NUTCH-237:
------------------------------

    Attachment: c2.patch
                svn-stat.txt

Note the two deleted files (I attached the result of svn stat). I didn't know how to include this info in the diff file, don't think it's possible with plain svn.

> Carrot2 clustering plugin upgrade.
> ----------------------------------
>
>          Key: NUTCH-237
>          URL: http://issues.apache.org/jira/browse/NUTCH-237
>      Project: Nutch
>         Type: Improvement
>     Reporter: Dawid Weiss
>     Priority: Trivial
>  Attachments: c2.patch, svn-stat.txt
>
> This is an upgrade of the clustering plugin to the newest release (1.0.2).

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-237) Carrot2 clustering plugin upgrade.

Michael Gibney (Jira)
In reply to this post by Michael Gibney (Jira)
     [ http://issues.apache.org/jira/browse/NUTCH-237?page=all ]

Dawid Weiss updated NUTCH-237:
------------------------------

    Attachment: libs.zip

Libraries that need to be replaced.

> Carrot2 clustering plugin upgrade.
> ----------------------------------
>
>          Key: NUTCH-237
>          URL: http://issues.apache.org/jira/browse/NUTCH-237
>      Project: Nutch
>         Type: Improvement
>     Reporter: Dawid Weiss
>     Priority: Trivial
>  Attachments: c2.patch, libs.zip, svn-stat.txt
>
> This is an upgrade of the clustering plugin to the newest release (1.0.2).

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-237) Carrot2 clustering plugin upgrade.

Michael Gibney (Jira)
In reply to this post by Michael Gibney (Jira)
    [ http://issues.apache.org/jira/browse/NUTCH-237?page=comments#action_12371606 ]

Andrzej Bialecki  commented on NUTCH-237:
-----------------------------------------

Hmm, I'm not sure I like this patch. It removes support for other languages than English. While I can agree with the argument that language detection in Carrot is relatively simplistic and we should use other mechanisms if available, this patch removed also stemmers and stopword lists for other languages, without the ability to provide them so that Lingo uses them when extracting text features... or perhaps I'm missing something?

> Carrot2 clustering plugin upgrade.
> ----------------------------------
>
>          Key: NUTCH-237
>          URL: http://issues.apache.org/jira/browse/NUTCH-237
>      Project: Nutch
>         Type: Improvement
>     Reporter: Dawid Weiss
>     Priority: Trivial
>  Attachments: c2.patch, libs.zip, svn-stat.txt
>
> This is an upgrade of the clustering plugin to the newest release (1.0.2).

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (NUTCH-237) Carrot2 clustering plugin upgrade.

Michael Gibney (Jira)
In reply to this post by Michael Gibney (Jira)
    [ http://issues.apache.org/jira/browse/NUTCH-237?page=comments#action_12371687 ]

Dawid Weiss commented on NUTCH-237:
-----------------------------------

Yes and no. I removed the "support" for foreign languages from the constructor code:

        // We initialize Lingo with English stemming and stopwords. Lingo has
        // a simple language detection filter, but you'll be better off hardcoding
        // the language according to your needs. If you have bilingual indices,
        // then there is a possibility of creating a more complex process that assigns
        // a language tag before the clustering is actually started.
        return new LingoLocalFilterComponent(
          new Language[] { new English() },
          defaults);
      }

Language detection is not really brilliant in the open source Lingo so I thought it wouldn't make sense to give people false hopes. Now, all the stemmers and stopword lists are still included in the release (look inside carrot2-util-tokenizer.jar$/com/dawidweiss/carrot/util/tokenizer/languages/...) so you can freely switch to another language by changing the instantiated language.

I have a better idea though -- how about if you apply this patch (because I\ve tested it and know it works) and I'll make the language configurable via ISO codes set in nutch configuration? The default would be English and you could set your own language in there if you wanted to. All right?

> Carrot2 clustering plugin upgrade.
> ----------------------------------
>
>          Key: NUTCH-237
>          URL: http://issues.apache.org/jira/browse/NUTCH-237
>      Project: Nutch
>         Type: Improvement
>     Reporter: Dawid Weiss
>     Priority: Trivial
>  Attachments: c2.patch, libs.zip, svn-stat.txt
>
> This is an upgrade of the clustering plugin to the newest release (1.0.2).

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Updated: (NUTCH-237) Carrot2 clustering plugin upgrade.

Michael Gibney (Jira)
In reply to this post by Michael Gibney (Jira)
     [ http://issues.apache.org/jira/browse/NUTCH-237?page=all ]

Dawid Weiss updated NUTCH-237:
------------------------------

    Attachment: NUTCH-237.DWEISS.patch.zip

Hi Andrzej. The ZIP file contains a patch and svn stat with the improved code:

- The primary language for hits without explicit langid and a list of enabled languages in the clustering component can be specified in the configuration file (readme.txt gives the details).

- by default all languages in Carrot2 (except for Polish) are enabled. English is the default.

- I removed the dependency on Neko in favor of the simpler routine we have in Carrot2 codebase anyway. The change shouldn't affect the results (I checked on my local installation and it seems to be fine).

I haven't played with the language identifier yet because I don't have a crawl with documents containing langid codes. The code should work without problems though -- details.getValue("lang") is converted to Carrot2's property RawDocument.PROPERTY_LANGUAGE and this is taken into account when clustering.

I couldn't delete previously attached files. This ZIP file contains only the patch and svnstat -- you'll have to remove a few JARs manually and replace other with their new counterparts from the ZIP file I've attached to this issue earlier (they haven't changed). Let me know if you need anything.


> Carrot2 clustering plugin upgrade.
> ----------------------------------
>
>          Key: NUTCH-237
>          URL: http://issues.apache.org/jira/browse/NUTCH-237
>      Project: Nutch
>         Type: Improvement

>     Reporter: Dawid Weiss
>     Priority: Trivial
>  Attachments: NUTCH-237.DWEISS.patch.zip, c2.patch, libs.zip, svn-stat.txt
>
> This is an upgrade of the clustering plugin to the newest release (1.0.2).

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira