Re: language identifier

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Re: language identifier

Jérôme Charron
After some long nights on benching and profiling the language
identifier plugin, I just attach a new patch to language identifier
plugin on Jira (http://issues.apache.org/jira/browse/NUTCH-60).
This patch provides some configuration options that enable to specify
the size of the data to use for language analysis and the NGrams sizes
to uses.
It also provides some optimizations that reduce the processing time
from 70% to 20%, depending on the configuration (size of data to
process), with an average gain of 25%.
I will provides more detailled results of my benchs on the Wiki as
soon as possible
(http://wiki.apache.org/nutch/LanguageIdentifierBenchs) and some
possible ways of improvements on
http://wiki.apache.org/nutch/NewLanguageIdentifier.

Jerome

--
http://motrech.free.fr/
http://frutch.free.fr/