Next Nutch release

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Next Nutch release

Sami Siren-2
Hello,

It has been a while from a previous release (0.8.1) and looking at the
great fixes done in trunk I'd start thinking about baking a new release
soon.

Looking at the jira roadmaps there are 1 blocking issues (fixing the
license headers) for 0.8.2 and two other blocking issues for 0.9.0 of
which I think NUTCH-233 is safe to put in.

The top 10 voted issues are currently:

NUTCH-61   Adaptive re-fetch interval. Detecting umodified content
NUTCH-48 "Did you mean" query enhancement/refignment feature
NUTCH-251 Administration GUI
NUTCH-289 CrawlDatum should store IP address
NUTCH-36 Chinese in Nutch
NUTCH-185 XMLParser is configurable xml parser plugin. NUTCH-59 meta
data support in webdb
NUTCH-92 DistributedSearch incorrectly scores results NUTCH-68 A
tool to generate arbitrary fetchlists NUTCH-87 Efficient
site-specific crawling for a large number of sites

Are there any opinions about issues that should go in before the next
release (Answering yes means that you are willing to provide a patch for
it).

--
 Sami Siren
Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Andrzej Białecki-2
Sami Siren wrote:

> Hello,
>
> It has been a while from a previous release (0.8.1) and looking at the
> great fixes done in trunk I'd start thinking about baking a new release
> soon.
>
> Looking at the jira roadmaps there are 1 blocking issues (fixing the
> license headers) for 0.8.2 and two other blocking issues for 0.9.0 of
> which I think NUTCH-233 is safe to put in.
>  

Agreed. The replacement regex mentioned in the original comment seems
safe enough, and simpler.

> The top 10 voted issues are currently:
>
> NUTCH-61   Adaptive re-fetch interval. Detecting umodified content
>  

Well ... I'm of a split mind on this. I can bring this patch up to date
and apply it before 0.9.0, if we understand that this is a "0" release
... ;) Otherwise I'd prefer to wait with it right after the release.

I would like also to proceed with NUTCH-339 (Fetcher2 patches + plus
some changes I made in the meantime), since I'd like to expose the new
fetcher to a broader audience, and it doesn't affect the existing
implementation.


> NUTCH-48 "Did you mean" query enhancement/refignment feature
> NUTCH-251 Administration GUI
> NUTCH-289 CrawlDatum should store IP address
>  

I'm still not entirely convinced about this - and there is already a
mechanism in place to support it if someone really wishes to keep this
particular info (CrawlDatum.metaData).

> NUTCH-36 Chinese in Nutch
> NUTCH-185 XMLParser is configurable xml parser plugin. NUTCH-59 meta
> data support in webdb
> NUTCH-92 DistributedSearch incorrectly scores results NUTCH-68

This is too intrusive to fix just before the release - and needs
additional discussion.


> NUTCH-68 A
> tool to generate arbitrary fetchlists

Easy to port this to 0.9.0 - I can do this.


> NUTCH-87 Efficient
> site-specific crawling for a large number of sites
>  



--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

thomasasta
In reply to this post by Sami Siren-2
Sami, Thanks a lot,

I would like to see a feature in, that a link to a webpage is sowing all areay indexed urls.

So other spiders can fetch this site and get the urls, the open souce natuch has already to provide.


So we need to start not to have open source coding the machine, but as well every node offering an open, downloadable database of urls,

And we need a list of urls, of other nutch domains. With this list, each Nutch can crawl the urls of the other nutch  providing them on a website.

As Million of urls are a lot, I suggest to have 26 websites from a-z to display all urls of the `word´ "a", all 25 urls links b-z as well on the page of the word-page "a".

then several Nutch nodes could use a small p2p feature and as well the sister yacy can fetch the urls from a central open source point: all nutch domains.

Would this be possible to generate a webpage-link somewhere on the nutch-homepage of the individual serverinstall with all urls?

Opensource has to found solidarity, so make the nutch url database open for as well open source search engine spiders from central points.

thanks

-------- Original-Nachricht --------
Datum: Tue, 16 Jan 2007 17:53:41 +0200
Von: Sami Siren <[hidden email]>
An: [hidden email]
Betreff: Next Nutch release

> Hello,
>
> It has been a while from a previous release (0.8.1) and looking at the
> great fixes done in trunk I'd start thinking about baking a new release
> soon.
>
> Looking at the jira roadmaps there are 1 blocking issues (fixing the
> license headers) for 0.8.2 and two other blocking issues for 0.9.0 of
> which I think NUTCH-233 is safe to put in.
>
> The top 10 voted issues are currently:
>
> NUTCH-61   Adaptive re-fetch interval. Detecting umodified content
> NUTCH-48 "Did you mean" query enhancement/refignment feature
> NUTCH-251 Administration GUI
> NUTCH-289 CrawlDatum should store IP address
> NUTCH-36 Chinese in Nutch
> NUTCH-185 XMLParser is configurable xml parser plugin. NUTCH-59 meta
> data support in webdb
> NUTCH-92 DistributedSearch incorrectly scores results NUTCH-68 A
> tool to generate arbitrary fetchlists NUTCH-87 Efficient
> site-specific crawling for a large number of sites
>
> Are there any opinions about issues that should go in before the next
> release (Answering yes means that you are willing to provide a patch for
> it).
>
> --
>  Sami Siren

--
"Feel free" - 5 GB Mailbox, 50 FreeSMS/Monat ...
Jetzt GMX ProMail testen: http://www.gmx.net/de/go/promail
Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

chrismattmann
In reply to this post by Andrzej Białecki-2
Folks,

 When would you like to make the release? I've been working on NUTCH-185,
but got a bit bogged down with other work. If there is interest in having
NUTCH-185 included in the release, I could make a push to get out a patch by
week's end...

 As for the rest, my +1 for NUTCH-61 being included sooner rather than
later. It seems that the patch has garnered enough use and attention that
folks would like to see it in the release. I think the email from the user
trying to manage a terabyte of data a few days back was particularly
telling.

Cheers,
  Chris



On 1/16/07 8:19 AM, "Andrzej Bialecki" <[hidden email]> wrote:

> Sami Siren wrote:
>> Hello,
>>
>> It has been a while from a previous release (0.8.1) and looking at the
>> great fixes done in trunk I'd start thinking about baking a new release
>> soon.
>>
>> Looking at the jira roadmaps there are 1 blocking issues (fixing the
>> license headers) for 0.8.2 and two other blocking issues for 0.9.0 of
>> which I think NUTCH-233 is safe to put in.
>>  
>
> Agreed. The replacement regex mentioned in the original comment seems
> safe enough, and simpler.
>
>> The top 10 voted issues are currently:
>>
>> NUTCH-61    Adaptive re-fetch interval. Detecting umodified content
>>  
>
> Well ... I'm of a split mind on this. I can bring this patch up to date
> and apply it before 0.9.0, if we understand that this is a "0" release
> ... ;) Otherwise I'd prefer to wait with it right after the release.
>
> I would like also to proceed with NUTCH-339 (Fetcher2 patches + plus
> some changes I made in the meantime), since I'd like to expose the new
> fetcher to a broader audience, and it doesn't affect the existing
> implementation.
>
>
>> NUTCH-48  "Did you mean" query enhancement/refignment feature
>> NUTCH-251  Administration GUI
>> NUTCH-289  CrawlDatum should store IP address
>>  
>
> I'm still not entirely convinced about this - and there is already a
> mechanism in place to support it if someone really wishes to keep this
> particular info (CrawlDatum.metaData).
>
>> NUTCH-36  Chinese in Nutch
>> NUTCH-185  XMLParser is configurable xml parser plugin.   NUTCH-59  meta
>> data support in webdb
>> NUTCH-92  DistributedSearch incorrectly scores results   NUTCH-68  
>
> This is too intrusive to fix just before the release - and needs
> additional discussion.
>
>
>> NUTCH-68 A
>> tool to generate arbitrary fetchlists  
>
> Easy to port this to 0.9.0 - I can do this.
>
>
>> NUTCH-87  Efficient
>> site-specific crawling for a large number of sites
>>  
>
>


Reply | Threaded
Open this post in threaded view
|

RE: Next Nutch release

Alan Tanaman
In reply to this post by Andrzej Białecki-2
All,

+5 on NUTCH-61
So far, we have been trying to use this patch with partial success on 0.8.1.
We would be happy to help with work on updating/testing this.

Obviously we are hardly impartial, and we would also like to have NUTCH-422
(index-extra plugin) incorporated (although we are aware that we still have
some cleanup to do and the provision of junit tests).

We have done some further work on NUTCH-185 (XMLParser is configurable xml
parser plugin), but haven't posted as yet because the work is perhaps too
highly-customized (we generate fields automatically without any need to
configure a specific Xpath).  We are still deliberating over the desired
configuration to do this without conflicting with those implementations
where it is necessary to specify which fields go into the index.

Apart from these, we would find the following candidates, which we hope to
use/work on very soon (but perhaps not soon enough for this release), very
useful:

NUTCH-48 "Did you mean" query enhancement/refinement feature
NUTCH-251 Administration GUI
NUTCH-36 Chinese in Nutch
NUTCH-92 DistributedSearch incorrectly scores results

Best regards,
Alan
_________________________
Alan Tanaman
iDNA Solutions
http://blog.idna-solutions.com

-----Original Message-----
From: Andrzej Bialecki [mailto:[hidden email]]
Sent: 16 January 2007 16:19
To: [hidden email]
Subject: Re: Next Nutch release

Sami Siren wrote:

> Hello,
>
> It has been a while from a previous release (0.8.1) and looking at the
> great fixes done in trunk I'd start thinking about baking a new release
> soon.
>
> Looking at the jira roadmaps there are 1 blocking issues (fixing the
> license headers) for 0.8.2 and two other blocking issues for 0.9.0 of
> which I think NUTCH-233 is safe to put in.
>  

Agreed. The replacement regex mentioned in the original comment seems
safe enough, and simpler.

> The top 10 voted issues are currently:
>
> NUTCH-61   Adaptive re-fetch interval. Detecting umodified content
>  

Well ... I'm of a split mind on this. I can bring this patch up to date
and apply it before 0.9.0, if we understand that this is a "0" release
... ;) Otherwise I'd prefer to wait with it right after the release.

I would like also to proceed with NUTCH-339 (Fetcher2 patches + plus
some changes I made in the meantime), since I'd like to expose the new
fetcher to a broader audience, and it doesn't affect the existing
implementation.


> NUTCH-48 "Did you mean" query enhancement/refignment feature
> NUTCH-251 Administration GUI
> NUTCH-289 CrawlDatum should store IP address
>  

I'm still not entirely convinced about this - and there is already a
mechanism in place to support it if someone really wishes to keep this
particular info (CrawlDatum.metaData).

> NUTCH-36 Chinese in Nutch
> NUTCH-185 XMLParser is configurable xml parser plugin.
NUTCH-59 meta
> data support in webdb
> NUTCH-92 DistributedSearch incorrectly scores results
NUTCH-68

This is too intrusive to fix just before the release - and needs
additional discussion.


> NUTCH-68 A
> tool to generate arbitrary fetchlists

Easy to port this to 0.9.0 - I can do this.


> NUTCH-87 Efficient
> site-specific crawling for a large number of sites
>  



--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com



Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Enis Soztutar
In reply to this post by Sami Siren-2
Hi all, for NUTCH-251:

I suppose that NUTCH-251 is relatively a significant issue by the votes.
Stafan has written a good plugin for the admin gui and i have updated it
to work with nutch-0.8, hadoop 0.4.

Some of the features in the patch is not appropriate for our use cases
and it requires hadoop changes, thus I am currently working on an
alternative implementation of the administration gui, which runs a
hadoop server( like JobTraker) to listen to submitted Jobs, an web Gui
to submit and track the jobs from the browser and a job runner.

The architechture details of the patch is as follows :

  - An interface AdminJob which is an abstract class representing a Job
in nutch.
  - various classes extending AdminJob. for ex FetchAdminJob, IndexAdminJob.
  - A queue which sorts the jobs in priority order, by a modified a
topological sort(jobs can be dependent).
  - an interface to submit Jobs
  - a rpc server to listen to job submissions
  - an extension point (basically same as the previous)
  - a web server to serve plugin jsp's

upon the features will be
    - submitting jobs from code, command line or web interface,
    - tracking jobs from the command line or web interface
    - scheduling jobs

I could send the code or details if anyone is interested in pretesting.
And i will appreciate any comments and suggestions on this. I am
planning to complete the patch and submit it to Jira ASAP.

Sami Siren wrote:

> Hello,
>
> It has been a while from a previous release (0.8.1) and looking at the
> great fixes done in trunk I'd start thinking about baking a new release
> soon.
>
> Looking at the jira roadmaps there are 1 blocking issues (fixing the
> license headers) for 0.8.2 and two other blocking issues for 0.9.0 of
> which I think NUTCH-233 is safe to put in.
>
> The top 10 voted issues are currently:
>
> NUTCH-61   Adaptive re-fetch interval. Detecting umodified content
> NUTCH-48 "Did you mean" query enhancement/refignment feature
> NUTCH-251 Administration GUI
> NUTCH-289 CrawlDatum should store IP address
> NUTCH-36 Chinese in Nutch
> NUTCH-185 XMLParser is configurable xml parser plugin. NUTCH-59 meta
> data support in webdb
> NUTCH-92 DistributedSearch incorrectly scores results NUTCH-68 A
> tool to generate arbitrary fetchlists NUTCH-87 Efficient
> site-specific crawling for a large number of sites
>
> Are there any opinions about issues that should go in before the next
> release (Answering yes means that you are willing to provide a patch for
> it).
>
> --
>  Sami Siren
>
>  

Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Sami Siren-2
2007/1/17, Enis Soztutar <[hidden email]>:
>
> Hi all, for NUTCH-251:
>
> I suppose that NUTCH-251 is relatively a significant issue by the votes.
> Stafan has written a good plugin for the admin gui and i have updated it
> to work with nutch-0.8, hadoop 0.4.


Good to hear someone is working on that! Why not target it to
trunk version of Nutch?

>  - a web server to serve plugin jsp's

Why not make it regular war? also please consider making a clean
separation of view/logic when you implement the web ui.

--
 Sami Siren
Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Sami Siren-2
In reply to this post by Andrzej Białecki-2
>
> > The top 10 voted issues are currently:
> >
> > NUTCH-61       Adaptive re-fetch interval. Detecting umodified content
> >
>
> Well ... I'm of a split mind on this. I can bring this patch up to date
> and apply it before 0.9.0, if we understand that this is a "0" release
> ... ;) Otherwise I'd prefer to wait with it right after the release.


+1 for putting it in after 0.9.0

I would like also to proceed with NUTCH-339 (Fetcher2 patches + plus
> some changes I made in the meantime), since I'd like to expose the new
> fetcher to a broader audience, and it doesn't affect the existing
> implementation.


+1 for putting it in before 0.9.0


> NUTCH-48      "Did you mean" query enhancement/refignment feature
> > NUTCH-251     Administration GUI
> > NUTCH-289     CrawlDatum should store IP address
> >
>
> I'm still not entirely convinced about this - and there is already a
> mechanism in place to support it if someone really wishes to keep this
> particular info (CrawlDatum.metaData).
>
> > NUTCH-36      Chinese in Nutch
> > NUTCH-185     XMLParser is configurable xml parser
> plugin.            NUTCH-59        meta
> > data support in webdb
> > NUTCH-92      DistributedSearch incorrectly scores
> results            NUTCH-68
>
> This is too intrusive to fix just before the release - and needs
> additional discussion.


+1

> NUTCH-68      A
> > tool to generate arbitrary fetchlists
>
> Easy to port this to 0.9.0 - I can do this.


cool.


I'll start working on the headers and stuff to get the blocking issue away.

--
 Sami Siren
Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Enis Soztutar
In reply to this post by Sami Siren-2
Sami Siren wrote:

> 2007/1/17, Enis Soztutar <[hidden email]>:
>>
>> Hi all, for NUTCH-251:
>>
>> I suppose that NUTCH-251 is relatively a significant issue by the votes.
>> Stafan has written a good plugin for the admin gui and i have updated it
>> to work with nutch-0.8, hadoop 0.4.
>
>
> Good to hear someone is working on that! Why not target it to
> trunk version of Nutch?
It is targetted to the trunk already. The previous was targetted to
nutch-0.8, hadoop 0.4, since back then that versions was the latest in
the trunk
>
>>  - a web server to serve plugin jsp's
>
> Why not make it regular war? also please consider making a clean
> separation of view/logic when you implement the web ui.
As Stafan's version used embedded Jetty server, I continued this way.
But i will consider that possibility also.

>
> --
> Sami Siren
>

Reply | Threaded
Open this post in threaded view
|

RE: Next Nutch release

Armel T. Nene-2
Hi guys,

 

I have been working on NUTCH-61 Adaptive re-fetch interval. Detecting
unmodified content applying it to Nutch 0.8.1. Here are some points:

 

1.    This feature is great for Nutch to have has it differentiate between
modified and unmodified content, therefore not indexing twice even if the
document fetch time has arrived.

a.    There are some performance issues here. Even with this patch, Nutch
still fetches the content and then checks its status against the last
modified time in the database. If it has to check for a 1000 files before
indexing the following 10 files, this will cause a real problem for those
that are after real time indexing.

 

2.    Since, I applied this patch to Nutch 0.8.1, when I try to parse xml
files with our modified version of the xmlparser /indexer plugin; the
fetcher throws the following exception:

 

WARN  fetcher.Fetcher - Error parsing:
file:/C:/880254/8802_583254_20051006_12.xml: failed(2,200):
java.lang.IllegalStateException: Root element not set

 

The system will not hang or crash but the xml file will be indexed without
any generated fields. The plugins works fine without the patch. I have
another parser that parses graphics and other formats that fails when used
with the patch. So far this problem occurs when using the file protocol.

 

3.    the patch works fine when indexing web site using the http protocol.

 

I am willing to work with Andrzej to make it stable as I understand it's the
architect of this patch. I have the possibility of testing it in a mix
environment in our computer lab. This patch can be the stepping stone for
other features such real time indexing and fetch queue for index updating as
opposed to creating a new index each time.

 

Best Regards,

 

Armel

 

-------------------------------------------------

Armel T. Nene

iDNA Solutions

Tel: +44 (207) 257 6124

Mobile: +44 (788) 695 0483

http://blog.idna-solutions.com

-----Original Message-----
From: Enis Soztutar [mailto:[hidden email]]
Sent: 17 January 2007 15:39
To: [hidden email]
Subject: Re: Next Nutch release

 

Sami Siren wrote:

> 2007/1/17, Enis Soztutar <[hidden email]>:

>>

>> Hi all, for NUTCH-251:

>>

>> I suppose that NUTCH-251 is relatively a significant issue by the votes.

>> Stafan has written a good plugin for the admin gui and i have updated it

>> to work with nutch-0.8, hadoop 0.4.

>

>

> Good to hear someone is working on that! Why not target it to

> trunk version of Nutch?

It is targetted to the trunk already. The previous was targetted to

nutch-0.8, hadoop 0.4, since back then that versions was the latest in

the trunk

>

>>  - a web server to serve plugin jsp's

>

> Why not make it regular war? also please consider making a clean

> separation of view/logic when you implement the web ui.

As Stafan's version used embedded Jetty server, I continued this way.

But i will consider that possibility also.

 

>

> --

> Sami Siren

>

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Andrzej Białecki-2
Armel T. Nene wrote:
> I am willing to work with Andrzej to make it stable as I understand it's the
> architect of this patch. I have the possibility of testing it in a mix
> environment in our computer lab. This patch can be the stepping stone for
> other features such real time indexing and fetch queue for index updating as
> opposed to creating a new index each time.
>  

Thanks for taking the initiative! I'll be glad to review the patch and
apply it right after the 0.9 release. The best way to keep the process
open would be to make svn diff and attach this new version of the patch
to the JIRA issue.

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Stefan Groschupf
In reply to this post by Enis Soztutar
Hi,

great to hear people still working on things. It shows once more  
getting something in early would save some effort. :)
Just some random comments.

We run the gui in several production environemnts with patched hadoop  
code - since this is from our point of view the clean approach.  
Everything else feels like a workaround to fix some strange hadoop  
behaviors. It is may be a long time ago that I spoke to Doug and some  
other Hadoop developers  but at this time I understand people that  
there is a general interest to have a nutch gui and support required  
functionality in hadoop.
I'm not sure if that is still the case or if I had a wrong impression.
In any case from my p.o.v. the clean way would be getting the  
required minor changes into hadoop (not critical simple stuff from my  
point of view) instead of implement working around in nutch. Since  
hadoop is a kind of child of nutch there should be a close relation  
at least to discuss things.
Anyway no strong option, just my 2 cents. In any case I'm very happy  
if people see now the need for a gui as well and someone is working  
on that since I'm kind of busy with other projects.

Thanks.
Stefan


On 17.01.2007, at 06:42, Enis Soztutar wrote:

> Hi all, for NUTCH-251:
>
> I suppose that NUTCH-251 is relatively a significant issue by the  
> votes. Stafan has written a good plugin for the admin gui and i  
> have updated it to work with nutch-0.8, hadoop 0.4.
>
> Some of the features in the patch is not appropriate for our use  
> cases and it requires hadoop changes, thus I am currently working  
> on an alternative implementation of the administration gui, which  
> runs a hadoop server( like JobTraker) to listen to submitted Jobs,  
> an web Gui to submit and track the jobs from the browser and a job  
> runner.
>
> The architechture details of the patch is as follows :
>
>  - An interface AdminJob which is an abstract class representing a  
> Job in nutch.
>  - various classes extending AdminJob. for ex FetchAdminJob,  
> IndexAdminJob.
>  - A queue which sorts the jobs in priority order, by a modified a  
> topological sort(jobs can be dependent).
>  - an interface to submit Jobs
>  - a rpc server to listen to job submissions
>  - an extension point (basically same as the previous)
>  - a web server to serve plugin jsp's
>
> upon the features will be
>    - submitting jobs from code, command line or web interface,
>    - tracking jobs from the command line or web interface
>    - scheduling jobs
>
> I could send the code or details if anyone is interested in  
> pretesting. And i will appreciate any comments and suggestions on  
> this. I am planning to complete the patch and submit it to Jira ASAP.
>
> Sami Siren wrote:
>> Hello,
>>
>> It has been a while from a previous release (0.8.1) and looking at  
>> the
>> great fixes done in trunk I'd start thinking about baking a new  
>> release
>> soon.
>>
>> Looking at the jira roadmaps there are 1 blocking issues (fixing the
>> license headers) for 0.8.2 and two other blocking issues for 0.9.0 of
>> which I think NUTCH-233 is safe to put in.
>>
>> The top 10 voted issues are currently:
>>
>> NUTCH-61   Adaptive re-fetch interval. Detecting umodified content
>> NUTCH-48 "Did you mean" query enhancement/refignment feature
>> NUTCH-251 Administration GUI
>> NUTCH-289 CrawlDatum should store IP address
>> NUTCH-36 Chinese in Nutch
>> NUTCH-185 XMLParser is configurable xml parser plugin. NUTCH-59  
>> meta
>> data support in webdb
>> NUTCH-92 DistributedSearch incorrectly scores results NUTCH-68 A
>> tool to generate arbitrary fetchlists NUTCH-87 Efficient
>> site-specific crawling for a large number of sites
>>
>> Are there any opinions about issues that should go in before the next
>> release (Answering yes means that you are willing to provide a  
>> patch for
>> it).
>>
>> --
>>  Sami Siren
>>
>>
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California
http://www.101tec.com



Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

scott green
In reply to this post by Enis Soztutar
Hi,

I just finished reading all source code about nutch gui. And
personally i don't like putting a lot of code snippets into jsp files
since it takes a lot time when refactoring. So how about to adopt
using velocity/freemarker with servlet?

On 1/17/07, Enis Soztutar <[hidden email]> wrote:

> Hi all, for NUTCH-251:
>
> I suppose that NUTCH-251 is relatively a significant issue by the votes.
> Stafan has written a good plugin for the admin gui and i have updated it
> to work with nutch-0.8, hadoop 0.4.
>
> Some of the features in the patch is not appropriate for our use cases
> and it requires hadoop changes, thus I am currently working on an
> alternative implementation of the administration gui, which runs a
> hadoop server( like JobTraker) to listen to submitted Jobs, an web Gui
> to submit and track the jobs from the browser and a job runner.
>
> The architechture details of the patch is as follows :
>
>   - An interface AdminJob which is an abstract class representing a Job
> in nutch.
>   - various classes extending AdminJob. for ex FetchAdminJob, IndexAdminJob.
>   - A queue which sorts the jobs in priority order, by a modified a
> topological sort(jobs can be dependent).
>   - an interface to submit Jobs
>   - a rpc server to listen to job submissions
>   - an extension point (basically same as the previous)
>   - a web server to serve plugin jsp's
>
> upon the features will be
>     - submitting jobs from code, command line or web interface,
>     - tracking jobs from the command line or web interface
>     - scheduling jobs
>
> I could send the code or details if anyone is interested in pretesting.
> And i will appreciate any comments and suggestions on this. I am
> planning to complete the patch and submit it to Jira ASAP.
>
> Sami Siren wrote:
> > Hello,
> >
> > It has been a while from a previous release (0.8.1) and looking at the
> > great fixes done in trunk I'd start thinking about baking a new release
> > soon.
> >
> > Looking at the jira roadmaps there are 1 blocking issues (fixing the
> > license headers) for 0.8.2 and two other blocking issues for 0.9.0 of
> > which I think NUTCH-233 is safe to put in.
> >
> > The top 10 voted issues are currently:
> >
> > NUTCH-61       Adaptive re-fetch interval. Detecting umodified content
> > NUTCH-48      "Did you mean" query enhancement/refignment feature
> > NUTCH-251     Administration GUI
> > NUTCH-289     CrawlDatum should store IP address
> > NUTCH-36      Chinese in Nutch
> > NUTCH-185     XMLParser is configurable xml parser plugin.            NUTCH-59        meta
> > data support in webdb
> > NUTCH-92      DistributedSearch incorrectly scores results            NUTCH-68        A
> > tool to generate arbitrary fetchlists                 NUTCH-87        Efficient
> > site-specific crawling for a large number of sites
> >
> > Are there any opinions about issues that should go in before the next
> > release (Answering yes means that you are willing to provide a patch for
> > it).
> >
> > --
> >  Sami Siren
> >
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Stefan Groschupf
Hi,
> I just finished reading all source code about nutch gui. And
> personally i don't like putting a lot of code snippets into jsp files
> since it takes a lot time when refactoring. So how about to adopt
> using velocity/freemarker with servlet?


In general I agree it is the view layer and should have as less as  
possible code, however the idea was to have as less as possible  
dependencies to thirdparty tools and libraries and also getting  
things realized with low tech (jsp).

Stefan



Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Doug Cutting
In reply to this post by Stefan Groschupf
Stefan Groschupf wrote:
> We run the gui in several production environemnts with patched hadoop
> code - since this is from our point of view the clean approach.
> Everything else feels like a workaround to fix some strange hadoop
> behaviors.

Are there issues in Hadoop's Jira for these?  If so, do they have
patches attached?  Are they linked to the corresponding issue in Nutch?

Doug
Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

scott green
In reply to this post by Stefan Groschupf
Stefan,

I also dived into contrib/web2 in nutch. The one and admin-gui are
both owns some plugins based on nutch plugin architecture. So I think
it is great if we extract something in high level and they should have
a lot commons.  Well, i dont know it is the right time to do this job.

On 1/19/07, Stefan Groschupf <[hidden email]> wrote:

> Hi,
> > I just finished reading all source code about nutch gui. And
> > personally i don't like putting a lot of code snippets into jsp files
> > since it takes a lot time when refactoring. So how about to adopt
> > using velocity/freemarker with servlet?
>
>
> In general I agree it is the view layer and should have as less as
> possible code, however the idea was to have as less as possible
> dependencies to thirdparty tools and libraries and also getting
> things realized with low tech (jsp).
>
> Stefan
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Stefan Groschupf
In reply to this post by Doug Cutting
Th old hadoop patch is here:
https://issues.apache.org/jira/browse/NUTCH-251
Also we had this conversation:
http://www.mail-archive.com/hadoop-dev@.../msg00314.html
I guess after this we missed to post the patches we use internally.

If someone feels strong about getting the gui working with hadoop he/
she should feel free to update the patch and post it in the hadoop jira.

Stefan







On 18.01.2007, at 15:39, Doug Cutting wrote:

> Stefan Groschupf wrote:
>> We run the gui in several production environemnts with patched  
>> hadoop code - since this is from our point of view the clean  
>> approach. Everything else feels like a workaround to fix some  
>> strange hadoop behaviors.
>
> Are there issues in Hadoop's Jira for these?  If so, do they have  
> patches attached?  Are they linked to the corresponding issue in  
> Nutch?
>
> Doug
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California
http://www.101tec.com



Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Stefan Groschupf
In reply to this post by scott green
Hi Scott,

feel free - I have no options on that.

 From my very little point of view the nutch > .8 source stream is a  
one way street.
In all my projects we move as far as possible away from nutch. I like  
hadoop a lot and writing customer tools on top of it is - that easy.
But nutch .8 was a proof of concept for the early hadoop.  There is  
only one serious developer left and wow how great he does his job -  
but nutch >.8 is just too monolithic, to difficult to extend, to  
difficult to debug, to difficult to integrate for a serious mission  
critical application.
I spend a signification part of my life daily working with nutch, but  
if someone would ask - I would answer don't use it.
May be one day we can get some developer together first think about a  
good extendable design and than start a 2.x stream or a new project.
And ... yes no opic and yes definitely no plugin architecture (I feel  
very sorry for all that wast so much life time because of my terrible  
complicate plugin system) but a clean IOC design with lightweight  
default interface implementations and a great test coverage.
Anyway just my *very little* point of view based on 3.5 years nutch  
experience.

Stefan





On 18.01.2007, at 21:33, Scott Green wrote:

> Stefan,
>
> I also dived into contrib/web2 in nutch. The one and admin-gui are
> both owns some plugins based on nutch plugin architecture. So I think
> it is great if we extract something in high level and they should have
> a lot commons.  Well, i dont know it is the right time to do this job.
>
> On 1/19/07, Stefan Groschupf <[hidden email]> wrote:
>> Hi,
>> > I just finished reading all source code about nutch gui. And
>> > personally i don't like putting a lot of code snippets into jsp  
>> files
>> > since it takes a lot time when refactoring. So how about to adopt
>> > using velocity/freemarker with servlet?
>>
>>
>> In general I agree it is the view layer and should have as less as
>> possible code, however the idea was to have as less as possible
>> dependencies to thirdparty tools and libraries and also getting
>> things realized with low tech (jsp).
>>
>> Stefan
>>
>>
>>
>>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California
http://www.101tec.com



Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Andrzej Białecki-2
Stefan Groschupf wrote:

> Hi Scott,
>
> feel free - I have no options on that.
>
> From my very little point of view the nutch > .8 source stream is a
> one way street.
> In all my projects we move as far as possible away from nutch. I like
> hadoop a lot and writing customer tools on top of it is - that easy.
> But nutch .8 was a proof of concept for the early hadoop.  There is
> only one serious developer left and wow how great he does his job -
> but nutch >.8 is just too monolithic, to difficult to extend, to
> difficult to debug, to difficult to integrate for a serious mission
> critical application.
> I spend a signification part of my life daily working with nutch, but
> if someone would ask - I would answer don't use it.

Let me comment on what you said:

* I guess that for each case where Nutch seems inappropriate I could
give you a counter-example of Nutch being used  commercially with much
success. I guess it depends on a particular application and the type of
customer.

* no doubt Nutch has its warts - the plugin system could be simpler, for
example ;) but hey, it's great that we have a plugin system at all! It
would be easier now to refactor Nutch to use a different plugin system
than it was to go from the completely monolithic design to the plugin
system ... As with any open source project - if you don't like it, fix
it and contribute the fix.

* things won't happen magically unless there is a greater involvement of
skilled developers. "One way road" - well, with limited resources that
this project has at the moment the only way is to gradually improve, we
cannot afford to abandon the current codebase and start from scratch.
Are you willing to spend the time and do the required refactoring?
Anyone else?

> May be one day we can get some developer together first think about a
> good extendable design and than start a 2.x stream or a new project.

I hope so too. But as Steve B. said once, what we need is "developers,
developers, developers ..." ;)

Nutch, as it is now, is not too well-focused, so that may be the reason
why it doesn't attract too many developers - and casual users find it
perhaps too difficult to get interested enough to dig deeper. On one end
of spectrum we have small desktop installations in mind, on the other
end we have scalable 1 bln page server farms ... it's hard to satisfy
everyone, and the current design is not that satisfactory for either
group. So, I think a better focus is needed, combined with design that
satisfies either one or the other group - or maybe two designs for each
group, assuming we can motivate enough people to participate in each
sub-project.

> And ... yes no opic and yes definitely no plugin architecture (I feel
> very sorry for all that wast so much life time

Ah, the more I study the theory behind PageRank calculation the more I
think OPIC is an excellent solution to this hard problem - but our
current implementation is broken. I'm slowly coming to a point where I
should be able to fix it - but let's not throw out the baby with the
water ...

> because of my terrible complicate plugin system) but a clean IOC
> design with lightweight default interface implementations and a great
> test coverage.
> Anyway just my *very little* point of view based on 3.5 years nutch
> experience.
I'm looking forward to your patches that implement the clean IOC design ;) Seriously - if you can show how to refactor a portion of Nutch to a clean IOC design, we will start refactoring the rest of it in this direction.

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Reply | Threaded
Open this post in threaded view
|

Re: Next Nutch release

Stefan Groschupf
Hi Andrzej,

thank you for taking the time to comment, I highly value your comments.

> * I guess that for each case where Nutch seems inappropriate I  
> could give you a counter-example of Nutch being used  commercially  
> with much success. I guess it depends on a particular application  
> and the type of customer.

Yes, it would be interesting to hear who use nutch .8 _successfully_  
in production.

> * no doubt Nutch has its warts - the plugin system could be  
> simpler, for example ;) but hey, it's great that we have a plugin  
> system at all! It would be easier now to refactor Nutch to use a  
> different plugin system than it was to go from the completely  
> monolithic design to the plugin system ... As with any open source  
> project - if you don't like it, fix it and contribute the fix.

Sure - I tried that more than once - but I do not want to start this  
discussion again.

> * things won't happen magically unless there is a greater  
> involvement of skilled developers. "One way road" - well, with  
> limited resources that this project has at the moment the only way  
> is to gradually improve, we cannot afford to abandon the current  
> codebase and start from scratch.

I agree - the problem are skilled developers, I remember more than  
one offer of different companies to dedicate developers to the  
project, but looks like there was no interest.

> Are you willing to spend the time and do the required refactoring?  
> Anyone else?

In general there was some emotional discussion about API changes.  
Since nutch is a 0.x and also a software and not a library more  
frequent refactorings had may be improved the maintainability of the  
code over the time.

Sure if we start a 2.x branch and if I'm not developing for the trash  
or "jira nirvana", I can imaging to contribute. I would rethink and  
rewrite some major parts (e.g. remove the reusage of objects with a  
complex states and endless if than else conditions no body can debug)  
and I believe that is not difficult. I'm not talking about the  
algorithm stuff here.

>> May be one day we can get some developer together first think  
>> about a good extendable design and than start a 2.x stream or a  
>> new project.
>
> I hope so too. But as Steve B. said once, what we need is  
> "developers, developers, developers ..." ;)

I agree, however it must be attractive for developers to spend time  
in a open source project. We saw many developers here. You are the  
only one left that does some serious development and I can't find  
words how much respect I have for your work. You are the only one  
that is able to fix serious bugs.

It is very less attractive to developers spending weeks to find a bug  
like the regular expression one. Than such a bug sits there for month  
in the jira being rejected. Sure if nobody of the contributors run  
nutch with a 500 mio url web db, than it might be difficult to  
reproduce such a bug. If you have a set of a such issues (another one  
is the gui etc.) you decide to run your very own  nutch brunch in  
your home svn. At least all of my customers did over the time. The  
result - no public nutch contributions, no developers.


> Nutch, as it is now, is not too well-focused, so that may be the  
> reason why it doesn't attract too many developers - and casual  
> users find it perhaps too difficult to get interested enough to dig  
> deeper.

I agree that is another issue, since nutch tries to solve to many  
problems at the same time the code is to difficult to understand for  
newbies.


> On one end of spectrum we have small desktop installations in mind,  
> on the other end we have scalable 1 bln page server farms ... it's  
> hard to satisfy everyone, and the current design is not that  
> satisfactory for either group. So, I think a better focus is  
> needed, combined with design that satisfies either one or the other  
> group - or maybe two designs for each group, assuming we can  
> motivate enough people to participate in each sub-project.

Sounds like a good idea! :-)

>> And ... yes no opic and yes definitely no plugin architecture (I  
>> feel very sorry for all that wast so much life time
>
> Ah, the more I study the theory behind PageRank calculation the  
> more I think OPIC is an excellent solution to this hard problem -  
> but our current implementation is broken.

Yes - very much, a search engine that need to recrawl from scratch  
each time to get sense-fully index scores - that is really broken.
However at least the "page rank" implementation in nutch .7 worked  
great for me, it just didn't scaled that well.


> I'm slowly coming to a point where I should be able to fix it - but  
> let's not throw out the baby with the water ...
Wow, I hold my finger crossed!

>> because of my terrible complicate plugin system) but a clean IOC  
>> design with lightweight default interface implementations and a  
>> great test coverage.
>> Anyway just my *very little* point of view based on 3.5 years  
>> nutch experience.
> I'm looking forward to your patches that implement the clean IOC  
> design ;) Seriously - if you can show how to refactor a portion of  
> Nutch to a clean IOC design, we will start refactoring the rest of  
> it in this direction.

No - sorry I'm personal too tiered doing patches. I already talked to  
a set of people and at least 2 good developers would be serious  
interested in writing a search engine from scratch based on hadoop by  
"reusing" as much nutch code as sense-fully. Another 2 would be  
interested. All those people worked with nutch or working in the IR  
research area or in vertical search companies. May be a interesting  
starting point for a nice summer project.
We might even find some company that would sponsor some work - I know  
at least 2 that would be interested. You might know one as well. :-)

Anyway don't count on that - I don't know - at least in the moment I  
can imaging more interesting things in my spear time than doing nutch  
patches.


I don't want to start a emotional discussion here, however talking  
about the problem in public might help.
Cheers,
Stefan

12