Whole-web crawling with the mapreduce branch

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Whole-web crawling with the mapreduce branch

Steffen Viken Valvåg
Hi,

I'm playing around with the mapreduce branch, and got it working for a
simple intranet crawl by following the nutch tutorial on
http://lucene.apache.org/nutch/tutorial.html.  The tutorial seems
inapplicable when it comes to whole-web crawling, though, as the "nutch
admin" command has been disabled, and the usage of the "nutch inject"
command seems to have changed.  I'm willing to read the source to get up to
speed, but if there is any other documentation on the mapreduce branch that
would obviously be helpful.  I would also greatly appreciate it if someone
took the time to give me a short bullet list of commands to get me started
on a whole-web crawl.

Thanks,
Steffen

Reply | Threaded
Open this post in threaded view
|

Re: Whole-web crawling with the mapreduce branch

Doug Cutting-2
For now, look at the source for crawl/Crawl.java.

I'll try to add some documentation ASAP.

Doug

Steffen Viken Valv?g wrote:

> Hi,
>
> I'm playing around with the mapreduce branch, and got it working for a
> simple intranet crawl by following the nutch tutorial on
> http://lucene.apache.org/nutch/tutorial.html.  The tutorial seems
> inapplicable when it comes to whole-web crawling, though, as the "nutch
> admin" command has been disabled, and the usage of the "nutch inject"
> command seems to have changed.  I'm willing to read the source to get up to
> speed, but if there is any other documentation on the mapreduce branch that
> would obviously be helpful.  I would also greatly appreciate it if someone
> took the time to give me a short bullet list of commands to get me started
> on a whole-web crawl.
>
> Thanks,
> Steffen
>
Reply | Threaded
Open this post in threaded view
|

RE: Whole-web crawling with the mapreduce branch

Steffen Viken Valvåg
Thanks,

That got me going.  Works like a charm :)

Steffen

-----Original Message-----
From: Doug Cutting [mailto:[hidden email]]
Sent: 15. september 2005 23:48
To: [hidden email]
Subject: Re: Whole-web crawling with the mapreduce branch

For now, look at the source for crawl/Crawl.java.

I'll try to add some documentation ASAP.

Doug

Steffen Viken Valvåg wrote:
> Hi,
>
> I'm playing around with the mapreduce branch, and got it working for a
> simple intranet crawl by following the nutch tutorial on
> http://lucene.apache.org/nutch/tutorial.html.  The tutorial seems
> inapplicable when it comes to whole-web crawling, though, as the
> "nutch admin" command has been disabled, and the usage of the "nutch
inject"
> command seems to have changed.  I'm willing to read the source to get
> up to speed, but if there is any other documentation on the mapreduce
> branch that would obviously be helpful.  I would also greatly
> appreciate it if someone took the time to give me a short bullet list
> of commands to get me started on a whole-web crawl.
>
> Thanks,
> Steffen
>