mapred: config parameters

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

mapred: config parameters

Michael Nebel
Hi,

the last days I gave the mapred-branch a try and I was impressed!

But I still have a problem with the incremental crawling. My setup: I
have 4 boxes (1x namenode/jobtracker - 3x datanode/tasktracker). Running
one round of "crawling" consists out of the steps:

- generate (I set a limit of "-topN 10000000")
- fetch
- update
- index
- invertlinks

For the first round, I injected a list of about 20.000 websites. When
running nutch, I expected, that the fetcher would be pretty busy and
went for a coffee. Ok: perhaps someone talked to wife and decided, I
should not drink so much coffee. But I think, I made a mistake.. But
after 100 URLs he stopped working.

After some tweaking I got the installation to fetch about 10.000 pages,
but this is still not what I expect. First guess was the url-filter, but
I see the urls in the tasktracker log. I looked at the mailinglist and
got many ideas, but I still get more confused.

I think, the following parameters have an influence on the number of
pages fetched (in the brackets are the values I selected):

- mapred.map.tasks     (100)
- mapred.reduce.tasks   (3)
- mapred.task.timeout   (3600000 [an other question])
- mapred.tasktracker.tasks.maximum (10)
- fetcher.threads.fetch (100)
- fetcher.server.delay (5.0)
- fetcher.threads.per.host        (10)
- generate.max.per.host (1000)
- http.content.limit                    (2000000)

I don't like my parameters, but so I got the most results. Looking at
the jobmanager, each "map task" fetched between 70 - 100 pages. Having
100 map.tasks: I have ~ 8000 new pages fetched in the end. That's nearly
the number the crawldb says too.

Which parameter has an influence on the number of pages ONE task
fetches. By my observations, I would guess it's "fetcher.threads.fetch".
Increasing this number further means, to blast the load on the
tasktrackers. So there must be an other problem.

Any help appreciated!

Regards

        Michael

--
Michael Nebel
http://www.nebel.de/
http://www.netluchs.de/

Reply | Threaded
Open this post in threaded view
|

Re: mapred: config parameters

Gal Nitzan
Hi Michael,

this question should be asked in the nutch-users list.

Take a look at a thread: So many Unfetched Pages using MapReduce

G.

On Tue, 2006-01-31 at 15:52 +0100, Michael Nebel wrote:

> Hi,
>
> the last days I gave the mapred-branch a try and I was impressed!
>
> But I still have a problem with the incremental crawling. My setup: I
> have 4 boxes (1x namenode/jobtracker - 3x datanode/tasktracker). Running
> one round of "crawling" consists out of the steps:
>
> - generate (I set a limit of "-topN 10000000")
> - fetch
> - update
> - index
> - invertlinks
>
> For the first round, I injected a list of about 20.000 websites. When
> running nutch, I expected, that the fetcher would be pretty busy and
> went for a coffee. Ok: perhaps someone talked to wife and decided, I
> should not drink so much coffee. But I think, I made a mistake.. But
> after 100 URLs he stopped working.
>
> After some tweaking I got the installation to fetch about 10.000 pages,
> but this is still not what I expect. First guess was the url-filter, but
> I see the urls in the tasktracker log. I looked at the mailinglist and
> got many ideas, but I still get more confused.
>
> I think, the following parameters have an influence on the number of
> pages fetched (in the brackets are the values I selected):
>
> - mapred.map.tasks     (100)
> - mapred.reduce.tasks   (3)
> - mapred.task.timeout   (3600000 [an other question])
> - mapred.tasktracker.tasks.maximum (10)
> - fetcher.threads.fetch (100)
> - fetcher.server.delay (5.0)
> - fetcher.threads.per.host        (10)
> - generate.max.per.host (1000)
> - http.content.limit                    (2000000)
>
> I don't like my parameters, but so I got the most results. Looking at
> the jobmanager, each "map task" fetched between 70 - 100 pages. Having
> 100 map.tasks: I have ~ 8000 new pages fetched in the end. That's nearly
> the number the crawldb says too.
>
> Which parameter has an influence on the number of pages ONE task
> fetches. By my observations, I would guess it's "fetcher.threads.fetch".
> Increasing this number further means, to blast the load on the
> tasktrackers. So there must be an other problem.
>
> Any help appreciated!
>
> Regards
>
> Michael
>