Incremental crawl again ... (Please explain)

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Incremental crawl again ... (Please explain)

zzcgiacomini
I am currently using the last nightly nutch-0.8-dev build and
I am really confused about how to proceed after I have done two
different "whole web" incremental crawl

The tutorial to me is not clear on how to merge the results after the
two crawls in order to be able to
make a search operation.

Could some one please give me an Hints on what is the right procedure ?!
here is what I am doing:

1. create an initial urls file  /tmp/dmoz/urls.txt
2. hadoop dfs -put /tmp/urls/ url
3. nutch inject test/crawldb dmoz
4. nutch generate test/crawldb test/segments
5. nutch fetch test/segments/20060522144050
6. nutch updatedb test/crawldb   test/segments/20060522144050
7. nutch invertlinks linkdb test/segments/20060522144050
8. nutch  index  test/indexes test/crawldb linkdb
test/segments/20060522144050
 
..and now I am able to search...

Now I run

 9. nutch generate test/crawldb test/segments -topN 1000

and I will end up to have a new segment  :   test/segments/20060522151957

10. nutch fetch test/segments/20060522151957
11. nutch updatedb test/crawldb test/segments/20060522151957


From this point on I cannot make any progresses much

A) I have tried to merge the two segments into a new one with the idea to rerun an invertlinks and index  on it but:

   nutch mergesegs test/segments -dir test/segments  

   whatever I specify as outputdir or outputsegment I get errors

B) I have also tried to make invertlinks on all test/segments with the goal to run nutch index command to produce a second
   indexes directory, let say test/indexes1, an finally run the merge index on index2

   nutch invertlinks  test/linkdb  -dir test/segments

   This as created a new linkdb directory *NOT* under test as specified but as <user>/linkdb-1108390519

   nutch index  test/indexes1 test/crawldb linkdb test/segments/20060522144050
   nutch merge index2 test/indexes test/indexes1

   now I am not sure what to do; If I rename test/index2 to be test/indexes after having removed test/indexes
   I will not able to search anymore.


-Corrado
   


Reply | Threaded
Open this post in threaded view
|

Re: Incremental crawl again ... (Please explain)

sudhendra seshachala
Please do follow the link below..
  http://www.mail-archive.com/nutch-user@.../msg03990.html
   
  I have been able to follow the threads as explained and merge multiple crawl.. It works like a champ.
   
  Thanks
  Sudhi

zzcgiacomini <[hidden email]> wrote:
  I am currently using the last nightly nutch-0.8-dev build and
I am really confused about how to proceed after I have done two
different "whole web" incremental crawl

The tutorial to me is not clear on how to merge the results after the
two crawls in order to be able to
make a search operation.

Could some one please give me an Hints on what is the right procedure ?!
here is what I am doing:

1. create an initial urls file /tmp/dmoz/urls.txt
2. hadoop dfs -put /tmp/urls/ url
3. nutch inject test/crawldb dmoz
4. nutch generate test/crawldb test/segments
5. nutch fetch test/segments/20060522144050
6. nutch updatedb test/crawldb test/segments/20060522144050
7. nutch invertlinks linkdb test/segments/20060522144050
8. nutch index test/indexes test/crawldb linkdb
test/segments/20060522144050

..and now I am able to search...

Now I run

9. nutch generate test/crawldb test/segments -topN 1000

and I will end up to have a new segment : test/segments/20060522151957

10. nutch fetch test/segments/20060522151957
11. nutch updatedb test/crawldb test/segments/20060522151957


From this point on I cannot make any progresses much

A) I have tried to merge the two segments into a new one with the idea to rerun an invertlinks and index on it but:

nutch mergesegs test/segments -dir test/segments

whatever I specify as outputdir or outputsegment I get errors

B) I have also tried to make invertlinks on all test/segments with the goal to run nutch index command to produce a second
indexes directory, let say test/indexes1, an finally run the merge index on index2

nutch invertlinks test/linkdb -dir test/segments

This as created a new linkdb directory *NOT* under test as specified but as /linkdb-1108390519

nutch index test/indexes1 test/crawldb linkdb test/segments/20060522144050
nutch merge index2 test/indexes test/indexes1

now I am not sure what to do; If I rename test/index2 to be test/indexes after having removed test/indexes
I will not able to search anymore.


-Corrado






  Sudhi Seshachala
  http://sudhilogs.blogspot.com/
   


 __________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 
Reply | Threaded
Open this post in threaded view
|

Re: Incremental crawl again ... (Please explain)

Jacob Brunson
I looked at the referenced messaged at
http://www.mail-archive.com/nutch-user@.../msg03990.html
but I am still having problems.

I am running the latest checkout from subversion.

These are the commands which I've run:
bin/nutch crawl myurls/ -dir crawl -threads 4 -depth 3 -topN 10000
bin/nutch generate crawl/crawldb crawl/segments -topN 500
lastsegment=`ls -d crawl/segments/2* | tail -1`
bin/nutch fetch $lastsegment
bin/nutch updatedb crawl/crawldb $lastsegment
bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb $lastsegment

This last command fails with a java.io.IOException saying: "Output
directory /home/nutch/nutch/crawl/indexes already exists"

So I'm confused because it seems like I did exactly what was described
in the referenced email, but it didn't work for me.  Can someone help
me figure out what I'm doing wrong or what I need to do instead?
Thanks,
Jacob


On 5/22/06, sudhendra seshachala <[hidden email]> wrote:

> Please do follow the link below..
>   http://www.mail-archive.com/nutch-user@.../msg03990.html
>
>   I have been able to follow the threads as explained and merge multiple crawl.. It works like a champ.
>
>   Thanks
>   Sudhi
>
> zzcgiacomini <[hidden email]> wrote:
>   I am currently using the last nightly nutch-0.8-dev build and
> I am really confused about how to proceed after I have done two
> different "whole web" incremental crawl
>
> The tutorial to me is not clear on how to merge the results after the
> two crawls in order to be able to
> make a search operation.
>
> Could some one please give me an Hints on what is the right procedure ?!
> here is what I am doing:
>
> 1. create an initial urls file /tmp/dmoz/urls.txt
> 2. hadoop dfs -put /tmp/urls/ url
> 3. nutch inject test/crawldb dmoz
> 4. nutch generate test/crawldb test/segments
> 5. nutch fetch test/segments/20060522144050
> 6. nutch updatedb test/crawldb test/segments/20060522144050
> 7. nutch invertlinks linkdb test/segments/20060522144050
> 8. nutch index test/indexes test/crawldb linkdb
> test/segments/20060522144050
>
> ..and now I am able to search...
>
> Now I run
>
> 9. nutch generate test/crawldb test/segments -topN 1000
>
> and I will end up to have a new segment : test/segments/20060522151957
>
> 10. nutch fetch test/segments/20060522151957
> 11. nutch updatedb test/crawldb test/segments/20060522151957
>
>
> From this point on I cannot make any progresses much
>
> A) I have tried to merge the two segments into a new one with the idea to rerun an invertlinks and index on it but:
>
> nutch mergesegs test/segments -dir test/segments
>
> whatever I specify as outputdir or outputsegment I get errors
>
> B) I have also tried to make invertlinks on all test/segments with the goal to run nutch index command to produce a second
> indexes directory, let say test/indexes1, an finally run the merge index on index2
>
> nutch invertlinks test/linkdb -dir test/segments
>
> This as created a new linkdb directory *NOT* under test as specified but as /linkdb-1108390519
>
> nutch index test/indexes1 test/crawldb linkdb test/segments/20060522144050
> nutch merge index2 test/indexes test/indexes1
>
> now I am not sure what to do; If I rename test/index2 to be test/indexes after having removed test/indexes
> I will not able to search anymore.
>
>
> -Corrado
>
>
>
>
>
>
>   Sudhi Seshachala
>   http://sudhilogs.blogspot.com/
>
>
>
>  __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>


--
http://JacobBrunson.com
Reply | Threaded
Open this post in threaded view
|

Re: Incremental crawl again ... (Please explain)

Zaheed Haque
On 5/25/06, Jacob Brunson <[hidden email]> wrote:
> I looked at the referenced messaged at
> http://www.mail-archive.com/nutch-user@.../msg03990.html
> but I am still having problems.
>
> I am running the latest checkout from subversion.
>
> These are the commands which I've run:

> bin/nutch crawl myurls/ -dir crawl -threads 4 -depth 3 -topN 10000

bin/nutch crawl - is a one shot command to fetch/generate/index a
nutch index. I would NOT recommend one to use this one shot command.

Please take the long route which will give you more control over your
tasks. The long route meaning - inject, generate, fetch, updatedb,
index, dedup, merge. Please see the following -
Whole web crawling...

http://lucene.apache.org/nutch/tutorial8.html#Whole-web+Crawling

Cheers

> bin/nutch generate crawl/crawldb crawl/segments -topN 500
> lastsegment=`ls -d crawl/segments/2* | tail -1`
> bin/nutch fetch $lastsegment
> bin/nutch updatedb crawl/crawldb $lastsegment
> bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb $lastsegment
>
> This last command fails with a java.io.IOException saying: "Output
> directory /home/nutch/nutch/crawl/indexes already exists"
>
> So I'm confused because it seems like I did exactly what was described
> in the referenced email, but it didn't work for me.  Can someone help
> me figure out what I'm doing wrong or what I need to do instead?
> Thanks,
> Jacob
>
>
> On 5/22/06, sudhendra seshachala <[hidden email]> wrote:
> > Please do follow the link below..
> >   http://www.mail-archive.com/nutch-user@.../msg03990.html
> >
> >   I have been able to follow the threads as explained and merge multiple crawl.. It works like a champ.
> >
> >   Thanks
> >   Sudhi
> >
> > zzcgiacomini <[hidden email]> wrote:
> >   I am currently using the last nightly nutch-0.8-dev build and
> > I am really confused about how to proceed after I have done two
> > different "whole web" incremental crawl
> >
> > The tutorial to me is not clear on how to merge the results after the
> > two crawls in order to be able to
> > make a search operation.
> >
> > Could some one please give me an Hints on what is the right procedure ?!
> > here is what I am doing:
> >
> > 1. create an initial urls file /tmp/dmoz/urls.txt
> > 2. hadoop dfs -put /tmp/urls/ url
> > 3. nutch inject test/crawldb dmoz
> > 4. nutch generate test/crawldb test/segments
> > 5. nutch fetch test/segments/20060522144050
> > 6. nutch updatedb test/crawldb test/segments/20060522144050
> > 7. nutch invertlinks linkdb test/segments/20060522144050
> > 8. nutch index test/indexes test/crawldb linkdb
> > test/segments/20060522144050
> >
> > ..and now I am able to search...
> >
> > Now I run
> >
> > 9. nutch generate test/crawldb test/segments -topN 1000
> >
> > and I will end up to have a new segment : test/segments/20060522151957
> >
> > 10. nutch fetch test/segments/20060522151957
> > 11. nutch updatedb test/crawldb test/segments/20060522151957
> >
> >
> > From this point on I cannot make any progresses much
> >
> > A) I have tried to merge the two segments into a new one with the idea to rerun an invertlinks and index on it but:
> >
> > nutch mergesegs test/segments -dir test/segments
> >
> > whatever I specify as outputdir or outputsegment I get errors
> >
> > B) I have also tried to make invertlinks on all test/segments with the goal to run nutch index command to produce a second
> > indexes directory, let say test/indexes1, an finally run the merge index on index2
> >
> > nutch invertlinks test/linkdb -dir test/segments
> >
> > This as created a new linkdb directory *NOT* under test as specified but as /linkdb-1108390519
> >
> > nutch index test/indexes1 test/crawldb linkdb test/segments/20060522144050
> > nutch merge index2 test/indexes test/indexes1
> >
> > now I am not sure what to do; If I rename test/index2 to be test/indexes after having removed test/indexes
> > I will not able to search anymore.
> >
> >
> > -Corrado
> >
> >
> >
> >
> >
> >
> >   Sudhi Seshachala
> >   http://sudhilogs.blogspot.com/
> >
> >
> >
> >  __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam protection around
> > http://mail.yahoo.com
> >
>
>
> --
> http://JacobBrunson.com
>
Reply | Threaded
Open this post in threaded view
|

Re: Incremental crawl again ... (Please explain)

Jacob Brunson
Addition comments and testcase below.

On 5/25/06, Zaheed Haque <[hidden email]> wrote:

> On 5/25/06, Jacob Brunson <[hidden email]> wrote:
> > I looked at the referenced messaged at
> > http://www.mail-archive.com/nutch-user@.../msg03990.html
> > but I am still having problems.
> >
> > I am running the latest checkout from subversion.
> >
> > These are the commands which I've run:
>
> > bin/nutch crawl myurls/ -dir crawl -threads 4 -depth 3 -topN 10000
>
> bin/nutch crawl - is a one shot command to fetch/generate/index a
> nutch index. I would NOT recommend one to use this one shot command.
Thats funny because when I look at the source code for crawling, it
does pretty much the same thing as the "whole web crawling" method.
>
> Please take the long route which will give you more control over your
> tasks. The long route meaning - inject, generate, fetch, updatedb,
> index, dedup, merge. Please see the following -
> Whole web crawling...
>
> http://lucene.apache.org/nutch/tutorial8.html#Whole-web+Crawling
>
Yes, I've gone though that tutorial also and followed it and I'm
having the same problem.  The tutorial does not describe how to add to
the original index.  If you can help me figure out this, I would be
glad to add to the tutorial and make it more complete.

Just to be perfectly clear, these are the complete set of steps I take
to get the error.  (I'm running Java1.5, only the urls/ directory
exists at the beginning):
$ svn update
$ ant
$ bin/nutch inject crawl.test/crawldb urls/
$ bin/nutch generate crawl.test/crawldb crawl.test/segments -topN 20
$ lastsegment=`ls -d crawl.test/segments/2* | tail -1`
$ bin/nutch fetch $lastsegment
$ bin/nutch updatedb crawl.test/crawldb $lastsegment
$ bin/nutch invertlinks crawl.test/linkdb $lastsegment
$ bin/nutch index crawl.test/indexes crawl.test/crawldb
crawl.test/linkdb $lastsegment
$ bin/nutch merge crawl.test/index crawl.test/indexes
$ bin/nutch generate crawl.test/crawldb crawl.test/segments -topN 20
$ lastsegment=`ls -d crawl.test/segments/2* | tail -1`
$ bin/nutch fetch $lastsegment
$ bin/nutch updatedb crawl.test/crawldb $lastsegment
$ bin/nutch invertlinks crawl.test/linkdb $lastsegment
$ bin/nutch index crawl.test/indexes crawl.test/crawldb
crawl.test/linkdb $lastsegment

And at this point, I have my problem.  I get the following output:
060525 171327 Indexer: adding segment: crawl.test/segments/20060525165518
Exception in thread "main" java.io.IOException: Output directory
/home/nutch/nutch/crawl.test/indexes already exists.
        at org.apache.hadoop.mapred.OutputFormatBase.checkOutputSpecs(OutputFormatBase.java:37)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:263)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:311)
        at org.apache.nutch.indexer.Indexer.index(Indexer.java:287)
        at org.apache.nutch.indexer.Indexer.main(Indexer.java:304)

So if you could help me figure out what I need to do differently, I
would be sure to update the tutorial on the on the wiki to help others
who might have the same problems as me.
Thanks,
Jacob
Reply | Threaded
Open this post in threaded view
|

Re: Incremental crawl again ... (Please explain)

zzcgiacomini
In reply to this post by Jacob Brunson
I am not at all a Nutch expert, I am just experimenting a little bit,
but  as far as I understood it
you can remove the indexes directory and re-index again the segments:
In may case ofter step 8 of the (see below) I have only one segment :
test/segments/20060522144050
after step 9 I will have a second segment
test/segments/20060522144050
Now what we can do is to remove the test/indexes directory and
re-index the two segments:
this what I did :
 
hadoop dfs -rm test/indexes
nutch index test/indexes test/crawldb linkdb
test/segments/20060522144050 test/segments/20060522144050

Hope it helps

-Corrqdo



Jacob Brunson wrote:

> I looked at the referenced messaged at
> http://www.mail-archive.com/nutch-user@.../msg03990.html
> but I am still having problems.
>
> I am running the latest checkout from subversion.
>
> These are the commands which I've run:
> bin/nutch crawl myurls/ -dir crawl -threads 4 -depth 3 -topN 10000
> bin/nutch generate crawl/crawldb crawl/segments -topN 500
> lastsegment=`ls -d crawl/segments/2* | tail -1`
> bin/nutch fetch $lastsegment
> bin/nutch updatedb crawl/crawldb $lastsegment
> bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb $lastsegment
>
> This last command fails with a java.io.IOException saying: "Output
> directory /home/nutch/nutch/crawl/indexes already exists"
>
> So I'm confused because it seems like I did exactly what was described
> in the referenced email, but it didn't work for me.  Can someone help
> me figure out what I'm doing wrong or what I need to do instead?
> Thanks,
> Jacob
>
>
> On 5/22/06, sudhendra seshachala <[hidden email]> wrote:
>> Please do follow the link below..
>>   http://www.mail-archive.com/nutch-user@.../msg03990.html
>>
>>   I have been able to follow the threads as explained and merge
>> multiple crawl.. It works like a champ.
>>
>>   Thanks
>>   Sudhi
>>
>> zzcgiacomini <[hidden email]> wrote:
>>   I am currently using the last nightly nutch-0.8-dev build and
>> I am really confused about how to proceed after I have done two
>> different "whole web" incremental crawl
>>
>> The tutorial to me is not clear on how to merge the results after the
>> two crawls in order to be able to
>> make a search operation.
>>
>> Could some one please give me an Hints on what is the right procedure ?!
>> here is what I am doing:
>>
>> 1. create an initial urls file /tmp/dmoz/urls.txt
>> 2. hadoop dfs -put /tmp/urls/ url
>> 3. nutch inject test/crawldb dmoz
>> 4. nutch generate test/crawldb test/segments
>> 5. nutch fetch test/segments/20060522144050
>> 6. nutch updatedb test/crawldb test/segments/20060522144050
>> 7. nutch invertlinks linkdb test/segments/20060522144050
>> 8. nutch index test/indexes test/crawldb linkdb
>> test/segments/20060522144050
>>
>> ..and now I am able to search...
>>
>> Now I run
>>
>> 9. nutch generate test/crawldb test/segments -topN 1000
>>
>> and I will end up to have a new segment : test/segments/20060522151957
>>
>> 10. nutch fetch test/segments/20060522151957
>> 11. nutch updatedb test/crawldb test/segments/20060522151957
>>
>>
>> From this point on I cannot make any progresses much
>>
>> A) I have tried to merge the two segments into a new one with the
>> idea to rerun an invertlinks and index on it but:
>>
>> nutch mergesegs test/segments -dir test/segments
>>
>> whatever I specify as outputdir or outputsegment I get errors
>>
>> B) I have also tried to make invertlinks on all test/segments with
>> the goal to run nutch index command to produce a second
>> indexes directory, let say test/indexes1, an finally run the merge
>> index on index2
>>
>> nutch invertlinks test/linkdb -dir test/segments
>>
>> This as created a new linkdb directory *NOT* under test as specified
>> but as /linkdb-1108390519
>>
>> nutch index test/indexes1 test/crawldb linkdb
>> test/segments/20060522144050
>> nutch merge index2 test/indexes test/indexes1
>>
>> now I am not sure what to do; If I rename test/index2 to be
>> test/indexes after having removed test/indexes
>> I will not able to search anymore.
>>
>>
>> -Corrado
>>
>>
>>
>>
>>
>>
>>   Sudhi Seshachala
>>   http://sudhilogs.blogspot.com/
>>
>>
>>
>>  __________________________________________________
>> Do You Yahoo!?
>> Tired of spam?  Yahoo! Mail has the best spam protection around
>> http://mail.yahoo.com
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Incremental crawl again ... (Please explain)

Jacob Brunson
Yes, I see what you mean about re-indexing again over all the
segments.  However, indexing takes a lot of time and I was hoping that
merging many smaller indexes would be a much more efficient method.
Besides, deleting the index and re-indexing just doesn't seem like
*The Right Thing(tm)*.

On 5/26/06, zzcgiacomini <[hidden email]> wrote:

> I am not at all a Nutch expert, I am just experimenting a little bit,
> but  as far as I understood it
> you can remove the indexes directory and re-index again the segments:
> In may case ofter step 8 of the (see below) I have only one segment :
> test/segments/20060522144050
> after step 9 I will have a second segment
> test/segments/20060522144050
> Now what we can do is to remove the test/indexes directory and
> re-index the two segments:
> this what I did :
>
> hadoop dfs -rm test/indexes
> nutch index test/indexes test/crawldb linkdb
> test/segments/20060522144050 test/segments/20060522144050
>
> Hope it helps
>
> -Corrqdo
>
>
>
> Jacob Brunson wrote:
> > I looked at the referenced messaged at
> > http://www.mail-archive.com/nutch-user@.../msg03990.html
> > but I am still having problems.
> >
> > I am running the latest checkout from subversion.
> >
> > These are the commands which I've run:
> > bin/nutch crawl myurls/ -dir crawl -threads 4 -depth 3 -topN 10000
> > bin/nutch generate crawl/crawldb crawl/segments -topN 500
> > lastsegment=`ls -d crawl/segments/2* | tail -1`
> > bin/nutch fetch $lastsegment
> > bin/nutch updatedb crawl/crawldb $lastsegment
> > bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb $lastsegment
> >
> > This last command fails with a java.io.IOException saying: "Output
> > directory /home/nutch/nutch/crawl/indexes already exists"
> >
> > So I'm confused because it seems like I did exactly what was described
> > in the referenced email, but it didn't work for me.  Can someone help
> > me figure out what I'm doing wrong or what I need to do instead?
> > Thanks,
> > Jacob
> >
> >
> > On 5/22/06, sudhendra seshachala <[hidden email]> wrote:
> >> Please do follow the link below..
> >>   http://www.mail-archive.com/nutch-user@.../msg03990.html
> >>
> >>   I have been able to follow the threads as explained and merge
> >> multiple crawl.. It works like a champ.
> >>
> >>   Thanks
> >>   Sudhi
> >>
> >> zzcgiacomini <[hidden email]> wrote:
> >>   I am currently using the last nightly nutch-0.8-dev build and
> >> I am really confused about how to proceed after I have done two
> >> different "whole web" incremental crawl
> >>
> >> The tutorial to me is not clear on how to merge the results after the
> >> two crawls in order to be able to
> >> make a search operation.
> >>
> >> Could some one please give me an Hints on what is the right procedure ?!
> >> here is what I am doing:
> >>
> >> 1. create an initial urls file /tmp/dmoz/urls.txt
> >> 2. hadoop dfs -put /tmp/urls/ url
> >> 3. nutch inject test/crawldb dmoz
> >> 4. nutch generate test/crawldb test/segments
> >> 5. nutch fetch test/segments/20060522144050
> >> 6. nutch updatedb test/crawldb test/segments/20060522144050
> >> 7. nutch invertlinks linkdb test/segments/20060522144050
> >> 8. nutch index test/indexes test/crawldb linkdb
> >> test/segments/20060522144050
> >>
> >> ..and now I am able to search...
> >>
> >> Now I run
> >>
> >> 9. nutch generate test/crawldb test/segments -topN 1000
> >>
> >> and I will end up to have a new segment : test/segments/20060522151957
> >>
> >> 10. nutch fetch test/segments/20060522151957
> >> 11. nutch updatedb test/crawldb test/segments/20060522151957
> >>
> >>
> >> From this point on I cannot make any progresses much
> >>
> >> A) I have tried to merge the two segments into a new one with the
> >> idea to rerun an invertlinks and index on it but:
> >>
> >> nutch mergesegs test/segments -dir test/segments
> >>
> >> whatever I specify as outputdir or outputsegment I get errors
> >>
> >> B) I have also tried to make invertlinks on all test/segments with
> >> the goal to run nutch index command to produce a second
> >> indexes directory, let say test/indexes1, an finally run the merge
> >> index on index2
> >>
> >> nutch invertlinks test/linkdb -dir test/segments
> >>
> >> This as created a new linkdb directory *NOT* under test as specified
> >> but as /linkdb-1108390519
> >>
> >> nutch index test/indexes1 test/crawldb linkdb
> >> test/segments/20060522144050
> >> nutch merge index2 test/indexes test/indexes1
> >>
> >> now I am not sure what to do; If I rename test/index2 to be
> >> test/indexes after having removed test/indexes
> >> I will not able to search anymore.
> >>
> >>
> >> -Corrado
> >>
> >>
> >>
> >>
> >>
> >>
> >>   Sudhi Seshachala
> >>   http://sudhilogs.blogspot.com/
> >>
> >>
> >>
> >>  __________________________________________________
> >> Do You Yahoo!?
> >> Tired of spam?  Yahoo! Mail has the best spam protection around
> >> http://mail.yahoo.com
> >>
> >
> >
>
>


--
http://JacobBrunson.com
Reply | Threaded
Open this post in threaded view
|

Re: Incremental crawl again ... (Please explain)

Stefan Neufeind
I haven't yet tried - but could you maybe:
- move the new segments somewhere independent of the existing ones
- create a separate linkdb for it (to my understanding the linkdb is
only needed when indexing)
- create a separate index on that
- then move segment into segments-dir and new index into indexes-dir as
"part-XXXX"
- just merge indexes (should work relatively fast)

In the long term your segments, indexes etc. add up - so in this case
you'd need to maybe think about merging segments etc.

Also, this is "only" my current understanding of the topic. It would be
nice to get feedback and maybe easier solutions from others as well.



Regards,
 Stefan

Jacob Brunson wrote:

> Yes, I see what you mean about re-indexing again over all the
> segments.  However, indexing takes a lot of time and I was hoping that
> merging many smaller indexes would be a much more efficient method.
> Besides, deleting the index and re-indexing just doesn't seem like
> *The Right Thing(tm)*.
>
> On 5/26/06, zzcgiacomini <[hidden email]> wrote:
>> I am not at all a Nutch expert, I am just experimenting a little bit,
>> but  as far as I understood it
>> you can remove the indexes directory and re-index again the segments:
>> In may case ofter step 8 of the (see below) I have only one segment :
>> test/segments/20060522144050
>> after step 9 I will have a second segment
>> test/segments/20060522144050
>> Now what we can do is to remove the test/indexes directory and
>> re-index the two segments:
>> this what I did :
>>
>> hadoop dfs -rm test/indexes
>> nutch index test/indexes test/crawldb linkdb
>> test/segments/20060522144050 test/segments/20060522144050
>>
>> Hope it helps
>>
>> -Corrqdo
>>
>>
>>
>> Jacob Brunson wrote:
>> > I looked at the referenced messaged at
>> > http://www.mail-archive.com/nutch-user@.../msg03990.html
>> > but I am still having problems.
>> >
>> > I am running the latest checkout from subversion.
>> >
>> > These are the commands which I've run:
>> > bin/nutch crawl myurls/ -dir crawl -threads 4 -depth 3 -topN 10000
>> > bin/nutch generate crawl/crawldb crawl/segments -topN 500
>> > lastsegment=`ls -d crawl/segments/2* | tail -1`
>> > bin/nutch fetch $lastsegment
>> > bin/nutch updatedb crawl/crawldb $lastsegment
>> > bin/nutch index crawl/indexes crawl/crawldb crawl/linkdb $lastsegment
>> >
>> > This last command fails with a java.io.IOException saying: "Output
>> > directory /home/nutch/nutch/crawl/indexes already exists"
>> >
>> > So I'm confused because it seems like I did exactly what was described
>> > in the referenced email, but it didn't work for me.  Can someone help
>> > me figure out what I'm doing wrong or what I need to do instead?
>> > Thanks,
>> > Jacob
>> >
>> >
>> > On 5/22/06, sudhendra seshachala <[hidden email]> wrote:
>> >> Please do follow the link below..
>> >>  
>> http://www.mail-archive.com/nutch-user@.../msg03990.html
>> >>
>> >>   I have been able to follow the threads as explained and merge
>> >> multiple crawl.. It works like a champ.
>> >>
>> >>   Thanks
>> >>   Sudhi
>> >>
>> >> zzcgiacomini <[hidden email]> wrote:
>> >>   I am currently using the last nightly nutch-0.8-dev build and
>> >> I am really confused about how to proceed after I have done two
>> >> different "whole web" incremental crawl
>> >>
>> >> The tutorial to me is not clear on how to merge the results after the
>> >> two crawls in order to be able to
>> >> make a search operation.
>> >>
>> >> Could some one please give me an Hints on what is the right
>> procedure ?!
>> >> here is what I am doing:
>> >>
>> >> 1. create an initial urls file /tmp/dmoz/urls.txt
>> >> 2. hadoop dfs -put /tmp/urls/ url
>> >> 3. nutch inject test/crawldb dmoz
>> >> 4. nutch generate test/crawldb test/segments
>> >> 5. nutch fetch test/segments/20060522144050
>> >> 6. nutch updatedb test/crawldb test/segments/20060522144050
>> >> 7. nutch invertlinks linkdb test/segments/20060522144050
>> >> 8. nutch index test/indexes test/crawldb linkdb
>> >> test/segments/20060522144050
>> >>
>> >> ..and now I am able to search...
>> >>
>> >> Now I run
>> >>
>> >> 9. nutch generate test/crawldb test/segments -topN 1000
>> >>
>> >> and I will end up to have a new segment : test/segments/20060522151957
>> >>
>> >> 10. nutch fetch test/segments/20060522151957
>> >> 11. nutch updatedb test/crawldb test/segments/20060522151957
>> >>
>> >>
>> >> From this point on I cannot make any progresses much
>> >>
>> >> A) I have tried to merge the two segments into a new one with the
>> >> idea to rerun an invertlinks and index on it but:
>> >>
>> >> nutch mergesegs test/segments -dir test/segments
>> >>
>> >> whatever I specify as outputdir or outputsegment I get errors
>> >>
>> >> B) I have also tried to make invertlinks on all test/segments with
>> >> the goal to run nutch index command to produce a second
>> >> indexes directory, let say test/indexes1, an finally run the merge
>> >> index on index2
>> >>
>> >> nutch invertlinks test/linkdb -dir test/segments
>> >>
>> >> This as created a new linkdb directory *NOT* under test as specified
>> >> but as /linkdb-1108390519
>> >>
>> >> nutch index test/indexes1 test/crawldb linkdb
>> >> test/segments/20060522144050
>> >> nutch merge index2 test/indexes test/indexes1
>> >>
>> >> now I am not sure what to do; If I rename test/index2 to be
>> >> test/indexes after having removed test/indexes
>> >> I will not able to search anymore.
Reply | Threaded
Open this post in threaded view
|

Re: Incremental crawl again ... (Please explain)

zzcgiacomini
I just tried this and it looks is working:

nutch index test/indexes1 test/crawldb linkdb test/segments/20060522181136
nutch index test/indexes2 test/crawldb linkdb test/segments/20060522181136
nutch merge test/index test/indexes1 test/indexes2

querying also works, I have setup searcher.dir in nutch-site.xml as "test"
and used  the following line to query :
/opt/nutch-0.8-dev/bin/nutch org.apache.nutch.searcher.NutchBean computer

I am just experimenting, so I do not know if is the right way to do things

-Corrado


Stefan Neufeind wrote:

> I haven't yet tried - but could you maybe:
> - move the new segments somewhere independent of the existing ones
> - create a separate linkdb for it (to my understanding the linkdb is
> only needed when indexing)
> - create a separate index on that
> - then move segment into segments-dir and new index into indexes-dir as
> "part-XXXX"
> - just merge indexes (should work relatively fast)
>
> In the long term your segments, indexes etc. add up - so in this case
> you'd need to maybe think about merging segments etc.
>
> Also, this is "only" my current understanding of the topic. It would be
> nice to get feedback and maybe easier solutions from others as well.
>
>
>
> Regards,
>  Stefan
>
>