Interleaved (parallel) fetch cycles

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Interleaved (parallel) fetch cycles

Andrzej Białecki-2
Hi,

I'm planning to work on adding support in 0.8 for interleaved fetch cycles.

What this means is that (within some limits) you can generate multiple
fetchlists, fetch them at different times, and then update the crawldb
not necessarily in the original sequence as they were generated. You can
also generate more fetchlists before any updatedb is run.

This functionality was supported in 0.7.x. When FetchListTool selected a
Page for fetching, its next fetch time was pushed 1 week in the future.
This was a simple and effective way to prevent the same Pages ending up
on the next fetchlist, but at the same time to have their waiting "time
out" after 1 week, if e.g. fetching failed, segment was lost or
whatever. Please note that this method requires modification of WebDB.

If fetching was completed and an updatedb was run, the original
fetchTime/fetchInterval could be recovered from a copy of the Page
inside the FetcherOutput.

Now, in 0.8 we do it differently. We don't modify CrawlDB, so we have no
way of recording which CrawlDatums end up on some fetchlist. This means
that two "generate" operations run in sequence, without intervening
updatedb, will produce exactly the same fetchlists.

Generator would have to be modified to use the same trick as in 0.7.
Unfortunately, this probably means that it will have to run a sort of
updatedb, using its output fetchlist to mark entries in CrawlDB. This
adds another map-reduce job to an already long-ish job (Generator
already uses two map-reduce jobs). This also means that Generator will
have to put a lock on CrawlDB for the duration of this job, so that no
other "generate" or "updatedb" can update it at the same time.

Then, when running an updatedb, the issue of scores and metadata comes
into question. We can imagine now that there were some other updatedb-s
run in the meantime, not necessarily with earlier fetchlists - so the
score and metadata info could be actually newer in the latest CrawlDB
than what we have inside the current segment. In such case, we will get
the following in CrawlDbReducer:

* "old" value from CrawlDb (which could be actually newer!). Even if
it's old, its fetchTime could be in the future due to the trick
described above. We could also get null here, if we just discovered a
new page.

* "original" value from CrawlDb, which was recorded in fetchlist. This,
for once, has a true fetch time, and its metadata and score are
snapshots of that information at the time of "generate".

* "new" value from Fetcher, with new score / metadata information. We
will also get "new" values from redirects, which might not match any of
the above values (i.e. they could use unique urls).

* "linked" values from parsers, with score / metadata contributions.

Now, the question is how to update the score, metadata, fetchTime and
fetchInterval information. We need a way to determine if the "new" value
we have is in fact newer or older than the "old" value - I'm not sure
how to do this, fetchTime and fetchInterval could have been modified so
they are not reliable... Perhaps we should add a "generation ID" to
CrawlDatum? Anyway, assuming we have a way to know this:

* if "new" is newer than "old", then we take all metadata from "old",
overwrite all info with the values from "new", and we keep "new".

* if "new" is older than "old", then we overwrite its metadata with all
values from "old". We do the same with fetchTime and fetchInterval. What
about the score? I think that for new score calculations we should take
the latest available score info from the "old" value.

Updatedb would also have to lock CrawlDB so that no other updatedb or
generate could run while we modify it.

That's probably all at the moment ... Any comments or suggestions
appreciated!

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Reply | Threaded
Open this post in threaded view
|

Re: Interleaved (parallel) fetch cycles

Doug Cutting
Andrzej Bialecki wrote:
> I'm planning to work on adding support in 0.8 for interleaved fetch cycles.

Great!

> Then, when running an updatedb, the issue of scores and metadata comes
> into question. We can imagine now that there were some other updatedb-s
> run in the meantime, not necessarily with earlier fetchlists - so the
> score and metadata info could be actually newer in the latest CrawlDB
> than what we have inside the current segment. In such case, we will get
> the following in CrawlDbReducer:
>
> * "old" value from CrawlDb (which could be actually newer!). Even if
> it's old, its fetchTime could be in the future due to the trick
> described above. We could also get null here, if we just discovered a
> new page.
>
> * "original" value from CrawlDb, which was recorded in fetchlist. This,
> for once, has a true fetch time, and its metadata and score are
> snapshots of that information at the time of "generate".
>
> * "new" value from Fetcher, with new score / metadata information. We
> will also get "new" values from redirects, which might not match any of
> the above values (i.e. they could use unique urls).
>
> * "linked" values from parsers, with score / metadata contributions.
>
> Now, the question is how to update the score, metadata, fetchTime and
> fetchInterval information. We need a way to determine if the "new" value
> we have is in fact newer or older than the "old" value - I'm not sure
> how to do this, fetchTime and fetchInterval could have been modified so
> they are not reliable... Perhaps we should add a "generation ID" to
> CrawlDatum?

Would it work to, when generating, set the fetch time for generated
items to the current time?  That way, the "new" value will always be a
bit after the "old" time.  In 0.7 we stored not the fetched-time but the
time-to-next-fetch, so we had to set it into the future.  But if we
instead just mark it as fetched now, so that it won't be re-generated
until its fetch interval has expired, that would resolve this, no?

>  Anyway, assuming we have a way to know this:
>
> * if "new" is newer than "old", then we take all metadata from "old",
> overwrite all info with the values from "new", and we keep "new".
>
> * if "new" is older than "old", then we overwrite its metadata with all
> values from "old". We do the same with fetchTime and fetchInterval.

That sounds right to me.  When is "original" used, if at all?

> What about the score? I think that for new score calculations we should take
> the latest available score info from the "old" value.

That also sounds right.  The crawl db should own the scores.  Scores
should not be updated by the fetcher, but only by crawldb updates.

> Updatedb would also have to lock CrawlDB so that no other updatedb or
> generate could run while we modify it.

Yes, that sounds right too.

Thanks for working on this!

Doug