Experiment on crawler behaviour

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Experiment on crawler behaviour

Andrzej Białecki-2
Hi,

I found this article pretty interesting:

    http://drunkmenworkhere.org/219

Could we come up with some codified rules, reverse-engineered from the
bots' behavior?

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com