Technorati

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Technorati

Paul Harrison-2
Does anyone know if Technorati is running on a Lucene/Nutch platform?

 

Thanks,

 

Paul

Reply | Threaded
Open this post in threaded view
|

User Agent

waterwheel
What should I be using for a user agent in the crawler?  We just tried
crawling a government site and if we leave the user agent set to nutch,
we get the crawl.  When I change it, I'm getting blocked with an error
about the user agent not being supported.  It seems that I should be
changing the user agent to something that identifies my site - shouldn't
I?  But that might limit my crawling ability.

What are others setting their user agent to?
Reply | Threaded
Open this post in threaded view
|

Re: Technorati

Erik Hatcher
In reply to this post by Paul Harrison-2
On Dec 9, 2005, at 2:50 PM, Paul Harrison wrote:
> Does anyone know if Technorati is running on a Lucene/Nutch platform?

        http://www.google.com/search?q=lucene%20technorati  :))


Reply | Threaded
Open this post in threaded view
|

Re: User Agent

gekkokid
In reply to this post by waterwheel
just make sure you have "Mozilla/5.0" at the front :)

----- Original Message -----
From: "Insurance Squared Inc." <[hidden email]>
To: <[hidden email]>
Sent: Friday, December 09, 2005 2:55 PM
Subject: User Agent


> What should I be using for a user agent in the crawler?  We just tried
> crawling a government site and if we leave the user agent set to nutch,
> we get the crawl.  When I change it, I'm getting blocked with an error
> about the user agent not being supported.  It seems that I should be
> changing the user agent to something that identifies my site - shouldn't
> I?  But that might limit my crawling ability.
>
> What are others setting their user agent to?
>
Reply | Threaded
Open this post in threaded view
|

Re: Technorati

Tim Archambault
In reply to this post by Paul Harrison-2
http://nutch.sourceforge.net/blog/2004/11/lucene-lecture.html



On 12/9/05, Paul Harrison <[hidden email]> wrote:

>
> Does anyone know if Technorati is running on a Lucene/Nutch platform?
>
>
>
> Thanks,
>
>
>
> Paul
>
>
>