Quantcast

SocketTimeOutException is coming even after increasing http.timeout

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

SocketTimeOutException is coming even after increasing http.timeout

suyashaoc
Hello Nutch-Users,

I crawled 1000 urls setting http.timeout to 7000. But more than 200 urls are throwing SocketTimeOut exception.

I also tried increasing number of fetcher threads. That also did not worked.

Please help how to fix.

Regards,
Suyash
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: SocketTimeOutException is coming even after increasing http.timeout

Markus Jelsma-2
Hello - http.timeout is in milliseconds. Apparently, those sites are not able to deliver a page in 7 seconds, for any reason. Increase to 30000 and it should work just fine for most sites. If you are blocked by a firewall, any timeout is never going to be high enough.

Regards,
Markus

 
 
-----Original message-----

> From:suyashaoc <[hidden email]>
> Sent: Saturday 18th March 2017 8:51
> To: [hidden email]
> Subject: SocketTimeOutException is coming even after increasing http.timeout
>
> Hello Nutch-Users,
>
> I crawled 1000 urls setting http.timeout to 7000. But more than 200 urls are
> throwing SocketTimeOut exception.
>
> I also tried increasing number of fetcher threads. That also did not worked.
>
> Please help how to fix.
>
> Regards,
> Suyash
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/SocketTimeOutException-is-coming-even-after-increasing-http-timeout-tp4325750.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
Loading...