crawling webpage results

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

crawling webpage results

Shanthoosh PV
Hi ,

                    I want to crawl a result obtained based upon  a user defined keyword search in a search engine . Is it possible to do it in nutch . Please provide useful insights , i tried searching in this forum and google but found nothing helpful .

                    The user may provide a search engine like google.com along with keyword to search for in that search engine . The results of this search should be crawled . Is it possibe to do in nutch , just providing the search engine url along with the keyword for search.



Shanthoosh
Reply | Threaded
Open this post in threaded view
|

Re: crawling webpage results

Alex McLintock
This should really be a user type question, not a dev question. But
what the heck.

The first thing which comes to mind is to do the search yourself and
provide the results of that search as seed pages.

But since you asked on the dev mailing list, you could possibly write
something which actually queried Google itself through its API - but
Nutch doesn't do that itself. If you do write it then consider
submitting it as a patch.

Goodluck

Alex

On 1 September 2010 09:14, Shanthoosh PV <[hidden email]> wrote:

> Hi ,
>
>                     I want to crawl a result obtained based upon  a user
> defined keyword search in a search engine . Is it possible to do it in nutch
> . Please provide useful insights , i tried searching in this forum and
> google but found nothing helpful .
>
>                     The user may provide a search engine like google.com
> along with keyword to search for in that search engine . The results of this
> search should be crawled . Is it possibe to do in nutch , just providing the
> search engine url along with the keyword for search.
>
>
>
> Shanthoosh
>