Converting Lucene Demo to search index in DFS

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Converting Lucene Demo to search index in DFS

howard chen
Hi,

I have been playing with Lucene & Hadoop.

The Lucene demo (Web) is a great tutorial for people to understand Lucene,

e.g. http://lucene.apache.org/java/docs/demo3.html

I want to ask

1. If I put the index in the DFS of Hadoop, is it easy to modify the
codes to search in the DFS, rather than local FS? (ignore abt
mapreduce first, I mean just search index in the DFS from web server)

2. More than (1), now if I want to search the index from serveral
running nodes using mapreduce, is the wordcount example a good
starting point?

Thanks for any comments and suggestions.

howa.
Reply | Threaded
Open this post in threaded view
|

Re: Converting Lucene Demo to search index in DFS

howard chen
On 12/4/06, howard chen <[hidden email]> wrote:

> Hi,
>
> I have been playing with Lucene & Hadoop.
>
> The Lucene demo (Web) is a great tutorial for people to understand Lucene,
>
> e.g. http://lucene.apache.org/java/docs/demo3.html
>
> I want to ask
>
> 1. If I put the index in the DFS of Hadoop, is it easy to modify the
> codes to search in the DFS, rather than local FS? (ignore abt
> mapreduce first, I mean just search index in the DFS from web server)
>
> 2. More than (1), now if I want to search the index from serveral
> running nodes using mapreduce, is the wordcount example a good
> starting point?
>
> Thanks for any comments and suggestions.
>
> howa.
>

in fact, the problem maybe just i don't know how to split the lucene
index, if i can split the index, the flow i suppose is similar to the
word count example.

can anyone experience in nutch can tell me how to deal with index splitting?

thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Converting Lucene Demo to search index in DFS

Otis Gospodnetic-2
In reply to this post by howard chen
Somebody just (2 days ago?) posted some code for splitting a Lucene index over on java-dev@lucene.... you may want to have a look.

Otis

----- Original Message ----
From: howard chen <[hidden email]>
To: [hidden email]
Sent: Thursday, December 7, 2006 1:04:08 PM
Subject: Re: Converting Lucene Demo to search index in DFS

On 12/4/06, howard chen <[hidden email]> wrote:

> Hi,
>
> I have been playing with Lucene & Hadoop.
>
> The Lucene demo (Web) is a great tutorial for people to understand Lucene,
>
> e.g. http://lucene.apache.org/java/docs/demo3.html
>
> I want to ask
>
> 1. If I put the index in the DFS of Hadoop, is it easy to modify the
> codes to search in the DFS, rather than local FS? (ignore abt
> mapreduce first, I mean just search index in the DFS from web server)
>
> 2. More than (1), now if I want to search the index from serveral
> running nodes using mapreduce, is the wordcount example a good
> starting point?
>
> Thanks for any comments and suggestions.
>
> howa.
>

in fact, the problem maybe just i don't know how to split the lucene
index, if i can split the index, the flow i suppose is similar to the
word count example.

can anyone experience in nutch can tell me how to deal with index splitting?

thanks.