Sending query to multiple servers and combine all Hits from them ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Sending query to multiple servers and combine all Hits from them ?

Vikas Khengare
Hi Friends.......
 
        I am doing search application which has following scenario.
 
Architecture ==>
1] Common GUI
2] When user enter one query then It should go to 4 searcher server (All servers are on remote machines)
3] After searching all 4 server should return results i.e. Hits ( All server return hits objects in diff formats i.e. all 4 servers return hit doc's in diff formats)
4] Combine all Hits and form only type of result
5] Show that result in Uniform way to user
 
Problems ==>
1] How do I send my search query to all 4 searcher server. ?
2] After searching How do I get all result (Hits) for combining them in one bundle; Because they all are in different format. ?
3] Shall I use AJAX for sending query across multiple server and getting results back from all servers( Or any other technology, if yes specify) ?
4] How do I combine all hits ?
 
 
 
Thanks..................
 
Best Regards
Reply | Threaded
Open this post in threaded view
|

RE: Sending query to multiple servers and combine all Hits from them ?

Daniel Pfeifer
You search all four servers by doing this (the QueryParser in this
example uses the Lucene 1.9 syntax):

 

Searchable[] searchables = new Searchable[]{(Searchable)
Naming.lookup(x1), (Searchable) Naming.lookup(x2), ...};

MultiSearcher multiSearcher = new MultiSearcher(searchables);

Hits hits = multiSearcher.search(new QueryParser("title", new
StandardAnalyzer()).parse("title:Ajax"));

 

/Daniel

 

________________________________

From: Vikas Khengare [mailto:[hidden email]]
Sent: den 2 februari 2006 06:30
To: [hidden email]; [hidden email];
[hidden email]; [hidden email];
[hidden email]
Subject: Sending query to multiple servers and combine all Hits from
them ?

 

Hi Friends.......

 

        I am doing search application which has following scenario.

 

Architecture ==>

1] Common GUI

2] When user enter one query then It should go to 4 searcher server (All
servers are on remote machines)

3] After searching all 4 server should return results i.e. Hits ( All
server return hits objects in diff formats i.e. all 4 servers return hit
doc's in diff formats)

4] Combine all Hits and form only type of result

5] Show that result in Uniform way to user

 

Problems ==>

1] How do I send my search query to all 4 searcher server. ?

2] After searching How do I get all result (Hits) for combining them in
one bundle; Because they all are in different format. ?

3] Shall I use AJAX for sending query across multiple server and getting
results back from all servers( Or any other technology, if yes specify)
?

4] How do I combine all hits ?

 

 

 



Thanks..................

 

Best Regards

[ [hidden email] ]

Reply | Threaded
Open this post in threaded view
|

Re: Sending query to multiple servers and combine all Hits from them ?

Chris Hostetter-3
In reply to this post by Vikas Khengare

Please do not cross posting to every java/lucene list you can find.  The
@jakarta lists are no longer used and forward their mail to the equivilent
@lucene lists.  The commits list exists only for people to recieve
notification about changes in the SVN repository, and people should never
send mail directly to it.

Unless you have a specific reason why you want to send mail to java-dev
regarding development of the internals of the lucene code base please send
your questions to java-user -- java-user is the correct place to send
questions about using the lucene code base.  Even if you think your
message may relate to a bug, or a possible change in the code base, do
not cross post to both lists.  Start by mailing java-user, and if people
think you have encountered a bug, or that your topic warrants discussion
about a change in API/Implimentation then the discussion can move to
java-dev.




: Date: Thu, 2 Feb 2006 10:59:42 +0530
: From: Vikas Khengare <[hidden email]>
: Reply-To: [hidden email]
: To: [hidden email], [hidden email],
:      [hidden email], [hidden email],
:      [hidden email]
: Subject: Sending query to multiple servers and combine all Hits from them


-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

index merging

Omar Didi
In reply to this post by Vikas Khengare
Hi all,

 I am trying to index in 2 different indexes with different mergefactors. so if one of the indexes is merging than I want to index only on the other index.
is there a way to know if an index is merging?.

thanks

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: index merging

Yonik Seeley
No, there isn't anything in the API to tell you that.
A merge may be triggered on any IndexWriter.add() call.

If you want to avoid merges, you can set the merge factor really high
so that merges will never happen, and set maxBufferedDocs to the size
of the segments you want.

-Yonik

On 2/2/06, Omar Didi <[hidden email]> wrote:
> Hi all,
>
>  I am trying to index in 2 different indexes with different mergefactors. so if one of the indexes is merging than I want to index only on the other index.
> is there a way to know if an index is merging?.
>
> thanks

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: index merging

Omar Didi
In reply to this post by Omar Didi
Thanks Yonik,
I can't set the merge factor too high because I will end up with the too many files open problem.

I have got this method from indexReader.java

  public static boolean isLocked(Directory directory) throws IOException {
    return
            directory.makeLock(IndexWriter.WRITE_LOCK_NAME).isLocked() ||
            directory.makeLock(IndexWriter.COMMIT_LOCK_NAME).isLocked();
  }

if I change this method to only check for directory.makeLock(IndexWriter.COMMIT_LOCK_NAME).isLocked(); will this tell me that a index is merging?

-----Original Message-----
From: Yonik Seeley [mailto:[hidden email]]
Sent: Thursday, February 02, 2006 3:53 PM
To: [hidden email]
Subject: Re: index merging


No, there isn't anything in the API to tell you that.
A merge may be triggered on any IndexWriter.add() call.

If you want to avoid merges, you can set the merge factor really high
so that merges will never happen, and set maxBufferedDocs to the size
of the segments you want.

-Yonik

On 2/2/06, Omar Didi <[hidden email]> wrote:
> Hi all,
>
>  I am trying to index in 2 different indexes with different mergefactors. so if one of the indexes is merging than I want to index only on the other index.
> is there a way to know if an index is merging?.
>
> thanks

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: index merging

Yonik Seeley
On 2/2/06, Omar Didi <[hidden email]> wrote:
> Thanks Yonik,
> I can't set the merge factor too high because I will end up with the too many files open problem.

Right.  I meant only for adding a lot of documents.  After a lot of
adds, then you could set the mergefactor back down to a reasonable
level and optimize the index.

> if I change this method to only check for directory.makeLock(IndexWriter.COMMIT_LOCK_NAME).isLocked(); will this tell me that a index is merging?

Yes, the commit lock will be locked when merging is going on, but you
will get some false positives because the lock is used for other
purposes.  That may still fit your need though.

-Yonik

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: index merging

Vanlerberghe, Luc
In reply to this post by Omar Didi
Sorry to contradict you Yonik, but I'm pretty sure the commit lock is
*not* locked during a merge, only while the "segments" file is being
updated.  

The merge process takes a set of 'old' segment files, writes new segment
files and 'registers' them in the "segments" file when they are ready to
be used.  During an optimize, this proces is repeated until there is
only one segment left.

Every process reading or writing the "segments" file grabs the commit
lock, reads the file, opens any other files it needs and releases the
lock.

While an IndexWriter or IndexReader is updating the index, the write
lock will be set though...
If this write lock is kept longer than expected you could assume a merge
is in progress.

A better way might be to check for segments files that are not
'registered' in the "segments" file yet.

Luc

-----Original Message-----
From: Yonik Seeley [mailto:[hidden email]]
Sent: donderdag 2 februari 2006 22:25
To: [hidden email]
Subject: Re: index merging

On 2/2/06, Omar Didi <[hidden email]> wrote:
> Thanks Yonik,
> I can't set the merge factor too high because I will end up with the
too many files open problem.

Right.  I meant only for adding a lot of documents.  After a lot of
adds, then you could set the mergefactor back down to a reasonable
level and optimize the index.

> if I change this method to only check for
directory.makeLock(IndexWriter.COMMIT_LOCK_NAME).isLocked(); will this
tell me that a index is merging?

Yes, the commit lock will be locked when merging is going on, but you
will get some false positives because the lock is used for other
purposes.  That may still fit your need though.

-Yonik

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: index merging

Yonik Seeley
On 2/6/06, Vanlerberghe, Luc <[hidden email]> wrote:
> Sorry to contradict you Yonik, but I'm pretty sure the commit lock is
> *not* locked during a merge, only while the "segments" file is being
> updated.

Oops, you're right.  Good thing too... if the commit lock was held
during merges, one couldn't even open up a new IndexReader.

-Yonik

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]