Run your Lucene Applications on Google AppEngine with GAELucene

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Run your Lucene Applications on Google AppEngine with GAELucene

Kerang Lv
Hi Lucene users,

Enlightened by the discussion "Can I run Lucene in google app engine? [http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html], I implemented a google datastore based Lucene component, GAELucene, which can help you to run search applications on google app engine.

The main clazz of GAELucene include:
    * GAEDirectory - a read only Directory based on google datastore.
    * GAEFile - stands for an index file, the file's byte content will be splited into multi GAEFileContent.
    * GAEFileContent - stands for a segment of index file.
    * GAECategory - the identifier of different indices.
    * GAEIndexInput - a memory-resident IndexInput? implementation like the RAMInputStream.
    * GAEIndexReader - wrapper for IndexReader? that cached in GAEIndexReaderPool
    * GAEIndexReaderPool - pool for GAEIndexReader

The following code snippet demonstrates the use of GAELucene do searching:
  Query queryObject = parserQuery(request);
  GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
  GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
  IndexSearcher searcher =newIndexSearcher(indexReader);
  Hits hits = searcher.search(queryObject);
  readerPool.returnReader(indexReader);

You can find more here 'http://code.google.com/p/gaelucene/'



     

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Run your Lucene Applications on Google AppEngine with GAELucene

Allahbaksh Mohammedali Asadullah

Hi,
This is great news and good work. I think I will try this today evening. I think we should put this as one of component in lucene-contrib. What do you say? Committer and owner please comment.
Regards,
Allahbaksh
 
 
 
-----Original Message-----
From: Kerang Lv [mailto:[hidden email]]
Sent: Monday, September 14, 2009 9:34 PM
To: [hidden email]
Subject: Run your Lucene Applications on Google AppEngine with GAELucene

Hi Lucene users,

Enlightened by the discussion "Can I run Lucene in google app engine? [http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html], I implemented a google datastore based Lucene component, GAELucene, which can help you to run search applications on google app engine.

The main clazz of GAELucene include:
    * GAEDirectory - a read only Directory based on google datastore.
    * GAEFile - stands for an index file, the file's byte content will be splited into multi GAEFileContent.
    * GAEFileContent - stands for a segment of index file.
    * GAECategory - the identifier of different indices.
    * GAEIndexInput - a memory-resident IndexInput? implementation like the RAMInputStream.
    * GAEIndexReader - wrapper for IndexReader? that cached in GAEIndexReaderPool
    * GAEIndexReaderPool - pool for GAEIndexReader

The following code snippet demonstrates the use of GAELucene do searching:
  Query queryObject = parserQuery(request);
  GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
  GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
  IndexSearcher searcher =newIndexSearcher(indexReader);
  Hits hits = searcher.search(queryObject);
  readerPool.returnReader(indexReader);

You can find morere 'http://code.google.com/p/gaelucene/'



     

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Run your Lucene Applications on Google AppEngine with GAELucene

Michael McCandless-2
On Tue, Sep 15, 2009 at 12:39 AM, Allahbaksh Mohammedali Asadullah
<[hidden email]> wrote:

> I think we should put this as one of component in lucene-contrib.

+1, this looks like a great contribution!

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Run your Lucene Applications on Google AppEngine with GAELucene

Erdinc Yilmazel-2
In reply to this post by Kerang Lv
This is great news! Are you happy with the the performance of the google
data store? Do you plan to support in memory indexes using the memcache api?

Thanks

On Mon, Sep 14, 2009 at 5:04 PM, Kerang Lv <[hidden email]> wrote:

> Hi Lucene users,
>
> Enlightened by the discussion "Can I run Lucene in google app engine? [
> http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html],
> I implemented a google datastore based Lucene component, GAELucene, which
> can help you to run search applications on google app engine.
>
> The main clazz of GAELucene include:
>    * GAEDirectory - a read only Directory based on google datastore.
>    * GAEFile - stands for an index file, the file's byte content will be
> splited into multi GAEFileContent.
>    * GAEFileContent - stands for a segment of index file.
>    * GAECategory - the identifier of different indices.
>    * GAEIndexInput - a memory-resident IndexInput? implementation like the
> RAMInputStream.
>    * GAEIndexReader - wrapper for IndexReader? that cached in
> GAEIndexReaderPool
>    * GAEIndexReaderPool - pool for GAEIndexReader
>
> The following code snippet demonstrates the use of GAELucene do searching:
>  Query queryObject = parserQuery(request);
>  GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
>  GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
>  IndexSearcher searcher =newIndexSearcher(indexReader);
>  Hits hits = searcher.search(queryObject);
>  readerPool.returnReader(indexReader);
>
> You can find more here 'http://code.google.com/p/gaelucene/'
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

RE: Run your Lucene Applications on Google AppEngine with GAELucene

Allahbaksh Mohammedali Asadullah
In reply to this post by Michael McCandless-2

HI Mike,
I think adding this in Lucene 3.0 contrib would be the best we could do. I think we could add it in Lucene 2.9 Release as it would grow the community and we would also able to find some nice practices, bugs, improvement and that would make it better in upcoming release.

Regards,
Allahbaksh

-----Original Message----
From: Michael McCandless [mailto:[hidden email]]
Sent: Tuesday, September 15, 2009 3:25 PM
To: [hidden email]
Subject: Re: Run your Lucene Applications on Google AppEngine with GAELucene

On Tue, Sep 15, 2009 at 12:39 AM, Allahbaksh Mohammedali Asadullah
<[hidden email]> wrote:

> I think we should put this as one of component in lucene-contrib.

+1, this looks like a great contribution!

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Run your Lucene Applications on Google AppEngine with GAELucene

Kerang Lv
In reply to this post by Allahbaksh Mohammedali Asadullah
>I think I will try this today evening.
Remember to update your local project from the svn, I fixed some mistakes just now. I apologize for my negligence.

>I think we should put this as one of component in lucene-contrib. What do you say?
Yes, that's a good news.



----- Original Message ----
From: Allahbaksh Mohammedali Asadullah <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Tuesday, September 15, 2009 12:39:16 PM
Subject: RE: Run your Lucene Applications on Google AppEngine with GAELucene


Hi,
This is great news and good work. I think I will try this today evening. I think we should put this as one of component in lucene-contrib. What do you say? Committer and owner please comment.
Regards,
Allahbaksh



-----Original Message-----
From: Kerang Lv [mailto:[hidden email]]
Sent: Monday, September 14, 2009 9:34 PM
To: [hidden email]
Subject: Run your Lucene Applications on Google AppEngine with GAELucene

Hi Lucene users,

Enlightened by the discussion "Can I run Lucene in google app engine? [http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html], I implemented a google datastore based Lucene component, GAELucene, which can help you to run search applications on google app engine.

The main clazz of GAELucene include:
    * GAEDirectory - a read only Directory based on google datastore.
    * GAEFile - stands for an index file, the file's byte content will be splited into multi GAEFileContent.
    * GAEFileContent - stands for a segment of index file.
    * GAECategory - the identifier of different indices.
    * GAEIndexInput - a memory-resident IndexInput? implementation like the RAMInputStream.
    * GAEIndexReader - wrapper for IndexReader? that cached in GAEIndexReaderPool
    * GAEIndexReaderPool - pool for GAEIndexReader

The following code snippet demonstrates the use of GAELucene do searching:
  Query queryObject = parserQuery(request);
  GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
  GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
  IndexSearcher searcher =newIndexSearcher(indexReader);
  Hits hits = searcher.search(queryObject);
  readerPool.returnReader(indexReader);

You can find morere 'http://code.google.com/p/gaelucene/'



     

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


     

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Run your Lucene Applications on Google AppEngine with GAELucene

Kerang Lv
In reply to this post by Erdinc Yilmazel-2
>Do you plan to support in memory indexes using the memcache api?
I'm afraid not, I prefer to do indexing on another machine before I got a plan that can finish indexing within 30s.



----- Original Message ----
From: Erdinc Yilmazel <[hidden email]>
To: [hidden email]
Sent: Tuesday, September 15, 2009 6:05:15 PM
Subject: Re: Run your Lucene Applications on Google AppEngine with GAELucene

This is great news! Are you happy with the the performance of the google
data store? Do you plan to support in memory indexes using the memcache api?

Thanks

On Mon, Sep 14, 2009 at 5:04 PM, Kerang Lv <[hidden email]> wrote:

> Hi Lucene users,
>
> Enlightened by the discussion "Can I run Lucene in google app engine? [
> http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html],
> I implemented a google datastore based Lucene component, GAELucene, which
> can help you to run search applications on google app engine.
>
> The main clazz of GAELucene include:
>    * GAEDirectory - a read only Directory based on google datastore.
>    * GAEFile - stands for an index file, the file's byte content will be
> splited into multi GAEFileContent.
>    * GAEFileContent - stands for a segment of index file.
>    * GAECategory - the identifier of different indices.
>    * GAEIndexInput - a memory-resident IndexInput? implementation like the
> RAMInputStream.
>    * GAEIndexReader - wrapper for IndexReader? that cached in
> GAEIndexReaderPool
>    * GAEIndexReaderPool - pool for GAEIndexReader
>
> The following code snippet demonstrates the use of GAELucene do searching:
>  Query queryObject = parserQuery(request);
>  GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
>  GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
>  IndexSearcher searcher =newIndexSearcher(indexReader);
>  Hits hits = searcher.search(queryObject);
>  readerPool.returnReader(indexReader);
>
> You can find more here 'http://code.google.com/p/gaelucene/'
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>



     

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Run your Lucene Applications on Google AppEngine with GAELucene

Daniel Shane-2
In reply to this post by Allahbaksh Mohammedali Asadullah
I question the performance of such an approach. For lucene to be fast,
disk access need to be fast, and the transaction stuff with google is
not that good.

I'll have to test it out to see, but I anticipate a huge performance hit
compared to lucene running with a real HDD access.

Daniel Shane

Allahbaksh Mohammedali Asadullah wrote:

> Hi,
> This is great news and good work. I think I will try this today evening. I think we should put this as one of component in lucene-contrib. What do you say? Committer and owner please comment.
> Regards,
> Allahbaksh
>  
>  
>  
> -----Original Message-----
> From: Kerang Lv [mailto:[hidden email]]
> Sent: Monday, September 14, 2009 9:34 PM
> To: [hidden email]
> Subject: Run your Lucene Applications on Google AppEngine with GAELucene
>
> Hi Lucene users,
>
> Enlightened by the discussion "Can I run Lucene in google app engine? [http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html], I implemented a google datastore based Lucene component, GAELucene, which can help you to run search applications on google app engine.
>
> The main clazz of GAELucene include:
>     * GAEDirectory - a read only Directory based on google datastore.
>     * GAEFile - stands for an index file, the file's byte content will be splited into multi GAEFileContent.
>     * GAEFileContent - stands for a segment of index file.
>     * GAECategory - the identifier of different indices.
>     * GAEIndexInput - a memory-resident IndexInput? implementation like the RAMInputStream.
>     * GAEIndexReader - wrapper for IndexReader? that cached in GAEIndexReaderPool
>     * GAEIndexReaderPool - pool for GAEIndexReader
>
> The following code snippet demonstrates the use of GAELucene do searching:
>   Query queryObject = parserQuery(request);
>   GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
>   GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
>   IndexSearcher searcher =newIndexSearcher(indexReader);
>   Hits hits = searcher.search(queryObject);
>   readerPool.returnReader(indexReader);
>
> You can find morere 'http://code.google.com/p/gaelucene/'
>
>
>
>      
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>  


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Run your Lucene Applications on Google AppEngine with GAELucene

Kerang Lv
Hi Daniel Shane,

The GAEFile/GAEIndexInput was a bit like the RAMFile/RAMInputStream, it can hold the file's byte data in memory, so the performance should be acceptable once most file segments were loaded from the google datastore.

Wish your test result.


----- Original Message ----
From: Daniel Shane <[hidden email]>
To: [hidden email]
Sent: Wednesday, September 16, 2009 10:33:32 PM
Subject: Re: Run your Lucene Applications on Google AppEngine with GAELucene

I question the performance of such an approach. For lucene to be fast, disk access need to be fast, and the transaction stuff with google is not that good.

I'll have to test it out to see, but I anticipate a huge performance hit compared to lucene running with a real HDD access.

Daniel Shane

Allahbaksh Mohammedali Asadullah wrote:

> Hi,
> This is great news and good work. I think I will try this today evening. I think we should put this as one of component in lucene-contrib. What do you say? Committer and owner please comment.
> Regards,
> Allahbaksh
>    -----Original Message-----
> From: Kerang Lv [mailto:[hidden email]] Sent: Monday, September 14, 2009 9:34 PM
> To: [hidden email]
> Subject: Run your Lucene Applications on Google AppEngine with GAELucene
>
> Hi Lucene users,
>
> Enlightened by the discussion "Can I run Lucene in google app engine? [http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html], I implemented a google datastore based Lucene component, GAELucene, which can help you to run search applications on google app engine.
>
> The main clazz of GAELucene include:    * GAEDirectory - a read only Directory based on google datastore.    * GAEFile - stands for an index file, the file's byte content will be splited into multi GAEFileContent.    * GAEFileContent - stands for a segment of index file.    * GAECategory - the identifier of different indices.    * GAEIndexInput - a memory-resident IndexInput? implementation like the RAMInputStream.    * GAEIndexReader - wrapper for IndexReader? that cached in GAEIndexReaderPool    * GAEIndexReaderPool - pool for GAEIndexReader
> The following code snippet demonstrates the use of GAELucene do searching:  Query queryObject = parserQuery(request);
>  GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
>  GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
>  IndexSearcher searcher =newIndexSearcher(indexReader);
>  Hits hits = searcher.search(queryObject);
>  readerPool.returnReader(indexReader);
>
> You can find morere 'http://code.google.com/p/gaelucene/'
>
>
>
>     
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]