Lucene and Javolution: A good mix ?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Lucene and Javolution: A good mix ?

Robichaud, Jean-Philippe-2
Hello Dear Lucene coders!

 

Some of you may remember, I'm using lucene for a product (and many other
internal utilities).  I'm also using another open source library called
Javolution (www.javolution.org <http://www.javolution.org/> ) which does
many things, one of them being to offer excellent replacements for
ArrayList/Map/... and a super good memory management extension to the
java language.

 

As I'm [trying to] follow the conversations on this list, I see that
many of you are working towards optimizing lucene in term of memory
footprint and speed.  I just finished optimizing my code (not lucene
itself, but my code written on top of it) using Javolution PoolContext
and the FastList/FastMap/... classes.  The resulting speedup is a 6
times faster code.

 

Javolution make it easy to recycle objects and do some object allocation
on the stack rather than on the head, which remove stress on the garbage
collector.  Javolution also offers 2 classes  (Text and TextBuilder) to
replace String/StringBuffer which are perfect for anything related to
string manipulation and some C "union/struct" equivalent for java.  The
thing is really great.

 

Would anyone be interested in doing Lucene a face lift and start using
javolution as a core lucene dependency?  I understand that right now,
lucene is free of any dependencies, which is quite great, but anyone
interested in doing fast/lean/stable java application should seriously
consider using javolution anyway.

 

Any thoughts?

 

Jp

Reply | Threaded
Open this post in threaded view
|

Re: Lucene and Javolution: A good mix ?

Robert Engels
I would suggest that the Javolution folks do their tests against  
modern JVM...

I have followed the Javolution project for some time, and while I  
agree that some of the techniques should improve things, I think that  
modern JVMs do most of this work for you (and the latest class  
libraries also help - StringBuilder and others).

I also think that when you start doing you own memory management you  
might as well write the code in C/C++ because you need to use similar  
techniques (similar to the resource management when using SWT).

Just my thoughts.

On Apr 4, 2007, at 8:54 PM, Jean-Philippe Robichaud wrote:

> Hello Dear Lucene coders!
>
>
>
> Some of you may remember, I'm using lucene for a product (and many  
> other
> internal utilities).  I'm also using another open source library  
> called
> Javolution (www.javolution.org <http://www.javolution.org/> ) which  
> does
> many things, one of them being to offer excellent replacements for
> ArrayList/Map/... and a super good memory management extension to the
> java language.
>
>
>
> As I'm [trying to] follow the conversations on this list, I see that
> many of you are working towards optimizing lucene in term of memory
> footprint and speed.  I just finished optimizing my code (not lucene
> itself, but my code written on top of it) using Javolution PoolContext
> and the FastList/FastMap/... classes.  The resulting speedup is a 6
> times faster code.
>
>
>
> Javolution make it easy to recycle objects and do some object  
> allocation
> on the stack rather than on the head, which remove stress on the  
> garbage
> collector.  Javolution also offers 2 classes  (Text and  
> TextBuilder) to
> replace String/StringBuffer which are perfect for anything related to
> string manipulation and some C "union/struct" equivalent for java.  
> The
> thing is really great.
>
>
>
> Would anyone be interested in doing Lucene a face lift and start using
> javolution as a core lucene dependency?  I understand that right now,
> lucene is free of any dependencies, which is quite great, but anyone
> interested in doing fast/lean/stable java application should seriously
> consider using javolution anyway.
>
>
>
> Any thoughts?
>
>
>
> Jp
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Lucene and Javolution: A good mix ?

Robichaud, Jean-Philippe-2
I understand your concerns!

I was a little skeptical at the beginning.  But even with the 1.5 jvm,
the improvements still holds.

Lucene creates a lots of "garbage" (strings, tokens, ...) either at
index time or query time. While the new garbage collector strategies did
seriously improve since java 1.4, the gains are still there as the
object "creation" is also a cost that javolution easily saves us from.  

What javolution requires to give the best is can is for you to make
certain critical classes extends the RealtimeObject class and implement
a Factory pattern inside.  Once this is done, you can now fully profit
from the Javolution features.  Even without doing that, we could still
benefit from the FastList/FastMap classes being already pooled and the
possibility to 'thread-safely' iterate lists/maps without creating any
objects.

Javolution is also released for gcj, which is great since it won't
interfere with the lucene's gcj effort.

From what I can foresee, the pros/cons would be:
Pros:
        Leaner memory footprint
        Saving many cpu cycles
Cons:
        Adding a dependency to lucene codebase
        Lucene developers must get familiar with the "Context" concepts


Jp
-----Original Message-----
From: robert engels [mailto:[hidden email]]
Sent: Wednesday, April 04, 2007 10:31 PM
To: [hidden email]
Subject: Re: Lucene and Javolution: A good mix ?

I would suggest that the Javolution folks do their tests against  
modern JVM...

I have followed the Javolution project for some time, and while I  
agree that some of the techniques should improve things, I think that  
modern JVMs do most of this work for you (and the latest class  
libraries also help - StringBuilder and others).

I also think that when you start doing you own memory management you  
might as well write the code in C/C++ because you need to use similar  
techniques (similar to the resource management when using SWT).

Just my thoughts.

On Apr 4, 2007, at 8:54 PM, Jean-Philippe Robichaud wrote:

> Hello Dear Lucene coders!
>
>
>
> Some of you may remember, I'm using lucene for a product (and many  
> other
> internal utilities).  I'm also using another open source library  
> called
> Javolution (www.javolution.org <http://www.javolution.org/> ) which  
> does
> many things, one of them being to offer excellent replacements for
> ArrayList/Map/... and a super good memory management extension to the
> java language.
>
>
>
> As I'm [trying to] follow the conversations on this list, I see that
> many of you are working towards optimizing lucene in term of memory
> footprint and speed.  I just finished optimizing my code (not lucene
> itself, but my code written on top of it) using Javolution PoolContext
> and the FastList/FastMap/... classes.  The resulting speedup is a 6
> times faster code.
>
>
>
> Javolution make it easy to recycle objects and do some object  
> allocation
> on the stack rather than on the head, which remove stress on the  
> garbage
> collector.  Javolution also offers 2 classes  (Text and  
> TextBuilder) to
> replace String/StringBuffer which are perfect for anything related to
> string manipulation and some C "union/struct" equivalent for java.  
> The
> thing is really great.
>
>
>
> Would anyone be interested in doing Lucene a face lift and start using
> javolution as a core lucene dependency?  I understand that right now,
> lucene is free of any dependencies, which is quite great, but anyone
> interested in doing fast/lean/stable java application should seriously
> consider using javolution anyway.
>
>
>
> Any thoughts?
>
>
>
> Jp
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene and Javolution: A good mix ?

Mike Klaas
On 4/4/07, Jean-Philippe Robichaud <[hidden email]> wrote:
> I understand your concerns!
>
> I was a little skeptical at the beginning.  But even with the 1.5 jvm,
> the improvements still holds.
>
> Lucene creates a lots of "garbage" (strings, tokens, ...) either at
> index time or query time. While the new garbage collector strategies did
> seriously improve since java 1.4, the gains are still there as the
> object "creation" is also a cost that javolution easily saves us from.

I think the best approach at convincing people would be to produce a
patch that implements some of the suggested changes, and benchmark it.
 As it stands, the positives are all hypothetical and the negatives
rather tangible.

-MIke

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene and Javolution: A good mix ?

Otis Gospodnetic-2
In reply to this post by Robichaud, Jean-Philippe-2
What Mike said.  Without seeing the Javalutionized Lucene in action we won't get very far.
jean-Philippe, are you interested in making the changes to Lucene and showing the performance improvement?
Note that you can use the super-nice and easy to use contrib/benchmark to compare the "vanilla Lucene" and the "Javalutionized Lucene".


Otis
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share

----- Original Message ----
From: Mike Klaas <[hidden email]>
To: [hidden email]
Sent: Thursday, April 5, 2007 1:58:38 PM
Subject: Re: Lucene and Javolution: A good mix ?

On 4/4/07, Jean-Philippe Robichaud <[hidden email]> wrote:
> I understand your concerns!
>
> I was a little skeptical at the beginning.  But even with the 1.5 jvm,
> the improvements still holds.
>
> Lucene creates a lots of "garbage" (strings, tokens, ...) either at
> index time or query time. While the new garbage collector strategies did
> seriously improve since java 1.4, the gains are still there as the
> object "creation" is also a cost that javolution easily saves us from.

I think the best approach at convincing people would be to produce a
patch that implements some of the suggested changes, and benchmark it.
 As it stands, the positives are all hypothetical and the negatives
rather tangible.

-MIke

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Lucene and Javolution: A good mix ?

Robichaud, Jean-Philippe-2
Yes, I believe enough in this approach to try it.  I'm already starting
to play with it.  I took the current trunk and I'm starting to play with
it.  That begin said, I'm quite busy right now so I can't promise any
steady progress.  Also, I won't apply patches that are already in JIRA,
so the numbers I'll get won't be the 'up-to-date" ones.

I understand that before this idea gets any traction, we must have an
idea of how much this could help.  But before going deep with this work,
I wanted to know if Lucene developers have any interest in this kind of
work.  If the gurus dislike the idea of adding a dependency to Lucene
(which is not the case for others Apache projects!), then I won't spend
too much time on this.

Jp

-----Original Message-----
From: Otis Gospodnetic [mailto:[hidden email]]
Sent: Thursday, April 05, 2007 3:01 PM
To: [hidden email]
Subject: Re: Lucene and Javolution: A good mix ?

What Mike said.  Without seeing the Javalutionized Lucene in action we
won't get very far.
jean-Philippe, are you interested in making the changes to Lucene and
showing the performance improvement?
Note that you can use the super-nice and easy to use contrib/benchmark
to compare the "vanilla Lucene" and the "Javalutionized Lucene".


Otis
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share

----- Original Message ----
From: Mike Klaas <[hidden email]>
To: [hidden email]
Sent: Thursday, April 5, 2007 1:58:38 PM
Subject: Re: Lucene and Javolution: A good mix ?

On 4/4/07, Jean-Philippe Robichaud <[hidden email]>
wrote:
> I understand your concerns!
>
> I was a little skeptical at the beginning.  But even with the 1.5 jvm,
> the improvements still holds.
>
> Lucene creates a lots of "garbage" (strings, tokens, ...) either at
> index time or query time. While the new garbage collector strategies
did
> seriously improve since java 1.4, the gains are still there as the
> object "creation" is also a cost that javolution easily saves us from.

I think the best approach at convincing people would be to produce a
patch that implements some of the suggested changes, and benchmark it.
 As it stands, the positives are all hypothetical and the negatives
rather tangible.

-MIke

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene and Javolution: A good mix ?

Otis Gospodnetic-2
In reply to this post by Robichaud, Jean-Philippe-2
I'm not in love with the dependency idea, though it's not that big of a deal for me.
However, I think you will want to get some of the performance patched (e.g. LUCENE-843) in first, so you can compare the latest and greatest version of Lucene with your Javalutionized version.  From what I gather from Mike's emails, he is doing a lot of object and array sharing and reusing in order to minimize object creation, memory allocation, and thus create less work for the garbage collector.

My 2.... pick a currency, say Levs.

Otis
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share

----- Original Message ----
From: Jean-Philippe Robichaud <[hidden email]>
To: [hidden email]
Sent: Thursday, April 5, 2007 3:19:51 PM
Subject: RE: Lucene and Javolution: A good mix ?

Yes, I believe enough in this approach to try it.  I'm already starting
to play with it.  I took the current trunk and I'm starting to play with
it.  That begin said, I'm quite busy right now so I can't promise any
steady progress.  Also, I won't apply patches that are already in JIRA,
so the numbers I'll get won't be the 'up-to-date" ones.

I understand that before this idea gets any traction, we must have an
idea of how much this could help.  But before going deep with this work,
I wanted to know if Lucene developers have any interest in this kind of
work.  If the gurus dislike the idea of adding a dependency to Lucene
(which is not the case for others Apache projects!), then I won't spend
too much time on this.

Jp

-----Original Message-----
From: Otis Gospodnetic [mailto:[hidden email]]
Sent: Thursday, April 05, 2007 3:01 PM
To: [hidden email]
Subject: Re: Lucene and Javolution: A good mix ?

What Mike said.  Without seeing the Javalutionized Lucene in action we
won't get very far.
jean-Philippe, are you interested in making the changes to Lucene and
showing the performance improvement?
Note that you can use the super-nice and easy to use contrib/benchmark
to compare the "vanilla Lucene" and the "Javalutionized Lucene".


Otis
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share

----- Original Message ----
From: Mike Klaas <[hidden email]>
To: [hidden email]
Sent: Thursday, April 5, 2007 1:58:38 PM
Subject: Re: Lucene and Javolution: A good mix ?

On 4/4/07, Jean-Philippe Robichaud <[hidden email]>
wrote:
> I understand your concerns!
>
> I was a little skeptical at the beginning.  But even with the 1.5 jvm,
> the improvements still holds.
>
> Lucene creates a lots of "garbage" (strings, tokens, ...) either at
> index time or query time. While the new garbage collector strategies
did
> seriously improve since java 1.4, the gains are still there as the
> object "creation" is also a cost that javolution easily saves us from.

I think the best approach at convincing people would be to produce a
patch that implements some of the suggested changes, and benchmark it.
 As it stands, the positives are all hypothetical and the negatives
rather tangible.

-MIke

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene and Javolution: A good mix ?

Grant Ingersoll-4
In reply to this post by Robichaud, Jean-Philippe-2
I'm not saying I'm against it, but one of the things that makes  
Lucene so great is it's lack of dependencies in the core.  It isn't  
necessarily a slippery slope, either, if we do add one dependency.

Javolution is BSD license, AFAICT.  I don't know if that is a good or  
bad license as far as Apache is concerned, but it should be looked  
into before you spend any time on it.

This is not meant to be a discouragement.  If it shows a significant  
improvement, people will notice and it will be taken seriously,  
especially if it is backward compatible, well tested and well  
documented.

-Grant

On Apr 5, 2007, at 3:19 PM, Jean-Philippe Robichaud wrote:

> Yes, I believe enough in this approach to try it.  I'm already  
> starting
> to play with it.  I took the current trunk and I'm starting to play  
> with
> it.  That begin said, I'm quite busy right now so I can't promise any
> steady progress.  Also, I won't apply patches that are already in  
> JIRA,
> so the numbers I'll get won't be the 'up-to-date" ones.
>
> I understand that before this idea gets any traction, we must have an
> idea of how much this could help.  But before going deep with this  
> work,
> I wanted to know if Lucene developers have any interest in this  
> kind of
> work.  If the gurus dislike the idea of adding a dependency to Lucene
> (which is not the case for others Apache projects!), then I won't  
> spend
> too much time on this.
>
> Jp
>
> -----Original Message-----
> From: Otis Gospodnetic [mailto:[hidden email]]
> Sent: Thursday, April 05, 2007 3:01 PM
> To: [hidden email]
> Subject: Re: Lucene and Javolution: A good mix ?
>
> What Mike said.  Without seeing the Javalutionized Lucene in action we
> won't get very far.
> jean-Philippe, are you interested in making the changes to Lucene and
> showing the performance improvement?
> Note that you can use the super-nice and easy to use contrib/benchmark
> to compare the "vanilla Lucene" and the "Javalutionized Lucene".
>
>
> Otis
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share
>
> ----- Original Message ----
> From: Mike Klaas <[hidden email]>
> To: [hidden email]
> Sent: Thursday, April 5, 2007 1:58:38 PM
> Subject: Re: Lucene and Javolution: A good mix ?
>
> On 4/4/07, Jean-Philippe Robichaud <Jean-
> [hidden email]>
> wrote:
>> I understand your concerns!
>>
>> I was a little skeptical at the beginning.  But even with the 1.5  
>> jvm,
>> the improvements still holds.
>>
>> Lucene creates a lots of "garbage" (strings, tokens, ...) either at
>> index time or query time. While the new garbage collector strategies
> did
>> seriously improve since java 1.4, the gains are still there as the
>> object "creation" is also a cost that javolution easily saves us  
>> from.
>
> I think the best approach at convincing people would be to produce a
> patch that implements some of the suggested changes, and benchmark it.
>  As it stands, the positives are all hypothetical and the negatives
> rather tangible.
>
> -MIke
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

------------------------------------------------------
Grant Ingersoll
http://www.grantingersoll.com/
http://lucene.grantingersoll.com
http://www.paperoftheweek.com/



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]