Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Yueyu Lin
Hi,
  We're building a server based Lucene. When doing the multiple threads performance test, we found a minor synchronized issue.

  We found if we were using 2 IndexSearcher, we would get 10% performance benefit.
  But if we increased the number of IndexSearcher from 2, the performance improvement became slight even worse.
  To find why it is so, we traced into the codes. Fortunately we got it.

  The class org.apache.lucene.index.TermInfosReader , as you know, every IndexSearcher will have one TermInfosReader. Every query, one method in the class must be called:
private synchronized void ensureIndexIsRead() throws IOException . Notice it, it's a synchronized function to ensure if the index is readable. Although, if it's initialized, it will return immediately, but it's still a lock. So if we use two IndexSearcher, the opportunity to wait will decrease a lot. In some conditions for an example, a busy system and the query is
relatively quick, the opportunity to wait will increase a lot.
  The attachement is our modification for that code. It might be helpful to increase the concurrency performance while querying. Can you read it and file that as a bug to fix in the next release?

  The main change is like below:
  If you accept it, the sysout can be removed :).
     ....
    private void ensureIndexIsRead() throws IOException {
    if (indexTerms != null)                       // index already read
      return;                                     // do nothing
    synchronized(this){
        System.out.println("Read index@--@");
        if(indexTerms != null){
            System.out.println ("Someone read it.return-_-");
            return ;
        }
        readIndex ();
    }
  }
 
  private synchronized void readIndex() throws IOException{
      Term[] m_indexTerms = null;
      try {
          int indexSize = (int)indexEnum.size;        // otherwise read index
          m_indexTerms = new Term[indexSize];
          indexInfos = new TermInfo[indexSize];
          indexPointers = new long[indexSize];

          for (int i = 0; indexEnum.next(); i++) {
            m_indexTerms[i] = indexEnum.term();
            indexInfos[i] = indexEnum.termInfo();
            indexPointers[i] = indexEnum.indexPointer;
          }
        } finally {
            indexEnum.close();
            indexEnum = null;
            indexTerms = m_indexTerms;
        }
  }
--
--
Yueyu Lin
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

TermInfosReader.java (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Chris Hostetter-3

:   We found if we were using 2 IndexSearcher, we would get 10% performance
: benefit.
:   But if we increased the number of IndexSearcher from 2, the performance
: improvement became slight even worse.

Why use more then 2 IndexSearchers?

Typically 1 is all you need, except for when you want to open and "warm
up" a new Searcher because you know your index has changed on disk and
you're ready for those changes to be visible.

(I'm not arguing against your change -- concurrancy isn't my forte so i
have no opinion on wether your suggesting is good or not, i'm just
questioning the goal)

Acctually .. i don't know a lot about the internals of IndexSearcher and
TermInfosReader, but according to your description of the problem...

:   The class org.apache.lucene.index.TermInfosReader , as you know, every
: IndexSearcher will have one TermInfosReader. Every query, one method in the
: class must be called:
: private synchronized void ensureIndexIsRead() throws IOException . Notice

If the method isn't static, then how can two differnet instances of
IndexSearcher, each with their own TermInfosReader, block one another?




-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Yueyu Lin
Please trace the codes into the Lucene when searching.
Here is a table about how invokations are called.
The trace log:   *Steps*
 *ClassName*
 *Functions*
 *Description*
  1.  org.apache.lucene.search.Searcher  public final Hits search(Query
query)  It will call another search function.   2.
org.apache.lucene.search.Searcher  public Hits search(Query query, Filter
filter)  Only one line code. It will new a Hits.
return new Hits(this, query, filter);   3.
org.apache.lucene.search.Hits Hits(Searcher s, Query q, Filter f)
Next, we will trace into the constructor to see what stuffs will be
done.  4.
org.apache.lucene.search.Hits  Hits(Searcher s, Query q, Filter f)
line 41 : weight = q.weight(s)  This call will rewrite the Query if
necessary, let us to see what will happen then.


  5.  org.apache.lucene.search.Query  public Weight weight(Searcher
searcher)
line 92: Query query = searcher.rewrite(this);  This call will begin to
rewrite the Query.   6.  *org.apache.lucene.search.IndexSearcher*  public
Query rewrite(Query original)  NOTE: we only have one IndexSearcher which
has one IndexReader. If there is any functioins that are synchronized, the
query process will be queued.   7.
org.apache.lucene.search.BooleanQuery public Query rewrite(IndexReader
reader)
line 396: Query query = c.getQuery().rewrite(reader);  Here, BooleanQuery
will get its subqueries and call their rewrite function. The function will
require to pass a parameter: *IndexReader* that we only have one instance.
From the codes we will notice *TermQuery* will not be rewrote and *
PrefixQuery* will be rewrote to several *TermQuery*s. So we ignore the *
TermQuery* and look into the *PrefixQuery*.   8.
org.apache.lucene.search.PrefixQuery  public Query rewrite(IndexReader
reader)
line 41: TermEnum enumerator = reader.terms(prefix);  Let's see what will
happen then.   9.  org.apache.lucene.index.SegmentReader  public TermEnum
terms(Term t)
line 277: return tis.terms(t);  SegmentReader is in fact an IndexReader's
implementation.   10.  org.apache.lucene.index.TermInfosReader  public
SegmentTermEnum terms(Term term)
line 211:get(term);

  11.  org.apache.lucene.index.TermInfosReader  TermInfo get(Term term)
line 136:ensureIndexIsRead();  We finally find it!   12.
org.apache.lucene.index.TermInfosReader  private synchronized void
ensureIndexIsRead()  Let's analyze the function and to see why it's
synchronized and how to improve it.

On 5/9/06, Chris Hostetter <[hidden email]> wrote:

>
>
> :   We found if we were using 2 IndexSearcher, we would get 10%
> performance
> : benefit.
> :   But if we increased the number of IndexSearcher from 2, the
> performance
> : improvement became slight even worse.
>
> Why use more then 2 IndexSearchers?
>
> Typically 1 is all you need, except for when you want to open and "warm
> up" a new Searcher because you know your index has changed on disk and
> you're ready for those changes to be visible.
>
> (I'm not arguing against your change -- concurrancy isn't my forte so i
> have no opinion on wether your suggesting is good or not, i'm just
> questioning the goal)
>
> Acctually .. i don't know a lot about the internals of IndexSearcher and
> TermInfosReader, but according to your description of the problem...
>
> :   The class org.apache.lucene.index.TermInfosReader , as you know, every
> : IndexSearcher will have one TermInfosReader. Every query, one method in
> the
> : class must be called:
> : private synchronized void ensureIndexIsRead() throws IOException .
> Notice
>
> If the method isn't static, then how can two differnet instances of
> IndexSearcher, each with their own TermInfosReader, block one another?
>
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
--
Yueyu Lin
Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Yueyu Lin
ooops, the table seems twisted.
Can you see that clearly?

On 5/9/06, yueyu lin <[hidden email]> wrote:

>
> Please trace the codes into the Lucene when searching.
> Here is a table about how invokations are called.
> The trace log:   *Steps*
>  *ClassName*
>  *Functions*
>  *Description*
>   1.  org.apache.lucene.search.Searcher  public final Hits search(Query
> query)  It will call another search function.   2.
> org.apache.lucene.search.Searcher  public Hits search(Query query, Filter
> filter)  Only one line code. It will new a Hits.
> return new Hits(this, query, filter);   3.  org.apache.lucene.search.Hits Hits(Searcher s, Query q, Filter f) Next, we will trace into the constructor to see what stuffs will be done.  4.
> org.apache.lucene.search.Hits  Hits(Searcher s, Query q, Filter f)
> line 41 : weight = q.weight(s)  This call will rewrite the Query if
> necessary, let us to see what will happen then.
>
>
>   5.  org.apache.lucene.search.Query  public Weight weight(Searcher
> searcher)
> line 92: Query query = searcher.rewrite(this);  This call will begin to
> rewrite the Query.   6.  *org.apache.lucene.search.IndexSearcher*  public
> Query rewrite(Query original)  NOTE: we only have one IndexSearcher which
> has one IndexReader. If there is any functioins that are synchronized, the
> query process will be queued.   7.  org.apache.lucene.search.BooleanQuery public Query rewrite(IndexReader reader)
> line 396: Query query = c.getQuery().rewrite(reader);  Here, BooleanQuery
> will get its subqueries and call their rewrite function. The function will
> require to pass a parameter: *IndexReader* that we only have one instance.
> From the codes we will notice *TermQuery* will not be rewrote and *
> PrefixQuery* will be rewrote to several *TermQuery*s. So we ignore the *
> TermQuery* and look into the *PrefixQuery*.   8.
> org.apache.lucene.search.PrefixQuery  public Query rewrite(IndexReader
> reader)
> line 41: TermEnum enumerator = reader.terms(prefix);  Let's see what will
> happen then.   9.  org.apache.lucene.index.SegmentReader  public TermEnum
> terms(Term t)
> line 277: return tis.terms(t);  SegmentReader is in fact an IndexReader's
> implementation.   10.
> org.apache.lucene.index.TermInfosReader
> public SegmentTermEnum terms(Term term)
> line 211:get(term);
>
>   11.
> org.apache.lucene.index.TermInfosReader
> TermInfo get(Term term)
> line 136:ensureIndexIsRead();  We finally find it!   12.
> org.apache.lucene.index.TermInfosReader
> private synchronized void ensureIndexIsRead()  Let's analyze the function
> and to see why it's synchronized and how to improve it.
>
> On 5/9/06, Chris Hostetter <[hidden email]> wrote:
> >
> >
> > :   We found if we were using 2 IndexSearcher, we would get 10%
> > performance
>
>
> : benefit.
> :   But if we increased the number of IndexSearcher from 2, the
> performance
> : improvement became slight even worse.
>
> Why use more then 2 IndexSearchers?
>
> Typically 1 is all you need, except for when you want to open and "warm
> up" a new Searcher because you know your index has changed on disk and
> you're ready for those changes to be visible.
>
> (I'm not arguing against your change -- concurrancy isn't my forte so i
> have no opinion on wether your suggesting is good or not, i'm just
> questioning the goal)
>
> Acctually .. i don't know a lot about the internals of IndexSearcher and
> TermInfosReader, but according to your description of the problem...
>
> :   The class org.apache.lucene.index.TermInfosReader , as you know, every
> : IndexSearcher will have one TermInfosReader. Every query, one method in
> the
> : class must be called:
> : private synchronized void ensureIndexIsRead() throws IOException .
> Notice
>
> If the method isn't static, then how can two differnet instances of
> IndexSearcher, each with their own TermInfosReader, block one another?
>
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
>
>
> --
> --
> Yueyu Lin
>



--
--
Yueyu Lin
Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Yueyu Lin
In reply to this post by Chris Hostetter-3
One IndexSearcher is one IndexSearcher instance. The instance has a lot of
functions. Unfortunately they will call another synchronized function in
other class's instance (TermInfosReader). That's the point why we need two
IndexSearchers. But two searchers will cost double cache memory. It's not
worthy. So if Lucene team can modify the codes slightly, the synchronization
problem will be gone.

On 5/9/06, Chris Hostetter <[hidden email]> wrote:

>
>
> :   We found if we were using 2 IndexSearcher, we would get 10%
> performance
> : benefit.
> :   But if we increased the number of IndexSearcher from 2, the
> performance
> : improvement became slight even worse.
>
> Why use more then 2 IndexSearchers?
>
> Typically 1 is all you need, except for when you want to open and "warm
> up" a new Searcher because you know your index has changed on disk and
> you're ready for those changes to be visible.
>
> (I'm not arguing against your change -- concurrancy isn't my forte so i
> have no opinion on wether your suggesting is good or not, i'm just
> questioning the goal)
>
> Acctually .. i don't know a lot about the internals of IndexSearcher and
> TermInfosReader, but according to your description of the problem...
>
> :   The class org.apache.lucene.index.TermInfosReader , as you know, every
> : IndexSearcher will have one TermInfosReader. Every query, one method in
> the
> : class must be called:
> : private synchronized void ensureIndexIsRead() throws IOException .
> Notice
>
> If the method isn't static, then how can two differnet instances of
> IndexSearcher, each with their own TermInfosReader, block one another?
>
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
--
Yueyu Lin
Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Otis Gospodnetic-2
In reply to this post by Yueyu Lin
Yueyu Lin,

From what I can tell from a quick look at the method, that method need to remain synchronized, so multiple threads don't accidentally re-read that 'indexTerms' (Term[] type).  Even though the method is synchronized, it looks like only the first invocation would enter that try/catch/finally block where term reading happens.  Subsequent calls to this method should exist quickly, because indexTerms != null.

Are you sure this is causing the bottleneck for you?
I believe the proper way to figure that out is to kill the JVM with a SIGnal that causes the JVM to dump thread information.  That would tell you where the code is blocking.

Also, if you have concrete suggestions for code changes, please post them to JIRA as diffs/patches.

Otis


----- Original Message ----
From: yueyu lin <[hidden email]>
To: [hidden email]
Sent: Tuesday, May 9, 2006 3:53:55 AM
Subject: Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Please trace the codes into the Lucene when searching.
Here is a table about how invokations are called.
The trace log:   *Steps*
 *ClassName*
 *Functions*
 *Description*
  1.  org.apache.lucene.search.Searcher  public final Hits search(Query
query)  It will call another search function.   2.
org.apache.lucene.search.Searcher  public Hits search(Query query, Filter
filter)  Only one line code. It will new a Hits.
return new Hits(this, query, filter);   3.
org.apache.lucene.search.Hits Hits(Searcher s, Query q, Filter f)
Next, we will trace into the constructor to see what stuffs will be
done.  4.
org.apache.lucene.search.Hits  Hits(Searcher s, Query q, Filter f)
line 41 : weight = q.weight(s)  This call will rewrite the Query if
necessary, let us to see what will happen then.


  5.  org.apache.lucene.search.Query  public Weight weight(Searcher
searcher)
line 92: Query query = searcher.rewrite(this);  This call will begin to
rewrite the Query.   6.  *org.apache.lucene.search.IndexSearcher*  public
Query rewrite(Query original)  NOTE: we only have one IndexSearcher which
has one IndexReader. If there is any functioins that are synchronized, the
query process will be queued.   7.
org.apache.lucene.search.BooleanQuery public Query rewrite(IndexReader
reader)
line 396: Query query = c.getQuery().rewrite(reader);  Here, BooleanQuery
will get its subqueries and call their rewrite function. The function will
require to pass a parameter: *IndexReader* that we only have one instance.
From the codes we will notice *TermQuery* will not be rewrote and *
PrefixQuery* will be rewrote to several *TermQuery*s. So we ignore the *
TermQuery* and look into the *PrefixQuery*.   8.
org.apache.lucene.search.PrefixQuery  public Query rewrite(IndexReader
reader)
line 41: TermEnum enumerator = reader.terms(prefix);  Let's see what will
happen then.   9.  org.apache.lucene.index.SegmentReader  public TermEnum
terms(Term t)
line 277: return tis.terms(t);  SegmentReader is in fact an IndexReader's
implementation.   10.  org.apache.lucene.index.TermInfosReader  public
SegmentTermEnum terms(Term term)
line 211:get(term);

  11.  org.apache.lucene.index.TermInfosReader  TermInfo get(Term term)
line 136:ensureIndexIsRead();  We finally find it!   12.
org.apache.lucene.index.TermInfosReader  private synchronized void
ensureIndexIsRead()  Let's analyze the function and to see why it's
synchronized and how to improve it.

On 5/9/06, Chris Hostetter <[hidden email]> wrote:

>
>
> :   We found if we were using 2 IndexSearcher, we would get 10%
> performance
> : benefit.
> :   But if we increased the number of IndexSearcher from 2, the
> performance
> : improvement became slight even worse.
>
> Why use more then 2 IndexSearchers?
>
> Typically 1 is all you need, except for when you want to open and "warm
> up" a new Searcher because you know your index has changed on disk and
> you're ready for those changes to be visible.
>
> (I'm not arguing against your change -- concurrancy isn't my forte so i
> have no opinion on wether your suggesting is good or not, i'm just
> questioning the goal)
>
> Acctually .. i don't know a lot about the internals of IndexSearcher and
> TermInfosReader, but according to your description of the problem...
>
> :   The class org.apache.lucene.index.TermInfosReader , as you know, every
> : IndexSearcher will have one TermInfosReader. Every query, one method in
> the
> : class must be called:
> : private synchronized void ensureIndexIsRead() throws IOException .
> Notice
>
> If the method isn't static, then how can two differnet instances of
> IndexSearcher, each with their own TermInfosReader, block one another?
>
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
--
Yueyu Lin




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Doug Cutting
In reply to this post by Yueyu Lin
The best search performance is achieved using a single IndexSearcher
shared by multiple threads.  Peter Keegan has demonstrated rates of up
to 400 searches per second on eight-CPU machines using this approach:

http://www.mail-archive.com/java-user@.../msg05074.html

So the synchronization is probably not hurting your performance.

Doug

yueyu lin wrote:

> One IndexSearcher is one IndexSearcher instance. The instance has a lot of
> functions. Unfortunately they will call another synchronized function in
> other class's instance (TermInfosReader). That's the point why we need two
> IndexSearchers. But two searchers will cost double cache memory. It's not
> worthy. So if Lucene team can modify the codes slightly, the
> synchronization
> problem will be gone.
>
> On 5/9/06, Chris Hostetter <[hidden email]> wrote:
>
>>
>>
>> :   We found if we were using 2 IndexSearcher, we would get 10%
>> performance
>> : benefit.
>> :   But if we increased the number of IndexSearcher from 2, the
>> performance
>> : improvement became slight even worse.
>>
>> Why use more then 2 IndexSearchers?
>>
>> Typically 1 is all you need, except for when you want to open and "warm
>> up" a new Searcher because you know your index has changed on disk and
>> you're ready for those changes to be visible.
>>
>> (I'm not arguing against your change -- concurrancy isn't my forte so i
>> have no opinion on wether your suggesting is good or not, i'm just
>> questioning the goal)
>>
>> Acctually .. i don't know a lot about the internals of IndexSearcher and
>> TermInfosReader, but according to your description of the problem...
>>
>> :   The class org.apache.lucene.index.TermInfosReader , as you know,
>> every
>> : IndexSearcher will have one TermInfosReader. Every query, one method in
>> the
>> : class must be called:
>> : private synchronized void ensureIndexIsRead() throws IOException .
>> Notice
>>
>> If the method isn't static, then how can two differnet instances of
>> IndexSearcher, each with their own TermInfosReader, block one another?
>>
>>
>>
>>
>> -Hoss
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
>
> --
> --
> Yueyu Lin
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Yueyu Lin
In reply to this post by Otis Gospodnetic-2
Yes, the modification is still synchronized and the first thread will be
responsible for reading first. And then other threads will not read and the
synchronization is unnecessary.
private void ensureIndexIsRead() throws IOException {
    if (indexTerms != null)                       // index already read
      return;                                     // do nothing
    synchronized(this){
        System.out.println("Read index@--@");
        if(indexTerms != null){
            System.out.println ("Someone read it.return-_-");
            return ;
        }
        readIndex ();
    }
  }

  private synchronized void readIndex() throws IOException{
      Term[] m_indexTerms = null;
      try {
          int indexSize = (int)indexEnum.size;        // otherwise read
index
          m_indexTerms = new Term[indexSize];
          indexInfos = new TermInfo[indexSize];
          indexPointers = new long[indexSize];

          for (int i = 0; indexEnum.next(); i++) {
            m_indexTerms[i] = indexEnum.term();
            indexInfos[i] = indexEnum.termInfo();
            indexPointers[i] = indexEnum.indexPointer;
          }
        } finally {
            indexEnum.close();
            indexEnum = null;
            indexTerms = m_indexTerms;
        }
  }

That's a small trick I learned when I was developing a busy stock system.

If the method ensureIndexIsRead() is synchronized, it will be blocked for a
while, although even 2 lines codes.


On 5/9/06, Otis Gospodnetic <[hidden email]> wrote:

>
> Yueyu Lin,
>
> From what I can tell from a quick look at the method, that method need to
> remain synchronized, so multiple threads don't accidentally re-read that
> 'indexTerms' (Term[] type).  Even though the method is synchronized, it
> looks like only the first invocation would enter that try/catch/finally
> block where term reading happens.  Subsequent calls to this method should
> exist quickly, because indexTerms != null.
>
> Are you sure this is causing the bottleneck for you?
> I believe the proper way to figure that out is to kill the JVM with a
> SIGnal that causes the JVM to dump thread information.  That would tell you
> where the code is blocking.
>
> Also, if you have concrete suggestions for code changes, please post them
> to JIRA as diffs/patches.
>
> Otis
>
>
> ----- Original Message ----
> From: yueyu lin <[hidden email]>
> To: [hidden email]
> Sent: Tuesday, May 9, 2006 3:53:55 AM
> Subject: Re: Multiple threads searching in Lucene and the synchronized
> issue. -- solution attached.
>
> Please trace the codes into the Lucene when searching.
> Here is a table about how invokations are called.
> The trace log:   *Steps*
> *ClassName*
> *Functions*
> *Description*
>   1.  org.apache.lucene.search.Searcher  public final Hits search(Query
> query)  It will call another search function.   2.
> org.apache.lucene.search.Searcher  public Hits search(Query query, Filter
> filter)  Only one line code. It will new a Hits.
> return new Hits(this, query, filter);   3.
> org.apache.lucene.search.Hits Hits(Searcher s, Query q, Filter f)
> Next, we will trace into the constructor to see what stuffs will be
> done.  4.
> org.apache.lucene.search.Hits  Hits(Searcher s, Query q, Filter f)
> line 41 : weight = q.weight(s)  This call will rewrite the Query if
> necessary, let us to see what will happen then.
>
>
>   5.  org.apache.lucene.search.Query  public Weight weight(Searcher
> searcher)
> line 92: Query query = searcher.rewrite(this);  This call will begin to
> rewrite the Query.   6.  *org.apache.lucene.search.IndexSearcher*  public
> Query rewrite(Query original)  NOTE: we only have one IndexSearcher which
> has one IndexReader. If there is any functioins that are synchronized, the
> query process will be queued.   7.
> org.apache.lucene.search.BooleanQuery public Query rewrite(IndexReader
> reader)
> line 396: Query query = c.getQuery().rewrite(reader);  Here, BooleanQuery
> will get its subqueries and call their rewrite function. The function will
> require to pass a parameter: *IndexReader* that we only have one instance.
> From the codes we will notice *TermQuery* will not be rewrote and *
> PrefixQuery* will be rewrote to several *TermQuery*s. So we ignore the *
> TermQuery* and look into the *PrefixQuery*.   8.
> org.apache.lucene.search.PrefixQuery  public Query rewrite(IndexReader
> reader)
> line 41: TermEnum enumerator = reader.terms(prefix);  Let's see what will
> happen then.   9.  org.apache.lucene.index.SegmentReader  public TermEnum
> terms(Term t)
> line 277: return tis.terms(t);  SegmentReader is in fact an IndexReader's
> implementation.   10.  org.apache.lucene.index.TermInfosReader  public
> SegmentTermEnum terms(Term term)
> line 211:get(term);
>
>   11.  org.apache.lucene.index.TermInfosReader  TermInfo get(Term term)
> line 136:ensureIndexIsRead();  We finally find it!   12.
> org.apache.lucene.index.TermInfosReader  private synchronized void
> ensureIndexIsRead()  Let's analyze the function and to see why it's
> synchronized and how to improve it.
>
> On 5/9/06, Chris Hostetter <[hidden email]> wrote:
> >
> >
> > :   We found if we were using 2 IndexSearcher, we would get 10%
> > performance
> > : benefit.
> > :   But if we increased the number of IndexSearcher from 2, the
> > performance
> > : improvement became slight even worse.
> >
> > Why use more then 2 IndexSearchers?
> >
> > Typically 1 is all you need, except for when you want to open and "warm
> > up" a new Searcher because you know your index has changed on disk and
> > you're ready for those changes to be visible.
> >
> > (I'm not arguing against your change -- concurrancy isn't my forte so i
> > have no opinion on wether your suggesting is good or not, i'm just
> > questioning the goal)
> >
> > Acctually .. i don't know a lot about the internals of IndexSearcher and
> > TermInfosReader, but according to your description of the problem...
> >
> > :   The class org.apache.lucene.index.TermInfosReader , as you know,
> every
> > : IndexSearcher will have one TermInfosReader. Every query, one method
> in
> > the
> > : class must be called:
> > : private synchronized void ensureIndexIsRead() throws IOException .
> > Notice
> >
> > If the method isn't static, then how can two differnet instances of
> > IndexSearcher, each with their own TermInfosReader, block one another?
> >
> >
> >
> >
> > -Hoss
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> --
> --
> Yueyu Lin
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
--
Yueyu Lin
Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Yueyu Lin
In reply to this post by Doug Cutting
My assumption is that every query is relatively quick. If the times lapsed
in other process when querying, the ensureIndexIsRead() function will not
cause a lot of problems. If not, the ensureIndexIsRead() function will be a
bottle neck.
I could understand that a lot of systems' queries are quiet complex, so the
problem may be gone. But for our system, more than 150 queries per seconds
on a dual CPUs linux box, that's a problem.

If the queries became more complicated, we would ignore it in most of time.

On 5/10/06, Doug Cutting <[hidden email]> wrote:

>
> The best search performance is achieved using a single IndexSearcher
> shared by multiple threads.  Peter Keegan has demonstrated rates of up
> to 400 searches per second on eight-CPU machines using this approach:
>
> http://www.mail-archive.com/java-user@.../msg05074.html
>
> So the synchronization is probably not hurting your performance.
>
> Doug
>
> yueyu lin wrote:
> > One IndexSearcher is one IndexSearcher instance. The instance has a lot
> of
> > functions. Unfortunately they will call another synchronized function in
> > other class's instance (TermInfosReader). That's the point why we need
> two
> > IndexSearchers. But two searchers will cost double cache memory. It's
> not
> > worthy. So if Lucene team can modify the codes slightly, the
> > synchronization
> > problem will be gone.
> >
> > On 5/9/06, Chris Hostetter <[hidden email]> wrote:
> >
> >>
> >>
> >> :   We found if we were using 2 IndexSearcher, we would get 10%
> >> performance
> >> : benefit.
> >> :   But if we increased the number of IndexSearcher from 2, the
> >> performance
> >> : improvement became slight even worse.
> >>
> >> Why use more then 2 IndexSearchers?
> >>
> >> Typically 1 is all you need, except for when you want to open and "warm
> >> up" a new Searcher because you know your index has changed on disk and
> >> you're ready for those changes to be visible.
> >>
> >> (I'm not arguing against your change -- concurrancy isn't my forte so i
> >> have no opinion on wether your suggesting is good or not, i'm just
> >> questioning the goal)
> >>
> >> Acctually .. i don't know a lot about the internals of IndexSearcher
> and
> >> TermInfosReader, but according to your description of the problem...
> >>
> >> :   The class org.apache.lucene.index.TermInfosReader , as you know,
> >> every
> >> : IndexSearcher will have one TermInfosReader. Every query, one method
> in
> >> the
> >> : class must be called:
> >> : private synchronized void ensureIndexIsRead() throws IOException .
> >> Notice
> >>
> >> If the method isn't static, then how can two differnet instances of
> >> IndexSearcher, each with their own TermInfosReader, block one another?
> >>
> >>
> >>
> >>
> >> -Hoss
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
> >
> > --
> > --
> > Yueyu Lin
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
--
Yueyu Lin
Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Yueyu Lin
In reply to this post by Otis Gospodnetic-2
Oh,please believe in me that I've forced the JVM to print the thread dump.
It waited here indeed.
I'll try to post the patch to JIRA.
I don't want to modify these codes by myself because that will break the
Lucene codes. So I wish you can do me the favor to check these codes and
make it availabe in the next release.
On 5/9/06, Otis Gospodnetic <[hidden email]> wrote:

>
> Yueyu Lin,
>
> From what I can tell from a quick look at the method, that method need to
> remain synchronized, so multiple threads don't accidentally re-read that
> 'indexTerms' (Term[] type).  Even though the method is synchronized, it
> looks like only the first invocation would enter that try/catch/finally
> block where term reading happens.  Subsequent calls to this method should
> exist quickly, because indexTerms != null.
>
> Are you sure this is causing the bottleneck for you?
> I believe the proper way to figure that out is to kill the JVM with a
> SIGnal that causes the JVM to dump thread information.  That would tell you
> where the code is blocking.
>
> Also, if you have concrete suggestions for code changes, please post them
> to JIRA as diffs/patches.
>
> Otis
>
>
> ----- Original Message ----
> From: yueyu lin <[hidden email]>
> To: [hidden email]
> Sent: Tuesday, May 9, 2006 3:53:55 AM
> Subject: Re: Multiple threads searching in Lucene and the synchronized
> issue. -- solution attached.
>
> Please trace the codes into the Lucene when searching.
> Here is a table about how invokations are called.
> The trace log:   *Steps*
> *ClassName*
> *Functions*
> *Description*
>   1.  org.apache.lucene.search.Searcher  public final Hits search(Query
> query)  It will call another search function.   2.
> org.apache.lucene.search.Searcher  public Hits search(Query query, Filter
> filter)  Only one line code. It will new a Hits.
> return new Hits(this, query, filter);   3.
> org.apache.lucene.search.Hits Hits(Searcher s, Query q, Filter f)
> Next, we will trace into the constructor to see what stuffs will be
> done.  4.
> org.apache.lucene.search.Hits  Hits(Searcher s, Query q, Filter f)
> line 41 : weight = q.weight(s)  This call will rewrite the Query if
> necessary, let us to see what will happen then.
>
>
>   5.  org.apache.lucene.search.Query  public Weight weight(Searcher
> searcher)
> line 92: Query query = searcher.rewrite(this);  This call will begin to
> rewrite the Query.   6.  *org.apache.lucene.search.IndexSearcher*  public
> Query rewrite(Query original)  NOTE: we only have one IndexSearcher which
> has one IndexReader. If there is any functioins that are synchronized, the
> query process will be queued.   7.
> org.apache.lucene.search.BooleanQuery public Query rewrite(IndexReader
> reader)
> line 396: Query query = c.getQuery().rewrite(reader);  Here, BooleanQuery
> will get its subqueries and call their rewrite function. The function will
> require to pass a parameter: *IndexReader* that we only have one instance.
> From the codes we will notice *TermQuery* will not be rewrote and *
> PrefixQuery* will be rewrote to several *TermQuery*s. So we ignore the *
> TermQuery* and look into the *PrefixQuery*.   8.
> org.apache.lucene.search.PrefixQuery  public Query rewrite(IndexReader
> reader)
> line 41: TermEnum enumerator = reader.terms(prefix);  Let's see what will
> happen then.   9.  org.apache.lucene.index.SegmentReader  public TermEnum
> terms(Term t)
> line 277: return tis.terms(t);  SegmentReader is in fact an IndexReader's
> implementation.   10.  org.apache.lucene.index.TermInfosReader  public
> SegmentTermEnum terms(Term term)
> line 211:get(term);
>
>   11.  org.apache.lucene.index.TermInfosReader  TermInfo get(Term term)
> line 136:ensureIndexIsRead();  We finally find it!   12.
> org.apache.lucene.index.TermInfosReader  private synchronized void
> ensureIndexIsRead()  Let's analyze the function and to see why it's
> synchronized and how to improve it.
>
> On 5/9/06, Chris Hostetter <[hidden email]> wrote:
> >
> >
> > :   We found if we were using 2 IndexSearcher, we would get 10%
> > performance
> > : benefit.
> > :   But if we increased the number of IndexSearcher from 2, the
> > performance
> > : improvement became slight even worse.
> >
> > Why use more then 2 IndexSearchers?
> >
> > Typically 1 is all you need, except for when you want to open and "warm
> > up" a new Searcher because you know your index has changed on disk and
> > you're ready for those changes to be visible.
> >
> > (I'm not arguing against your change -- concurrancy isn't my forte so i
> > have no opinion on wether your suggesting is good or not, i'm just
> > questioning the goal)
> >
> > Acctually .. i don't know a lot about the internals of IndexSearcher and
> > TermInfosReader, but according to your description of the problem...
> >
> > :   The class org.apache.lucene.index.TermInfosReader , as you know,
> every
> > : IndexSearcher will have one TermInfosReader. Every query, one method
> in
> > the
> > : class must be called:
> > : private synchronized void ensureIndexIsRead() throws IOException .
> > Notice
> >
> > If the method isn't static, then how can two differnet instances of
> > IndexSearcher, each with their own TermInfosReader, block one another?
> >
> >
> >
> >
> > -Hoss
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> --
> --
> Yueyu Lin
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
--
Yueyu Lin
Reply | Threaded
Open this post in threaded view
|

RE: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Robert Engels
In reply to this post by Yueyu Lin
I am interested in the exact performance difference in ms per query removing
the synchronized block?

I can see that after a while when using your code, the JIT will probably
inline the 'non-reading' path.

Even then...

I would not think that 2 lines of synchronized code would contribute much
when even the simplest of queries needs to execute thousands of lines of
code (and probably at least a few calls to the OS, except when using a pure
memory index).

-----Original Message-----
From: yueyu lin [mailto:[hidden email]]
Sent: Tuesday, May 09, 2006 8:40 PM
To: [hidden email]
Subject: Re: Multiple threads searching in Lucene and the synchronized
issue. -- solution attached.


My assumption is that every query is relatively quick. If the times lapsed
in other process when querying, the ensureIndexIsRead() function will not
cause a lot of problems. If not, the ensureIndexIsRead() function will be a
bottle neck.
I could understand that a lot of systems' queries are quiet complex, so the
problem may be gone. But for our system, more than 150 queries per seconds
on a dual CPUs linux box, that's a problem.

If the queries became more complicated, we would ignore it in most of time.

On 5/10/06, Doug Cutting <[hidden email]> wrote:

>
> The best search performance is achieved using a single IndexSearcher
> shared by multiple threads.  Peter Keegan has demonstrated rates of up
> to 400 searches per second on eight-CPU machines using this approach:
>
> http://www.mail-archive.com/java-user@.../msg05074.html
>
> So the synchronization is probably not hurting your performance.
>
> Doug
>
> yueyu lin wrote:
> > One IndexSearcher is one IndexSearcher instance. The instance has a lot
> of
> > functions. Unfortunately they will call another synchronized function in
> > other class's instance (TermInfosReader). That's the point why we need
> two
> > IndexSearchers. But two searchers will cost double cache memory. It's
> not
> > worthy. So if Lucene team can modify the codes slightly, the
> > synchronization
> > problem will be gone.
> >
> > On 5/9/06, Chris Hostetter <[hidden email]> wrote:
> >
> >>
> >>
> >> :   We found if we were using 2 IndexSearcher, we would get 10%
> >> performance
> >> : benefit.
> >> :   But if we increased the number of IndexSearcher from 2, the
> >> performance
> >> : improvement became slight even worse.
> >>
> >> Why use more then 2 IndexSearchers?
> >>
> >> Typically 1 is all you need, except for when you want to open and "warm
> >> up" a new Searcher because you know your index has changed on disk and
> >> you're ready for those changes to be visible.
> >>
> >> (I'm not arguing against your change -- concurrancy isn't my forte so i
> >> have no opinion on wether your suggesting is good or not, i'm just
> >> questioning the goal)
> >>
> >> Acctually .. i don't know a lot about the internals of IndexSearcher
> and
> >> TermInfosReader, but according to your description of the problem...
> >>
> >> :   The class org.apache.lucene.index.TermInfosReader , as you know,
> >> every
> >> : IndexSearcher will have one TermInfosReader. Every query, one method
> in
> >> the
> >> : class must be called:
> >> : private synchronized void ensureIndexIsRead() throws IOException .
> >> Notice
> >>
> >> If the method isn't static, then how can two differnet instances of
> >> IndexSearcher, each with their own TermInfosReader, block one another?
> >>
> >>
> >>
> >>
> >> -Hoss
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
> >
> > --
> > --
> > Yueyu Lin
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
--
Yueyu Lin


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Robert Engels
In reply to this post by Yueyu Lin
I think your basic problem is that you are using multiple IndexSearchers?
And creating new instances during runtime? If so, you will be reading the
index information far too often. This is not a good configuration.

-----Original Message-----
From: yueyu lin [mailto:[hidden email]]
Sent: Tuesday, May 09, 2006 8:46 PM
To: [hidden email]; Otis Gospodnetic
Subject: Re: Multiple threads searching in Lucene and the synchronized
issue. -- solution attached.


Oh,please believe in me that I've forced the JVM to print the thread dump.
It waited here indeed.
I'll try to post the patch to JIRA.
I don't want to modify these codes by myself because that will break the
Lucene codes. So I wish you can do me the favor to check these codes and
make it availabe in the next release.
On 5/9/06, Otis Gospodnetic <[hidden email]> wrote:

>
> Yueyu Lin,
>
> From what I can tell from a quick look at the method, that method need to
> remain synchronized, so multiple threads don't accidentally re-read that
> 'indexTerms' (Term[] type).  Even though the method is synchronized, it
> looks like only the first invocation would enter that try/catch/finally
> block where term reading happens.  Subsequent calls to this method should
> exist quickly, because indexTerms != null.
>
> Are you sure this is causing the bottleneck for you?
> I believe the proper way to figure that out is to kill the JVM with a
> SIGnal that causes the JVM to dump thread information.  That would tell
you

> where the code is blocking.
>
> Also, if you have concrete suggestions for code changes, please post them
> to JIRA as diffs/patches.
>
> Otis
>
>
> ----- Original Message ----
> From: yueyu lin <[hidden email]>
> To: [hidden email]
> Sent: Tuesday, May 9, 2006 3:53:55 AM
> Subject: Re: Multiple threads searching in Lucene and the synchronized
> issue. -- solution attached.
>
> Please trace the codes into the Lucene when searching.
> Here is a table about how invokations are called.
> The trace log:   *Steps*
> *ClassName*
> *Functions*
> *Description*
>   1.  org.apache.lucene.search.Searcher  public final Hits search(Query
> query)  It will call another search function.   2.
> org.apache.lucene.search.Searcher  public Hits search(Query query, Filter
> filter)  Only one line code. It will new a Hits.
> return new Hits(this, query, filter);   3.
> org.apache.lucene.search.Hits Hits(Searcher s, Query q, Filter f)
> Next, we will trace into the constructor to see what stuffs will be
> done.  4.
> org.apache.lucene.search.Hits  Hits(Searcher s, Query q, Filter f)
> line 41 : weight = q.weight(s)  This call will rewrite the Query if
> necessary, let us to see what will happen then.
>
>
>   5.  org.apache.lucene.search.Query  public Weight weight(Searcher
> searcher)
> line 92: Query query = searcher.rewrite(this);  This call will begin to
> rewrite the Query.   6.  *org.apache.lucene.search.IndexSearcher*  public
> Query rewrite(Query original)  NOTE: we only have one IndexSearcher which
> has one IndexReader. If there is any functioins that are synchronized, the
> query process will be queued.   7.
> org.apache.lucene.search.BooleanQuery public Query rewrite(IndexReader
> reader)
> line 396: Query query = c.getQuery().rewrite(reader);  Here, BooleanQuery
> will get its subqueries and call their rewrite function. The function will
> require to pass a parameter: *IndexReader* that we only have one instance.
> From the codes we will notice *TermQuery* will not be rewrote and *
> PrefixQuery* will be rewrote to several *TermQuery*s. So we ignore the *
> TermQuery* and look into the *PrefixQuery*.   8.
> org.apache.lucene.search.PrefixQuery  public Query rewrite(IndexReader
> reader)
> line 41: TermEnum enumerator = reader.terms(prefix);  Let's see what will
> happen then.   9.  org.apache.lucene.index.SegmentReader  public TermEnum
> terms(Term t)
> line 277: return tis.terms(t);  SegmentReader is in fact an IndexReader's
> implementation.   10.  org.apache.lucene.index.TermInfosReader  public
> SegmentTermEnum terms(Term term)
> line 211:get(term);
>
>   11.  org.apache.lucene.index.TermInfosReader  TermInfo get(Term term)
> line 136:ensureIndexIsRead();  We finally find it!   12.
> org.apache.lucene.index.TermInfosReader  private synchronized void
> ensureIndexIsRead()  Let's analyze the function and to see why it's
> synchronized and how to improve it.
>
> On 5/9/06, Chris Hostetter <[hidden email]> wrote:
> >
> >
> > :   We found if we were using 2 IndexSearcher, we would get 10%
> > performance
> > : benefit.
> > :   But if we increased the number of IndexSearcher from 2, the
> > performance
> > : improvement became slight even worse.
> >
> > Why use more then 2 IndexSearchers?
> >
> > Typically 1 is all you need, except for when you want to open and "warm
> > up" a new Searcher because you know your index has changed on disk and
> > you're ready for those changes to be visible.
> >
> > (I'm not arguing against your change -- concurrancy isn't my forte so i
> > have no opinion on wether your suggesting is good or not, i'm just
> > questioning the goal)
> >
> > Acctually .. i don't know a lot about the internals of IndexSearcher and
> > TermInfosReader, but according to your description of the problem...
> >
> > :   The class org.apache.lucene.index.TermInfosReader , as you know,
> every
> > : IndexSearcher will have one TermInfosReader. Every query, one method
> in
> > the
> > : class must be called:
> > : private synchronized void ensureIndexIsRead() throws IOException .
> > Notice
> >
> > If the method isn't static, then how can two differnet instances of
> > IndexSearcher, each with their own TermInfosReader, block one another?
> >
> >
> >
> >
> > -Hoss
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> --
> --
> Yueyu Lin
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
--
Yueyu Lin


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Yueyu Lin
o, I think I didn't express it clearly.
First, I only have one IndexSearcher and multiple threads will share it.
Then I found the performance is not so good like I expect in a dual CPUs
machine.
So I forced the JVM to print thread dump and I found the threads are waiting
here.

After that, I traced the Lucene in Runtime using IDE's debug mode. I found
the codes.

To resolve the problem, first I try to modify the codes and rebuild another
Lucene jar.
That's a bad idea, I didn't want to maintain my custom Lucene package.

So I tried to use 2 IndexSearchers and expect to reduce the possiblity to
wait. In this test
I found the wait behavior almost disappeared.

Two lines codes seem small, but it's still a problem in a busy system. Here
what I emphasize is
they are indeed a problem in our system although only 10% performance or so.

That is to say, if you are using the original Lucene jar, we need 11 ms to
finish a query, the modified
Lucene jar will finish a query in 10 ms. But that's not true for all
conditioins. If a query will cost 100ms, maybe
the new version will only cost 99ms.

On 5/10/06, Robert Engels <[hidden email]> wrote:

>
> I think your basic problem is that you are using multiple IndexSearchers?
> And creating new instances during runtime? If so, you will be reading the
> index information far too often. This is not a good configuration.
>
> -----Original Message-----
> From: yueyu lin [mailto:[hidden email]]
> Sent: Tuesday, May 09, 2006 8:46 PM
> To: [hidden email]; Otis Gospodnetic
> Subject: Re: Multiple threads searching in Lucene and the synchronized
> issue. -- solution attached.
>
>
> Oh,please believe in me that I've forced the JVM to print the thread dump.
> It waited here indeed.
> I'll try to post the patch to JIRA.
> I don't want to modify these codes by myself because that will break the
> Lucene codes. So I wish you can do me the favor to check these codes and
> make it availabe in the next release.
> On 5/9/06, Otis Gospodnetic <[hidden email]> wrote:
> >
> > Yueyu Lin,
> >
> > From what I can tell from a quick look at the method, that method need
> to
> > remain synchronized, so multiple threads don't accidentally re-read that
> > 'indexTerms' (Term[] type).  Even though the method is synchronized, it
> > looks like only the first invocation would enter that try/catch/finally
> > block where term reading happens.  Subsequent calls to this method
> should
> > exist quickly, because indexTerms != null.
> >
> > Are you sure this is causing the bottleneck for you?
> > I believe the proper way to figure that out is to kill the JVM with a
> > SIGnal that causes the JVM to dump thread information.  That would tell
> you
> > where the code is blocking.
> >
> > Also, if you have concrete suggestions for code changes, please post
> them
> > to JIRA as diffs/patches.
> >
> > Otis
> >
> >
> > ----- Original Message ----
> > From: yueyu lin <[hidden email]>
> > To: [hidden email]
> > Sent: Tuesday, May 9, 2006 3:53:55 AM
> > Subject: Re: Multiple threads searching in Lucene and the synchronized
> > issue. -- solution attached.
> >
> > Please trace the codes into the Lucene when searching.
> > Here is a table about how invokations are called.
> > The trace log:   *Steps*
> > *ClassName*
> > *Functions*
> > *Description*
> >   1.  org.apache.lucene.search.Searcher  public final Hits search(Query
> > query)  It will call another search function.   2.
> > org.apache.lucene.search.Searcher  public Hits search(Query query,
> Filter
> > filter)  Only one line code. It will new a Hits.
> > return new Hits(this, query, filter);   3.
> > org.apache.lucene.search.Hits Hits(Searcher s, Query q, Filter f)
> > Next, we will trace into the constructor to see what stuffs will be
> > done.  4.
> > org.apache.lucene.search.Hits  Hits(Searcher s, Query q, Filter f)
> > line 41 : weight = q.weight(s)  This call will rewrite the Query if
> > necessary, let us to see what will happen then.
> >
> >
> >   5.  org.apache.lucene.search.Query  public Weight weight(Searcher
> > searcher)
> > line 92: Query query = searcher.rewrite(this);  This call will begin to
> > rewrite the Query.   6.  *org.apache.lucene.search.IndexSearcher*
>   public
> > Query rewrite(Query original)  NOTE: we only have one IndexSearcher
> which
> > has one IndexReader. If there is any functioins that are synchronized,
> the
> > query process will be queued.   7.
> > org.apache.lucene.search.BooleanQuery public Query rewrite(IndexReader
> > reader)
> > line 396: Query query = c.getQuery().rewrite(reader);  Here,
> BooleanQuery
> > will get its subqueries and call their rewrite function. The function
> will
> > require to pass a parameter: *IndexReader* that we only have one
> instance.
> > From the codes we will notice *TermQuery* will not be rewrote and *
> > PrefixQuery* will be rewrote to several *TermQuery*s. So we ignore the *
> > TermQuery* and look into the *PrefixQuery*.   8.
> > org.apache.lucene.search.PrefixQuery  public Query rewrite(IndexReader
> > reader)
> > line 41: TermEnum enumerator = reader.terms(prefix);  Let's see what
> will
> > happen then.   9.  org.apache.lucene.index.SegmentReader  public
> TermEnum
> > terms(Term t)
> > line 277: return tis.terms(t);  SegmentReader is in fact an
> IndexReader's
> > implementation.   10.  org.apache.lucene.index.TermInfosReader  public
> > SegmentTermEnum terms(Term term)
> > line 211:get(term);
> >
> >   11.  org.apache.lucene.index.TermInfosReader  TermInfo get(Term term)
> > line 136:ensureIndexIsRead();  We finally find it!   12.
> > org.apache.lucene.index.TermInfosReader  private synchronized void
> > ensureIndexIsRead()  Let's analyze the function and to see why it's
> > synchronized and how to improve it.
> >
> > On 5/9/06, Chris Hostetter <[hidden email]> wrote:
> > >
> > >
> > > :   We found if we were using 2 IndexSearcher, we would get 10%
> > > performance
> > > : benefit.
> > > :   But if we increased the number of IndexSearcher from 2, the
> > > performance
> > > : improvement became slight even worse.
> > >
> > > Why use more then 2 IndexSearchers?
> > >
> > > Typically 1 is all you need, except for when you want to open and
> "warm
> > > up" a new Searcher because you know your index has changed on disk and
> > > you're ready for those changes to be visible.
> > >
> > > (I'm not arguing against your change -- concurrancy isn't my forte so
> i
> > > have no opinion on wether your suggesting is good or not, i'm just
> > > questioning the goal)
> > >
> > > Acctually .. i don't know a lot about the internals of IndexSearcher
> and
> > > TermInfosReader, but according to your description of the problem...
> > >
> > > :   The class org.apache.lucene.index.TermInfosReader , as you know,
> > every
> > > : IndexSearcher will have one TermInfosReader. Every query, one method
> > in
> > > the
> > > : class must be called:
> > > : private synchronized void ensureIndexIsRead() throws IOException .
> > > Notice
> > >
> > > If the method isn't static, then how can two differnet instances of
> > > IndexSearcher, each with their own TermInfosReader, block one another?
> > >
> > >
> > >
> > >
> > > -Hoss
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> > >
> > >
> >
> >
> > --
> > --
> > Yueyu Lin
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> --
> --
> Yueyu Lin
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
--
Yueyu Lin
Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Yonik Seeley
In reply to this post by Yueyu Lin
Yueyu Lin,

Your patch below looks suspiciously like the double-checked locking
anti-pattern, and is not guaranteed to work.
There really isn't a way to safely lazily initialize without using
synchronized or volatile.

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

On 5/9/06, yueyu lin <[hidden email]> wrote:

> Yes, the modification is still synchronized and the first thread will be
> responsible for reading first. And then other threads will not read and the
> synchronization is unnecessary.
> private void ensureIndexIsRead() throws IOException {
>     if (indexTerms != null)                       // index already read
>       return;                                     // do nothing
>     synchronized(this){
>         System.out.println("Read index@--@");
>         if(indexTerms != null){
>             System.out.println ("Someone read it.return-_-");
>             return ;
>         }
>         readIndex ();
>     }
>   }
>
>   private synchronized void readIndex() throws IOException{
>       Term[] m_indexTerms = null;
>       try {
>           int indexSize = (int)indexEnum.size;        // otherwise read
> index
>           m_indexTerms = new Term[indexSize];
>           indexInfos = new TermInfo[indexSize];
>           indexPointers = new long[indexSize];
>
>           for (int i = 0; indexEnum.next(); i++) {
>             m_indexTerms[i] = indexEnum.term();
>             indexInfos[i] = indexEnum.termInfo();
>             indexPointers[i] = indexEnum.indexPointer;
>           }
>         } finally {
>             indexEnum.close();
>             indexEnum = null;
>             indexTerms = m_indexTerms;
>         }
>   }
>
> That's a small trick I learned when I was developing a busy stock system.
>
> If the method ensureIndexIsRead() is synchronized, it will be blocked for a
> while, although even 2 lines codes.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Yueyu Lin
In java, call a synchronized function in a synchronized block, if they have
the same mutex object, nothing will happen.
If they have different mutex objects, something may be screwed up.

On 5/10/06, Yonik Seeley <[hidden email]> wrote:

>
> Yueyu Lin,
>
> Your patch below looks suspiciously like the double-checked locking
> anti-pattern, and is not guaranteed to work.
> There really isn't a way to safely lazily initialize without using
> synchronized or volatile.
>
> -Yonik
> http://incubator.apache.org/solr Solr, the open-source Lucene search
> server
>
> On 5/9/06, yueyu lin <[hidden email]> wrote:
> > Yes, the modification is still synchronized and the first thread will be
> > responsible for reading first. And then other threads will not read and
> the
> > synchronization is unnecessary.
> > private void ensureIndexIsRead() throws IOException {
> >     if (indexTerms != null)                       // index already read
> >       return;                                     // do nothing
> >     synchronized(this){
> >         System.out.println("Read index@--@");
> >         if(indexTerms != null){
> >             System.out.println ("Someone read it.return-_-");
> >             return ;
> >         }
> >         readIndex ();
> >     }
> >   }
> >
> >   private synchronized void readIndex() throws IOException{
> >       Term[] m_indexTerms = null;
> >       try {
> >           int indexSize = (int)indexEnum.size;        // otherwise read
> > index
> >           m_indexTerms = new Term[indexSize];
> >           indexInfos = new TermInfo[indexSize];
> >           indexPointers = new long[indexSize];
> >
> >           for (int i = 0; indexEnum.next(); i++) {
> >             m_indexTerms[i] = indexEnum.term();
> >             indexInfos[i] = indexEnum.termInfo();
> >             indexPointers[i] = indexEnum.indexPointer;
> >           }
> >         } finally {
> >             indexEnum.close();
> >             indexEnum = null;
> >             indexTerms = m_indexTerms;
> >         }
> >   }
> >
> > That's a small trick I learned when I was developing a busy stock
> system.
> >
> > If the method ensureIndexIsRead() is synchronized, it will be blocked
> for a
> > while, although even 2 lines codes.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
--
Yueyu Lin
Reply | Threaded
Open this post in threaded view
|

RE: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Robert Engels
In reply to this post by Yonik Seeley
I am fairly certain his code is ok, since it rechecks the initialized state
in the synchronized block before initializing.

Worst case, during the initial checks when the initialization is occurring
there may be some unneeded checking, but after that, the code should perform
better since it will never enter a synchronized block.

I just doubt that this change makes any real difference if the IndexSearcher
is long-lived.

-----Original Message-----
From: Yonik Seeley [mailto:[hidden email]]
Sent: Tuesday, May 09, 2006 9:04 PM
To: [hidden email]
Cc: Otis Gospodnetic
Subject: Re: Multiple threads searching in Lucene and the synchronized
issue. -- solution attached.


Yueyu Lin,

Your patch below looks suspiciously like the double-checked locking
anti-pattern, and is not guaranteed to work.
There really isn't a way to safely lazily initialize without using
synchronized or volatile.

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

On 5/9/06, yueyu lin <[hidden email]> wrote:
> Yes, the modification is still synchronized and the first thread will be
> responsible for reading first. And then other threads will not read and
the

> synchronization is unnecessary.
> private void ensureIndexIsRead() throws IOException {
>     if (indexTerms != null)                       // index already read
>       return;                                     // do nothing
>     synchronized(this){
>         System.out.println("Read index@--@");
>         if(indexTerms != null){
>             System.out.println ("Someone read it.return-_-");
>             return ;
>         }
>         readIndex ();
>     }
>   }
>
>   private synchronized void readIndex() throws IOException{
>       Term[] m_indexTerms = null;
>       try {
>           int indexSize = (int)indexEnum.size;        // otherwise read
> index
>           m_indexTerms = new Term[indexSize];
>           indexInfos = new TermInfo[indexSize];
>           indexPointers = new long[indexSize];
>
>           for (int i = 0; indexEnum.next(); i++) {
>             m_indexTerms[i] = indexEnum.term();
>             indexInfos[i] = indexEnum.termInfo();
>             indexPointers[i] = indexEnum.indexPointer;
>           }
>         } finally {
>             indexEnum.close();
>             indexEnum = null;
>             indexTerms = m_indexTerms;
>         }
>   }
>
> That's a small trick I learned when I was developing a busy stock system.
>
> If the method ensureIndexIsRead() is synchronized, it will be blocked for
a
> while, although even 2 lines codes.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Yonik Seeley
In reply to this post by Yueyu Lin
Here is a reference to double-checked locking.  Many people have tried
to get around synchronization during lazy initialization - AFAIK, none
have succeeded.  With the new memory model in Java5, you can get away
with just volatile, which is like half a synchronization (a read
barrier + a write barrier).

http://www.google.com/search?q=double+checked+locking

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server


On 5/9/06, yueyu lin <[hidden email]> wrote:

> In java, call a synchronized function in a synchronized block, if they have
> the same mutex object, nothing will happen.
> If they have different mutex objects, something may be screwed up.
>
> On 5/10/06, Yonik Seeley <[hidden email]> wrote:
> >
> > Yueyu Lin,
> >
> > Your patch below looks suspiciously like the double-checked locking
> > anti-pattern, and is not guaranteed to work.
> > There really isn't a way to safely lazily initialize without using
> > synchronized or volatile.
> >
> > -Yonik
> > http://incubator.apache.org/solr Solr, the open-source Lucene search
> > server
> >
> > On 5/9/06, yueyu lin <[hidden email]> wrote:
> > > Yes, the modification is still synchronized and the first thread will be
> > > responsible for reading first. And then other threads will not read and
> > the
> > > synchronization is unnecessary.
> > > private void ensureIndexIsRead() throws IOException {
> > >     if (indexTerms != null)                       // index already read
> > >       return;                                     // do nothing
> > >     synchronized(this){
> > >         System.out.println("Read index@--@");
> > >         if(indexTerms != null){
> > >             System.out.println ("Someone read it.return-_-");
> > >             return ;
> > >         }
> > >         readIndex ();
> > >     }
> > >   }
> > >
> > >   private synchronized void readIndex() throws IOException{
> > >       Term[] m_indexTerms = null;
> > >       try {
> > >           int indexSize = (int)indexEnum.size;        // otherwise read
> > > index
> > >           m_indexTerms = new Term[indexSize];
> > >           indexInfos = new TermInfo[indexSize];
> > >           indexPointers = new long[indexSize];
> > >
> > >           for (int i = 0; indexEnum.next(); i++) {
> > >             m_indexTerms[i] = indexEnum.term();
> > >             indexInfos[i] = indexEnum.termInfo();
> > >             indexPointers[i] = indexEnum.indexPointer;
> > >           }
> > >         } finally {
> > >             indexEnum.close();
> > >             indexEnum = null;
> > >             indexTerms = m_indexTerms;
> > >         }
> > >   }
> > >
> > > That's a small trick I learned when I was developing a busy stock
> > system.
> > >
> > > If the method ensureIndexIsRead() is synchronized, it will be blocked
> > for a
> > > while, although even 2 lines codes.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> --
> --
> Yueyu Lin
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Yonik Seeley
In reply to this post by Robert Engels
On 5/9/06, Robert Engels <[hidden email]> wrote:
> I am fairly certain his code is ok, since it rechecks the initialized state
> in the synchronized block before initializing.

That "recheck" is why the pattern (or anti-pattern) is called
double-checked locking :-)

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Chris Hostetter-3

: > I am fairly certain his code is ok, since it rechecks the initialized state
: > in the synchronized block before initializing.
:
: That "recheck" is why the pattern (or anti-pattern) is called
: double-checked locking :-)

More specificly, this is functionally half way between example labeled
'Broken multithreaded version' and the example labeled '(Still) Broken
multithreaded version'

http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html

An optimizing compiler can freely optimize away the m_indexTerms variable
so the new readIndex function directly modifies this.indexTerms, and the
JVM can allocate RAM for the array and set the variable to point at this
RAM before the array is fully populated.



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

Otis Gospodnetic-2
In reply to this post by Yueyu Lin
Yueyu Lin,

Sorry, I don't follow this part:
"To resolve the problem, first I try to modify the codes and rebuild another
Lucene jar.
That's a bad idea, I didn't want to maintain my custom Lucene package."

Are you saying you _did_ make the code changes and _did_ run your application with a modified Jar?
If so, did you measure the performance difference?  What was the difference?

If you have not modified the code and have not run your application with the modified Jar, then you should try that first and see if your change makes any substantial difference.

We're all very interested in improving Lucene's performance, so please let us know what you find.

Thanks,
Otis

----- Original Message ----
From: yueyu lin <[hidden email]>
To: [hidden email]; [hidden email]
Sent: Tuesday, May 9, 2006 10:01:55 PM
Subject: Re: Multiple threads searching in Lucene and the synchronized issue. -- solution attached.

o, I think I didn't express it clearly.
First, I only have one IndexSearcher and multiple threads will share it.
Then I found the performance is not so good like I expect in a dual CPUs
machine.
So I forced the JVM to print thread dump and I found the threads are waiting
here.

After that, I traced the Lucene in Runtime using IDE's debug mode. I found
the codes.

To resolve the problem, first I try to modify the codes and rebuild another
Lucene jar.
That's a bad idea, I didn't want to maintain my custom Lucene package.

So I tried to use 2 IndexSearchers and expect to reduce the possiblity to
wait. In this test
I found the wait behavior almost disappeared.

Two lines codes seem small, but it's still a problem in a busy system. Here
what I emphasize is
they are indeed a problem in our system although only 10% performance or so.

That is to say, if you are using the original Lucene jar, we need 11 ms to
finish a query, the modified
Lucene jar will finish a query in 10 ms. But that's not true for all
conditioins. If a query will cost 100ms, maybe
the new version will only cost 99ms.

On 5/10/06, Robert Engels <[hidden email]> wrote:

>
> I think your basic problem is that you are using multiple IndexSearchers?
> And creating new instances during runtime? If so, you will be reading the
> index information far too often. This is not a good configuration.
>
> -----Original Message-----
> From: yueyu lin [mailto:[hidden email]]
> Sent: Tuesday, May 09, 2006 8:46 PM
> To: [hidden email]; Otis Gospodnetic
> Subject: Re: Multiple threads searching in Lucene and the synchronized
> issue. -- solution attached.
>
>
> Oh,please believe in me that I've forced the JVM to print the thread dump.
> It waited here indeed.
> I'll try to post the patch to JIRA.
> I don't want to modify these codes by myself because that will break the
> Lucene codes. So I wish you can do me the favor to check these codes and
> make it availabe in the next release.
> On 5/9/06, Otis Gospodnetic <[hidden email]> wrote:
> >
> > Yueyu Lin,
> >
> > From what I can tell from a quick look at the method, that method need
> to
> > remain synchronized, so multiple threads don't accidentally re-read that
> > 'indexTerms' (Term[] type).  Even though the method is synchronized, it
> > looks like only the first invocation would enter that try/catch/finally
> > block where term reading happens.  Subsequent calls to this method
> should
> > exist quickly, because indexTerms != null.
> >
> > Are you sure this is causing the bottleneck for you?
> > I believe the proper way to figure that out is to kill the JVM with a
> > SIGnal that causes the JVM to dump thread information.  That would tell
> you
> > where the code is blocking.
> >
> > Also, if you have concrete suggestions for code changes, please post
> them
> > to JIRA as diffs/patches.
> >
> > Otis
> >
> >
> > ----- Original Message ----
> > From: yueyu lin <[hidden email]>
> > To: [hidden email]
> > Sent: Tuesday, May 9, 2006 3:53:55 AM
> > Subject: Re: Multiple threads searching in Lucene and the synchronized
> > issue. -- solution attached.
> >
> > Please trace the codes into the Lucene when searching.
> > Here is a table about how invokations are called.
> > The trace log:   *Steps*
> > *ClassName*
> > *Functions*
> > *Description*
> >   1.  org.apache.lucene.search.Searcher  public final Hits search(Query
> > query)  It will call another search function.   2.
> > org.apache.lucene.search.Searcher  public Hits search(Query query,
> Filter
> > filter)  Only one line code. It will new a Hits.
> > return new Hits(this, query, filter);   3.
> > org.apache.lucene.search.Hits Hits(Searcher s, Query q, Filter f)
> > Next, we will trace into the constructor to see what stuffs will be
> > done.  4.
> > org.apache.lucene.search.Hits  Hits(Searcher s, Query q, Filter f)
> > line 41 : weight = q.weight(s)  This call will rewrite the Query if
> > necessary, let us to see what will happen then.
> >
> >
> >   5.  org.apache.lucene.search.Query  public Weight weight(Searcher
> > searcher)
> > line 92: Query query = searcher.rewrite(this);  This call will begin to
> > rewrite the Query.   6.  *org.apache.lucene.search.IndexSearcher*
>   public
> > Query rewrite(Query original)  NOTE: we only have one IndexSearcher
> which
> > has one IndexReader. If there is any functioins that are synchronized,
> the
> > query process will be queued.   7.
> > org.apache.lucene.search.BooleanQuery public Query rewrite(IndexReader
> > reader)
> > line 396: Query query = c.getQuery().rewrite(reader);  Here,
> BooleanQuery
> > will get its subqueries and call their rewrite function. The function
> will
> > require to pass a parameter: *IndexReader* that we only have one
> instance.
> > From the codes we will notice *TermQuery* will not be rewrote and *
> > PrefixQuery* will be rewrote to several *TermQuery*s. So we ignore the *
> > TermQuery* and look into the *PrefixQuery*.   8.
> > org.apache.lucene.search.PrefixQuery  public Query rewrite(IndexReader
> > reader)
> > line 41: TermEnum enumerator = reader.terms(prefix);  Let's see what
> will
> > happen then.   9.  org.apache.lucene.index.SegmentReader  public
> TermEnum
> > terms(Term t)
> > line 277: return tis.terms(t);  SegmentReader is in fact an
> IndexReader's
> > implementation.   10.  org.apache.lucene.index.TermInfosReader  public
> > SegmentTermEnum terms(Term term)
> > line 211:get(term);
> >
> >   11.  org.apache.lucene.index.TermInfosReader  TermInfo get(Term term)
> > line 136:ensureIndexIsRead();  We finally find it!   12.
> > org.apache.lucene.index.TermInfosReader  private synchronized void
> > ensureIndexIsRead()  Let's analyze the function and to see why it's
> > synchronized and how to improve it.
> >
> > On 5/9/06, Chris Hostetter <[hidden email]> wrote:
> > >
> > >
> > > :   We found if we were using 2 IndexSearcher, we would get 10%
> > > performance
> > > : benefit.
> > > :   But if we increased the number of IndexSearcher from 2, the
> > > performance
> > > : improvement became slight even worse.
> > >
> > > Why use more then 2 IndexSearchers?
> > >
> > > Typically 1 is all you need, except for when you want to open and
> "warm
> > > up" a new Searcher because you know your index has changed on disk and
> > > you're ready for those changes to be visible.
> > >
> > > (I'm not arguing against your change -- concurrancy isn't my forte so
> i
> > > have no opinion on wether your suggesting is good or not, i'm just
> > > questioning the goal)
> > >
> > > Acctually .. i don't know a lot about the internals of IndexSearcher
> and
> > > TermInfosReader, but according to your description of the problem...
> > >
> > > :   The class org.apache.lucene.index.TermInfosReader , as you know,
> > every
> > > : IndexSearcher will have one TermInfosReader. Every query, one method
> > in
> > > the
> > > : class must be called:
> > > : private synchronized void ensureIndexIsRead() throws IOException .
> > > Notice
> > >
> > > If the method isn't static, then how can two differnet instances of
> > > IndexSearcher, each with their own TermInfosReader, block one another?
> > >
> > >
> > >
> > >
> > > -Hoss
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> > >
> > >
> >
> >
> > --
> > --
> > Yueyu Lin
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> --
> --
> Yueyu Lin
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
--
Yueyu Lin




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12