Re: SearcherManager not seeing changes in IndexWriteral and

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: SearcherManager not seeing changes in IndexWriteral and

Michael Sokolov-4
I'm not seeing anything there that would synchronize, or serialize, the
read after the write and commit. Did you expect that for some reason?

On Fri, Nov 9, 2018, 6:00 AM Boris Petrov <[hidden email] wrote:

> Hi all,
>
> I'm using Lucene version 7.5.0. We have a test that does something like:
>
> Thread 1:
>
>             Field idStringField = new StringField("id", id,
> Field.Store.YES);
>             Field contentsField = new TextField("contents", reader);
>             Document document = new Document();
>             document.add(idStringField);
>             document.add(contentsField);
>
>             writer.updateDocument(new Term(ID_FIELD, id), document);
>             writer.flush(); // not sure this flush is needed?
>             writer.commit();
>
> Thread 2:
>
>             searchManager.maybeRefreshBlocking();
>             IndexSearcher searcher = searchManager.acquire();
>             try {
>                 QueryParser parser = new QueryParser("contents", analyzer);
>                 Query luceneQuery = parser.parse(queryText);
>                 ScoreDoc[] hits = searcher.search(luceneQuery,
> 50).scoreDocs;
>             } finally {
>                 searchManager.release(searcher);
>             }
>
> Thread 1 happens before Thread 2.
>
> Sometimes, only sometimes, the commit from thread 1 is not *immediately*
> visible in Thread 2. If I put a "Thread.sleep(1000)" it always works.
> Without it, sometimes the search is empty. I'm not sure if I'm doing
> something wrong or this is a bug?
>
> Thanks!
>
>
Reply | Threaded
Open this post in threaded view
|

Re: SearcherManager not seeing changes in IndexWriteral and

Boris Petrov
If you mean the synchronization of the threads, it is not in the
example, but Thread 2 is *started* after Thread 1 finished executing the
code that I gave as an example. So there is happens-before between them.
If you mean synchronization on the Lucene level - isn't that what
"maybeRefreshBlocking" should do?

On 11/9/18 3:29 PM, Michael Sokolov wrote:

> I'm not seeing anything there that would synchronize, or serialize, the
> read after the write and commit. Did you expect that for some reason?
>
> On Fri, Nov 9, 2018, 6:00 AM Boris Petrov <[hidden email] wrote:
>
>> Hi all,
>>
>> I'm using Lucene version 7.5.0. We have a test that does something like:
>>
>> Thread 1:
>>
>>             Field idStringField = new StringField("id", id,
>> Field.Store.YES);
>>             Field contentsField = new TextField("contents", reader);
>>             Document document = new Document();
>>             document.add(idStringField);
>>             document.add(contentsField);
>>
>>             writer.updateDocument(new Term(ID_FIELD, id), document);
>>             writer.flush(); // not sure this flush is needed?
>>             writer.commit();
>>
>> Thread 2:
>>
>>             searchManager.maybeRefreshBlocking();
>>             IndexSearcher searcher = searchManager.acquire();
>>             try {
>>                 QueryParser parser = new QueryParser("contents", analyzer);
>>                 Query luceneQuery = parser.parse(queryText);
>>                 ScoreDoc[] hits = searcher.search(luceneQuery,
>> 50).scoreDocs;
>>             } finally {
>>                 searchManager.release(searcher);
>>             }
>>
>> Thread 1 happens before Thread 2.
>>
>> Sometimes, only sometimes, the commit from thread 1 is not *immediately*
>> visible in Thread 2. If I put a "Thread.sleep(1000)" it always works.
>> Without it, sometimes the search is empty. I'm not sure if I'm doing
>> something wrong or this is a bug?
>>
>> Thanks!
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: SearcherManager not seeing changes in IndexWriteral and

Michael Sokolov-4
That should work, I think, but if you are serializing these threads so
that they cannot run concurrently, maybe try running both operations
in a single thread, at least as a test.
On Fri, Nov 9, 2018 at 9:16 AM Boris Petrov <[hidden email]> wrote:

>
> If you mean the synchronization of the threads, it is not in the
> example, but Thread 2 is *started* after Thread 1 finished executing the
> code that I gave as an example. So there is happens-before between them.
> If you mean synchronization on the Lucene level - isn't that what
> "maybeRefreshBlocking" should do?
>
> On 11/9/18 3:29 PM, Michael Sokolov wrote:
> > I'm not seeing anything there that would synchronize, or serialize, the
> > read after the write and commit. Did you expect that for some reason?
> >
> > On Fri, Nov 9, 2018, 6:00 AM Boris Petrov <[hidden email] wrote:
> >
> >> Hi all,
> >>
> >> I'm using Lucene version 7.5.0. We have a test that does something like:
> >>
> >> Thread 1:
> >>
> >>             Field idStringField = new StringField("id", id,
> >> Field.Store.YES);
> >>             Field contentsField = new TextField("contents", reader);
> >>             Document document = new Document();
> >>             document.add(idStringField);
> >>             document.add(contentsField);
> >>
> >>             writer.updateDocument(new Term(ID_FIELD, id), document);
> >>             writer.flush(); // not sure this flush is needed?
> >>             writer.commit();
> >>
> >> Thread 2:
> >>
> >>             searchManager.maybeRefreshBlocking();
> >>             IndexSearcher searcher = searchManager.acquire();
> >>             try {
> >>                 QueryParser parser = new QueryParser("contents", analyzer);
> >>                 Query luceneQuery = parser.parse(queryText);
> >>                 ScoreDoc[] hits = searcher.search(luceneQuery,
> >> 50).scoreDocs;
> >>             } finally {
> >>                 searchManager.release(searcher);
> >>             }
> >>
> >> Thread 1 happens before Thread 2.
> >>
> >> Sometimes, only sometimes, the commit from thread 1 is not *immediately*
> >> visible in Thread 2. If I put a "Thread.sleep(1000)" it always works.
> >> Without it, sometimes the search is empty. I'm not sure if I'm doing
> >> something wrong or this is a bug?
> >>
> >> Thanks!
> >>
> >>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: SearcherManager not seeing changes in IndexWriteral and

Boris Petrov
Well, it's a bit involved to try it in a single thread as I've
oversimplified the example. But as far as I understand this should work,
right? So something else is wrong? Committing the writer and then
"maybeRefreshBlocking" should be enough to have the changes visible, yes?

On 11/9/18 4:45 PM, Michael Sokolov wrote:

> That should work, I think, but if you are serializing these threads so
> that they cannot run concurrently, maybe try running both operations
> in a single thread, at least as a test.
> On Fri, Nov 9, 2018 at 9:16 AM Boris Petrov <[hidden email]> wrote:
>> If you mean the synchronization of the threads, it is not in the
>> example, but Thread 2 is *started* after Thread 1 finished executing the
>> code that I gave as an example. So there is happens-before between them.
>> If you mean synchronization on the Lucene level - isn't that what
>> "maybeRefreshBlocking" should do?
>>
>> On 11/9/18 3:29 PM, Michael Sokolov wrote:
>>> I'm not seeing anything there that would synchronize, or serialize, the
>>> read after the write and commit. Did you expect that for some reason?
>>>
>>> On Fri, Nov 9, 2018, 6:00 AM Boris Petrov <[hidden email] wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'm using Lucene version 7.5.0. We have a test that does something like:
>>>>
>>>> Thread 1:
>>>>
>>>>             Field idStringField = new StringField("id", id,
>>>> Field.Store.YES);
>>>>             Field contentsField = new TextField("contents", reader);
>>>>             Document document = new Document();
>>>>             document.add(idStringField);
>>>>             document.add(contentsField);
>>>>
>>>>             writer.updateDocument(new Term(ID_FIELD, id), document);
>>>>             writer.flush(); // not sure this flush is needed?
>>>>             writer.commit();
>>>>
>>>> Thread 2:
>>>>
>>>>             searchManager.maybeRefreshBlocking();
>>>>             IndexSearcher searcher = searchManager.acquire();
>>>>             try {
>>>>                 QueryParser parser = new QueryParser("contents", analyzer);
>>>>                 Query luceneQuery = parser.parse(queryText);
>>>>                 ScoreDoc[] hits = searcher.search(luceneQuery,
>>>> 50).scoreDocs;
>>>>             } finally {
>>>>                 searchManager.release(searcher);
>>>>             }
>>>>
>>>> Thread 1 happens before Thread 2.
>>>>
>>>> Sometimes, only sometimes, the commit from thread 1 is not *immediately*
>>>> visible in Thread 2. If I put a "Thread.sleep(1000)" it always works.
>>>> Without it, sometimes the search is empty. I'm not sure if I'm doing
>>>> something wrong or this is a bug?
>>>>
>>>> Thanks!
>>>>
>>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: SearcherManager not seeing changes in IndexWriteral and

Erick Erickson
If it's hard to do in a single thread, how about timestamping the
events to insure that they happen in the expected order?

That would verify the sequencing is happening as you expect and
_still_ not see the expected docs...

Assuming it does fail, do you think you could reduce it to a
multi-threaded test case?

Best,
Erick
On Fri, Nov 9, 2018 at 7:03 AM Boris Petrov <[hidden email]> wrote:

>
> Well, it's a bit involved to try it in a single thread as I've
> oversimplified the example. But as far as I understand this should work,
> right? So something else is wrong? Committing the writer and then
> "maybeRefreshBlocking" should be enough to have the changes visible, yes?
>
> On 11/9/18 4:45 PM, Michael Sokolov wrote:
> > That should work, I think, but if you are serializing these threads so
> > that they cannot run concurrently, maybe try running both operations
> > in a single thread, at least as a test.
> > On Fri, Nov 9, 2018 at 9:16 AM Boris Petrov <[hidden email]> wrote:
> >> If you mean the synchronization of the threads, it is not in the
> >> example, but Thread 2 is *started* after Thread 1 finished executing the
> >> code that I gave as an example. So there is happens-before between them.
> >> If you mean synchronization on the Lucene level - isn't that what
> >> "maybeRefreshBlocking" should do?
> >>
> >> On 11/9/18 3:29 PM, Michael Sokolov wrote:
> >>> I'm not seeing anything there that would synchronize, or serialize, the
> >>> read after the write and commit. Did you expect that for some reason?
> >>>
> >>> On Fri, Nov 9, 2018, 6:00 AM Boris Petrov <[hidden email] wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> I'm using Lucene version 7.5.0. We have a test that does something like:
> >>>>
> >>>> Thread 1:
> >>>>
> >>>>             Field idStringField = new StringField("id", id,
> >>>> Field.Store.YES);
> >>>>             Field contentsField = new TextField("contents", reader);
> >>>>             Document document = new Document();
> >>>>             document.add(idStringField);
> >>>>             document.add(contentsField);
> >>>>
> >>>>             writer.updateDocument(new Term(ID_FIELD, id), document);
> >>>>             writer.flush(); // not sure this flush is needed?
> >>>>             writer.commit();
> >>>>
> >>>> Thread 2:
> >>>>
> >>>>             searchManager.maybeRefreshBlocking();
> >>>>             IndexSearcher searcher = searchManager.acquire();
> >>>>             try {
> >>>>                 QueryParser parser = new QueryParser("contents", analyzer);
> >>>>                 Query luceneQuery = parser.parse(queryText);
> >>>>                 ScoreDoc[] hits = searcher.search(luceneQuery,
> >>>> 50).scoreDocs;
> >>>>             } finally {
> >>>>                 searchManager.release(searcher);
> >>>>             }
> >>>>
> >>>> Thread 1 happens before Thread 2.
> >>>>
> >>>> Sometimes, only sometimes, the commit from thread 1 is not *immediately*
> >>>> visible in Thread 2. If I put a "Thread.sleep(1000)" it always works.
> >>>> Without it, sometimes the search is empty. I'm not sure if I'm doing
> >>>> something wrong or this is a bug?
> >>>>
> >>>> Thanks!
> >>>>
> >>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: SearcherManager not seeing changes in IndexWriteral and

Boris Petrov
Well, while debugging I put a bunch of println's which showed the
expected order. And, besides, I've written the code, I know that writing
to the index happens way before searching in it - the test makes sure of
that.

If you think there indeed might be some problem, I'll try to reproduce
it in a small example. But I will try only the obvious (which perhaps
you've tried a million times and have tests for) - I'll have one thread
write to the index, another (which starts after the first) search in it
and I'll create a bash script that runs the program until it fails (what
I currently do with our test). I'll do this beginning of next week.

Thank you for the support!

On 11/9/18 5:37 PM, Erick Erickson wrote:

> If it's hard to do in a single thread, how about timestamping the
> events to insure that they happen in the expected order?
>
> That would verify the sequencing is happening as you expect and
> _still_ not see the expected docs...
>
> Assuming it does fail, do you think you could reduce it to a
> multi-threaded test case?
>
> Best,
> Erick
> On Fri, Nov 9, 2018 at 7:03 AM Boris Petrov <[hidden email]> wrote:
>> Well, it's a bit involved to try it in a single thread as I've
>> oversimplified the example. But as far as I understand this should work,
>> right? So something else is wrong? Committing the writer and then
>> "maybeRefreshBlocking" should be enough to have the changes visible, yes?
>>
>> On 11/9/18 4:45 PM, Michael Sokolov wrote:
>>> That should work, I think, but if you are serializing these threads so
>>> that they cannot run concurrently, maybe try running both operations
>>> in a single thread, at least as a test.
>>> On Fri, Nov 9, 2018 at 9:16 AM Boris Petrov <[hidden email]> wrote:
>>>> If you mean the synchronization of the threads, it is not in the
>>>> example, but Thread 2 is *started* after Thread 1 finished executing the
>>>> code that I gave as an example. So there is happens-before between them.
>>>> If you mean synchronization on the Lucene level - isn't that what
>>>> "maybeRefreshBlocking" should do?
>>>>
>>>> On 11/9/18 3:29 PM, Michael Sokolov wrote:
>>>>> I'm not seeing anything there that would synchronize, or serialize, the
>>>>> read after the write and commit. Did you expect that for some reason?
>>>>>
>>>>> On Fri, Nov 9, 2018, 6:00 AM Boris Petrov <[hidden email] wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I'm using Lucene version 7.5.0. We have a test that does something like:
>>>>>>
>>>>>> Thread 1:
>>>>>>
>>>>>>             Field idStringField = new StringField("id", id,
>>>>>> Field.Store.YES);
>>>>>>             Field contentsField = new TextField("contents", reader);
>>>>>>             Document document = new Document();
>>>>>>             document.add(idStringField);
>>>>>>             document.add(contentsField);
>>>>>>
>>>>>>             writer.updateDocument(new Term(ID_FIELD, id), document);
>>>>>>             writer.flush(); // not sure this flush is needed?
>>>>>>             writer.commit();
>>>>>>
>>>>>> Thread 2:
>>>>>>
>>>>>>             searchManager.maybeRefreshBlocking();
>>>>>>             IndexSearcher searcher = searchManager.acquire();
>>>>>>             try {
>>>>>>                 QueryParser parser = new QueryParser("contents", analyzer);
>>>>>>                 Query luceneQuery = parser.parse(queryText);
>>>>>>                 ScoreDoc[] hits = searcher.search(luceneQuery,
>>>>>> 50).scoreDocs;
>>>>>>             } finally {
>>>>>>                 searchManager.release(searcher);
>>>>>>             }
>>>>>>
>>>>>> Thread 1 happens before Thread 2.
>>>>>>
>>>>>> Sometimes, only sometimes, the commit from thread 1 is not *immediately*
>>>>>> visible in Thread 2. If I put a "Thread.sleep(1000)" it always works.
>>>>>> Without it, sometimes the search is empty. I'm not sure if I'm doing
>>>>>> something wrong or this is a bug?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: SearcherManager not seeing changes in IndexWriteral and

Erick Erickson
You might be able to do this with a couple of threads in a single
program, but certainly up to you.

Best,
Erick
On Fri, Nov 9, 2018 at 7:47 AM Boris Petrov <[hidden email]> wrote:

>
> Well, while debugging I put a bunch of println's which showed the
> expected order. And, besides, I've written the code, I know that writing
> to the index happens way before searching in it - the test makes sure of
> that.
>
> If you think there indeed might be some problem, I'll try to reproduce
> it in a small example. But I will try only the obvious (which perhaps
> you've tried a million times and have tests for) - I'll have one thread
> write to the index, another (which starts after the first) search in it
> and I'll create a bash script that runs the program until it fails (what
> I currently do with our test). I'll do this beginning of next week.
>
> Thank you for the support!
>
> On 11/9/18 5:37 PM, Erick Erickson wrote:
> > If it's hard to do in a single thread, how about timestamping the
> > events to insure that they happen in the expected order?
> >
> > That would verify the sequencing is happening as you expect and
> > _still_ not see the expected docs...
> >
> > Assuming it does fail, do you think you could reduce it to a
> > multi-threaded test case?
> >
> > Best,
> > Erick
> > On Fri, Nov 9, 2018 at 7:03 AM Boris Petrov <[hidden email]> wrote:
> >> Well, it's a bit involved to try it in a single thread as I've
> >> oversimplified the example. But as far as I understand this should work,
> >> right? So something else is wrong? Committing the writer and then
> >> "maybeRefreshBlocking" should be enough to have the changes visible, yes?
> >>
> >> On 11/9/18 4:45 PM, Michael Sokolov wrote:
> >>> That should work, I think, but if you are serializing these threads so
> >>> that they cannot run concurrently, maybe try running both operations
> >>> in a single thread, at least as a test.
> >>> On Fri, Nov 9, 2018 at 9:16 AM Boris Petrov <[hidden email]> wrote:
> >>>> If you mean the synchronization of the threads, it is not in the
> >>>> example, but Thread 2 is *started* after Thread 1 finished executing the
> >>>> code that I gave as an example. So there is happens-before between them.
> >>>> If you mean synchronization on the Lucene level - isn't that what
> >>>> "maybeRefreshBlocking" should do?
> >>>>
> >>>> On 11/9/18 3:29 PM, Michael Sokolov wrote:
> >>>>> I'm not seeing anything there that would synchronize, or serialize, the
> >>>>> read after the write and commit. Did you expect that for some reason?
> >>>>>
> >>>>> On Fri, Nov 9, 2018, 6:00 AM Boris Petrov <[hidden email] wrote:
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I'm using Lucene version 7.5.0. We have a test that does something like:
> >>>>>>
> >>>>>> Thread 1:
> >>>>>>
> >>>>>>             Field idStringField = new StringField("id", id,
> >>>>>> Field.Store.YES);
> >>>>>>             Field contentsField = new TextField("contents", reader);
> >>>>>>             Document document = new Document();
> >>>>>>             document.add(idStringField);
> >>>>>>             document.add(contentsField);
> >>>>>>
> >>>>>>             writer.updateDocument(new Term(ID_FIELD, id), document);
> >>>>>>             writer.flush(); // not sure this flush is needed?
> >>>>>>             writer.commit();
> >>>>>>
> >>>>>> Thread 2:
> >>>>>>
> >>>>>>             searchManager.maybeRefreshBlocking();
> >>>>>>             IndexSearcher searcher = searchManager.acquire();
> >>>>>>             try {
> >>>>>>                 QueryParser parser = new QueryParser("contents", analyzer);
> >>>>>>                 Query luceneQuery = parser.parse(queryText);
> >>>>>>                 ScoreDoc[] hits = searcher.search(luceneQuery,
> >>>>>> 50).scoreDocs;
> >>>>>>             } finally {
> >>>>>>                 searchManager.release(searcher);
> >>>>>>             }
> >>>>>>
> >>>>>> Thread 1 happens before Thread 2.
> >>>>>>
> >>>>>> Sometimes, only sometimes, the commit from thread 1 is not *immediately*
> >>>>>> visible in Thread 2. If I put a "Thread.sleep(1000)" it always works.
> >>>>>> Without it, sometimes the search is empty. I'm not sure if I'm doing
> >>>>>> something wrong or this is a bug?
> >>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: SearcherManager not seeing changes in IndexWriteral and

Boris Petrov
Hello,

OK, so actually this appears to be a bug in our code - Lucene is searching correctly, we were doing something wrong with the result after that. Sorry all for the confusion! Thanks for the support!
________________________________
From: Erick Erickson <[hidden email]>
Sent: Friday, November 9, 2018 6:16 PM
To: java-user
Subject: Re: SearcherManager not seeing changes in IndexWriteral and

You might be able to do this with a couple of threads in a single
program, but certainly up to you.

Best,
Erick
On Fri, Nov 9, 2018 at 7:47 AM Boris Petrov <[hidden email]> wrote:

>
> Well, while debugging I put a bunch of println's which showed the
> expected order. And, besides, I've written the code, I know that writing
> to the index happens way before searching in it - the test makes sure of
> that.
>
> If you think there indeed might be some problem, I'll try to reproduce
> it in a small example. But I will try only the obvious (which perhaps
> you've tried a million times and have tests for) - I'll have one thread
> write to the index, another (which starts after the first) search in it
> and I'll create a bash script that runs the program until it fails (what
> I currently do with our test). I'll do this beginning of next week.
>
> Thank you for the support!
>
> On 11/9/18 5:37 PM, Erick Erickson wrote:
> > If it's hard to do in a single thread, how about timestamping the
> > events to insure that they happen in the expected order?
> >
> > That would verify the sequencing is happening as you expect and
> > _still_ not see the expected docs...
> >
> > Assuming it does fail, do you think you could reduce it to a
> > multi-threaded test case?
> >
> > Best,
> > Erick
> > On Fri, Nov 9, 2018 at 7:03 AM Boris Petrov <[hidden email]> wrote:
> >> Well, it's a bit involved to try it in a single thread as I've
> >> oversimplified the example. But as far as I understand this should work,
> >> right? So something else is wrong? Committing the writer and then
> >> "maybeRefreshBlocking" should be enough to have the changes visible, yes?
> >>
> >> On 11/9/18 4:45 PM, Michael Sokolov wrote:
> >>> That should work, I think, but if you are serializing these threads so
> >>> that they cannot run concurrently, maybe try running both operations
> >>> in a single thread, at least as a test.
> >>> On Fri, Nov 9, 2018 at 9:16 AM Boris Petrov <[hidden email]> wrote:
> >>>> If you mean the synchronization of the threads, it is not in the
> >>>> example, but Thread 2 is *started* after Thread 1 finished executing the
> >>>> code that I gave as an example. So there is happens-before between them.
> >>>> If you mean synchronization on the Lucene level - isn't that what
> >>>> "maybeRefreshBlocking" should do?
> >>>>
> >>>> On 11/9/18 3:29 PM, Michael Sokolov wrote:
> >>>>> I'm not seeing anything there that would synchronize, or serialize, the
> >>>>> read after the write and commit. Did you expect that for some reason?
> >>>>>
> >>>>> On Fri, Nov 9, 2018, 6:00 AM Boris Petrov <[hidden email] wrote:
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I'm using Lucene version 7.5.0. We have a test that does something like:
> >>>>>>
> >>>>>> Thread 1:
> >>>>>>
> >>>>>>             Field idStringField = new StringField("id", id,
> >>>>>> Field.Store.YES);
> >>>>>>             Field contentsField = new TextField("contents", reader);
> >>>>>>             Document document = new Document();
> >>>>>>             document.add(idStringField);
> >>>>>>             document.add(contentsField);
> >>>>>>
> >>>>>>             writer.updateDocument(new Term(ID_FIELD, id), document);
> >>>>>>             writer.flush(); // not sure this flush is needed?
> >>>>>>             writer.commit();
> >>>>>>
> >>>>>> Thread 2:
> >>>>>>
> >>>>>>             searchManager.maybeRefreshBlocking();
> >>>>>>             IndexSearcher searcher = searchManager.acquire();
> >>>>>>             try {
> >>>>>>                 QueryParser parser = new QueryParser("contents", analyzer);
> >>>>>>                 Query luceneQuery = parser.parse(queryText);
> >>>>>>                 ScoreDoc[] hits = searcher.search(luceneQuery,
> >>>>>> 50).scoreDocs;
> >>>>>>             } finally {
> >>>>>>                 searchManager.release(searcher);
> >>>>>>             }
> >>>>>>
> >>>>>> Thread 1 happens before Thread 2.
> >>>>>>
> >>>>>> Sometimes, only sometimes, the commit from thread 1 is not *immediately*
> >>>>>> visible in Thread 2. If I put a "Thread.sleep(1000)" it always works.
> >>>>>> Without it, sometimes the search is empty. I'm not sure if I'm doing
> >>>>>> something wrong or this is a bug?
> >>>>>>
> >>>>>> Thanks!
> >>>>>>
> >>>>>>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: SearcherManager not seeing changes in IndexWriteral and

Michael McCandless-2
Thanks for bringing closure, Boris.

Mike McCandless

http://blog.mikemccandless.com


On Mon, Nov 12, 2018 at 7:13 AM Boris Petrov <[hidden email]> wrote:

> Hello,
>
> OK, so actually this appears to be a bug in our code - Lucene is searching
> correctly, we were doing something wrong with the result after that. Sorry
> all for the confusion! Thanks for the support!
> ________________________________
> From: Erick Erickson <[hidden email]>
> Sent: Friday, November 9, 2018 6:16 PM
> To: java-user
> Subject: Re: SearcherManager not seeing changes in IndexWriteral and
>
> You might be able to do this with a couple of threads in a single
> program, but certainly up to you.
>
> Best,
> Erick
> On Fri, Nov 9, 2018 at 7:47 AM Boris Petrov <[hidden email]> wrote:
> >
> > Well, while debugging I put a bunch of println's which showed the
> > expected order. And, besides, I've written the code, I know that writing
> > to the index happens way before searching in it - the test makes sure of
> > that.
> >
> > If you think there indeed might be some problem, I'll try to reproduce
> > it in a small example. But I will try only the obvious (which perhaps
> > you've tried a million times and have tests for) - I'll have one thread
> > write to the index, another (which starts after the first) search in it
> > and I'll create a bash script that runs the program until it fails (what
> > I currently do with our test). I'll do this beginning of next week.
> >
> > Thank you for the support!
> >
> > On 11/9/18 5:37 PM, Erick Erickson wrote:
> > > If it's hard to do in a single thread, how about timestamping the
> > > events to insure that they happen in the expected order?
> > >
> > > That would verify the sequencing is happening as you expect and
> > > _still_ not see the expected docs...
> > >
> > > Assuming it does fail, do you think you could reduce it to a
> > > multi-threaded test case?
> > >
> > > Best,
> > > Erick
> > > On Fri, Nov 9, 2018 at 7:03 AM Boris Petrov <[hidden email]>
> wrote:
> > >> Well, it's a bit involved to try it in a single thread as I've
> > >> oversimplified the example. But as far as I understand this should
> work,
> > >> right? So something else is wrong? Committing the writer and then
> > >> "maybeRefreshBlocking" should be enough to have the changes visible,
> yes?
> > >>
> > >> On 11/9/18 4:45 PM, Michael Sokolov wrote:
> > >>> That should work, I think, but if you are serializing these threads
> so
> > >>> that they cannot run concurrently, maybe try running both operations
> > >>> in a single thread, at least as a test.
> > >>> On Fri, Nov 9, 2018 at 9:16 AM Boris Petrov <[hidden email]>
> wrote:
> > >>>> If you mean the synchronization of the threads, it is not in the
> > >>>> example, but Thread 2 is *started* after Thread 1 finished
> executing the
> > >>>> code that I gave as an example. So there is happens-before between
> them.
> > >>>> If you mean synchronization on the Lucene level - isn't that what
> > >>>> "maybeRefreshBlocking" should do?
> > >>>>
> > >>>> On 11/9/18 3:29 PM, Michael Sokolov wrote:
> > >>>>> I'm not seeing anything there that would synchronize, or
> serialize, the
> > >>>>> read after the write and commit. Did you expect that for some
> reason?
> > >>>>>
> > >>>>> On Fri, Nov 9, 2018, 6:00 AM Boris Petrov <[hidden email]
> wrote:
> > >>>>>
> > >>>>>> Hi all,
> > >>>>>>
> > >>>>>> I'm using Lucene version 7.5.0. We have a test that does
> something like:
> > >>>>>>
> > >>>>>> Thread 1:
> > >>>>>>
> > >>>>>>             Field idStringField = new StringField("id", id,
> > >>>>>> Field.Store.YES);
> > >>>>>>             Field contentsField = new TextField("contents",
> reader);
> > >>>>>>             Document document = new Document();
> > >>>>>>             document.add(idStringField);
> > >>>>>>             document.add(contentsField);
> > >>>>>>
> > >>>>>>             writer.updateDocument(new Term(ID_FIELD, id),
> document);
> > >>>>>>             writer.flush(); // not sure this flush is needed?
> > >>>>>>             writer.commit();
> > >>>>>>
> > >>>>>> Thread 2:
> > >>>>>>
> > >>>>>>             searchManager.maybeRefreshBlocking();
> > >>>>>>             IndexSearcher searcher = searchManager.acquire();
> > >>>>>>             try {
> > >>>>>>                 QueryParser parser = new QueryParser("contents",
> analyzer);
> > >>>>>>                 Query luceneQuery = parser.parse(queryText);
> > >>>>>>                 ScoreDoc[] hits = searcher.search(luceneQuery,
> > >>>>>> 50).scoreDocs;
> > >>>>>>             } finally {
> > >>>>>>                 searchManager.release(searcher);
> > >>>>>>             }
> > >>>>>>
> > >>>>>> Thread 1 happens before Thread 2.
> > >>>>>>
> > >>>>>> Sometimes, only sometimes, the commit from thread 1 is not
> *immediately*
> > >>>>>> visible in Thread 2. If I put a "Thread.sleep(1000)" it always
> works.
> > >>>>>> Without it, sometimes the search is empty. I'm not sure if I'm
> doing
> > >>>>>> something wrong or this is a bug?
> > >>>>>>
> > >>>>>> Thanks!
> > >>>>>>
> > >>>>>>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: [hidden email]
> > >> For additional commands, e-mail: [hidden email]
> > >>
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>