[lucy-user] C library - DECREFing PolySearcher

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[lucy-user] C library - DECREFing PolySearcher

serkanmulayim@gmail.com
Hi guys,

I have a question regarding the searchers (PolySearcher and IndexSearcher).

I wrote an application to do searching and indexing for 2 cases: 1- unified
index(searched via IndexSearcher) 2- splitted index (there are around 12
indexes to be searched via PolySearcher). Both indexes have the same number
of documents in total.

In order to make the application simpler, for every search query I create
the related searcher, do the search, and then close it (I do closing by
DECREF). So basically a searcher is created and removed.

For the queries that do not have any results, for the IndexSearcher, I do
not see any issues. For the same query with Poly search, I receive the
following LLDB output. If I wait in the LLDB with a breakpoint before
DECREFing the PolySearcher this issue does not happen.

Process 44678 stopped
* thread #1: tid = 0x325b47, 0x0000000100004c0a
test3searchs`cfish_dec_refcount(vself=0x0000000103abe060) + 10 at
clownfish.c:100, queue = 'com.apple.main-thread', stop reason =
EXC_BAD_ACCESS (code=1, address=0x28)
    frame #0: 0x0000000100004c0a
test3searchs`cfish_dec_refcount(vself=0x0000000103abe060) + 10 at
clownfish.c:100
   97   cfish_dec_refcount(void *vself) {
   98      cfish_Obj *self = (Obj*)vself;
   99      cfish_Class *klass = self->klass;
-> 100    if (klass->flags & CFISH_fREFCOUNTSPECIAL) {
   101        if (SI_immortal(klass)) {
   102            return (uint32_t)self->refcount;
   103        }


I believe it is not directly related to the number of results returned. But
something happens when we try to DECREF very quickly. I see that we need
pthread. This makes me think that search and DECREF does not happen in the
same thread, but I do not want to be confusing so no more comments
regarding the threads :).

Has anyone seen this issue? Do you have any suggestions? Is there a concern
in creating a new searcher and destroying it by DECREF every time (other
than a possible slowness)? Is there a better way to destroy a searcher?

Thanks,
Serkan
Reply | Threaded
Open this post in threaded view
|

[lucy-user] Re: C library - DECREFing PolySearcher

serkanmulayim@gmail.com
Update: I can reproduce it with IndexSearcher too. Previously I had said
that it was only happening in PolySearcher...

On Thu, Jan 5, 2017 at 4:31 PM, Serkan Mulayim <[hidden email]>
wrote:

> Hi guys,
>
> I have a question regarding the searchers (PolySearcher and IndexSearcher).
>
> I wrote an application to do searching and indexing for 2 cases: 1-
> unified index(searched via IndexSearcher) 2- splitted index (there are
> around 12 indexes to be searched via PolySearcher). Both indexes have the
> same number of documents in total.
>
> In order to make the application simpler, for every search query I create
> the related searcher, do the search, and then close it (I do closing by
> DECREF). So basically a searcher is created and removed.
>
> For the queries that do not have any results, for the IndexSearcher, I do
> not see any issues. For the same query with Poly search, I receive the
> following LLDB output. If I wait in the LLDB with a breakpoint before
> DECREFing the PolySearcher this issue does not happen.
>
> Process 44678 stopped
> * thread #1: tid = 0x325b47, 0x0000000100004c0a test3searchs`cfish_dec_
> refcount(vself=0x0000000103abe060) + 10 at clownfish.c:100, queue =
> 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x28)
>     frame #0: 0x0000000100004c0a test3searchs`cfish_dec_refcount(vself=0x0000000103abe060)
> + 10 at clownfish.c:100
>    97   cfish_dec_refcount(void *vself) {
>    98      cfish_Obj *self = (Obj*)vself;
>    99      cfish_Class *klass = self->klass;
> -> 100    if (klass->flags & CFISH_fREFCOUNTSPECIAL) {
>    101        if (SI_immortal(klass)) {
>    102            return (uint32_t)self->refcount;
>    103        }
>
>
> I believe it is not directly related to the number of results returned.
> But something happens when we try to DECREF very quickly. I see that we
> need pthread. This makes me think that search and DECREF does not happen in
> the same thread, but I do not want to be confusing so no more comments
> regarding the threads :).
>
> Has anyone seen this issue? Do you have any suggestions? Is there a
> concern in creating a new searcher and destroying it by DECREF every time
> (other than a possible slowness)? Is there a better way to destroy a
> searcher?
>
> Thanks,
> Serkan
>
>
Reply | Threaded
Open this post in threaded view
|

[lucy-user] Re: C library - DECREFing PolySearcher

serkanmulayim@gmail.com
Sorry guys, 1 more update: when I build the app with -O2 and without -g,
the issue is gone. This makes me think that there is something wrong with
the timing, and related to threads. Just speculating :)

On Thu, Jan 5, 2017 at 5:08 PM, Serkan Mulayim <[hidden email]>
wrote:

> Update: I can reproduce it with IndexSearcher too. Previously I had said
> that it was only happening in PolySearcher...
>
> On Thu, Jan 5, 2017 at 4:31 PM, Serkan Mulayim <[hidden email]>
> wrote:
>
>> Hi guys,
>>
>> I have a question regarding the searchers (PolySearcher and
>> IndexSearcher).
>>
>> I wrote an application to do searching and indexing for 2 cases: 1-
>> unified index(searched via IndexSearcher) 2- splitted index (there are
>> around 12 indexes to be searched via PolySearcher). Both indexes have the
>> same number of documents in total.
>>
>> In order to make the application simpler, for every search query I create
>> the related searcher, do the search, and then close it (I do closing by
>> DECREF). So basically a searcher is created and removed.
>>
>> For the queries that do not have any results, for the IndexSearcher, I do
>> not see any issues. For the same query with Poly search, I receive the
>> following LLDB output. If I wait in the LLDB with a breakpoint before
>> DECREFing the PolySearcher this issue does not happen.
>>
>> Process 44678 stopped
>> * thread #1: tid = 0x325b47, 0x0000000100004c0a
>> test3searchs`cfish_dec_refcount(vself=0x0000000103abe060) + 10 at
>> clownfish.c:100, queue = 'com.apple.main-thread', stop reason =
>> EXC_BAD_ACCESS (code=1, address=0x28)
>>     frame #0: 0x0000000100004c0a test3searchs`cfish_dec_refcount(vself=0x0000000103abe060)
>> + 10 at clownfish.c:100
>>    97   cfish_dec_refcount(void *vself) {
>>    98      cfish_Obj *self = (Obj*)vself;
>>    99      cfish_Class *klass = self->klass;
>> -> 100    if (klass->flags & CFISH_fREFCOUNTSPECIAL) {
>>    101        if (SI_immortal(klass)) {
>>    102            return (uint32_t)self->refcount;
>>    103        }
>>
>>
>> I believe it is not directly related to the number of results returned.
>> But something happens when we try to DECREF very quickly. I see that we
>> need pthread. This makes me think that search and DECREF does not happen in
>> the same thread, but I do not want to be confusing so no more comments
>> regarding the threads :).
>>
>> Has anyone seen this issue? Do you have any suggestions? Is there a
>> concern in creating a new searcher and destroying it by DECREF every time
>> (other than a possible slowness)? Is there a better way to destroy a
>> searcher?
>>
>> Thanks,
>> Serkan
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] C library - DECREFing PolySearcher

Nick Wellnhofer
In reply to this post by serkanmulayim@gmail.com
On 06/01/2017 01:31, Serkan Mulayim wrote:
> I believe it is not directly related to the number of results returned. But
> something happens when we try to DECREF very quickly. I see that we need
> pthread. This makes me think that search and DECREF does not happen in the
> same thread, but I do not want to be confusing so no more comments
> regarding the threads :).

Lucy doesn't use threads internally.

> Has anyone seen this issue? Do you have any suggestions? Is there a concern
> in creating a new searcher and destroying it by DECREF every time (other
> than a possible slowness)? Is there a better way to destroy a searcher?

Recreating searchers is perfectly fine and DECREF is the only way to destroy a
searcher. The only suggestion I have is to test with Valgrind or ASan.

Nick

Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] C library - DECREFing PolySearcher

serkanmulayim@gmail.com
Thanks Nick,

Then can there be a bug s.t. you try to decref the searcher after getting 0
results?

Thanks again,
Serkan

On Fri, Jan 6, 2017 at 4:20 AM, Nick Wellnhofer <[hidden email]> wrote:

> On 06/01/2017 01:31, Serkan Mulayim wrote:
>
>> I believe it is not directly related to the number of results returned.
>> But
>> something happens when we try to DECREF very quickly. I see that we need
>> pthread. This makes me think that search and DECREF does not happen in the
>> same thread, but I do not want to be confusing so no more comments
>> regarding the threads :).
>>
>
> Lucy doesn't use threads internally.
>
> Has anyone seen this issue? Do you have any suggestions? Is there a concern
>> in creating a new searcher and destroying it by DECREF every time (other
>> than a possible slowness)? Is there a better way to destroy a searcher?
>>
>
> Recreating searchers is perfectly fine and DECREF is the only way to
> destroy a searcher. The only suggestion I have is to test with Valgrind or
> ASan.
>
> Nick
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] C library - DECREFing PolySearcher

Nick Wellnhofer
On 06/01/2017 18:55, Serkan Mulayim wrote:
> Then can there be a bug s.t. you try to decref the searcher after getting 0
> results?

That's unlikely but I can't tell without a test case.

Nick

Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] C library - DECREFing PolySearcher

serkanmulayim@gmail.com
Thank you Nick, I think there might be a problem in my code which is not
related to this. I am working on verifying it. I am sorry if I caused if
any troubles. Thanks again...

On Fri, Jan 6, 2017 at 10:08 AM, Nick Wellnhofer <[hidden email]>
wrote:

> On 06/01/2017 18:55, Serkan Mulayim wrote:
>
>> Then can there be a bug s.t. you try to decref the searcher after getting
>> 0
>> results?
>>
>
> That's unlikely but I can't tell without a test case.
>
> Nick
>
>