[lucy-user] Lucy unable to find an index file

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[lucy-user] Lucy unable to find an index file

Edwin Crockford
Hi guys,

Can anyone help. I'm getting the following type of error message while
attempting to rebuild a Lucy index:

Can't open '/home/ipacs/ipacs/index/webdisk/seg_943/lextemp': No such file or directory
         lucy_FSFolder_local_open_filehandle at /root/.local/share/.cpan/build/Lucy-0.3.2-oCiVA7/core/Lucy/Store/FSFolder.c line 102
         lucy_Folder_local_open_in at /root/.local/share/.cpan/build/Lucy-0.3.2-oCiVA7/core/Lucy/Store/Folder.c line 97
         lucy_Folder_open_in at /root/.local/share/.cpan/build/Lucy-0.3.2-oCiVA7/core/Lucy/Store/Folder.c line 72
         lucy_PostPool_flip at /root/.local/share/.cpan/build/Lucy-0.3.2-oCiVA7/core/Lucy/Index/PostingPool.c line 179

Anyone seen this before and have some ideas of what is going wrong?

Thanks

Edwin Crockford


Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] Lucy unable to find an index file

Marvin Humphrey
On Wed, Aug 7, 2013 at 2:32 AM, Edwin Crockford <[hidden email]> wrote:

> I'm getting the following type of error message while
> attempting to rebuild a Lucy index:
>
> Can't open '/home/ipacs/ipacs/index/webdisk/seg_943/lextemp': No such file
> or directory
>         lucy_FSFolder_local_open_filehandle at
> /root/.local/share/.cpan/build/Lucy-0.3.2-oCiVA7/core/Lucy/Store/FSFolder.c
> line 102
>         lucy_Folder_local_open_in at
> /root/.local/share/.cpan/build/Lucy-0.3.2-oCiVA7/core/Lucy/Store/Folder.c
> line 97
>         lucy_Folder_open_in at
> /root/.local/share/.cpan/build/Lucy-0.3.2-oCiVA7/core/Lucy/Store/Folder.c
> line 72
>         lucy_PostPool_flip at
> /root/.local/share/.cpan/build/Lucy-0.3.2-oCiVA7/core/Lucy/Index/PostingPool.c
> line 179
>
> Anyone seen this before and have some ideas of what is going wrong?

I don't recall having seen that reported as a trouble spot before.

That particular error would occur after all docs have been added and while the
new segment is being finished.  The Indexer is looking for a temporary file
related to the lexicon which should have been written while adding documents.

Some questions to help with troubleshooting:

*   Is the error reproducible?
*   Anything noteworthy about the file system -- e.g. is this an NFS volume?
*   What do you mean by "reindexing"?
*   Any permissions issues?
*   What do you see in /home/ipacs/ipacs/index/webdisk/seg_943/ after the
    crash?
*   How many docs are being added before calling Indexer#commit?
*   Any fields that have no data?  I'm thinking something like a document that
    contains a `foo => undef` pair.  If that's the case, what happens if you
    don't supply that key?

Marvin Humphrey
Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] Lucy unable to find an index file

Edwin Crockford
On 07/08/2013 14:11, Marvin Humphrey wrote:

> On Wed, Aug 7, 2013 at 2:32 AM, Edwin Crockford <[hidden email]> wrote:
>> I'm getting the following type of error message while
>> attempting to rebuild a Lucy index:
>>
>> Can't open '/home/ipacs/ipacs/index/webdisk/seg_943/lextemp': No such file
>> or directory
>>          lucy_FSFolder_local_open_filehandle at
>> /root/.local/share/.cpan/build/Lucy-0.3.2-oCiVA7/core/Lucy/Store/FSFolder.c
>> line 102
>>          lucy_Folder_local_open_in at
>> /root/.local/share/.cpan/build/Lucy-0.3.2-oCiVA7/core/Lucy/Store/Folder.c
>> line 97
>>          lucy_Folder_open_in at
>> /root/.local/share/.cpan/build/Lucy-0.3.2-oCiVA7/core/Lucy/Store/Folder.c
>> line 72
>>          lucy_PostPool_flip at
>> /root/.local/share/.cpan/build/Lucy-0.3.2-oCiVA7/core/Lucy/Index/PostingPool.c
>> line 179
>>
>> Anyone seen this before and have some ideas of what is going wrong?
> I don't recall having seen that reported as a trouble spot before.
>
> That particular error would occur after all docs have been added and while the
> new segment is being finished.  The Indexer is looking for a temporary file
> related to the lexicon which should have been written while adding documents.
>
> Some questions to help with troubleshooting:
>
> *   Is the error reproducible?
> *   Anything noteworthy about the file system -- e.g. is this an NFS volume?
> *   What do you mean by "reindexing"?
> *   Any permissions issues?
> *   What do you see in /home/ipacs/ipacs/index/webdisk/seg_943/ after the
>      crash?
> *   How many docs are being added before calling Indexer#commit?
> *   Any fields that have no data?  I'm thinking something like a document that
>      contains a `foo => undef` pair.  If that's the case, what happens if you
>      don't supply that key?
>
> Marvin Humphrey
Thanks for coming back to me and sorry to be slow in replying.  I
believe we have traced the issue back to an interaction between two
different systems (one doing bulk updates and another doing on the fly
single document indexing) attempting updates at the same time. I think
there was a way around the locking that caused the problem, does that
seem plausable?

The error was reproducible to some extent but went away when we shut
down one of the systems.
File system is standard UNIX, Centos 6
Reindexing refers to the bulk indexer (mentioned above) that rebuilds
the index from scratch.
No permissions issue that are obvious, both indexers were running as the
same user.
The docs are added in blocks of around 10 by the bulk indexer, we always
populate all the fields

Any ideas would be helpful

Thanks

Edwin

Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] Lucy unable to find an index file

Marvin Humphrey
On Fri, Aug 9, 2013 at 5:29 AM, Edwin Crockford <[hidden email]> wrote:
> I believe we
> have traced the issue back to an interaction between two different systems
> (one doing bulk updates and another doing on the fly single document
> indexing) attempting updates at the same time. I think there was a way
> around the locking that caused the problem, does that seem plausable?

Yes, that makes sense.  The error can be explained by having two Indexers
trying to write to the same segment.  One of them happens to delete the temp
file "lextemp" first, and then the other can't find it and throws an
exception.

Only one Indexer may operate on a given index at a time.  A BackgroundMerger
may operate at the same time as an Indexer, but even it must acquire the write
lock briefly (once at the start of its run and once again at the end).  While
Lucy's locking APIs provide the technical capacity to disable the locking
mechanism, it is not generally possible to get around the need for locking in
order to coordinate write access to an index.

> The error was reproducible to some extent but went away when we shut down
> one of the systems.
>
> File system is standard UNIX, Centos 6
>
> Reindexing refers to the bulk indexer (mentioned above) that rebuilds the
> index from scratch.
>
> No permissions issue that are obvious, both indexers were running as the
> same user.
>
> The docs are added in blocks of around 10 by the bulk indexer, we always
> populate all the fields
>
> Any ideas would be helpful

If the "bulk indexer" does not change any content, you might look into
BackgroundMerger.

If it changes content, then perhaps consider building the replacement index
off to the side and queueing new docs, then adding the new docs after the
bulk update finishes and finally moving the index into place.

Marvin Humphrey
Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] Lucy unable to find an index file

Edwin Crockford
On 09/08/2013 15:48, Marvin Humphrey wrote:

> On Fri, Aug 9, 2013 at 5:29 AM, Edwin Crockford <[hidden email]> wrote:
>> I believe we
>> have traced the issue back to an interaction between two different systems
>> (one doing bulk updates and another doing on the fly single document
>> indexing) attempting updates at the same time. I think there was a way
>> around the locking that caused the problem, does that seem plausable?
> Yes, that makes sense.  The error can be explained by having two Indexers
> trying to write to the same segment.  One of them happens to delete the temp
> file "lextemp" first, and then the other can't find it and throws an
> exception.
>
> Only one Indexer may operate on a given index at a time.  A BackgroundMerger
> may operate at the same time as an Indexer, but even it must acquire the write
> lock briefly (once at the start of its run and once again at the end).  While
> Lucy's locking APIs provide the technical capacity to disable the locking
> mechanism, it is not generally possible to get around the need for locking in
> order to coordinate write access to an index.
>
>> The error was reproducible to some extent but went away when we shut down
>> one of the systems.
>>
>> File system is standard UNIX, Centos 6
>>
>> Reindexing refers to the bulk indexer (mentioned above) that rebuilds the
>> index from scratch.
>>
>> No permissions issue that are obvious, both indexers were running as the
>> same user.
>>
>> The docs are added in blocks of around 10 by the bulk indexer, we always
>> populate all the fields
>>
>> Any ideas would be helpful
> If the "bulk indexer" does not change any content, you might look into
> BackgroundMerger.
>
> If it changes content, then perhaps consider building the replacement index
> off to the side and queueing new docs, then adding the new docs after the
> bulk update finishes and finally moving the index into place.
>
> Marvin Humphrey
Thanks Marvin, sounds like I am on the right sort of track. Will look
into the options mentioned

Edwin