[lucy-user] Postings.skip error dring updates

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[lucy-user] Postings.skip error dring updates

Thomas den Braber-2
Hallo,

I experience a strange error when doing a lot of update in the index and at
the same time run the background merger.

it is /path/to/index/seg_20u/postings.skip past EOF (14 > 0)

I am using Lucy 0.3.3


Any ideas ?

--
Thomas den Braber
Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] Postings.skip error dring updates

Marvin Humphrey
On Fri, Sep 26, 2014 at 4:29 AM, Thomas den Braber
<[hidden email]> wrote:

> Hallo,
>
> I experience a strange error when doing a lot of update in the index and at
> the same time run the background merger.
>
> it is /path/to/index/seg_20u/postings.skip past EOF (14 > 0)
>
> I am using Lucy 0.3.3
>
> Any ideas ?

This kind of error typically manifests when two write processes have
stomped on each other trying to write the same segment. When an index
is being modified, content is originally written to temporary files;
these temporary files are then consolidated into a "compound" file
before the commit completes.  If one of the temporary files has been
overwritten by another process and is an unexpected length, the
consolidation will fail with an error message as above.

Two situations which have in the past led to such errors are locking
problems and internal failure to clean up detritus from a crashed
writing session.  Is there anything unusual about your locking setup
-- are you disabling locking, running on NFS, etc?

Marvin Humphrey
Reply | Threaded
Open this post in threaded view
|

Re: [lucy-user] Postings.skip error dring updates

Thomas den Braber
Marvin,

Maybe it is something with my locking mechanism. I have a cleanup process
that cleans left over lock files when for some reason the indexer crashes
and leaves a lock file behind. It could be that this process accidentally
cleared a lock file while it was not needed.

I will do some more test.

Thanks for the advise.

--
Thomas
-----Original Message-----
From: Marvin Humphrey <[hidden email]>
To: [hidden email]
Date: Fri, 26 Sep 2014 06:55:34 -0700
Subject: Re: [lucy-user] Postings.skip error dring updates


On Fri, Sep 26, 2014 at 4:29 AM, Thomas den Braber
<[hidden email] [mailto:thomas%40mediafiler.com]> wrote:
> Hallo,
>
> I experience a strange error when doing a lot of update in the index and
at
> the same time run the background merger.
>
> it is /path/to/index/seg_20u/postings.skip past EOF (14 > 0)
>
> I am using Lucy 0.3.3
>
> Any ideas ?

This kind of error typically manifests when two write processes have
stomped on each other trying to write the same segment. When an index
is being modified, content is originally written to temporary files;
these temporary files are then consolidated into a "compound" file
before the commit completes.  If one of the temporary files has been
overwritten by another process and is an unexpected length, the
consolidation will fail with an error message as above.

Two situations which have in the past led to such errors are locking
problems and internal failure to clean up detritus from a crashed
writing session.  Is there anything unusual about your locking setup
-- are you disabling locking, running on NFS, etc?

Marvin Humphrey