NoSuchFileException errors common on version 5.5.0

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

NoSuchFileException errors common on version 5.5.0

Shawn Heisey-2
I have a dev system running 5.5.0.  I am seeing a lot of
NoSuchFileException errors (for segments_XXXfilenames).

Here's a log excerpt:

2016-03-10 09:52:00.054 INFO  (qtp1012570586-821) [   x:inclive]
org.apache.solr.core.SolrCore.Request [inclive]  webapp=/solr
path=/admin/luke
params={qt=/admin/luke&show=schema&wt=javabin&version=2} status=500 QTime=1
2016-03-10 09:52:00.055 ERROR (qtp1012570586-821) [   x:inclive]
org.apache.solr.servlet.HttpSolrCall
null:java.nio.file.NoSuchFileException:
/index/solr5/data/data/inc_0/index/segments_ias
        at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
        at
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
        at
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
        at java.nio.file.Files.readAttributes(Files.java:1737)
        at java.nio.file.Files.size(Files.java:2332)
        at
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:209)
<snip>

I did not include the full stacktrace, only up to the first Lucene/Solr
class.

Most of the error logs are preceded by a request to the /admin/luke
handler, like you see above, but there are also entries where a failed
request is not logged right before the error.  My index maintenance
program calls /admin/luke to programmatically determine the uniqueKey
for the index.

These errors do not seem to actually interfere with Solr operation, but
they do concern me.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: NoSuchFileException errors common on version 5.5.0

Kevin Risden
This sounds related to SOLR-8587 and there is a fix in SOLR-8793 that isn't
out in a release since it was fixed after 5.5 went out.

Kevin Risden
Hadoop Tech Lead | Avalon Consulting, LLC <http://www.avalonconsult.com/>
M: 732 213 8417
LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> | Google+
<http://www.google.com/+AvalonConsultingLLC> | Twitter
<https://twitter.com/avalonconsult>

-------------------------------------------------------------------------------------------------------------
This message (including any attachments) contains confidential information
intended for a specific individual and purpose, and is protected by law. If
you are not the intended recipient, you should delete this message. Any
disclosure, copying, or distribution of this message, or the taking of any
action based on it, is strictly prohibited.

On Thu, Mar 10, 2016 at 11:02 AM, Shawn Heisey <[hidden email]> wrote:

> I have a dev system running 5.5.0.  I am seeing a lot of
> NoSuchFileException errors (for segments_XXXfilenames).
>
> Here's a log excerpt:
>
> 2016-03-10 09:52:00.054 INFO  (qtp1012570586-821) [   x:inclive]
> org.apache.solr.core.SolrCore.Request [inclive]  webapp=/solr
> path=/admin/luke
> params={qt=/admin/luke&show=schema&wt=javabin&version=2} status=500 QTime=1
> 2016-03-10 09:52:00.055 ERROR (qtp1012570586-821) [   x:inclive]
> org.apache.solr.servlet.HttpSolrCall
> null:java.nio.file.NoSuchFileException:
> /index/solr5/data/data/inc_0/index/segments_ias
>         at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>         at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>         at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>         at
>
> sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
>         at
>
> sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
>         at
>
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
>         at java.nio.file.Files.readAttributes(Files.java:1737)
>         at java.nio.file.Files.size(Files.java:2332)
>         at
> org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:209)
> <snip>
>
> I did not include the full stacktrace, only up to the first Lucene/Solr
> class.
>
> Most of the error logs are preceded by a request to the /admin/luke
> handler, like you see above, but there are also entries where a failed
> request is not logged right before the error.  My index maintenance
> program calls /admin/luke to programmatically determine the uniqueKey
> for the index.
>
> These errors do not seem to actually interfere with Solr operation, but
> they do concern me.
>
> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: NoSuchFileException errors common on version 5.5.0

Shawn Heisey-2
On 3/10/2016 10:09 AM, Kevin Risden wrote:
> This sounds related to SOLR-8587 and there is a fix in SOLR-8793 that isn't
> out in a release since it was fixed after 5.5 went out.

Thanks for that info.

I pulled down branch_5_5 and installed a 5.5.1 snapshot.  Had to edit
lucene/version.properties to get it to be 5.5.1.  I also had to edit the
SolrIdentifierValidator class to allow hyphens, since I have them in
some of my core names.  The NoSuchFileException errors are gone now.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: NoSuchFileException errors common on version 5.5.0

Shawn Heisey-2
On 3/10/2016 12:18 PM, Shawn Heisey wrote:
> I pulled down branch_5_5 and installed a 5.5.1 snapshot.  Had to edit
> lucene/version.properties to get it to be 5.5.1.  I also had to edit the
> SolrIdentifierValidator class to allow hyphens, since I have them in
> some of my core names.  The NoSuchFileException errors are gone now.

Spoke too soon.

The log message did change a little bit.  Now it's only one log entry on
LukeRequestHandler instead of two separate log entries, and it's a WARN
instead of ERROR.

2016-03-10 14:35:00.038 WARN  (qtp1012570586-11405) [   x:spark3live]
org.apache.solr.handler.admin.LukeRequestHandler Error getting file
length for [segments_c5t]
java.nio.file.NoSuchFileException:
/index/solr5/data/data/spark3_0/index/segments_c5t
        at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
        at
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
        at
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
        at java.nio.file.Files.readAttributes(Files.java:1737)
        at java.nio.file.Files.size(Files.java:2332)
        at
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:210)

Something else to note:  It wasn't 5.5.0 that I had installed, it was
5.5.0-SNAPSHOT -- I installed it some time before 5.5.0 was released.
Looks like I did the install of that version on January 29th.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

RE: NoSuchFileException errors common on version 5.5.0

Markus Jelsma-2
Hm, it happens on one of our nodes quite frequently, but that was 5.4, maybe even 5.3.

org.apache.solr.common.SolrException: Error handling 'status' action
        at org.apache.solr.handler.admin.CoreAdminOperation$4.call(CoreAdminOperation.java:192)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:354)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:153)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:676)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:439)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.NoSuchFileException: /var/lib/solr/logs_shard1_replica1/data/index/segments_47ne
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
        at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
        at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
        at java.nio.file.Files.readAttributes(Files.java:1737)
        at java.nio.file.Files.size(Files.java:2332)
        at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:210)
        at org.apache.lucene.store.NRTCachingDirectory.fileLength(NRTCachingDirectory.java:127)
        at org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:592)
        at org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:886)
        at org.apache.solr.handler.admin.CoreAdminOperation$4.call(CoreAdminOperation.java:180)
        ... 27 more

 
 
-----Original message-----

> From:Shawn Heisey <[hidden email]>
> Sent: Thursday 10th March 2016 23:21
> To: [hidden email]
> Subject: Re: NoSuchFileException errors common on version 5.5.0
>
> On 3/10/2016 12:18 PM, Shawn Heisey wrote:
> > I pulled down branch_5_5 and installed a 5.5.1 snapshot.  Had to edit
> > lucene/version.properties to get it to be 5.5.1.  I also had to edit the
> > SolrIdentifierValidator class to allow hyphens, since I have them in
> > some of my core names.  The NoSuchFileException errors are gone now.
>
> Spoke too soon.
>
> The log message did change a little bit.  Now it's only one log entry on
> LukeRequestHandler instead of two separate log entries, and it's a WARN
> instead of ERROR.
>
> 2016-03-10 14:35:00.038 WARN  (qtp1012570586-11405) [   x:spark3live]
> org.apache.solr.handler.admin.LukeRequestHandler Error getting file
> length for [segments_c5t]
> java.nio.file.NoSuchFileException:
> /index/solr5/data/data/spark3_0/index/segments_c5t
>         at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>         at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>         at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>         at
> sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
>         at
> sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
>         at
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
>         at java.nio.file.Files.readAttributes(Files.java:1737)
>         at java.nio.file.Files.size(Files.java:2332)
>         at
> org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:210)
>
> Something else to note:  It wasn't 5.5.0 that I had installed, it was
> 5.5.0-SNAPSHOT -- I installed it some time before 5.5.0 was released.
> Looks like I did the install of that version on January 29th.
>
> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: NoSuchFileException errors common on version 5.5.0

Shai Erera
In reply to this post by Shawn Heisey-2
Hey Shawn,

I added segments file information (name and size) to Core admin status API.
Turns out that you might get into NoSuchFileException if indexing happens
and the commit point has changed, but the IndexReader LukeRequestHandler
receives hasn't picked up the new commit yet, in which case the old
segments_N file was deleted and computing its size resulted in that
exception.

I pushed a fix for it which will be released in any one of future releases,
including 5.5.1 if we'll have any. The fix includes logging the exception
and returning -1 as the file size.

Shai

On Fri, Mar 11, 2016 at 12:21 AM Shawn Heisey <[hidden email]> wrote:

> On 3/10/2016 12:18 PM, Shawn Heisey wrote:
> > I pulled down branch_5_5 and installed a 5.5.1 snapshot.  Had to edit
> > lucene/version.properties to get it to be 5.5.1.  I also had to edit the
> > SolrIdentifierValidator class to allow hyphens, since I have them in
> > some of my core names.  The NoSuchFileException errors are gone now.
>
> Spoke too soon.
>
> The log message did change a little bit.  Now it's only one log entry on
> LukeRequestHandler instead of two separate log entries, and it's a WARN
> instead of ERROR.
>
> 2016-03-10 14:35:00.038 WARN  (qtp1012570586-11405) [   x:spark3live]
> org.apache.solr.handler.admin.LukeRequestHandler Error getting file
> length for [segments_c5t]
> java.nio.file.NoSuchFileException:
> /index/solr5/data/data/spark3_0/index/segments_c5t
>         at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>         at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>         at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>         at
>
> sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
>         at
>
> sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
>         at
>
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
>         at java.nio.file.Files.readAttributes(Files.java:1737)
>         at java.nio.file.Files.size(Files.java:2332)
>         at
> org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:210)
>
> Something else to note:  It wasn't 5.5.0 that I had installed, it was
> 5.5.0-SNAPSHOT -- I installed it some time before 5.5.0 was released.
> Looks like I did the install of that version on January 29th.
>
> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: NoSuchFileException errors common on version 5.5.0

Jay Potharaju-2
Hi.
I am seeing lot of these errors in my current 5.5.0 dev install. Would it
make sense to use 5.5 in production or a different version is recommended ?
I am using DIH, not sure if that matters in this case.

Thanks


On Fri, Mar 11, 2016 at 3:57 AM, Shai Erera <[hidden email]> wrote:

> Hey Shawn,
>
> I added segments file information (name and size) to Core admin status API.
> Turns out that you might get into NoSuchFileException if indexing happens
> and the commit point has changed, but the IndexReader LukeRequestHandler
> receives hasn't picked up the new commit yet, in which case the old
> segments_N file was deleted and computing its size resulted in that
> exception.
>
> I pushed a fix for it which will be released in any one of future releases,
> including 5.5.1 if we'll have any. The fix includes logging the exception
> and returning -1 as the file size.
>
> Shai
>
> On Fri, Mar 11, 2016 at 12:21 AM Shawn Heisey <[hidden email]> wrote:
>
> > On 3/10/2016 12:18 PM, Shawn Heisey wrote:
> > > I pulled down branch_5_5 and installed a 5.5.1 snapshot.  Had to edit
> > > lucene/version.properties to get it to be 5.5.1.  I also had to edit
> the
> > > SolrIdentifierValidator class to allow hyphens, since I have them in
> > > some of my core names.  The NoSuchFileException errors are gone now.
> >
> > Spoke too soon.
> >
> > The log message did change a little bit.  Now it's only one log entry on
> > LukeRequestHandler instead of two separate log entries, and it's a WARN
> > instead of ERROR.
> >
> > 2016-03-10 14:35:00.038 WARN  (qtp1012570586-11405) [   x:spark3live]
> > org.apache.solr.handler.admin.LukeRequestHandler Error getting file
> > length for [segments_c5t]
> > java.nio.file.NoSuchFileException:
> > /index/solr5/data/data/spark3_0/index/segments_c5t
> >         at
> > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> >         at
> > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> >         at
> > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> >         at
> >
> >
> sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> >         at
> >
> >
> sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
> >         at
> >
> >
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> >         at java.nio.file.Files.readAttributes(Files.java:1737)
> >         at java.nio.file.Files.size(Files.java:2332)
> >         at
> > org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:210)
> >
> > Something else to note:  It wasn't 5.5.0 that I had installed, it was
> > 5.5.0-SNAPSHOT -- I installed it some time before 5.5.0 was released.
> > Looks like I did the install of that version on January 29th.
> >
> > Thanks,
> > Shawn
> >
> >
>



--
Thanks
Jay Potharaju