Frequent Index Replication Failure in solr.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Frequent Index Replication Failure in solr.

Parshant Kumar
Hi team,

We are having solr architecture as *master->repeater-> 3 slave servers.*

We are doing incremental indexing on the master server(every 20 min) .
Replication of index is done from master to repeater server(every 10 mins)
and from repeater to 3 slave servers (every 3 hours).
*We are facing the frequent replication failure between master to repeater
server  as well as between repeater  to slave servers.*
On checking logs found that every time one of the below  exceptions
occurred whenever the replication has failed .

1)WARN : Error in fetching file: _4rnu_t.liv (downloaded 0 of 11505507
bytes)
java.io.EOFException: Unexpected end of ZLIB input stream
        at
java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
        at
java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
        at
org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
        at
org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
        at
org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:139)
        at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:166)
        at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
        at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1443)
        at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1409)


2)
WARN : Error getting file length for [segments_568]
java.nio.file.NoSuchFileException:
/data/solr/search/application/core-conf/im-search/data/index.20200711012319226/segments_568
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.Files.size(Files.java:2332)
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
at
org.apache.solr.handler.admin.LukeRequestHandler.getFileLength(LukeRequestHandler.java:615)
at
org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:588)
at
org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:335)

3)
WARN : Error in fetching file: _4nji.nvd (downloaded 507510784 of 555377795
bytes)
org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk
        at
org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:255)
        at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:227)
        at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:186)
        at
org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
        at
java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
        at
java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
        at
org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
        at
org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:128)
        at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:166)
        at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1458)
        at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1409)
        at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1390)
        at
org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:872)
        at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:438)
        at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:254)

*Replication configuration of master,repeater,slave's is given below:*

<requestHandler name="/replication" class="solr.ReplicationHandler">
        <lst name="master">
            <str name="enable">${enable.master:false}</str>
            <str name="replicateAfter">commit</str>
            <str name="replicateAfter">startup</str>
            <str name="commitReserveDuration">00:00:10</str>
        </lst>


*Commit Configuration master,repeater,slave's is given below :*

 <autoCommit><maxDocs>100000</maxDocs><openSearcher>false</openSearcher></autoCommit>


Please help in finding the root cause of replication failure.Let me
know for any queries.

Thanks

Parshant kumar









<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

--

Reply | Threaded
Open this post in threaded view
|

Re: Frequent Index Replication Failure in solr.

Parshant Kumar
All,please help on this

On Tue, Nov 3, 2020, 6:01 PM Parshant Kumar <[hidden email]>
wrote:

> Hi team,
>
> We are having solr architecture as *master->repeater-> 3 slave servers.*
>
> We are doing incremental indexing on the master server(every 20 min) .
> Replication of index is done from master to repeater server(every 10 mins)
> and from repeater to 3 slave servers (every 3 hours).
> *We are facing the frequent replication failure between master to repeater
> server  as well as between repeater  to slave servers.*
> On checking logs found that every time one of the below  exceptions
> occurred whenever the replication has failed .
>
> 1)WARN : Error in fetching file: _4rnu_t.liv (downloaded 0 of 11505507
> bytes)
> java.io.EOFException: Unexpected end of ZLIB input stream
>         at
> java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
>         at
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
>         at
> org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
>         at
> org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
>         at
> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:139)
>         at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:166)
>         at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
>         at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1443)
>         at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1409)
>
>
> 2)
> WARN : Error getting file length for [segments_568]
> java.nio.file.NoSuchFileException:
> /data/solr/search/application/core-conf/im-search/data/index.20200711012319226/segments_568
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at
> sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
> at
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.nio.file.Files.readAttributes(Files.java:1737)
> at java.nio.file.Files.size(Files.java:2332)
> at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
> at
> org.apache.solr.handler.admin.LukeRequestHandler.getFileLength(LukeRequestHandler.java:615)
> at
> org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:588)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:335)
>
> 3)
> WARN : Error in fetching file: _4nji.nvd (downloaded 507510784 of
> 555377795 bytes)
> org.apache.http.MalformedChunkCodingException: CRLF expected at end of
> chunk
>         at
> org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:255)
>         at
> org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:227)
>         at
> org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:186)
>         at
> org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
>         at
> java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
>         at
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
>         at
> org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
>         at
> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:128)
>         at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:166)
>         at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1458)
>         at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1409)
>         at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1390)
>         at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:872)
>         at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:438)
>         at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:254)
>
> *Replication configuration of master,repeater,slave's is given below:*
>
> <requestHandler name="/replication" class="solr.ReplicationHandler">
>         <lst name="master">
>             <str name="enable">${enable.master:false}</str>
>             <str name="replicateAfter">commit</str>
>             <str name="replicateAfter">startup</str>
>             <str name="commitReserveDuration">00:00:10</str>
>         </lst>
>
>
> *Commit Configuration master,repeater,slave's is given below :*
>
>  <autoCommit><maxDocs>100000</maxDocs><openSearcher>false</openSearcher></autoCommit>
>
>
> Please help in finding the root cause of replication failure.Let me know for any queries.
>
> Thanks
>
> Parshant kumar
>
>
>
>
>
>
>
>
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_5505033333099918512_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>

--

Reply | Threaded
Open this post in threaded view
|

Re: Frequent Index Replication Failure in solr.

David Hastings
looks like youre repeater is grabbing a file that the master merged into a
different file, why not lower how often you go from master->repeater,
and/or dont commit so often so you can make the index faster

On Fri, Nov 13, 2020 at 12:13 PM Parshant Kumar
<[hidden email]> wrote:

> All,please help on this
>
> On Tue, Nov 3, 2020, 6:01 PM Parshant Kumar <[hidden email]>
> wrote:
>
> > Hi team,
> >
> > We are having solr architecture as *master->repeater-> 3 slave servers.*
> >
> > We are doing incremental indexing on the master server(every 20 min) .
> > Replication of index is done from master to repeater server(every 10
> mins)
> > and from repeater to 3 slave servers (every 3 hours).
> > *We are facing the frequent replication failure between master to
> repeater
> > server  as well as between repeater  to slave servers.*
> > On checking logs found that every time one of the below  exceptions
> > occurred whenever the replication has failed .
> >
> > 1)WARN : Error in fetching file: _4rnu_t.liv (downloaded 0 of 11505507
> > bytes)
> > java.io.EOFException: Unexpected end of ZLIB input stream
> >         at
> > java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
> >         at
> > java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
> >         at
> >
> org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
> >         at
> >
> org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
> >         at
> >
> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:139)
> >         at
> >
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:166)
> >         at
> >
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> >         at
> >
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1443)
> >         at
> >
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1409)
> >
> >
> > 2)
> > WARN : Error getting file length for [segments_568]
> > java.nio.file.NoSuchFileException:
> >
> /data/solr/search/application/core-conf/im-search/data/index.20200711012319226/segments_568
> > at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> > at
> >
> sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> > at
> >
> sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
> > at
> >
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> > at java.nio.file.Files.readAttributes(Files.java:1737)
> > at java.nio.file.Files.size(Files.java:2332)
> > at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
> > at
> >
> org.apache.solr.handler.admin.LukeRequestHandler.getFileLength(LukeRequestHandler.java:615)
> > at
> >
> org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:588)
> > at
> >
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:335)
> >
> > 3)
> > WARN : Error in fetching file: _4nji.nvd (downloaded 507510784 of
> > 555377795 bytes)
> > org.apache.http.MalformedChunkCodingException: CRLF expected at end of
> > chunk
> >         at
> > org.apache.http.impl.io
> .ChunkedInputStream.getChunkSize(ChunkedInputStream.java:255)
> >         at
> > org.apache.http.impl.io
> .ChunkedInputStream.nextChunk(ChunkedInputStream.java:227)
> >         at
> > org.apache.http.impl.io
> .ChunkedInputStream.read(ChunkedInputStream.java:186)
> >         at
> >
> org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
> >         at
> > java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
> >         at
> > java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
> >         at
> >
> org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
> >         at
> >
> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:128)
> >         at
> >
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:166)
> >         at
> >
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1458)
> >         at
> >
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1409)
> >         at
> >
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1390)
> >         at
> >
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:872)
> >         at
> >
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:438)
> >         at
> >
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:254)
> >
> > *Replication configuration of master,repeater,slave's is given below:*
> >
> > <requestHandler name="/replication" class="solr.ReplicationHandler">
> >         <lst name="master">
> >             <str name="enable">${enable.master:false}</str>
> >             <str name="replicateAfter">commit</str>
> >             <str name="replicateAfter">startup</str>
> >             <str name="commitReserveDuration">00:00:10</str>
> >         </lst>
> >
> >
> > *Commit Configuration master,repeater,slave's is given below :*
> >
> >
> <autoCommit><maxDocs>100000</maxDocs><openSearcher>false</openSearcher></autoCommit>
> >
> >
> > Please help in finding the root cause of replication failure.Let me know
> for any queries.
> >
> > Thanks
> >
> > Parshant kumar
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > <
> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> Virus-free.
> > www.avast.com
> > <
> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
> >
> > <#m_5505033333099918512_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >
>
> --
>
>