Index Replication Failure

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Index Replication Failure

Parshant Kumar
Hi all,

We are having solr architecture as below.



We are facing the frequent replication failure between master to repeater server  as well as between repeater  to slave servers.
On checking logs found every time one of the below  exceptions occurred whenever the replication have failed.

1)
2)

3)


The replication configuration of master,repeater,slave's is given below:



Commit Configuration master,repeater,slave's is given below :


Replication between master and repeater occurs every 10 mins.
Replication between repeater and slave servers occurs every 15 mins between 4-7 am and after that in every 3 hours.

Thanks,
Parshant Kumar






Reply | Threaded
Open this post in threaded view
|

Re: Index Replication Failure

Parshant Kumar
Architecture image: If not visible in previous mail




On Sat, Oct 17, 2020 at 2:38 PM Parshant Kumar <[hidden email]> wrote:
Hi all,

We are having solr architecture as below.



We are facing the frequent replication failure between master to repeater server  as well as between repeater  to slave servers.
On checking logs found every time one of the below  exceptions occurred whenever the replication have failed.

1)
2)

3)


The replication configuration of master,repeater,slave's is given below:



Commit Configuration master,repeater,slave's is given below :


Replication between master and repeater occurs every 10 mins.
Replication between repeater and slave servers occurs every 15 mins between 4-7 am and after that in every 3 hours.

Thanks,
Parshant Kumar






Reply | Threaded
Open this post in threaded view
|

Re: Index Replication Failure

Erick Erickson
None of your images made it through the mail server. You’ll
have to put them somewhere and provide a link.

> On Oct 17, 2020, at 5:17 AM, Parshant Kumar <[hidden email]> wrote:
>
> Architecture image: If not visible in previous mail
>
>
>
>
> On Sat, Oct 17, 2020 at 2:38 PM Parshant Kumar <[hidden email]> wrote:
> Hi all,
>
> We are having solr architecture as below.
>
>
>
> We are facing the frequent replication failure between master to repeater server  as well as between repeater  to slave servers.
> On checking logs found every time one of the below  exceptions occurred whenever the replication have failed.
>
> 1)
>
> 2)
>
>
> 3)
>
>
> The replication configuration of master,repeater,slave's is given below:
>
>
>
> Commit Configuration master,repeater,slave's is given below :
>
>
>
> Replication between master and repeater occurs every 10 mins.
> Replication between repeater and slave servers occurs every 15 mins between 4-7 am and after that in every 3 hours.
>
> Thanks,
> Parshant Kumar
>
>
>
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Index Replication Failure

Parshant Kumar
*Architecture is master->repeater->slave servers in hierarchy.*

*One of the Below exceptions are occuring whenever replication fails.*

1)WARN : Error in fetching file: _4rnu_t.liv (downloaded 0 of 11505507
bytes)
java.io.EOFException: Unexpected end of ZLIB input stream
        at
java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
        at
java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
        at
org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
        at
org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
        at
org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:139)
        at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:166)
        at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
        at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1443)
        at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1409)

2)
WARN : Error getting file length for [segments_568]
java.nio.file.NoSuchFileException:
/data/solr/search/application/core-conf/im-search/data/index.20200711012319226/segments_568
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.Files.size(Files.java:2332)
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
at
org.apache.solr.handler.admin.LukeRequestHandler.getFileLength(LukeRequestHandler.java:615)
at
org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:588)
at
org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:335)


3)
WARN : Error in fetching file: _4nji.nvd (downloaded 507510784 of 555377795
bytes)
org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk
        at
org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:255)
        at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:227)
        at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:186)
        at
org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
        at
java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
        at
java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
        at
org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
        at
org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:128)
        at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:166)
        at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1458)
        at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1409)
        at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1390)
        at
org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:872)
        at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:438)
        at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:254)


*Replication configuration of master,repeater,slave's is given below:*

 <requestHandler name="/replication" class="solr.ReplicationHandler">
        <lst name="master">
            <str name="enable">${enable.master:false}</str>
            <str name="replicateAfter">commit</str>
            <str name="replicateAfter">startup</str>
            <str name="commitReserveDuration">00:00:10</str>
        </lst>


*Commit Configuration master,repeater,slave's is given below :*

 <autoCommit><maxDocs>100000</maxDocs><openSearcher>false</openSearcher></autoCommit>






On Sat, Oct 17, 2020 at 5:12 PM Erick Erickson <[hidden email]>
wrote:

> None of your images made it through the mail server. You’ll
> have to put them somewhere and provide a link.
>
> > On Oct 17, 2020, at 5:17 AM, Parshant Kumar <
> [hidden email]> wrote:
> >
> > Architecture image: If not visible in previous mail
> >
> >
> >
> >
> > On Sat, Oct 17, 2020 at 2:38 PM Parshant Kumar <
> [hidden email]> wrote:
> > Hi all,
> >
> > We are having solr architecture as below.
> >
> >
> >
> > We are facing the frequent replication failure between master to
> repeater server  as well as between repeater  to slave servers.
> > On checking logs found every time one of the below  exceptions occurred
> whenever the replication have failed.
> >
> > 1)
> >
> > 2)
> >
> >
> > 3)
> >
> >
> > The replication configuration of master,repeater,slave's is given below:
> >
> >
> >
> > Commit Configuration master,repeater,slave's is given below :
> >
> >
> >
> > Replication between master and repeater occurs every 10 mins.
> > Replication between repeater and slave servers occurs every 15 mins
> between 4-7 am and after that in every 3 hours.
> >
> > Thanks,
> > Parshant Kumar
> >
> >
> >
> >
> >
> >
> >
>
>

--

Reply | Threaded
Open this post in threaded view
|

Re: Index Replication Failure

Parshant Kumar
Hi all, please check the details

On Sat, Oct 17, 2020 at 5:52 PM Parshant Kumar <[hidden email]>
wrote:

>
>
> *Architecture is master->repeater->slave servers in hierarchy.*
>
> *One of the Below exceptions are occuring whenever replication fails.*
>
> 1)WARN : Error in fetching file: _4rnu_t.liv (downloaded 0 of 11505507
> bytes)
> java.io.EOFException: Unexpected end of ZLIB input stream
>         at
> java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
>         at
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
>         at
> org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
>         at
> org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:88)
>         at
> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:139)
>         at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:166)
>         at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
>         at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1443)
>         at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1409)
>
> 2)
> WARN : Error getting file length for [segments_568]
> java.nio.file.NoSuchFileException:
> /data/solr/search/application/core-conf/im-search/data/index.20200711012319226/segments_568
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at
> sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
> at
> sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.nio.file.Files.readAttributes(Files.java:1737)
> at java.nio.file.Files.size(Files.java:2332)
> at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
> at
> org.apache.solr.handler.admin.LukeRequestHandler.getFileLength(LukeRequestHandler.java:615)
> at
> org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:588)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:335)
>
>
> 3)
> WARN : Error in fetching file: _4nji.nvd (downloaded 507510784 of
> 555377795 bytes)
> org.apache.http.MalformedChunkCodingException: CRLF expected at end of
> chunk
>         at
> org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:255)
>         at
> org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:227)
>         at
> org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:186)
>         at
> org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
>         at
> java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
>         at
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
>         at
> org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:79)
>         at
> org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:128)
>         at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:166)
>         at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1458)
>         at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1409)
>         at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1390)
>         at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:872)
>         at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:438)
>         at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:254)
>
>
> *Replication configuration of master,repeater,slave's is given below:*
>
>  <requestHandler name="/replication" class="solr.ReplicationHandler">
>         <lst name="master">
>             <str name="enable">${enable.master:false}</str>
>             <str name="replicateAfter">commit</str>
>             <str name="replicateAfter">startup</str>
>             <str name="commitReserveDuration">00:00:10</str>
>         </lst>
>
>
> *Commit Configuration master,repeater,slave's is given below :*
>
>  <autoCommit><maxDocs>100000</maxDocs><openSearcher>false</openSearcher></autoCommit>
>
>
>
>
>
>
> On Sat, Oct 17, 2020 at 5:12 PM Erick Erickson <[hidden email]>
> wrote:
>
>> None of your images made it through the mail server. You’ll
>> have to put them somewhere and provide a link.
>>
>> > On Oct 17, 2020, at 5:17 AM, Parshant Kumar <
>> [hidden email]> wrote:
>> >
>> > Architecture image: If not visible in previous mail
>> >
>> >
>> >
>> >
>> > On Sat, Oct 17, 2020 at 2:38 PM Parshant Kumar <
>> [hidden email]> wrote:
>> > Hi all,
>> >
>> > We are having solr architecture as below.
>> >
>> >
>> >
>> > We are facing the frequent replication failure between master to
>> repeater server  as well as between repeater  to slave servers.
>> > On checking logs found every time one of the below  exceptions occurred
>> whenever the replication have failed.
>> >
>> > 1)
>> >
>> > 2)
>> >
>> >
>> > 3)
>> >
>> >
>> > The replication configuration of master,repeater,slave's is given below:
>> >
>> >
>> >
>> > Commit Configuration master,repeater,slave's is given below :
>> >
>> >
>> >
>> > Replication between master and repeater occurs every 10 mins.
>> > Replication between repeater and slave servers occurs every 15 mins
>> between 4-7 am and after that in every 3 hours.
>> >
>> > Thanks,
>> > Parshant Kumar
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>>
>>

--