solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

yaswanthcse
Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
configurations with all the certs in place, but the issue what I am seeing
is when trying to hit /update api on non-leader solr node , its throwing an
error

configured 2 solr nodes with 1 zookeeper.

metadata":[
"error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
"root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
"msg":"Async exception during distributed update:
javax.crypto.BadPaddingException: RSA private key operation failed",
"trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update:
javax.crypto.BadPaddingException: RSA private key operation failed\n\tat
org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.finish........

Strangely this is happening when we try to hit a non-leader node, hitting
leader node its working fine without any issue and getting the data indexed.

Not able to track down where the exact issue is happening.

Thanks,

--
Thanks & Regards,
Yaswanth Kumar Konathala.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

yaswanthcse
team, can someone help me on the above topic?

On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <[hidden email]>
wrote:

> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
> configurations with all the certs in place, but the issue what I am seeing
> is when trying to hit /update api on non-leader solr node , its throwing an
> error
>
> configured 2 solr nodes with 1 zookeeper.
>
> metadata":[
>
> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>
> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
> "msg":"Async exception during distributed update:
> javax.crypto.BadPaddingException: RSA private key operation failed",
> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> Async exception during distributed update:
> javax.crypto.BadPaddingException: RSA private key operation failed\n\tat
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>
> Strangely this is happening when we try to hit a non-leader node, hitting
> leader node its working fine without any issue and getting the data indexed.
>
> Not able to track down where the exact issue is happening.
>
> Thanks,
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> [hidden email]
>


--
Thanks & Regards,
Yaswanth Kumar Konathala.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

Jörn Franke
Have you looked in the logfiles?

Keystore Type correctly defined  on all nodes?

Have you configured the truststore on all nodes correctly?

Have you set clusterprop urlScheme to htttps in ZK?

https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper



> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <[hidden email]>:
>
> team, can someone help me on the above topic?
>
>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <[hidden email]>
>> wrote:
>>
>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>> configurations with all the certs in place, but the issue what I am seeing
>> is when trying to hit /update api on non-leader solr node , its throwing an
>> error
>>
>> configured 2 solr nodes with 1 zookeeper.
>>
>> metadata":[
>>
>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>>
>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>> "msg":"Async exception during distributed update:
>> javax.crypto.BadPaddingException: RSA private key operation failed",
>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>> Async exception during distributed update:
>> javax.crypto.BadPaddingException: RSA private key operation failed\n\tat
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>>
>> Strangely this is happening when we try to hit a non-leader node, hitting
>> leader node its working fine without any issue and getting the data indexed.
>>
>> Not able to track down where the exact issue is happening.
>>
>> Thanks,
>>
>> --
>> Thanks & Regards,
>> Yaswanth Kumar Konathala.
>> [hidden email]
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

yaswanthcse
Thanks Franke, but yes for all these questions I did configured it
properly, I made sure to include

<Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
default="JKS"/></Set>
  <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
default="JKS"/></Set>
in the jetty-ssl.xml along with the path keystore and truststore.

Also I have made sure that trusstore exists on all nodes and also I am
using the same file for both keystore and truststore as below
 <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
default="./etc/solr-keystore.jks"/></Set>
  <Set name="KeyStorePassword"><Property
name="solr.jetty.keystore.password" default="xxxx"/></Set>
  <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
default="./etc/solr-keystore.jks"/></Set>
  <Set name="TrustStorePassword"><Property
name="solr.jetty.truststore.password" default="xxxx"/></Set>

also urlScheme for ZK is set to https


Also the main error that I posted is the one that I am seeing as a return
response where as the below one is what I see from solr logs

2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
null:org.apache.solr.update.processor.Distr$
        at
org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
        at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
        at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
        at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
        at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
        at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
        at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
        at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
        at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
        at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
        at
org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
        at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
        at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
        at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
        at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)


One strange observation is that when I hit update api on the leader node
its working without any error, and now immediately if I hit non-leader its
working fine (only once or twice), but if I keep on trying to hit this node
again and again its then throwing the above error and once the error
started happening , its consistent again.

Please let me know if you need more information or if I am missing
something else

Thanks,

On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <[hidden email]> wrote:

> Have you looked in the logfiles?
>
> Keystore Type correctly defined  on all nodes?
>
> Have you configured the truststore on all nodes correctly?
>
> Have you set clusterprop urlScheme to htttps in ZK?
>
>
> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>
>
>
> > Am 02.06.2020 um 18:57 schrieb yaswanth kumar <[hidden email]>:
> >
> > team, can someone help me on the above topic?
> >
> >> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <[hidden email]>
> >> wrote:
> >>
> >> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
> >> configurations with all the certs in place, but the issue what I am
> seeing
> >> is when trying to hit /update api on non-leader solr node , its
> throwing an
> >> error
> >>
> >> configured 2 solr nodes with 1 zookeeper.
> >>
> >> metadata":[
> >>
> >>
> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
> >>
> >>
> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
> >> "msg":"Async exception during distributed update:
> >> javax.crypto.BadPaddingException: RSA private key operation failed",
> >>
> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >> Async exception during distributed update:
> >> javax.crypto.BadPaddingException: RSA private key operation failed\n\tat
> >>
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
> >>
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
> >>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
> >> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
> >>
> >> Strangely this is happening when we try to hit a non-leader node,
> hitting
> >> leader node its working fine without any issue and getting the data
> indexed.
> >>
> >> Not able to track down where the exact issue is happening.
> >>
> >> Thanks,
> >>
> >> --
> >> Thanks & Regards,
> >> Yaswanth Kumar Konathala.
> >> [hidden email]
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Yaswanth Kumar Konathala.
> > [hidden email]
>


--
Thanks & Regards,
Yaswanth Kumar Konathala.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

yaswanthcse
also I am seeing the below error as a parent one from solr.log

 at org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:366)
org.apache.solr.common.SolrException: javax.crypto.BadPaddingException: RSA
private key operation failed
        at
org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:366)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -
ishan - 2020-01-10 1$
        at
org.apache.solr.security.PKIAuthenticationPlugin.generateToken(PKIAuthenticationPlugin.java:305)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d0$
        at
org.apache.solr.security.PKIAuthenticationPlugin.access$200(PKIAuthenticationPlugin.java:61)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -$
        at
org.apache.solr.security.PKIAuthenticationPlugin$2.onQueued(PKIAuthenticationPlugin.java:239)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 $
        at
org.apache.solr.client.solrj.impl.Http2SolrClient.decorateRequest(Http2SolrClient.java:469)
~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -$
        at
org.apache.solr.client.solrj.impl.Http2SolrClient.initOutStream(Http2SolrClient.java:324)
~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 - i$
        at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:227)
~[solr-solrj-8.4.1.jar:8.4.1 83$
        at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd918709$
        at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
~[metrics-core-4.0.5.jar:4.0.5]
        at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf6978$
        at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/0000000000000000.run(Unknown
Source) ~[?:?]
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.crypto.BadPaddingException: RSA private key operation
failed
        at
sun.security.rsa.NativeRSACore.crtCrypt_Native(NativeRSACore.java:149)
~[?:?]
        at sun.security.rsa.NativeRSACore.rsa(NativeRSACore.java:91) ~[?:?]
        at sun.security.rsa.RSACore.rsa(RSACore.java:149) ~[?:?]
        at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:355)
~[?:?]
        at
com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:392) ~[?:?]
        at javax.crypto.Cipher.doFinal(Cipher.java:2260) ~[?:?]
        at
org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:364)
~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -
ishan - 2020-01-10 1$
        ... 13 more

On Tue, Jun 2, 2020 at 6:37 PM yaswanth kumar <[hidden email]> wrote:

> Thanks Franke, but yes for all these questions I did configured it
> properly, I made sure to include
>
> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
> default="JKS"/></Set>
>   <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
> default="JKS"/></Set>
> in the jetty-ssl.xml along with the path keystore and truststore.
>
> Also I have made sure that trusstore exists on all nodes and also I am
> using the same file for both keystore and truststore as below
>  <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
> default="./etc/solr-keystore.jks"/></Set>
>   <Set name="KeyStorePassword"><Property
> name="solr.jetty.keystore.password" default="xxxx"/></Set>
>   <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
> default="./etc/solr-keystore.jks"/></Set>
>   <Set name="TrustStorePassword"><Property
> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>
> also urlScheme for ZK is set to https
>
>
> Also the main error that I posted is the one that I am seeing as a return
> response where as the below one is what I see from solr logs
>
> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
> null:org.apache.solr.update.processor.Distr$
>         at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
>         at
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
>         at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
>         at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>         at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>         at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>         at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>         at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>         at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>         at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>         at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>         at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
>         at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>         at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>         at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>         at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>         at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>         at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>         at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>         at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>         at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>
>
> One strange observation is that when I hit update api on the leader node
> its working without any error, and now immediately if I hit non-leader its
> working fine (only once or twice), but if I keep on trying to hit this node
> again and again its then throwing the above error and once the error
> started happening , its consistent again.
>
> Please let me know if you need more information or if I am missing
> something else
>
> Thanks,
>
> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <[hidden email]> wrote:
>
>> Have you looked in the logfiles?
>>
>> Keystore Type correctly defined  on all nodes?
>>
>> Have you configured the truststore on all nodes correctly?
>>
>> Have you set clusterprop urlScheme to htttps in ZK?
>>
>>
>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>
>>
>>
>> > Am 02.06.2020 um 18:57 schrieb yaswanth kumar <[hidden email]>:
>> >
>> > team, can someone help me on the above topic?
>> >
>> >> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <[hidden email]>
>> >> wrote:
>> >>
>> >> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>> >> configurations with all the certs in place, but the issue what I am
>> seeing
>> >> is when trying to hit /update api on non-leader solr node , its
>> throwing an
>> >> error
>> >>
>> >> configured 2 solr nodes with 1 zookeeper.
>> >>
>> >> metadata":[
>> >>
>> >>
>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>> >>
>> >>
>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>> >> "msg":"Async exception during distributed update:
>> >> javax.crypto.BadPaddingException: RSA private key operation failed",
>> >>
>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>> >> Async exception during distributed update:
>> >> javax.crypto.BadPaddingException: RSA private key operation
>> failed\n\tat
>> >>
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>> >>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>> >>
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>> >>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>> >> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>> >>
>> >> Strangely this is happening when we try to hit a non-leader node,
>> hitting
>> >> leader node its working fine without any issue and getting the data
>> indexed.
>> >>
>> >> Not able to track down where the exact issue is happening.
>> >>
>> >> Thanks,
>> >>
>> >> --
>> >> Thanks & Regards,
>> >> Yaswanth Kumar Konathala.
>> >> [hidden email]
>> >>
>> >
>> >
>> > --
>> > Thanks & Regards,
>> > Yaswanth Kumar Konathala.
>> > [hidden email]
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> [hidden email]
>


--
Thanks & Regards,
Yaswanth Kumar Konathala.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

yaswanthcse
also forgot to update before that I have enabled basicauthentication and
provided the details in security.json and uploaded it via zookeeper.

Thanks,

On Tue, Jun 2, 2020 at 6:42 PM yaswanth kumar <[hidden email]> wrote:

> also I am seeing the below error as a parent one from solr.log
>
>  at org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:366)
> org.apache.solr.common.SolrException: javax.crypto.BadPaddingException:
> RSA private key operation failed
>         at
> org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:366)
> ~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -
> ishan - 2020-01-10 1$
>         at
> org.apache.solr.security.PKIAuthenticationPlugin.generateToken(PKIAuthenticationPlugin.java:305)
> ~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d0$
>         at
> org.apache.solr.security.PKIAuthenticationPlugin.access$200(PKIAuthenticationPlugin.java:61)
> ~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -$
>         at
> org.apache.solr.security.PKIAuthenticationPlugin$2.onQueued(PKIAuthenticationPlugin.java:239)
> ~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 $
>         at
> org.apache.solr.client.solrj.impl.Http2SolrClient.decorateRequest(Http2SolrClient.java:469)
> ~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -$
>         at
> org.apache.solr.client.solrj.impl.Http2SolrClient.initOutStream(Http2SolrClient.java:324)
> ~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 - i$
>         at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:227)
> ~[solr-solrj-8.4.1.jar:8.4.1 83$
>         at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
> ~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd918709$
>         at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> ~[metrics-core-4.0.5.jar:4.0.5]
>         at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> ~[solr-solrj-8.4.1.jar:8.4.1 832bf13dd9187095831caf6978$
>         at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/0000000000000000.run(Unknown
> Source) ~[?:?]
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> ~[?:?]
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> ~[?:?]
>         at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: javax.crypto.BadPaddingException: RSA private key operation
> failed
>         at
> sun.security.rsa.NativeRSACore.crtCrypt_Native(NativeRSACore.java:149)
> ~[?:?]
>         at sun.security.rsa.NativeRSACore.rsa(NativeRSACore.java:91) ~[?:?]
>         at sun.security.rsa.RSACore.rsa(RSACore.java:149) ~[?:?]
>         at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:355)
> ~[?:?]
>         at
> com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:392) ~[?:?]
>         at javax.crypto.Cipher.doFinal(Cipher.java:2260) ~[?:?]
>         at
> org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:364)
> ~[solr-core-8.4.1.jar:8.4.1 832bf13dd9187095831caf69783179d41059d013 -
> ishan - 2020-01-10 1$
>         ... 13 more
>
> On Tue, Jun 2, 2020 at 6:37 PM yaswanth kumar <[hidden email]>
> wrote:
>
>> Thanks Franke, but yes for all these questions I did configured it
>> properly, I made sure to include
>>
>> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
>> default="JKS"/></Set>
>>   <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
>> default="JKS"/></Set>
>> in the jetty-ssl.xml along with the path keystore and truststore.
>>
>> Also I have made sure that trusstore exists on all nodes and also I am
>> using the same file for both keystore and truststore as below
>>  <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
>> default="./etc/solr-keystore.jks"/></Set>
>>   <Set name="KeyStorePassword"><Property
>> name="solr.jetty.keystore.password" default="xxxx"/></Set>
>>   <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
>> default="./etc/solr-keystore.jks"/></Set>
>>   <Set name="TrustStorePassword"><Property
>> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>>
>> also urlScheme for ZK is set to https
>>
>>
>> Also the main error that I posted is the one that I am seeing as a return
>> response where as the below one is what I see from solr logs
>>
>> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
>> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
>> null:org.apache.solr.update.processor.Distr$
>>         at
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
>>         at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
>>         at
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>         at
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>         at
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
>>         at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>>         at
>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>>         at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>         at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>         at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>         at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>         at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>
>>
>> One strange observation is that when I hit update api on the leader node
>> its working without any error, and now immediately if I hit non-leader its
>> working fine (only once or twice), but if I keep on trying to hit this node
>> again and again its then throwing the above error and once the error
>> started happening , its consistent again.
>>
>> Please let me know if you need more information or if I am missing
>> something else
>>
>> Thanks,
>>
>> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <[hidden email]> wrote:
>>
>>> Have you looked in the logfiles?
>>>
>>> Keystore Type correctly defined  on all nodes?
>>>
>>> Have you configured the truststore on all nodes correctly?
>>>
>>> Have you set clusterprop urlScheme to htttps in ZK?
>>>
>>>
>>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>>
>>>
>>>
>>> > Am 02.06.2020 um 18:57 schrieb yaswanth kumar <[hidden email]>:
>>> >
>>> > team, can someone help me on the above topic?
>>> >
>>> >> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <[hidden email]
>>> >
>>> >> wrote:
>>> >>
>>> >> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>>> >> configurations with all the certs in place, but the issue what I am
>>> seeing
>>> >> is when trying to hit /update api on non-leader solr node , its
>>> throwing an
>>> >> error
>>> >>
>>> >> configured 2 solr nodes with 1 zookeeper.
>>> >>
>>> >> metadata":[
>>> >>
>>> >>
>>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>>> >>
>>> >>
>>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>>> >> "msg":"Async exception during distributed update:
>>> >> javax.crypto.BadPaddingException: RSA private key operation failed",
>>> >>
>>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>>> >> Async exception during distributed update:
>>> >> javax.crypto.BadPaddingException: RSA private key operation
>>> failed\n\tat
>>> >>
>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>>> >>
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>>> >>
>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>>> >>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>>> >> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>>> >>
>>> >> Strangely this is happening when we try to hit a non-leader node,
>>> hitting
>>> >> leader node its working fine without any issue and getting the data
>>> indexed.
>>> >>
>>> >> Not able to track down where the exact issue is happening.
>>> >>
>>> >> Thanks,
>>> >>
>>> >> --
>>> >> Thanks & Regards,
>>> >> Yaswanth Kumar Konathala.
>>> >> [hidden email]
>>> >>
>>> >
>>> >
>>> > --
>>> > Thanks & Regards,
>>> > Yaswanth Kumar Konathala.
>>> > [hidden email]
>>>
>>
>>
>> --
>> Thanks & Regards,
>> Yaswanth Kumar Konathala.
>> [hidden email]
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> [hidden email]
>


--
Thanks & Regards,
Yaswanth Kumar Konathala.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

Jörn Franke
In reply to this post by yaswanthcse
Why in the jetty-ssl.xml?

Should this not be configured in the solr.in.sh?

> Am 03.06.2020 um 00:38 schrieb yaswanth kumar <[hidden email]>:
>
> Thanks Franke, but yes for all these questions I did configured it
> properly, I made sure to include
>
> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
> default="JKS"/></Set>
>  <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
> default="JKS"/></Set>
> in the jetty-ssl.xml along with the path keystore and truststore.
>
> Also I have made sure that trusstore exists on all nodes and also I am
> using the same file for both keystore and truststore as below
> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
> default="./etc/solr-keystore.jks"/></Set>
>  <Set name="KeyStorePassword"><Property
> name="solr.jetty.keystore.password" default="xxxx"/></Set>
>  <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
> default="./etc/solr-keystore.jks"/></Set>
>  <Set name="TrustStorePassword"><Property
> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>
> also urlScheme for ZK is set to https
>
>
> Also the main error that I posted is the one that I am seeing as a return
> response where as the below one is what I see from solr logs
>
> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
> null:org.apache.solr.update.processor.Distr$
>        at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
>        at
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
>        at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
>        at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>        at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>        at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>        at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>        at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>        at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>        at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>        at
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>        at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
>        at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>        at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>        at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>        at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>        at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>        at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>        at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>        at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>        at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>        at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>
>
> One strange observation is that when I hit update api on the leader node
> its working without any error, and now immediately if I hit non-leader its
> working fine (only once or twice), but if I keep on trying to hit this node
> again and again its then throwing the above error and once the error
> started happening , its consistent again.
>
> Please let me know if you need more information or if I am missing
> something else
>
> Thanks,
>
>> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <[hidden email]> wrote:
>>
>> Have you looked in the logfiles?
>>
>> Keystore Type correctly defined  on all nodes?
>>
>> Have you configured the truststore on all nodes correctly?
>>
>> Have you set clusterprop urlScheme to htttps in ZK?
>>
>>
>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>
>>
>>
>>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <[hidden email]>:
>>>
>>> team, can someone help me on the above topic?
>>>
>>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <[hidden email]>
>>>> wrote:
>>>>
>>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>>>> configurations with all the certs in place, but the issue what I am
>> seeing
>>>> is when trying to hit /update api on non-leader solr node , its
>> throwing an
>>>> error
>>>>
>>>> configured 2 solr nodes with 1 zookeeper.
>>>>
>>>> metadata":[
>>>>
>>>>
>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>>>>
>>>>
>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>>>> "msg":"Async exception during distributed update:
>>>> javax.crypto.BadPaddingException: RSA private key operation failed",
>>>>
>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>>>> Async exception during distributed update:
>>>> javax.crypto.BadPaddingException: RSA private key operation failed\n\tat
>>>>
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>>>>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>>>>
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>>>>
>>>> Strangely this is happening when we try to hit a non-leader node,
>> hitting
>>>> leader node its working fine without any issue and getting the data
>> indexed.
>>>>
>>>> Not able to track down where the exact issue is happening.
>>>>
>>>> Thanks,
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Yaswanth Kumar Konathala.
>>>> [hidden email]
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Yaswanth Kumar Konathala.
>>> [hidden email]
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

yaswanthcse
thanks Franke,

I now made the use of the default jetty-ssl.xml that comes with the solr
package, but the issue is still happening when I try to push data to a
non-leader node.

Do you still think if its something to do with the configurations ??

Thanks,

On Wed, Jun 3, 2020 at 12:29 AM Jörn Franke <[hidden email]> wrote:

> Why in the jetty-ssl.xml?
>
> Should this not be configured in the solr.in.sh?
>
> > Am 03.06.2020 um 00:38 schrieb yaswanth kumar <[hidden email]>:
> >
> > Thanks Franke, but yes for all these questions I did configured it
> > properly, I made sure to include
> >
> > <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
> > default="JKS"/></Set>
> >  <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
> > default="JKS"/></Set>
> > in the jetty-ssl.xml along with the path keystore and truststore.
> >
> > Also I have made sure that trusstore exists on all nodes and also I am
> > using the same file for both keystore and truststore as below
> > <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
> > default="./etc/solr-keystore.jks"/></Set>
> >  <Set name="KeyStorePassword"><Property
> > name="solr.jetty.keystore.password" default="xxxx"/></Set>
> >  <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
> > default="./etc/solr-keystore.jks"/></Set>
> >  <Set name="TrustStorePassword"><Property
> > name="solr.jetty.truststore.password" default="xxxx"/></Set>
> >
> > also urlScheme for ZK is set to https
> >
> >
> > Also the main error that I posted is the one that I am seeing as a return
> > response where as the below one is what I see from solr logs
> >
> > 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
> > r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
> > null:org.apache.solr.update.processor.Distr$
> >        at
> >
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
> >        at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
> >        at
> >
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
> >        at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >        at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >        at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >        at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >        at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >        at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >        at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >        at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >        at
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
> >        at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> >        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
> >        at
> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
> >        at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
> >        at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
> >        at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
> >        at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> >        at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> >        at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> >        at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> >        at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> >        at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> >
> >
> > One strange observation is that when I hit update api on the leader node
> > its working without any error, and now immediately if I hit non-leader
> its
> > working fine (only once or twice), but if I keep on trying to hit this
> node
> > again and again its then throwing the above error and once the error
> > started happening , its consistent again.
> >
> > Please let me know if you need more information or if I am missing
> > something else
> >
> > Thanks,
> >
> >> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <[hidden email]>
> wrote:
> >>
> >> Have you looked in the logfiles?
> >>
> >> Keystore Type correctly defined  on all nodes?
> >>
> >> Have you configured the truststore on all nodes correctly?
> >>
> >> Have you set clusterprop urlScheme to htttps in ZK?
> >>
> >>
> >>
> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
> >>
> >>
> >>
> >>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <[hidden email]
> >:
> >>>
> >>> team, can someone help me on the above topic?
> >>>
> >>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <[hidden email]
> >
> >>>> wrote:
> >>>>
> >>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
> >>>> configurations with all the certs in place, but the issue what I am
> >> seeing
> >>>> is when trying to hit /update api on non-leader solr node , its
> >> throwing an
> >>>> error
> >>>>
> >>>> configured 2 solr nodes with 1 zookeeper.
> >>>>
> >>>> metadata":[
> >>>>
> >>>>
> >>
> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
> >>>>
> >>>>
> >>
> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
> >>>> "msg":"Async exception during distributed update:
> >>>> javax.crypto.BadPaddingException: RSA private key operation failed",
> >>>>
> >>
> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >>>> Async exception during distributed update:
> >>>> javax.crypto.BadPaddingException: RSA private key operation
> failed\n\tat
> >>>>
> >>
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
> >>>>
> >>
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
> >>>>
> >>
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
> >>>>
> >>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
> >>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
> >>>>
> >>>> Strangely this is happening when we try to hit a non-leader node,
> >> hitting
> >>>> leader node its working fine without any issue and getting the data
> >> indexed.
> >>>>
> >>>> Not able to track down where the exact issue is happening.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> --
> >>>> Thanks & Regards,
> >>>> Yaswanth Kumar Konathala.
> >>>> [hidden email]
> >>>>
> >>>
> >>>
> >>> --
> >>> Thanks & Regards,
> >>> Yaswanth Kumar Konathala.
> >>> [hidden email]
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Yaswanth Kumar Konathala.
> > [hidden email]
>


--
Thanks & Regards,
Yaswanth Kumar Konathala.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

yaswanthcse
Hi Franke,

I suspect its because of the certificate encryption ?? But will wait for
you to confirm the same. We are trying to generate a certs with RSA 2048
and finally combining them to a single JKS and that's what we are referring
as a keystore and truststore, let me know if it doesn't work or if there is
a standard procedure to do this certs.

Thanks,

On Wed, Jun 3, 2020 at 8:25 AM yaswanth kumar <[hidden email]> wrote:

> thanks Franke,
>
> I now made the use of the default jetty-ssl.xml that comes with the solr
> package, but the issue is still happening when I try to push data to a
> non-leader node.
>
> Do you still think if its something to do with the configurations ??
>
> Thanks,
>
> On Wed, Jun 3, 2020 at 12:29 AM Jörn Franke <[hidden email]> wrote:
>
>> Why in the jetty-ssl.xml?
>>
>> Should this not be configured in the solr.in.sh?
>>
>> > Am 03.06.2020 um 00:38 schrieb yaswanth kumar <[hidden email]>:
>> >
>> > Thanks Franke, but yes for all these questions I did configured it
>> > properly, I made sure to include
>> >
>> > <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
>> > default="JKS"/></Set>
>> >  <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
>> > default="JKS"/></Set>
>> > in the jetty-ssl.xml along with the path keystore and truststore.
>> >
>> > Also I have made sure that trusstore exists on all nodes and also I am
>> > using the same file for both keystore and truststore as below
>> > <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
>> > default="./etc/solr-keystore.jks"/></Set>
>> >  <Set name="KeyStorePassword"><Property
>> > name="solr.jetty.keystore.password" default="xxxx"/></Set>
>> >  <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
>> > default="./etc/solr-keystore.jks"/></Set>
>> >  <Set name="TrustStorePassword"><Property
>> > name="solr.jetty.truststore.password" default="xxxx"/></Set>
>> >
>> > also urlScheme for ZK is set to https
>> >
>> >
>> > Also the main error that I posted is the one that I am seeing as a
>> return
>> > response where as the below one is what I see from solr logs
>> >
>> > 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
>> > r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
>> > null:org.apache.solr.update.processor.Distr$
>> >        at
>> >
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
>> >        at
>> >
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
>> >        at
>> >
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
>> >        at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> >        at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> >        at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> >        at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> >        at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> >        at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> >        at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> >        at
>> >
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>> >        at
>> >
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
>> >        at
>> >
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>> >        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>> >        at
>> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>> >        at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>> >        at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>> >        at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>> >        at
>> >
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>> >        at
>> >
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>> >        at
>> >
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>> >        at
>> >
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>> >        at
>> >
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> >        at
>> >
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>> >
>> >
>> > One strange observation is that when I hit update api on the leader node
>> > its working without any error, and now immediately if I hit non-leader
>> its
>> > working fine (only once or twice), but if I keep on trying to hit this
>> node
>> > again and again its then throwing the above error and once the error
>> > started happening , its consistent again.
>> >
>> > Please let me know if you need more information or if I am missing
>> > something else
>> >
>> > Thanks,
>> >
>> >> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <[hidden email]>
>> wrote:
>> >>
>> >> Have you looked in the logfiles?
>> >>
>> >> Keystore Type correctly defined  on all nodes?
>> >>
>> >> Have you configured the truststore on all nodes correctly?
>> >>
>> >> Have you set clusterprop urlScheme to htttps in ZK?
>> >>
>> >>
>> >>
>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>> >>
>> >>
>> >>
>> >>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <[hidden email]
>> >:
>> >>>
>> >>> team, can someone help me on the above topic?
>> >>>
>> >>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <
>> [hidden email]>
>> >>>> wrote:
>> >>>>
>> >>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>> >>>> configurations with all the certs in place, but the issue what I am
>> >> seeing
>> >>>> is when trying to hit /update api on non-leader solr node , its
>> >> throwing an
>> >>>> error
>> >>>>
>> >>>> configured 2 solr nodes with 1 zookeeper.
>> >>>>
>> >>>> metadata":[
>> >>>>
>> >>>>
>> >>
>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>> >>>>
>> >>>>
>> >>
>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>> >>>> "msg":"Async exception during distributed update:
>> >>>> javax.crypto.BadPaddingException: RSA private key operation failed",
>> >>>>
>> >>
>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>> >>>> Async exception during distributed update:
>> >>>> javax.crypto.BadPaddingException: RSA private key operation
>> failed\n\tat
>> >>>>
>> >>
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>> >>>>
>> >>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>> >>>>
>> >>
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>> >>>>
>> >>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>> >>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>> >>>>
>> >>>> Strangely this is happening when we try to hit a non-leader node,
>> >> hitting
>> >>>> leader node its working fine without any issue and getting the data
>> >> indexed.
>> >>>>
>> >>>> Not able to track down where the exact issue is happening.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> --
>> >>>> Thanks & Regards,
>> >>>> Yaswanth Kumar Konathala.
>> >>>> [hidden email]
>> >>>>
>> >>>
>> >>>
>> >>> --
>> >>> Thanks & Regards,
>> >>> Yaswanth Kumar Konathala.
>> >>> [hidden email]
>> >>
>> >
>> >
>> > --
>> > Thanks & Regards,
>> > Yaswanth Kumar Konathala.
>> > [hidden email]
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> [hidden email]
>


--
Thanks & Regards,
Yaswanth Kumar Konathala.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

Jörn Franke
I think you should not do it in the Jetty xml
Follow the official reference guide.
It should be in solr.in.sh

https://lucene.apache.org/solr/guide/8_4/enabling-ssl.html




> Am 04.06.2020 um 06:48 schrieb yaswanth kumar <[hidden email]>:
>
> Hi Franke,
>
> I suspect its because of the certificate encryption ?? But will wait for
> you to confirm the same. We are trying to generate a certs with RSA 2048
> and finally combining them to a single JKS and that's what we are referring
> as a keystore and truststore, let me know if it doesn't work or if there is
> a standard procedure to do this certs.
>
> Thanks,
>
>> On Wed, Jun 3, 2020 at 8:25 AM yaswanth kumar <[hidden email]> wrote:
>>
>> thanks Franke,
>>
>> I now made the use of the default jetty-ssl.xml that comes with the solr
>> package, but the issue is still happening when I try to push data to a
>> non-leader node.
>>
>> Do you still think if its something to do with the configurations ??
>>
>> Thanks,
>>
>>> On Wed, Jun 3, 2020 at 12:29 AM Jörn Franke <[hidden email]> wrote:
>>>
>>> Why in the jetty-ssl.xml?
>>>
>>> Should this not be configured in the solr.in.sh?
>>>
>>>> Am 03.06.2020 um 00:38 schrieb yaswanth kumar <[hidden email]>:
>>>>
>>>> Thanks Franke, but yes for all these questions I did configured it
>>>> properly, I made sure to include
>>>>
>>>> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
>>>> default="JKS"/></Set>
>>>> <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
>>>> default="JKS"/></Set>
>>>> in the jetty-ssl.xml along with the path keystore and truststore.
>>>>
>>>> Also I have made sure that trusstore exists on all nodes and also I am
>>>> using the same file for both keystore and truststore as below
>>>> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
>>>> default="./etc/solr-keystore.jks"/></Set>
>>>> <Set name="KeyStorePassword"><Property
>>>> name="solr.jetty.keystore.password" default="xxxx"/></Set>
>>>> <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
>>>> default="./etc/solr-keystore.jks"/></Set>
>>>> <Set name="TrustStorePassword"><Property
>>>> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>>>>
>>>> also urlScheme for ZK is set to https
>>>>
>>>>
>>>> Also the main error that I posted is the one that I am seeing as a
>>> return
>>>> response where as the below one is what I see from solr logs
>>>>
>>>> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
>>>> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
>>>> null:org.apache.solr.update.processor.Distr$
>>>>       at
>>>>
>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
>>>>       at
>>>>
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>>>>       at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>>>>       at
>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>>>>       at
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>>>>       at
>>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>>>>       at
>>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>>>>       at
>>>>
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>>>       at
>>>>
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>>>       at
>>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>>>       at
>>>>
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>>>       at
>>>>
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>>       at
>>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>>>
>>>>
>>>> One strange observation is that when I hit update api on the leader node
>>>> its working without any error, and now immediately if I hit non-leader
>>> its
>>>> working fine (only once or twice), but if I keep on trying to hit this
>>> node
>>>> again and again its then throwing the above error and once the error
>>>> started happening , its consistent again.
>>>>
>>>> Please let me know if you need more information or if I am missing
>>>> something else
>>>>
>>>> Thanks,
>>>>
>>>>> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <[hidden email]>
>>> wrote:
>>>>>
>>>>> Have you looked in the logfiles?
>>>>>
>>>>> Keystore Type correctly defined  on all nodes?
>>>>>
>>>>> Have you configured the truststore on all nodes correctly?
>>>>>
>>>>> Have you set clusterprop urlScheme to htttps in ZK?
>>>>>
>>>>>
>>>>>
>>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>>>>
>>>>>
>>>>>
>>>>>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <[hidden email]
>>>> :
>>>>>>
>>>>>> team, can someone help me on the above topic?
>>>>>>
>>>>>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <
>>> [hidden email]>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>>>>>>> configurations with all the certs in place, but the issue what I am
>>>>> seeing
>>>>>>> is when trying to hit /update api on non-leader solr node , its
>>>>> throwing an
>>>>>>> error
>>>>>>>
>>>>>>> configured 2 solr nodes with 1 zookeeper.
>>>>>>>
>>>>>>> metadata":[
>>>>>>>
>>>>>>>
>>>>>
>>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>>>>>>>
>>>>>>>
>>>>>
>>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>>>>>>> "msg":"Async exception during distributed update:
>>>>>>> javax.crypto.BadPaddingException: RSA private key operation failed",
>>>>>>>
>>>>>
>>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>>>>>>> Async exception during distributed update:
>>>>>>> javax.crypto.BadPaddingException: RSA private key operation
>>> failed\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>>>>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>>>>>>>
>>>>>>> Strangely this is happening when we try to hit a non-leader node,
>>>>> hitting
>>>>>>> leader node its working fine without any issue and getting the data
>>>>> indexed.
>>>>>>>
>>>>>>> Not able to track down where the exact issue is happening.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> --
>>>>>>> Thanks & Regards,
>>>>>>> Yaswanth Kumar Konathala.
>>>>>>> [hidden email]
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Yaswanth Kumar Konathala.
>>>>>> [hidden email]
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Yaswanth Kumar Konathala.
>>>> [hidden email]
>>>
>>
>>
>> --
>> Thanks & Regards,
>> Yaswanth Kumar Konathala.
>> [hidden email]
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

yaswanthcse
I haven't done any changes on jetty xml , I am just using what it comes
with the solr package. just doing it in solr.in.sh but I am still seeing
the same issue.

Thanks,

On Thu, Jun 4, 2020 at 12:23 PM Jörn Franke <[hidden email]> wrote:

> I think you should not do it in the Jetty xml
> Follow the official reference guide.
> It should be in solr.in.sh
>
> https://lucene.apache.org/solr/guide/8_4/enabling-ssl.html
>
>
>
>
> > Am 04.06.2020 um 06:48 schrieb yaswanth kumar <[hidden email]>:
> >
> > Hi Franke,
> >
> > I suspect its because of the certificate encryption ?? But will wait for
> > you to confirm the same. We are trying to generate a certs with RSA 2048
> > and finally combining them to a single JKS and that's what we are
> referring
> > as a keystore and truststore, let me know if it doesn't work or if there
> is
> > a standard procedure to do this certs.
> >
> > Thanks,
> >
> >> On Wed, Jun 3, 2020 at 8:25 AM yaswanth kumar <[hidden email]>
> wrote:
> >>
> >> thanks Franke,
> >>
> >> I now made the use of the default jetty-ssl.xml that comes with the solr
> >> package, but the issue is still happening when I try to push data to a
> >> non-leader node.
> >>
> >> Do you still think if its something to do with the configurations ??
> >>
> >> Thanks,
> >>
> >>> On Wed, Jun 3, 2020 at 12:29 AM Jörn Franke <[hidden email]>
> wrote:
> >>>
> >>> Why in the jetty-ssl.xml?
> >>>
> >>> Should this not be configured in the solr.in.sh?
> >>>
> >>>> Am 03.06.2020 um 00:38 schrieb yaswanth kumar <[hidden email]
> >:
> >>>>
> >>>> Thanks Franke, but yes for all these questions I did configured it
> >>>> properly, I made sure to include
> >>>>
> >>>> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
> >>>> default="JKS"/></Set>
> >>>> <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
> >>>> default="JKS"/></Set>
> >>>> in the jetty-ssl.xml along with the path keystore and truststore.
> >>>>
> >>>> Also I have made sure that trusstore exists on all nodes and also I am
> >>>> using the same file for both keystore and truststore as below
> >>>> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
> >>>> default="./etc/solr-keystore.jks"/></Set>
> >>>> <Set name="KeyStorePassword"><Property
> >>>> name="solr.jetty.keystore.password" default="xxxx"/></Set>
> >>>> <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
> >>>> default="./etc/solr-keystore.jks"/></Set>
> >>>> <Set name="TrustStorePassword"><Property
> >>>> name="solr.jetty.truststore.password" default="xxxx"/></Set>
> >>>>
> >>>> also urlScheme for ZK is set to https
> >>>>
> >>>>
> >>>> Also the main error that I posted is the one that I am seeing as a
> >>> return
> >>>> response where as the below one is what I see from solr logs
> >>>>
> >>>> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
> >>>> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
> >>>> null:org.apache.solr.update.processor.Distr$
> >>>>       at
> >>>>
> >>>
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> >>>>       at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
> >>>>       at
> >>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
> >>>>       at
> >>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
> >>>>       at
> >>>>
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
> >>>>       at
> >>>>
> >>>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> >>>>       at
> >>>>
> >>>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> >>>>       at
> >>>>
> >>>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> >>>>       at
> >>>>
> >>>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> >>>>       at
> >>>>
> >>>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> >>>>       at
> >>>>
> >>>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> >>>>
> >>>>
> >>>> One strange observation is that when I hit update api on the leader
> node
> >>>> its working without any error, and now immediately if I hit non-leader
> >>> its
> >>>> working fine (only once or twice), but if I keep on trying to hit this
> >>> node
> >>>> again and again its then throwing the above error and once the error
> >>>> started happening , its consistent again.
> >>>>
> >>>> Please let me know if you need more information or if I am missing
> >>>> something else
> >>>>
> >>>> Thanks,
> >>>>
> >>>>> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <[hidden email]>
> >>> wrote:
> >>>>>
> >>>>> Have you looked in the logfiles?
> >>>>>
> >>>>> Keystore Type correctly defined  on all nodes?
> >>>>>
> >>>>> Have you configured the truststore on all nodes correctly?
> >>>>>
> >>>>> Have you set clusterprop urlScheme to htttps in ZK?
> >>>>>
> >>>>>
> >>>>>
> >>>
> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
> >>>>>
> >>>>>
> >>>>>
> >>>>>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <
> [hidden email]
> >>>> :
> >>>>>>
> >>>>>> team, can someone help me on the above topic?
> >>>>>>
> >>>>>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <
> >>> [hidden email]>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the
> ssl
> >>>>>>> configurations with all the certs in place, but the issue what I am
> >>>>> seeing
> >>>>>>> is when trying to hit /update api on non-leader solr node , its
> >>>>> throwing an
> >>>>>>> error
> >>>>>>>
> >>>>>>> configured 2 solr nodes with 1 zookeeper.
> >>>>>>>
> >>>>>>> metadata":[
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>
> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>
> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
> >>>>>>> "msg":"Async exception during distributed update:
> >>>>>>> javax.crypto.BadPaddingException: RSA private key operation
> failed",
> >>>>>>>
> >>>>>
> >>>
> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> >>>>>>> Async exception during distributed update:
> >>>>>>> javax.crypto.BadPaddingException: RSA private key operation
> >>> failed\n\tat
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
> >>>>>>>
> >>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
> >>>>>>>
> >>>>>>> Strangely this is happening when we try to hit a non-leader node,
> >>>>> hitting
> >>>>>>> leader node its working fine without any issue and getting the data
> >>>>> indexed.
> >>>>>>>
> >>>>>>> Not able to track down where the exact issue is happening.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> --
> >>>>>>> Thanks & Regards,
> >>>>>>> Yaswanth Kumar Konathala.
> >>>>>>> [hidden email]
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Thanks & Regards,
> >>>>>> Yaswanth Kumar Konathala.
> >>>>>> [hidden email]
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Thanks & Regards,
> >>>> Yaswanth Kumar Konathala.
> >>>> [hidden email]
> >>>
> >>
> >>
> >> --
> >> Thanks & Regards,
> >> Yaswanth Kumar Konathala.
> >> [hidden email]
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Yaswanth Kumar Konathala.
> > [hidden email]
>


--
Thanks & Regards,
Yaswanth Kumar Konathala.
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

Jörn Franke
If the keystore and/or truststore is encrypted you need to provide the Passwort in solr.in.sh

> Am 04.06.2020 um 18:38 schrieb yaswanth kumar <[hidden email]>:
>
> I haven't done any changes on jetty xml , I am just using what it comes
> with the solr package. just doing it in solr.in.sh but I am still seeing
> the same issue.
>
> Thanks,
>
>> On Thu, Jun 4, 2020 at 12:23 PM Jörn Franke <[hidden email]> wrote:
>>
>> I think you should not do it in the Jetty xml
>> Follow the official reference guide.
>> It should be in solr.in.sh
>>
>> https://lucene.apache.org/solr/guide/8_4/enabling-ssl.html
>>
>>
>>
>>
>>>> Am 04.06.2020 um 06:48 schrieb yaswanth kumar <[hidden email]>:
>>>
>>> Hi Franke,
>>>
>>> I suspect its because of the certificate encryption ?? But will wait for
>>> you to confirm the same. We are trying to generate a certs with RSA 2048
>>> and finally combining them to a single JKS and that's what we are
>> referring
>>> as a keystore and truststore, let me know if it doesn't work or if there
>> is
>>> a standard procedure to do this certs.
>>>
>>> Thanks,
>>>
>>>> On Wed, Jun 3, 2020 at 8:25 AM yaswanth kumar <[hidden email]>
>> wrote:
>>>>
>>>> thanks Franke,
>>>>
>>>> I now made the use of the default jetty-ssl.xml that comes with the solr
>>>> package, but the issue is still happening when I try to push data to a
>>>> non-leader node.
>>>>
>>>> Do you still think if its something to do with the configurations ??
>>>>
>>>> Thanks,
>>>>
>>>>> On Wed, Jun 3, 2020 at 12:29 AM Jörn Franke <[hidden email]>
>> wrote:
>>>>>
>>>>> Why in the jetty-ssl.xml?
>>>>>
>>>>> Should this not be configured in the solr.in.sh?
>>>>>
>>>>>> Am 03.06.2020 um 00:38 schrieb yaswanth kumar <[hidden email]
>>> :
>>>>>>
>>>>>> Thanks Franke, but yes for all these questions I did configured it
>>>>>> properly, I made sure to include
>>>>>>
>>>>>> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
>>>>>> default="JKS"/></Set>
>>>>>> <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
>>>>>> default="JKS"/></Set>
>>>>>> in the jetty-ssl.xml along with the path keystore and truststore.
>>>>>>
>>>>>> Also I have made sure that trusstore exists on all nodes and also I am
>>>>>> using the same file for both keystore and truststore as below
>>>>>> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
>>>>>> default="./etc/solr-keystore.jks"/></Set>
>>>>>> <Set name="KeyStorePassword"><Property
>>>>>> name="solr.jetty.keystore.password" default="xxxx"/></Set>
>>>>>> <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
>>>>>> default="./etc/solr-keystore.jks"/></Set>
>>>>>> <Set name="TrustStorePassword"><Property
>>>>>> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>>>>>>
>>>>>> also urlScheme for ZK is set to https
>>>>>>
>>>>>>
>>>>>> Also the main error that I posted is the one that I am seeing as a
>>>>> return
>>>>>> response where as the below one is what I see from solr logs
>>>>>>
>>>>>> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
>>>>>> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
>>>>>> null:org.apache.solr.update.processor.Distr$
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>>>>>>      at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>>>>>>      at
>>>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>>>>>>      at
>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>>>>>>      at
>>>>>>
>>>>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>>>>>>      at
>>>>>>
>>>>>
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>>>>>      at
>>>>>>
>>>>>
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>>>>>      at
>>>>>>
>>>>>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>>>>>      at
>>>>>>
>>>>>
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>>>>>      at
>>>>>>
>>>>>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>>>>      at
>>>>>>
>>>>>
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>>>>>
>>>>>>
>>>>>> One strange observation is that when I hit update api on the leader
>> node
>>>>>> its working without any error, and now immediately if I hit non-leader
>>>>> its
>>>>>> working fine (only once or twice), but if I keep on trying to hit this
>>>>> node
>>>>>> again and again its then throwing the above error and once the error
>>>>>> started happening , its consistent again.
>>>>>>
>>>>>> Please let me know if you need more information or if I am missing
>>>>>> something else
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <[hidden email]>
>>>>> wrote:
>>>>>>>
>>>>>>> Have you looked in the logfiles?
>>>>>>>
>>>>>>> Keystore Type correctly defined  on all nodes?
>>>>>>>
>>>>>>> Have you configured the truststore on all nodes correctly?
>>>>>>>
>>>>>>> Have you set clusterprop urlScheme to htttps in ZK?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <
>> [hidden email]
>>>>>> :
>>>>>>>>
>>>>>>>> team, can someone help me on the above topic?
>>>>>>>>
>>>>>>>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <
>>>>> [hidden email]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the
>> ssl
>>>>>>>>> configurations with all the certs in place, but the issue what I am
>>>>>>> seeing
>>>>>>>>> is when trying to hit /update api on non-leader solr node , its
>>>>>>> throwing an
>>>>>>>>> error
>>>>>>>>>
>>>>>>>>> configured 2 solr nodes with 1 zookeeper.
>>>>>>>>>
>>>>>>>>> metadata":[
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>>>>>>>>> "msg":"Async exception during distributed update:
>>>>>>>>> javax.crypto.BadPaddingException: RSA private key operation
>> failed",
>>>>>>>>>
>>>>>>>
>>>>>
>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>>>>>>>>> Async exception during distributed update:
>>>>>>>>> javax.crypto.BadPaddingException: RSA private key operation
>>>>> failed\n\tat
>>>>>>>>>
>>>>>>>
>>>>>
>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>>>>>>>>>
>>>>>>>
>>>>>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>>>>>>>>>
>>>>>>>
>>>>>
>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>>>>>>>>>
>>>>>>>
>>>>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>>>>>>>>>
>>>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>>>>>>>>>
>>>>>>>>> Strangely this is happening when we try to hit a non-leader node,
>>>>>>> hitting
>>>>>>>>> leader node its working fine without any issue and getting the data
>>>>>>> indexed.
>>>>>>>>>
>>>>>>>>> Not able to track down where the exact issue is happening.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks & Regards,
>>>>>>>>> Yaswanth Kumar Konathala.
>>>>>>>>> [hidden email]
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks & Regards,
>>>>>>>> Yaswanth Kumar Konathala.
>>>>>>>> [hidden email]
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Yaswanth Kumar Konathala.
>>>>>> [hidden email]
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Yaswanth Kumar Konathala.
>>>> [hidden email]
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Yaswanth Kumar Konathala.
>>> [hidden email]
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: solr 8.4.1 with ssl tls1.2 creating an issue with non-leader node

Jörn Franke
In reply to this post by yaswanthcse
You need to separate keystore and truststore.

I would leave the stores in their original format and provide the type in solr.in.sh

There is no need to convert them to JKS, PKCS12 is perfectly supported

> Am 04.06.2020 um 06:48 schrieb yaswanth kumar <[hidden email]>:
>
> Hi Franke,
>
> I suspect its because of the certificate encryption ?? But will wait for
> you to confirm the same. We are trying to generate a certs with RSA 2048
> and finally combining them to a single JKS and that's what we are referring
> as a keystore and truststore, let me know if it doesn't work or if there is
> a standard procedure to do this certs.
>
> Thanks,
>
>> On Wed, Jun 3, 2020 at 8:25 AM yaswanth kumar <[hidden email]> wrote:
>>
>> thanks Franke,
>>
>> I now made the use of the default jetty-ssl.xml that comes with the solr
>> package, but the issue is still happening when I try to push data to a
>> non-leader node.
>>
>> Do you still think if its something to do with the configurations ??
>>
>> Thanks,
>>
>>> On Wed, Jun 3, 2020 at 12:29 AM Jörn Franke <[hidden email]> wrote:
>>>
>>> Why in the jetty-ssl.xml?
>>>
>>> Should this not be configured in the solr.in.sh?
>>>
>>>> Am 03.06.2020 um 00:38 schrieb yaswanth kumar <[hidden email]>:
>>>>
>>>> Thanks Franke, but yes for all these questions I did configured it
>>>> properly, I made sure to include
>>>>
>>>> <Set name="KeyStoreType"><Property name="solr.jetty.keystore.type"
>>>> default="JKS"/></Set>
>>>> <Set name="TrustStoreType"><Property name="solr.jetty.truststore.type"
>>>> default="JKS"/></Set>
>>>> in the jetty-ssl.xml along with the path keystore and truststore.
>>>>
>>>> Also I have made sure that trusstore exists on all nodes and also I am
>>>> using the same file for both keystore and truststore as below
>>>> <Set name="KeyStorePath"><Property name="solr.jetty.keystore"
>>>> default="./etc/solr-keystore.jks"/></Set>
>>>> <Set name="KeyStorePassword"><Property
>>>> name="solr.jetty.keystore.password" default="xxxx"/></Set>
>>>> <Set name="TrustStorePath"><Property name="solr.jetty.truststore"
>>>> default="./etc/solr-keystore.jks"/></Set>
>>>> <Set name="TrustStorePassword"><Property
>>>> name="solr.jetty.truststore.password" default="xxxx"/></Set>
>>>>
>>>> also urlScheme for ZK is set to https
>>>>
>>>>
>>>> Also the main error that I posted is the one that I am seeing as a
>>> return
>>>> response where as the below one is what I see from solr logs
>>>>
>>>> 2020-06-02 22:32:04.472 ERROR (qtp984876512-93) [c:default s:shard1
>>>> r:core_node3 x:default_shard1_replica_n1] o.a.s.s.HttpSolrCall
>>>> null:org.apache.solr.update.processor.Distr$
>>>>       at
>>>>
>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)
>>>>       at
>>>>
>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
>>>>       at
>>>>
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>>>>       at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>>>>       at
>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>>>>       at
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>>>>       at
>>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>>>>       at
>>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>>>>       at
>>>>
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>>>       at
>>>>
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>>>       at
>>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>>>       at
>>>>
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>>>       at
>>>>
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>>       at
>>>>
>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>>>
>>>>
>>>> One strange observation is that when I hit update api on the leader node
>>>> its working without any error, and now immediately if I hit non-leader
>>> its
>>>> working fine (only once or twice), but if I keep on trying to hit this
>>> node
>>>> again and again its then throwing the above error and once the error
>>>> started happening , its consistent again.
>>>>
>>>> Please let me know if you need more information or if I am missing
>>>> something else
>>>>
>>>> Thanks,
>>>>
>>>>> On Tue, Jun 2, 2020 at 4:59 PM Jörn Franke <[hidden email]>
>>> wrote:
>>>>>
>>>>> Have you looked in the logfiles?
>>>>>
>>>>> Keystore Type correctly defined  on all nodes?
>>>>>
>>>>> Have you configured the truststore on all nodes correctly?
>>>>>
>>>>> Have you set clusterprop urlScheme to htttps in ZK?
>>>>>
>>>>>
>>>>>
>>> https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html#configure-zookeeper
>>>>>
>>>>>
>>>>>
>>>>>>> Am 02.06.2020 um 18:57 schrieb yaswanth kumar <[hidden email]
>>>> :
>>>>>>
>>>>>> team, can someone help me on the above topic?
>>>>>>
>>>>>>> On Mon, Jun 1, 2020 at 10:00 PM yaswanth kumar <
>>> [hidden email]>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Trying to setup solr 8.4.1 + open jdk 11 on centos , enabled the ssl
>>>>>>> configurations with all the certs in place, but the issue what I am
>>>>> seeing
>>>>>>> is when trying to hit /update api on non-leader solr node , its
>>>>> throwing an
>>>>>>> error
>>>>>>>
>>>>>>> configured 2 solr nodes with 1 zookeeper.
>>>>>>>
>>>>>>> metadata":[
>>>>>>>
>>>>>>>
>>>>>
>>> "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException",
>>>>>>>
>>>>>>>
>>>>>
>>> "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"],
>>>>>>> "msg":"Async exception during distributed update:
>>>>>>> javax.crypto.BadPaddingException: RSA private key operation failed",
>>>>>>>
>>>>>
>>> "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
>>>>>>> Async exception during distributed update:
>>>>>>> javax.crypto.BadPaddingException: RSA private key operation
>>> failed\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1189)\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1096)\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)\n\tat
>>>>>>>
>>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:80)\n\tat
>>>>>>>
>>> org.apache.solr.update.processor.UpdateRequestProcessor.finish........
>>>>>>>
>>>>>>> Strangely this is happening when we try to hit a non-leader node,
>>>>> hitting
>>>>>>> leader node its working fine without any issue and getting the data
>>>>> indexed.
>>>>>>>
>>>>>>> Not able to track down where the exact issue is happening.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> --
>>>>>>> Thanks & Regards,
>>>>>>> Yaswanth Kumar Konathala.
>>>>>>> [hidden email]
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Yaswanth Kumar Konathala.
>>>>>> [hidden email]
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Yaswanth Kumar Konathala.
>>>> [hidden email]
>>>
>>
>>
>> --
>> Thanks & Regards,
>> Yaswanth Kumar Konathala.
>> [hidden email]
>>
>
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> [hidden email]