Solr 6.6.2 Master/Slave SSL Replication Error

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr 6.6.2 Master/Slave SSL Replication Error

kway
I need to use SSL in my Master/Slave Solr 6.6.2 environment. I had created a
localhost SSL Cert on the Master (works on the Master because it’s local),
but this won’t work for the Slave which has replication based on the IP of
the Master server. I then changed it to a self-signed cert that uses the
machine name which seems to be a better fit for accessing the site.

However, I can't get replication to work when using SSL/HTTPS. It throws IO
Communication errors as it can’t resolve the https connection to a localhost
certificate on the Master. The error is as follows:

Master at: https://mastercomputername:8983/solr/core_index is not available.
Index fetch failed by exception: org.apache.solr.common.SolrException:
IOException occured when talking to server at:
https://mastercomputername:8983/solr/core_index

Is there a setting in Solr I need to enable to allow replication to occur? I
already installed the machine named cert from the Master server on the Slave
server.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr 6.6.2 Master/Slave SSL Replication Error

Shawn Heisey-2
On 4/21/2018 10:24 AM, kway wrote:
> However, I can't get replication to work when using SSL/HTTPS. It throws IO
> Communication errors as it can’t resolve the https connection to a localhost
> certificate on the Master. The error is as follows:
>
> Master at: https://mastercomputername:8983/solr/core_index is not available.
> Index fetch failed by exception: org.apache.solr.common.SolrException:
> IOException occured when talking to server at:
> https://mastercomputername:8983/solr/core_index

Can you share the full error?  It is likely to be a couple dozen lines,
possibly longer.  Without that, we won't know exactly what is failing,
so it's difficult to offer any ideas.  For correct interpretation, we
also need to know the exact Solr version that produced the error.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Solr 6.6.2 Master/Slave SSL Replication Error

kway
Thanks Shawn,

Here is what I get from the logs:

2018-04-20 18:03:57.805 WARN  (indexFetcher-19-thread-1) [  
x:XP1Prod_core_index_rebuild] o.a.s.h.IndexFetcher Master at:
https://mastercomputername:8983/solr/XP1Prod_core_index_rebuild is not
available. Index fetch failed by exception:
org.apache.solr.common.SolrException: IOException occured when talking to
server at: https://mastercomputername:8983/solr/XP1Prod_core_index_rebuild

Regards,

Kelly



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr 6.6.2 Master/Slave SSL Replication Error

kway
This post was updated on .
In reply to this post by Shawn Heisey-2
Thanks Shawn,

Here is what I get from the logs. I am using Solr 6.6.2:

2018-04-20 18:03:57.805 WARN  (indexFetcher-19-thread-1) [  
x:XP1Prod_core_index_rebuild] o.a.s.h.IndexFetcher Master at:
https://mastercomputername:8983/solr/XP1Prod_core_index_rebuild is not
available. Index fetch failed by exception:
org.apache.solr.common.SolrException: IOException occured when talking to
server at: https://mastercomputername:8983/solr/XP1Prod_core_index_rebuild

        at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:626)
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.handler.IndexFetcher.getDetails(IndexFetcher.java:1747)
        at
org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:923)
        at
org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:315)
        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
        at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
        at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
        at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
        at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
        at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
        at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find
valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
        at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
        at
sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
        at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
        at
org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
        at
org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
        at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
        at
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
        at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
        at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
        at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)

Regards,

Kelly



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr 6.6.2 Master/Slave SSL Replication Error

kway
This post was updated on .
... looking at this line, I am wondering if this is an issue because I am
using a Self-Signed Certificate. I am also running Solr on Windows (governance as all other servers are Windows in the environment)

Caused by: javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find
valid certification path to requested target

How would I get this to work with a self-signed cert?

Regards,

Kelly



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr 6.6.2 Master/Slave SSL Replication Error

Chris Hostetter-3

You need to configure Solr to use a "truststore" that contains the
certificate you want it to trust.  With a solr cloud setup, that usually
involves configuring the "keystore" and the "truststore" to both contain
the same keys...

https://lucene.apache.org/solr/guide/6_6/enabling-ssl.html


: Date: Sat, 21 Apr 2018 14:40:08 -0700 (MST)
: From: kway <[hidden email]>
: Reply-To: [hidden email]
: To: [hidden email]
: Subject: Re: Solr 6.6.2 Master/Slave SSL Replication Error
:
: ... looking at this line, I am wondering if this is an issue because I am
: using a Self-Signed Certificate:
:
: Caused by: javax.net.ssl.SSLHandshakeException:
: sun.security.validator.ValidatorException: PKIX path building failed:
: sun.security.provider.certpath.SunCertPathBuilderException: unable to find
: valid certification path to requested target
:
: How would I get this to work with a self-signed cert?
:
: Regards,
:
: Kelly
:
:
:
: --
: Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
:

-Hoss
http://www.lucidworks.com/
Reply | Threaded
Open this post in threaded view
|

Re: Solr 6.6.2 Master/Slave SSL Replication Error

kway
Thanks Chris,

I already have a key store/trust store and my settings are as follows:

set SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.jks
set SOLR_SSL_KEY_STORE_PASSWORD=secret
set SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.jks
set SOLR_SSL_TRUST_STORE_PASSWORD=secret
REM Require clients to authenticate
set SOLR_SSL_NEED_CLIENT_AUTH=false
REM Enable clients to authenticate (but not require)
set SOLR_SSL_WANT_CLIENT_AUTH=false

I am using a Master/Slave config, not a SolrCloud.

How would I add the self-signed Cert I created on my Master node to the Slave node? Is that what you are recommending?

Regards,

Kelly
_____________________________
From: Chris Hostetter <[hidden email]>
Sent: Sunday, April 22, 2018 5:43 PM
Subject: Re: Solr 6.6.2 Master/Slave SSL Replication Error
To: <[hidden email]>



You need to configure Solr to use a "truststore" that contains the
certificate you want it to trust. With a solr cloud setup, that usually
involves configuring the "keystore" and the "truststore" to both contain
the same keys...

https://lucene.apache.org/solr/guide/6_6/enabling-ssl.html


: Date: Sat, 21 Apr 2018 14:40:08 -0700 (MST)
: From: kway <[hidden email]>
: Reply-To: [hidden email]
: To: [hidden email]
: Subject: Re: Solr 6.6.2 Master/Slave SSL Replication Error
:
: ... looking at this line, I am wondering if this is an issue because I am
: using a Self-Signed Certificate:
:
: Caused by: javax.net.ssl.SSLHandshakeException:
: sun.security.validator.ValidatorException: PKIX path building failed:
: sun.security.provider.certpath.SunCertPathBuilderException: unable to find
: valid certification path to requested target
:
: How would I get this to work with a self-signed cert?
:
: Regards,
:
: Kelly
:
:
:
: --
: Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
:

-Hoss
http://www.lucidworks.com/



Reply | Threaded
Open this post in threaded view
|

Re: Solr 6.6.2 Master/Slave SSL Replication Error

Shawn Heisey-2
On 4/22/2018 4:40 PM, Kelly Rusk wrote:

> I already have a key store/trust store and my settings are as follows:
>
> set SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.jks
> set SOLR_SSL_KEY_STORE_PASSWORD=secret
> set SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.jks
> set SOLR_SSL_TRUST_STORE_PASSWORD=secret
> REM Require clients to authenticate
> set SOLR_SSL_NEED_CLIENT_AUTH=false
> REM Enable clients to authenticate (but not require)
> set SOLR_SSL_WANT_CLIENT_AUTH=false
>
> I am using a Master/Slave config, not a SolrCloud.
>
> How would I add the self-signed Cert I created on my Master node to the Slave node? Is that what you are recommending?

You will need the same SSL config, including both the key store and the
the trust store, on all Solr servers.  Put the keystore file and the
config above on all of them.  This should allow everything to work.

I'm assuming that the keystore file contains just the self-signed cert
and its private key?

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Solr 6.6.2 Master/Slave SSL Replication Error

kway
Hi Shawn,

Thanks for the assistance. The Master Server has a self-signed Cert with its machine name, and the Slave has a self-signed Cert with its machine name.

They have identical configurations, and I created a keystore per server. Should I import the self-signed Cert into each other’s keystore? Or are you stating that I need to copy the keystore over to the Slave instead of having the one I created?

Regards,

Kelly
_____________________________
From: Shawn Heisey <[hidden email]>
Sent: Sunday, April 22, 2018 7:56 PM
Subject: Re: Solr 6.6.2 Master/Slave SSL Replication Error
To: <[hidden email]>


On 4/22/2018 4:40 PM, Kelly Rusk wrote:

> I already have a key store/trust store and my settings are as follows:
>
> set SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.jks
> set SOLR_SSL_KEY_STORE_PASSWORD=secret
> set SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.jks
> set SOLR_SSL_TRUST_STORE_PASSWORD=secret
> REM Require clients to authenticate
> set SOLR_SSL_NEED_CLIENT_AUTH=false
> REM Enable clients to authenticate (but not require)
> set SOLR_SSL_WANT_CLIENT_AUTH=false
>
> I am using a Master/Slave config, not a SolrCloud.
>
> How would I add the self-signed Cert I created on my Master node to the Slave node? Is that what you are recommending?

You will need the same SSL config, including both the key store and the
the trust store, on all Solr servers.  Put the keystore file and the
config above on all of them.  This should allow everything to work.

I'm assuming that the keystore file contains just the self-signed cert
and its private key?

Thanks,
Shawn




Reply | Threaded
Open this post in threaded view
|

Re: Solr 6.6.2 Master/Slave SSL Replication Error

Shawn Heisey-2
On 4/22/2018 6:27 PM, Kelly Rusk wrote:
> Thanks for the assistance. The Master Server has a self-signed Cert with its machine name, and the Slave has a self-signed Cert with its machine name.
>
> They have identical configurations, and I created a keystore per server. Should I import the self-signed Cert into each other’s keystore? Or are you stating that I need to copy the keystore over to the Slave instead of having the one I created?

For the way you have it now, the trust store will need all of the
certificates of all of the servers.  It's the remote certificate that
must be validated, so having just the local certificate in the trust
store doesn't do you any good.

A better option would be to have one certificate that covers all of the
names you're using, and have all the servers set up identically.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Solr 6.6.2 Master/Slave SSL Replication Error

kway
Makes perfect sense! Should I use the key tool to import the Certs? If so, do you have an example you prefer or should I just pull from the docs?

Regards,

Kelly
_____________________________
From: Shawn Heisey <[hidden email]>
Sent: Sunday, April 22, 2018 8:40 PM
Subject: Re: Solr 6.6.2 Master/Slave SSL Replication Error
To: <[hidden email]>


On 4/22/2018 6:27 PM, Kelly Rusk wrote:
> Thanks for the assistance. The Master Server has a self-signed Cert with its machine name, and the Slave has a self-signed Cert with its machine name.
>
> They have identical configurations, and I created a keystore per server. Should I import the self-signed Cert into each other’s keystore? Or are you stating that I need to copy the keystore over to the Slave instead of having the one I created?

For the way you have it now, the trust store will need all of the
certificates of all of the servers.  It's the remote certificate that
must be validated, so having just the local certificate in the trust
store doesn't do you any good.

A better option would be to have one certificate that covers all of the
names you're using, and have all the servers set up identically.

Thanks,
Shawn




Reply | Threaded
Open this post in threaded view
|

RE: Solr 6.6.2 Master/Slave SSL Replication Error

kway
Hello all,

I added the incorrect certificate and can clearly see the certificate in my keystore when I run the following command:

keytool -list -v -keystore D:\Solr\solr-6.6.2\server\etc\solr-ssl.keystore.pfx -storepass mypass

However, I can't remove it as this command states "keytool error: java.lang.Exception: Alias <MyCert> does not exist":

keytool -delete -alias "MyCert" -keystore D:\Solr\solr-6.6.2\server\etc\solr-ssl.keystore.pfx -storepass mypass

How can it show it in the store, but not delete it? If I try to import it again, it says it can't import because it already exists in the store!

Thanks,

Kelly

-----Original Message-----
From: Kelly Rusk [mailto:[hidden email]]
Sent: Sunday, April 22, 2018 8:51 PM
To: [hidden email]; [hidden email]
Subject: Re: Solr 6.6.2 Master/Slave SSL Replication Error

Makes perfect sense! Should I use the key tool to import the Certs? If so, do you have an example you prefer or should I just pull from the docs?

Regards,

Kelly
_____________________________
From: Shawn Heisey <[hidden email]>
Sent: Sunday, April 22, 2018 8:40 PM
Subject: Re: Solr 6.6.2 Master/Slave SSL Replication Error
To: <[hidden email]>


On 4/22/2018 6:27 PM, Kelly Rusk wrote:
> Thanks for the assistance. The Master Server has a self-signed Cert with its machine name, and the Slave has a self-signed Cert with its machine name.
>
> They have identical configurations, and I created a keystore per server. Should I import the self-signed Cert into each other's keystore? Or are you stating that I need to copy the keystore over to the Slave instead of having the one I created?

For the way you have it now, the trust store will need all of the certificates of all of the servers.  It's the remote certificate that must be validated, so having just the local certificate in the trust store doesn't do you any good.

A better option would be to have one certificate that covers all of the names you're using, and have all the servers set up identically.

Thanks,
Shawn