We are trying to use Hadoop authentication with Kerberos in Solr 8.4.1 and encountered a problem. We’re using a Hadoop 3.1.1 based fork. We are using JDK8 so we fall back to HTTP/1.1 but also tested with JDK11 (HTTP/2) and we got the same error.
We have debugged the problem and found that curl can send the information to the node, and the internode TOLEADER request fails, because we don’t answer to the 401 challenge that is part of the SPNEGO mechanism:
HTTP/1.1 401 Unauthorized access
Set-Cookie: hadoop.auth=; HttpOnly
Checking the code shows that ConcurrentUpdateHttp2SolrClient calls Http2SolrClient.initOutStream which creates an OutputStreamContentProvider where the value of the isReproducible flag is false and jetty’s AuthenticationProtocolHandler will not continue the authentication in this case
We have also found a workaround. If we send a simple successfully authenticated message before Http2SolrClient.initOutStream in ConcurrentUpdateHttp2SolrClient the authentication works correctly. Not only for the simple message but also for the upcoming requests. So right now we send an OPTIONS request here and just ignore the answer.
Sending the OPTIONS request happens before setting up the update stream, so if we send multiple documents in a single update, only one OPTIONS will be sent to each leader.
Although this workaround works for us, we are not sure that this is the best place to ensure pre-authentication between the nodes. Does anybody have a better place to handle it?
Is there anybody here who is successfully using Solr8 with Hadoop Authentication and Kerberos?