Problem with glassfish and zookeeper 3.4.5

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with glassfish and zookeeper 3.4.5

kaustubh147
Hi,

Glassfish 3.1.2.2
Solr 4.5
Zookeeper 3.4.5

We have set up a SolrCloud with 4 Solr nodes and 3 zookeeper instances. It seems to be working fine from Solr admin page.

but when I am trying to connect it to web application using Solrj 4.5.
I am creating my Solr Cloud Server as suggested on the wiki page

LBHttpSolrServer lbHttpSolrServer = new LBHttpSolrServer(
                                        SOLR_INSTANCE01,
                                        SOLR_INSTANCE02,
                                        SOLR_INSTANCE03,
                                        SOLR_INSTANCE04);
solrServer = new CloudSolrServer(zk1:p1, zk2:p1, zk3:p1, lbHttpSolrServer);
solrServer.setDefaultCollection(collection);


It seems to be working fine for a while even though I am getting a WARNING as below
-----
SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'XYZ_path/SolrCloud_04/config/login.conf'. Will continue connection to Zookeeper server without SASL authentication,​ if Zookeeper server allows it.
--------------

The application is deployed on a single node cluster on glassfish.

as soon as my application has made some queries to the Solr server it will start throwing error in the solrServer.runQuery() method. The reason of the error is not clear..

Application logs shows following error trace many times...

-----------------------------------------
[#|2013-10-24T14:07:53.750-0700|WARNING|glassfish3.1.2|org.apache.zookeeper.ClientCnxn|_ThreadID=1434;_ThreadName=Thread-2;|SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'XYZ_PATH/config/login.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.|#]

[#|2013-10-24T14:07:53.750-0700|INFO|glassfish3.1.2|org.apache.zookeeper.ClientCnxn|_ThreadID=1434;_ThreadName=Thread-2;|Opening socket connection to server server_name/IP3:2181|#]

[#|2013-10-24T14:07:53.750-0700|INFO|glassfish3.1.2|org.apache.solr.common.cloud.ConnectionManager|_ThreadID=1435;_ThreadName=Thread-2;|Watcher org.apache.solr.common.cloud.ConnectionManager@187eaada name:ZooKeeperConnection Watcher:IP1:2181,IP2:2181,IP3:2181 got event WatchedEvent state:AuthFailed type:None path:null path:null type:None|#]

[#|2013-10-24T14:07:53.750-0700|INFO|glassfish3.1.2|org.apache.solr.common.cloud.ConnectionManager|_ThreadID=1435;_ThreadName=Thread-2;|Client->ZooKeeper status change trigger but we are already closed|#]

[#|2013-10-24T14:07:53.751-0700|INFO|glassfish3.1.2|org.apache.zookeeper.ClientCnxn|_ThreadID=1434;_ThreadName=Thread-2;|Socket connection established to server_name/IP3:2181, initiating session|#]

[#|2013-10-24T14:07:53.751-0700|INFO|glassfish3.1.2|org.apache.solr.common.cloud.ConnectionManager|_ThreadID=1420;_ThreadName=Thread-2;|Watcher org.apache.solr.common.cloud.ConnectionManager@4ba50169 name:ZooKeeperConnection Watcher:IP1:2181,IP2:2181,IP3:2181 got event WatchedEvent state:Disconnected type:None path:null path:null type:None|#]

[#|2013-10-24T14:07:53.751-0700|WARNING|glassfish3.1.2|org.apache.zookeeper.ClientCnxn|_ThreadID=1434;_ThreadName=Thread-2;|Session 0x0 for serverserver_name/IP3:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
        at sun.nio.ch.IOUtil.read(IOUtil.java:166)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
|#]


------------------------------------------

before this happen the zookeeper logs on all the 3 instances starts showing following warning

2013-10-24 14:05:55,200 [myid:3] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /IP_APPLICATION_SEVER - max is 200

it means that my application is making too many connections with the zookeeper and it is exceeding the limit which is set to 200.


Is there a way I can control the number of connections my application is making with the zookeeper.
The only component which is connecting to zookeeper in my application is CloudSolrServer object.

As per my investigation SASL warning is related to a existing bug in Zookeeper 3.4.5 and is being solved for Zookeeper 3.5 and it should not cause this issue....

I need help and guidance..

Thanks,
Kaustubh









Reply | Threaded
Open this post in threaded view
|

Re: Problem with glassfish and zookeeper 3.4.5

Shawn Heisey-4
On 10/24/2013 4:30 PM, kaustubh147 wrote:

> Glassfish 3.1.2.2
> Solr 4.5
> Zookeeper 3.4.5
>
> We have set up a SolrCloud with 4 Solr nodes and 3 zookeeper instances. It
> seems to be working fine from Solr admin page.
>
> but when I am trying to connect it to web application using Solrj 4.5.
> I am creating my Solr Cloud Server as suggested on the wiki page
>
> LBHttpSolrServer lbHttpSolrServer = new LBHttpSolrServer(
> SOLR_INSTANCE01,
> SOLR_INSTANCE02,
> SOLR_INSTANCE03,
> SOLR_INSTANCE04);
> solrServer = new CloudSolrServer(zk1:p1, zk2:p1, zk3:p1, lbHttpSolrServer);
> solrServer.setDefaultCollection(collection);

If this is what you are seeing as instructions for connecting from SolrJ
to SolrCloud, then something's really screwy.  Can you give me the URL
that shows this, so I can see about getting it changed?  The following
code example is how you should be doing that.  For this example,
zookeeper is using the default port of 2181 and the zookeeper hosts are
zoo1, zoo2, and zoo3.

String zkHost = "zoo1:2181,zoo2:2181,zoo3:2181";
// If you are using a chroot, use something like this instead:
// String zkHost = "zoo1:2181,zoo2:2181,zoo3:2181/chroot";
CloudSolrServer server = new CloudSolrServer(zkHost);
server.setDefaultCollection("collection1");

> It seems to be working fine for a while even though I am getting a WARNING
> as below
> -----
> SASL configuration failed: javax.security.auth.login.LoginException: No JAAS
> configuration section named 'Client' was found in specified JAAS
> configuration file: 'XYZ_path/SolrCloud_04/config/login.conf'. Will continue
> connection to Zookeeper server without SASL authentication,​ if Zookeeper
> server allows it.
> --------------

Later you said this sounds like a bug you saw in ZK 3.4.5. It may be
that glassfish turns on some system-wide setting related to
authentication that zookeeper picks up on.  I would tend to agree that
this probably is not related to the other problems mentioned below.

> before this happen the zookeeper logs on all the 3 instances starts showing
> following warning
>
> 2013-10-24 14:05:55,200 [myid:3] - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too
> many connections from /IP_APPLICATION_SEVER - max is 200
>
> it means that my application is making too many connections with the
> zookeeper and it is exceeding the limit which is set to 200.

Are you creating one CloudSolrServer object (static would be OK) and
using it for all interaction with SolrCloud, or are you creating many
CloudSolrServer objects over the life of your application?  Is there
more than one thread or instance of your application running, and each
one has its own CloudSolrServer object?  It is strongly recommended that
you only create one object for your entire application and use it for
all queries, updates, etc.  You can set the "collection" parameter on
each query or request object that you use, if you need to use more than one.

If you *DO* create many CloudSolrServer objects over the life of your
application and cannot immediately change your code so that it uses one
object, be sure to shutdown() each one when it is no longer required.  
Depending on the exact nature of your application, you may also need to
increase the maximum number of connections allowed in your zookeeper config.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Problem with glassfish and zookeeper 3.4.5

kaustubh147
Thanks Shawn,

I found a bug in my code, it was creating too many CloudSolrServer objects.

Thanks
Kaustubh