Docker and Solr Indexing

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Docker and Solr Indexing

solrnoobie
This post was updated on .
So we have a dockerized aws environment with the solr (6.6.3) docker container having
only 4 gigs for max ram.

Our problem is whenever we index, the container containing the leader shard
will restart after around 2 or less minutes of index time (batch is 50 docs
per batch with 3 threads in our app thread pool). Because of the container
restart, indexing will fail because solrJ will throw an invalid content type
exception because of the quick container restart.

What can possible cause the issues above?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Docker and Solr Indexing

Walter Underwood
4 Gb is very small for Solr.

Solr is not designed for Dockerized, fail-often use.

We use a LOT of Docker ECS, but all of our Solr servers are on EC2
instances. That’s about sixty instances in several clusters.

We run an 8 Gb heap for all our Solr instances. Instances in our biggest
cluster (in terms of index size and doc count) are c4.8xlarge, with 36 vCPU
and 60 Gb of RAM.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Sep 10, 2018, at 11:05 PM, solrnoobie <[hidden email]> wrote:
>
> So we have a dockerized aws environment with the solr docker container having
> only 4 gigs for max ram.
>
> Our problem is whenever we index, the container containing the leader shard
> will restart after around 2 or less minutes of index time (batch is 50 docs
> per batch with 3 threads in our app thread pool). Because of the container
> restart, indexing will fail because solrJ will throw an invalid content type
> exception because of the quick container restart.
>
> What can possible casue the issues above?
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Reply | Threaded
Open this post in threaded view
|

Re: Docker and Solr Indexing

Jan Høydahl / Cominvent
In reply to this post by solrnoobie
You have not shed any light on what the reason for the container restart was, and there is too little information about your setup and Solr usage to guess what goes on. Whether 4Gb is sufficient or not depends on how much data and queries you plan for each shard to handle, how much heap you give to Solr out of those 4G and many other factors.

Jan

> 11. sep. 2018 kl. 08:05 skrev solrnoobie <[hidden email]>:
>
> So we have a dockerized aws environment with the solr docker container having
> only 4 gigs for max ram.
>
> Our problem is whenever we index, the container containing the leader shard
> will restart after around 2 or less minutes of index time (batch is 50 docs
> per batch with 3 threads in our app thread pool). Because of the container
> restart, indexing will fail because solrJ will throw an invalid content type
> exception because of the quick container restart.
>
> What can possible casue the issues above?
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Docker and Solr Indexing

solrnoobie
Thank you all for the kind and timely reply.

So what we did is we upgraded the instances to 16 gigs and we rarely
encounter this now.

So what we did was to increase the batch size to 500 instead of 50 and it
worked for our test data. But when we tried 1000 batch size, the invalid
content type error returned. Can you guys shed some light on why this is
happening? I don't think that a thousand per batch is too much (although we
have documents with many fields and child documents) so I am not really sure
what's causing this aside from a docker containter restart.

Thanks!



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Docker and Solr Indexing

Shawn Heisey-2
On 9/11/2018 9:20 PM, solrnoobie wrote:
> So what we did is we upgraded the instances to 16 gigs and we rarely
> encounter this now.
>
> So what we did was to increase the batch size to 500 instead of 50 and it
> worked for our test data. But when we tried 1000 batch size, the invalid
> content type error returned. Can you guys shed some light on why this is
> happening? I don't think that a thousand per batch is too much (although we
> have documents with many fields and child documents) so I am not really sure
> what's causing this aside from a docker containter restart.

At no point in this thread have you shared the actual error messages. 
Without those and the exact version of Solr, it's difficult to help
you.  Saying that you got a "content type error" doesn't mean anything. 
We need to see the actual error, complete with all stacktrace data.  The
best information will be found in the logfile -- solr.log.

Solr (as packaged by this project) is not designed to restart itself
automatically.  If the JVM encounters an OutOfMemoryError exception and
the platform is NOT Windows, then Solr is designed to kill itself ...
but it will NOT automatically restart without outside intervention or a
change to its startup scripts.  This is done because program operation
is completely unpredictable when OOME hits, so the best course of action
is to self-terminate and let the admin fix the problem that cause the OOME.

The publicly available Solr docker container is NOT an official product
of this project.  It is third-party, so problems specific to the docker
container may need to be handled by the project that created it.  If the
docker container is set up to automatically restart Solr when it dies, I
would consider that to be a bug. About the only reason that Solr will
ever die is the OOME self-termination that I already described ... and
since the OOME is likely to occur again after restart, it's usually
better for the software to stay offline until the admin fixes the problem.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Docker and Solr Indexing

Dominique Bejean
Hi,

Are you aware about issues in Java applications in Docker if java version
is not 10 ?
https://blog.docker.com/2018/04/improved-docker-container-integration-with-java-10/

Regards.

Dominique


Le mer. 12 sept. 2018 à 05:42, Shawn Heisey <[hidden email]> a écrit :

> On 9/11/2018 9:20 PM, solrnoobie wrote:
> > So what we did is we upgraded the instances to 16 gigs and we rarely
> > encounter this now.
> >
> > So what we did was to increase the batch size to 500 instead of 50 and it
> > worked for our test data. But when we tried 1000 batch size, the invalid
> > content type error returned. Can you guys shed some light on why this is
> > happening? I don't think that a thousand per batch is too much (although
> we
> > have documents with many fields and child documents) so I am not really
> sure
> > what's causing this aside from a docker containter restart.
>
> At no point in this thread have you shared the actual error messages.
> Without those and the exact version of Solr, it's difficult to help
> you.  Saying that you got a "content type error" doesn't mean anything.
> We need to see the actual error, complete with all stacktrace data.  The
> best information will be found in the logfile -- solr.log.
>
> Solr (as packaged by this project) is not designed to restart itself
> automatically.  If the JVM encounters an OutOfMemoryError exception and
> the platform is NOT Windows, then Solr is designed to kill itself ...
> but it will NOT automatically restart without outside intervention or a
> change to its startup scripts.  This is done because program operation
> is completely unpredictable when OOME hits, so the best course of action
> is to self-terminate and let the admin fix the problem that cause the OOME.
>
> The publicly available Solr docker container is NOT an official product
> of this project.  It is third-party, so problems specific to the docker
> container may need to be handled by the project that created it.  If the
> docker container is set up to automatically restart Solr when it dies, I
> would consider that to be a bug. About the only reason that Solr will
> ever die is the OOME self-termination that I already described ... and
> since the OOME is likely to occur again after restart, it's usually
> better for the software to stay offline until the admin fixes the problem.
>
> Thanks,
> Shawn
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Docker and Solr Indexing

Shawn Heisey-2
On 9/12/2018 7:43 AM, Dominique Bejean wrote:
> Are you aware about issues in Java applications in Docker if java version
> is not 10 ?
> https://blog.docker.com/2018/04/improved-docker-container-integration-with-java-10/

Solr explicitly sets heap size when it starts, so Java is *NOT*
determining the heap size automatically.

As for CPUs, if the container isn't sized appropriately, then I guess
you might have an issue there.

The latest version of Solr should start and run just fine in Java 10. 
Some earlier versions of 7.x have problems starting in Java 10, but
should *run* fine after the script is fixed to detect the version
correctly.  Solr 6.x is not qualified for Java 9, and therefore not
qualified for Java 10.

Thanks,
Shawn