Quantcast

SOLR 4 Alpha Out Of Mem Err

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

SOLR 4 Alpha Out Of Mem Err

nick.koton
I have been experiencing out of memory errors when indexing via solrj into a
4 alpha cluster.  It seems when I delegate commits to the server (either
auto commit or commit within) there is nothing to throttle the solrj clients
and the server struggles to fan out the work.  However, when I handle
commits entirely within the client, the indexing rate is very restricted.

Any suggestions would be appreciated

Nick Cotton
[hidden email]





Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

Mark Miller-3
Can you give more info? How much RAM are you giving Solr with Xmx?

Can you be more specific about the behavior you are seeing with auto commit vs client commit?

How often are you trying to commit? With the client? With auto commit?

Are you doing soft commits? Std commits? A mix?

What's the stack trace for the OOM?

What OS are you using?

Anything else you can add?

--
Mark Miller



On Saturday, July 14, 2012 at 4:21 PM, Nick Koton wrote:

> I have been experiencing out of memory errors when indexing via solrj into a
> 4 alpha cluster. It seems when I delegate commits to the server (either
> auto commit or commit within) there is nothing to throttle the solrj clients
> and the server struggles to fan out the work. However, when I handle
> commits entirely within the client, the indexing rate is very restricted.
>
> Any suggestions would be appreciated
>
> Nick Cotton
> [hidden email]
>
>


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: SOLR 4 Alpha Out Of Mem Err

nick.koton
> Can you give more info? How much RAM are you giving Solr with Xmx?
RHEL 5.4
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
java  -Xmx16384m -Xms8192m
SOLR shards=6, each with a replica (i.e. total 12 JVMs)
Solrj multi-threaded client sends several 1,000 docs/sec with auto commit or commit within
Same client with explicit client commit tops out around 1,000 / sec

> How often are you trying to commit? With the client? With auto commit?
With auto commit or commit within, I've tried things in the range of 10,000-60,000 ms.  I have tried both with and without soft commit.  When I tried soft, it was in the range of 500-5,000 ms.  With explicit client commit I have tried in the range of 10,000 to 200,000 documents.

>Can you be more specific about the behavior you are seeing with auto commit vs client commit?
With auto commit, the client threads run without pausing and quickly ramp up to several 1,000s docs per second.  After a variable number of minutes, but seldom as long as an hour, the server(s) to which the client(s) are attached get exceptions.  I have included one stack trace below.  Clients continue to run without error.

With explicit client commit, the client will pause for a few seconds at the commit with a single thread and hit overall index rates around 500/sec.  With a second thread, the commit is around 5 seconds and overall rate is around 1,000/sec.  Adding a third thread causes the commit time to increase to a minute or more.  Overall rate increases only slightly with the third and subsequent threads.

> Anything else you can add?
I have seen this with a configuration as small as a single shard and a replica, but I've been always working with SolrCloud.  When there is a single shard without a replica, I have not seen the problem.


INFO: [shipment] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2
Jul 14, 2012 9:20:57 PM org.apache.solr.core.SolrCore execute
INFO: [shipment] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3
Jul 14, 2012 9:20:57 PM org.apache.solr.common.SolrException log
SEVERE: null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
        at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:456)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:284)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
        at org.eclipse.jetty.server.Server.handle(Server.java:351)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
        at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:954)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
        at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
        at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:152)
        at org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:340)
        at org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:296)
        at org.apache.solr.update.SolrCmdDistributor.flushAdds(SolrCmdDistributor.java:228)
        at org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:101)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:329)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:952)
        at org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:176)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1561)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:442)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:263)
        ... 25 more

Jul 14, 2012 9:20:57 PM org.apache.solr.common.SolrException log
-----Original Message-----
From: Mark Miller [mailto:[hidden email]]
Sent: Saturday, July 14, 2012 2:44 PM
To: [hidden email]
Subject: Re: SOLR 4 Alpha Out Of Mem Err

Can you give more info? How much RAM are you giving Solr with Xmx?

Can you be more specific about the behavior you are seeing with auto commit vs client commit?

How often are you trying to commit? With the client? With auto commit?

Are you doing soft commits? Std commits? A mix?

What's the stack trace for the OOM?

What OS are you using?

Anything else you can add?

--
Mark Miller



On Saturday, July 14, 2012 at 4:21 PM, Nick Koton wrote:

> I have been experiencing out of memory errors when indexing via solrj into a
> 4 alpha cluster. It seems when I delegate commits to the server (either
> auto commit or commit within) there is nothing to throttle the solrj clients
> and the server struggles to fan out the work. However, when I handle
> commits entirely within the client, the indexing rate is very restricted.
>
> Any suggestions would be appreciated
>
> Nick Cotton
> [hidden email]
>
>



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

Yonik Seeley-2-2
Do you have the following hard autoCommit in your config (as the stock
server does)?

     <autoCommit>
       <maxTime>15000</maxTime>
       <openSearcher>false</openSearcher>
     </autoCommit>

This is now fairly important since Solr now tracks information on
every uncommitted document added.
At some point we should probably hardcode some mechanism based on
number of documents or time.

-Yonik
http://lucidimagination.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: SOLR 4 Alpha Out Of Mem Err

nick.koton
> Do you have the following hard autoCommit in your config (as the stock
server does)?
>     <autoCommit>
>       <maxTime>15000</maxTime>
>       <openSearcher>false</openSearcher>
>     </autoCommit>

I have tried with and without that setting.  When I described running with
auto commit, that setting is what I mean.  I have varied the time in the
range 10,000-60,000 msec.  I have tried this setting with and without soft
commit in the server config file.

I have tried without this setting, but specifying the commit within time in
the solrj client in the add method.

In both these cases, the client seems to overrun the server and out of
memory in the server results.  One clarification I should make is that after
the server gets out of memory, the solrj client does NOT receive an error.
However, the documents indexed do not reliably appear to queries.

Approach #3 is to remove the autocommit in the server config, issue the add
method without commit within, but issue commits in the solrj client with
wait for sync and searcher set to true.  In case #3, I do not see the out of
memory in the server.  However, document index rates are restricted to about
1,000 per second.

 Nick

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
Sent: Sunday, July 15, 2012 5:15 AM
To: [hidden email]
Subject: Re: SOLR 4 Alpha Out Of Mem Err

Do you have the following hard autoCommit in your config (as the stock
server does)?

     <autoCommit>
       <maxTime>15000</maxTime>
       <openSearcher>false</openSearcher>
     </autoCommit>

This is now fairly important since Solr now tracks information on every
uncommitted document added.
At some point we should probably hardcode some mechanism based on number of
documents or time.

-Yonik
http://lucidimagination.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

Jack Krupansky-2
Maybe your rate of update is so high that the commit never gets a chance to
run. So, maybe all these uncommitted updates are buffered up and using
excess memory.

Try explicit commits from SolrJ, but less frequently. Or maybe if you just
pause your updates periodically (every 30 seconds or so) the auto-commit
would get a chance to occur. Although I have no idea how long a pause might
be needed.

-- Jack Krupansky

-----Original Message-----
From: Nick Koton
Sent: Sunday, July 15, 2012 10:52 AM
To: [hidden email] ; [hidden email]
Subject: RE: SOLR 4 Alpha Out Of Mem Err

> Do you have the following hard autoCommit in your config (as the stock
server does)?
>     <autoCommit>
>       <maxTime>15000</maxTime>
>       <openSearcher>false</openSearcher>
>     </autoCommit>

I have tried with and without that setting.  When I described running with
auto commit, that setting is what I mean.  I have varied the time in the
range 10,000-60,000 msec.  I have tried this setting with and without soft
commit in the server config file.

I have tried without this setting, but specifying the commit within time in
the solrj client in the add method.

In both these cases, the client seems to overrun the server and out of
memory in the server results.  One clarification I should make is that after
the server gets out of memory, the solrj client does NOT receive an error.
However, the documents indexed do not reliably appear to queries.

Approach #3 is to remove the autocommit in the server config, issue the add
method without commit within, but issue commits in the solrj client with
wait for sync and searcher set to true.  In case #3, I do not see the out of
memory in the server.  However, document index rates are restricted to about
1,000 per second.

Nick

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
Sent: Sunday, July 15, 2012 5:15 AM
To: [hidden email]
Subject: Re: SOLR 4 Alpha Out Of Mem Err

Do you have the following hard autoCommit in your config (as the stock
server does)?

     <autoCommit>
       <maxTime>15000</maxTime>
       <openSearcher>false</openSearcher>
     </autoCommit>

This is now fairly important since Solr now tracks information on every
uncommitted document added.
At some point we should probably hardcode some mechanism based on number of
documents or time.

-Yonik
http://lucidimagination.com 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

Yonik Seeley-2-2
In reply to this post by nick.koton
On Sun, Jul 15, 2012 at 11:52 AM, Nick Koton <[hidden email]> wrote:
>> Do you have the following hard autoCommit in your config (as the stock
> server does)?
>>     <autoCommit>
>>       <maxTime>15000</maxTime>
>>       <openSearcher>false</openSearcher>
>>     </autoCommit>
>
> I have tried with and without that setting.  When I described running with
> auto commit, that setting is what I mean.

OK cool.  You should be able to run the stock server (i.e. with this
autocommit) and blast in updates all day long - it looks like you have
more than enough memory.  If you can't, we need to fix something.  You
shouldn't need explicit commits unless you want the docs to be
searchable at that point.

> Solrj multi-threaded client sends several 1,000 docs/sec

Can you expand on that?  How many threads at once are sending docs to
solr?  Is each request a single doc or multiple?

-Yonik
http://lucidimagination.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

Yonik Seeley-2-2
In reply to this post by Jack Krupansky-2
On Sun, Jul 15, 2012 at 12:52 PM, Jack Krupansky
<[hidden email]> wrote:
> Maybe your rate of update is so high that the commit never gets a chance to
> run.

I don't believe that is possible.  If it is, it should be fixed.

-Yonik
http://lucidimagination.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

Jack Krupansky-2
Agreed. That's why I say "maybe". Clearly something sounds amiss here.

-- Jack Krupansky

-----Original Message-----
From: Yonik Seeley
Sent: Sunday, July 15, 2012 12:06 PM
To: [hidden email]
Subject: Re: SOLR 4 Alpha Out Of Mem Err

On Sun, Jul 15, 2012 at 12:52 PM, Jack Krupansky
<[hidden email]> wrote:
> Maybe your rate of update is so high that the commit never gets a chance
> to
> run.

I don't believe that is possible.  If it is, it should be fixed.

-Yonik
http://lucidimagination.com 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: SOLR 4 Alpha Out Of Mem Err

nick.koton
In reply to this post by Yonik Seeley-2-2
>> Solrj multi-threaded client sends several 1,000 docs/sec

>Can you expand on that?  How many threads at once are sending docs to solr?
Is each request a single doc or multiple?
I realize, after the fact, that my solrj client is much like
org.apache.solr.client.solrj.LargeVolumeTestBase.  The number of threads is
configurable at run time as are the various commit parameters.  Most of the
test have been in the 4-16 threads range.  Most of my testing has been with
the single document SolrServer::add(SolrInputDocument doc )method.  When I
realized what LargeVolumeTestBase is doing, I converted my program to use
the SolrServer::add(Collection<SolrInputDocument> docs) method with 100
documents in each add batch.  Unfortunately, the out of memory errors still
occur without client side commits.

If you agree my three approaches to committing are logical, would it be
useful for me to try to reproduce this with "example" schema in a small
cloud configuration using LargeVolumeTestBase or the like?  It will take me
a couple days to work it in.  Or perhaps this sort of test is already run?

Best
Nick

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
Sent: Sunday, July 15, 2012 11:05 AM
To: Nick Koton
Cc: [hidden email]
Subject: Re: SOLR 4 Alpha Out Of Mem Err

On Sun, Jul 15, 2012 at 11:52 AM, Nick Koton <[hidden email]> wrote:

>> Do you have the following hard autoCommit in your config (as the
>> stock
> server does)?
>>     <autoCommit>
>>       <maxTime>15000</maxTime>
>>       <openSearcher>false</openSearcher>
>>     </autoCommit>
>
> I have tried with and without that setting.  When I described running
> with auto commit, that setting is what I mean.

OK cool.  You should be able to run the stock server (i.e. with this
autocommit) and blast in updates all day long - it looks like you have more
than enough memory.  If you can't, we need to fix something.  You shouldn't
need explicit commits unless you want the docs to be searchable at that
point.

> Solrj multi-threaded client sends several 1,000 docs/sec

Can you expand on that?  How many threads at once are sending docs to solr?
Is each request a single doc or multiple?

-Yonik
http://lucidimagination.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

Michael Della Bitta-2
"unable to create new native thread"

That suggests you're running out of threads, not RAM. Possibly you're
using a multithreaded collector, and it's pushing you over the top of
how many threads your OS lets a single process allocate? Or somehow
the thread stack size is set too high?

More here: http://stackoverflow.com/questions/763579/how-many-threads-can-a-java-vm-support

Michael Della Bitta

------------------------------------------------
Appinions, Inc. -- Where Influence Isn’t a Game.
http://www.appinions.com


On Sun, Jul 15, 2012 at 2:45 PM, Nick Koton <[hidden email]> wrote:

>>> Solrj multi-threaded client sends several 1,000 docs/sec
>
>>Can you expand on that?  How many threads at once are sending docs to solr?
> Is each request a single doc or multiple?
> I realize, after the fact, that my solrj client is much like
> org.apache.solr.client.solrj.LargeVolumeTestBase.  The number of threads is
> configurable at run time as are the various commit parameters.  Most of the
> test have been in the 4-16 threads range.  Most of my testing has been with
> the single document SolrServer::add(SolrInputDocument doc )method.  When I
> realized what LargeVolumeTestBase is doing, I converted my program to use
> the SolrServer::add(Collection<SolrInputDocument> docs) method with 100
> documents in each add batch.  Unfortunately, the out of memory errors still
> occur without client side commits.
>
> If you agree my three approaches to committing are logical, would it be
> useful for me to try to reproduce this with "example" schema in a small
> cloud configuration using LargeVolumeTestBase or the like?  It will take me
> a couple days to work it in.  Or perhaps this sort of test is already run?
>
> Best
> Nick
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
> Sent: Sunday, July 15, 2012 11:05 AM
> To: Nick Koton
> Cc: [hidden email]
> Subject: Re: SOLR 4 Alpha Out Of Mem Err
>
> On Sun, Jul 15, 2012 at 11:52 AM, Nick Koton <[hidden email]> wrote:
>>> Do you have the following hard autoCommit in your config (as the
>>> stock
>> server does)?
>>>     <autoCommit>
>>>       <maxTime>15000</maxTime>
>>>       <openSearcher>false</openSearcher>
>>>     </autoCommit>
>>
>> I have tried with and without that setting.  When I described running
>> with auto commit, that setting is what I mean.
>
> OK cool.  You should be able to run the stock server (i.e. with this
> autocommit) and blast in updates all day long - it looks like you have more
> than enough memory.  If you can't, we need to fix something.  You shouldn't
> need explicit commits unless you want the docs to be searchable at that
> point.
>
>> Solrj multi-threaded client sends several 1,000 docs/sec
>
> Can you expand on that?  How many threads at once are sending docs to solr?
> Is each request a single doc or multiple?
>
> -Yonik
> http://lucidimagination.com
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

Mark Miller-3
In reply to this post by nick.koton

On Jul 15, 2012, at 2:45 PM, Nick Koton wrote:

> I converted my program to use
> the SolrServer::add(Collection<SolrInputDocument> docs) method with 100
> documents in each add batch.  Unfortunately, the out of memory errors still
> occur without client side commits.

This won't change much unfortunately - currently, each host has 10 add and 10 deletes buffered for it before it will flush. There are some recovery implications that have kept that buffer size low so far - but what it ends up meaning is that when you stream docs, every 10 docs is sent off on a thread. Generally, you might be able to keep up with this - but the commit cost appears to perhaps cause a small resource drop that backs things up a bit - and some of those threads take a little longer to finish while new threads fire off to keep servicing the constantly coming new documents. What appears will happen is large momentary spikes in the number of threads. Each thread needs a bit of space on the heap, and it would seem with a high enough spike you could get an OOM. In my testing, I have not triggered that yet, but I have seen large thread count spikes.

Raising the add doc buffer to 100 docs makes those thread bursts much, much less severe. I can't remember all of the implications of that buffer size though - need to talk to Yonik about it.

We could limit the number of threads for that executor, but I think that comes with some negatives as well.

You could try lowering -Xss so that each thread uses less RAM (if possible) as a shorter term (possible) workaround.

You could also use multiple threads with the std HttpSolrServer - it won't be quite as fast probably, but it can get close(ish).

My guess is that your client commits help because a commit will cause a wait on all outstanding requests - so that the commit is in logical order - this probably is like releasing a pressure valve - the system has a chance to catch up and reclaim lots of threads.

We will keep looking into what the best improvement is.

- Mark Miller
lucidimagination.com











Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: SOLR 4 Alpha Out Of Mem Err

nick.koton
In reply to this post by Michael Della Bitta-2
> That suggests you're running out of threads
Michael,
Thanks for this useful observation.  What I found just prior to the "problem
situation" was literally thousands of threads in the server JVM.  I have
pasted a few samples below obtained from the admin GUI.  I spent some time
today using this barometer, but I don't have enough to share right now.  I'm
looking at the difference between ConcurrentUpdateSolrServer and
HttpSolrServer and how my client may be misusing them.  I'll assume my
client is misbehaving and driving the server crazy for now.  If I figure out
how, I will share it so perhaps a safe guard can be put in place.

Nick


Server threads - very roughly 0.1 %:
cmdDistribExecutor-9-thread-7161 (10096)
java.util.concurrent.SynchronousQueue$TransferStack@17b90c55
. sun.misc.Unsafe.park(Native Method)
.
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
.
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Synchronous
Queue.java:424)
.
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueu
e.java:323)
.
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
.
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
.
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
07)
. java.lang.Thread.run(Thread.java:662)
-0.0000ms
-0.0000ms cmdDistribExecutor-9-thread-7160 (10086)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5509b5
6
. sun.misc.Unsafe.park(Native Method)
. java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
.
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1987)
.
org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158)
.
org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByR
oute.java:403)
.
org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRou
te.java:300)
.
org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(
ThreadSafeClientConnManager.java:224)
.
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDir
ector.java:401)
.
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:820)
.
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:754)
.
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:732)
.
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java
:351)
.
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java
:182)
.
org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:325
)
.
org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306
)
. java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
. java.util.concurrent.FutureTask.run(FutureTask.java:138)
.
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
. java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
. java.util.concurrent.FutureTask.run(FutureTask.java:138)
.
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
.
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
. java.lang.Thread.run(Thread.java:662)
20.0000ms
20.0000ms cmdDistribExecutor-9-thread-7159 (10085)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f062d
d3
. sun.misc.Unsafe.park(Native Method)
. java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
.
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1987)
.
org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158)
.
org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByR
oute.java:403)
.
org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRou
te.java:300)
.
org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(
ThreadSafeClientConnManager.java:224)
.
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDir
ector.java:401)
.
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:820)
.
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:754)
.
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:732)
.
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java
:351)
.
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java
:182)
.
org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:325
)
.
org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306
)
. java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
. java.util.concurrent.FutureTask.run(FutureTask.java:138)
.
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
. java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
. java.util.concurrent.FutureTask.run(FutureTask.java:138)
.
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
.
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
. java.lang.Thread.run(Thread.java:662)


-----Original Message-----
From: Michael Della Bitta [mailto:[hidden email]]
Sent: Sunday, July 15, 2012 4:46 PM
To: [hidden email]
Subject: Re: SOLR 4 Alpha Out Of Mem Err

"unable to create new native thread"

That suggests you're running out of threads, not RAM. Possibly you're using
a multithreaded collector, and it's pushing you over the top of how many
threads your OS lets a single process allocate? Or somehow the thread stack
size is set too high?

More here:
http://stackoverflow.com/questions/763579/how-many-threads-can-a-java-vm-sup
port

Michael Della Bitta

------------------------------------------------
Appinions, Inc. -- Where Influence Isn't a Game.
http://www.appinions.com


On Sun, Jul 15, 2012 at 2:45 PM, Nick Koton <[hidden email]> wrote:
>>> Solrj multi-threaded client sends several 1,000 docs/sec
>
>>Can you expand on that?  How many threads at once are sending docs to
solr?

> Is each request a single doc or multiple?
> I realize, after the fact, that my solrj client is much like
> org.apache.solr.client.solrj.LargeVolumeTestBase.  The number of
> threads is configurable at run time as are the various commit
> parameters.  Most of the test have been in the 4-16 threads range.  
> Most of my testing has been with the single document
> SolrServer::add(SolrInputDocument doc )method.  When I realized what
> LargeVolumeTestBase is doing, I converted my program to use the
> SolrServer::add(Collection<SolrInputDocument> docs) method with 100
> documents in each add batch.  Unfortunately, the out of memory errors
still occur without client side commits.
>
> If you agree my three approaches to committing are logical, would it
> be useful for me to try to reproduce this with "example" schema in a
> small cloud configuration using LargeVolumeTestBase or the like?  It
> will take me a couple days to work it in.  Or perhaps this sort of test is
already run?

>
> Best
> Nick
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik
> Seeley
> Sent: Sunday, July 15, 2012 11:05 AM
> To: Nick Koton
> Cc: [hidden email]
> Subject: Re: SOLR 4 Alpha Out Of Mem Err
>
> On Sun, Jul 15, 2012 at 11:52 AM, Nick Koton <[hidden email]> wrote:
>>> Do you have the following hard autoCommit in your config (as the
>>> stock
>> server does)?
>>>     <autoCommit>
>>>       <maxTime>15000</maxTime>
>>>       <openSearcher>false</openSearcher>
>>>     </autoCommit>
>>
>> I have tried with and without that setting.  When I described running
>> with auto commit, that setting is what I mean.
>
> OK cool.  You should be able to run the stock server (i.e. with this
> autocommit) and blast in updates all day long - it looks like you have
> more than enough memory.  If you can't, we need to fix something.  You
> shouldn't need explicit commits unless you want the docs to be
> searchable at that point.
>
>> Solrj multi-threaded client sends several 1,000 docs/sec
>
> Can you expand on that?  How many threads at once are sending docs to
solr?
> Is each request a single doc or multiple?
>
> -Yonik
> http://lucidimagination.com
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: SOLR 4 Alpha Out Of Mem Err

nick.koton
In reply to this post by Michael Della Bitta-2
After trying a number of things, I am successful in allowing the server to
auto commit and without having it hit thread/memory errors.  I have isolated
the required client change to replacing ConcurrentUpdateSolrServer with
HttpSolrServer.  I am able to maintain index rates of 3,000 documents/sec
with 6 shards and two servers per shard.  The servers receiving the index
requests hit steady state with approximately 800 threads per server.

So could there be something amiss in the server side implementation of
ConcurrentUpdateSolrServer?

Best regards,
Nick

-----Original Message-----
From: Nick Koton [mailto:[hidden email]]
Sent: Monday, July 16, 2012 5:53 PM
To: '[hidden email]'
Subject: RE: SOLR 4 Alpha Out Of Mem Err

> That suggests you're running out of threads
Michael,
Thanks for this useful observation.  What I found just prior to the "problem
situation" was literally thousands of threads in the server JVM.  I have
pasted a few samples below obtained from the admin GUI.  I spent some time
today using this barometer, but I don't have enough to share right now.  I'm
looking at the difference between ConcurrentUpdateSolrServer and
HttpSolrServer and how my client may be misusing them.  I'll assume my
client is misbehaving and driving the server crazy for now.  If I figure out
how, I will share it so perhaps a safe guard can be put in place.

Nick


Server threads - very roughly 0.1 %:
cmdDistribExecutor-9-thread-7161 (10096)
java.util.concurrent.SynchronousQueue$TransferStack@17b90c55
. sun.misc.Unsafe.park(Native Method)
.
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
.
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Synchronous
Queue.java:424)
.
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueu
e.java:323)
.
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
.
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
.
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
07)
. java.lang.Thread.run(Thread.java:662)
-0.0000ms
-0.0000ms cmdDistribExecutor-9-thread-7160 (10086)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5509b5
6
. sun.misc.Unsafe.park(Native Method)
. java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
.
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1987)
.
org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158)
.
org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByR
oute.java:403)
.
org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRou
te.java:300)
.
org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(
ThreadSafeClientConnManager.java:224)
.
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDir
ector.java:401)
.
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:820)
.
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:754)
.
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:732)
.
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java
:351)
.
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java
:182)
.
org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:325
)
.
org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306
)
. java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
. java.util.concurrent.FutureTask.run(FutureTask.java:138)
.
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
. java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
. java.util.concurrent.FutureTask.run(FutureTask.java:138)
.
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
.
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
. java.lang.Thread.run(Thread.java:662)
20.0000ms
20.0000ms cmdDistribExecutor-9-thread-7159 (10085)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f062d
d3
. sun.misc.Unsafe.park(Native Method)
. java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
.
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1987)
.
org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158)
.
org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByR
oute.java:403)
.
org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRou
te.java:300)
.
org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(
ThreadSafeClientConnManager.java:224)
.
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDir
ector.java:401)
.
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:820)
.
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:754)
.
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:732)
.
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java
:351)
.
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java
:182)
.
org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:325
)
.
org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306
)
. java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
. java.util.concurrent.FutureTask.run(FutureTask.java:138)
.
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
. java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
. java.util.concurrent.FutureTask.run(FutureTask.java:138)
.
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
.
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
. java.lang.Thread.run(Thread.java:662)


-----Original Message-----
From: Michael Della Bitta [mailto:[hidden email]]
Sent: Sunday, July 15, 2012 4:46 PM
To: [hidden email]
Subject: Re: SOLR 4 Alpha Out Of Mem Err

"unable to create new native thread"

That suggests you're running out of threads, not RAM. Possibly you're using
a multithreaded collector, and it's pushing you over the top of how many
threads your OS lets a single process allocate? Or somehow the thread stack
size is set too high?

More here:
http://stackoverflow.com/questions/763579/how-many-threads-can-a-java-vm-sup
port

Michael Della Bitta

------------------------------------------------
Appinions, Inc. -- Where Influence Isn't a Game.
http://www.appinions.com


On Sun, Jul 15, 2012 at 2:45 PM, Nick Koton <[hidden email]> wrote:
>>> Solrj multi-threaded client sends several 1,000 docs/sec
>
>>Can you expand on that?  How many threads at once are sending docs to
solr?

> Is each request a single doc or multiple?
> I realize, after the fact, that my solrj client is much like
> org.apache.solr.client.solrj.LargeVolumeTestBase.  The number of
> threads is configurable at run time as are the various commit
> parameters.  Most of the test have been in the 4-16 threads range.
> Most of my testing has been with the single document
> SolrServer::add(SolrInputDocument doc )method.  When I realized what
> LargeVolumeTestBase is doing, I converted my program to use the
> SolrServer::add(Collection<SolrInputDocument> docs) method with 100
> documents in each add batch.  Unfortunately, the out of memory errors
still occur without client side commits.
>
> If you agree my three approaches to committing are logical, would it
> be useful for me to try to reproduce this with "example" schema in a
> small cloud configuration using LargeVolumeTestBase or the like?  It
> will take me a couple days to work it in.  Or perhaps this sort of test is
already run?

>
> Best
> Nick
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik
> Seeley
> Sent: Sunday, July 15, 2012 11:05 AM
> To: Nick Koton
> Cc: [hidden email]
> Subject: Re: SOLR 4 Alpha Out Of Mem Err
>
> On Sun, Jul 15, 2012 at 11:52 AM, Nick Koton <[hidden email]> wrote:
>>> Do you have the following hard autoCommit in your config (as the
>>> stock
>> server does)?
>>>     <autoCommit>
>>>       <maxTime>15000</maxTime>
>>>       <openSearcher>false</openSearcher>
>>>     </autoCommit>
>>
>> I have tried with and without that setting.  When I described running
>> with auto commit, that setting is what I mean.
>
> OK cool.  You should be able to run the stock server (i.e. with this
> autocommit) and blast in updates all day long - it looks like you have
> more than enough memory.  If you can't, we need to fix something.  You
> shouldn't need explicit commits unless you want the docs to be
> searchable at that point.
>
>> Solrj multi-threaded client sends several 1,000 docs/sec
>
> Can you expand on that?  How many threads at once are sending docs to
solr?
> Is each request a single doc or multiple?
>
> -Yonik
> http://lucidimagination.com
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

Mark Miller-3

On Jul 17, 2012, at 8:08 PM, Nick Koton wrote:

> So could there be something amiss in the server side implementation of
> ConcurrentUpdateSolrServer?

See my earlier email. Once we decide on the appropriate change, we will address it.

- Mark Miller
lucidimagination.com











Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

Yonik Seeley-2-2
In reply to this post by Mark Miller-3
I think what makes the most sense is to limit the number of
connections to another host.  A host only has so many CPU resources,
and beyond a certain point throughput would start to suffer anyway
(and then only make the problem worse).  It also makes sense in that a
client could generate documents faster than we can index them (either
for a short period of time, or on average) and having flow control to
prevent unlimited buffering (which is essentially what this is) makes
sense.

Nick - when you switched to HttpSolrServer, things worked because this
added an explicit flow control mechanism.
A single request (i.e. an add with one or more documents) is fully
indexed to all endpoints before the response is returned.  Hence if
you have 10 indexing threads and are adding documents in batches of
100, there can be only 1000 documents buffered in the system at any
one time.

-Yonik
http://lucidimagination.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

solrman
In reply to this post by nick.koton
Nick,

to solve out of memory issue, i think you can make below change:
1) in solrsconfig.xml, reduce ramBufferSizeMB (there are two, change both)
2) in solrsconfig.xml, reduce documentCache value

to solve call commit slow down index issue, i think you can change new search default queyr:
in solrsconfig.xml, search for
<listener event="newSearcher" class="solr.QuerySenderListener">
change
<str name="q">content:*</str> <str name="start">0</str> <str name="rows">10</str>
to
<str name="q">content:notexist</str> <str name="start">0</str> <str name="rows">10</str>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

sausarkar
In reply to this post by Mark Miller-3
Hi Mark,

I am am also facing the same issue when trying to index in SolrCloud using DIH running on a non-leader server. The DIH server is creating around 10k threads and then OOM cannot create thread error.

Do you know when or which version this issue will be solved. I think a workaround for this issue is to find the leader from zookeeper and run the DIH on the leader.

Sauvik
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

Mark Miller-3
In reply to this post by solrman
>
> Hi Mark,
> I am am also facing the same issue when trying to index in SolrCloud using
> DIH running on a non-leader server. The DIH server is creating around 10k
> threads and then OOM cannot create thread error.
> Do you know when or which version this issue will be solved. I think a
> workaround for this issue is to find the leader from zookeeper and run the
> DIH on the leader.
> Sauvik


Oddly, I didn't get the above email that I can find - just found it on
markmail...odd.

I've made a JIRA issue for this:
https://issues.apache.org/jira/browse/SOLR-3658

I'm working on / testing a fix.

Yes, it affects both httpconcurrent server and dih. Anything that adds docs
with a single UpdateProcessor chain rather than a new one for each doc - as
long as it's adding fast enough.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: SOLR 4 Alpha Out Of Mem Err

sausarkar
Hello Mark,

Has this issue been fixed in the BETA release?

- Sauvik
Loading...