[jira] [Reopened] (SOLR-11277) Add auto hard commit setting based on tlog size

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Reopened] (SOLR-11277) Add auto hard commit setting based on tlog size

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/SOLR-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley reopened SOLR-11277:
---------------------------------
      Assignee:     (was: Anshum Gupta)

I recently tried beasting TestStressVersions, and I got a couple of failures that look like they are due to this patch.  Presumably the easiest fix would be to add synchronized(this) in UpdateLog.getCurrentLogSizeFromStream.

{code}
  2> 12269 ERROR (WRITER11) [    ] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Exception writing document id 1 to the index; possible analysis error.
  2>    at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:246)
  2>    at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
  2>    at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
  2>    at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:950)
  2>    at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1163)
  2>    at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:633)
  2>    at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
  2>    at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleAdds(JsonLoader.java:501)
  2>    at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:145)
  2>    at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:121)
  2>    at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:84)
  2>    at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
  2>    at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
  2>    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
  2>    at org.apache.solr.core.SolrCore.execute(SolrCore.java:2510)
  2>    at org.apache.solr.servlet.DirectSolrConnection.request(DirectSolrConnection.java:125)
  2>    at org.apache.solr.SolrTestCaseJ4.updateJ(SolrTestCaseJ4.java:1286)
  2>    at org.apache.solr.SolrTestCaseJ4.addAndGetVersion(SolrTestCaseJ4.java:1451)
  2>    at org.apache.solr.search.TestStressVersions$1.run(TestStressVersions.java:175)
  2> Caused by: java.lang.NullPointerException
  2>    at org.apache.solr.update.UpdateLog.getCurrentLogSizeFromStream(UpdateLog.java:299)
  2>    at org.apache.solr.update.DirectUpdateHandler2.getCurrentTLogSize(DirectUpdateHandler2.java:1007)
  2>    at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:291)
  2>    at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:234)
  2>    ... 18 more
{code}

> Add auto hard commit setting based on tlog size
> -----------------------------------------------
>
>                 Key: SOLR-11277
>                 URL: https://issues.apache.org/jira/browse/SOLR-11277
>             Project: Solr
>          Issue Type: New Feature
>      Security Level: Public(Default Security Level. Issues are Public)
>            Reporter: Rupa Shankar
>            Priority: Major
>             Fix For: 7.4, master (8.0)
>
>         Attachments: SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, max_size_auto_commit.patch
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> When indexing documents of variable sizes and at variable schedules, it can be hard to estimate the optimal auto hard commit maxDocs or maxTime settings. We’ve had some occurrences of really huge tlogs, resulting in serious issues, so in an attempt to avoid this, it would be great to have a “maxSize” setting based on the tlog size on disk.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]