[jira] [Commented] (SOLR-11277) Add auto hard commit setting based on tlog size

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-11277) Add auto hard commit setting based on tlog size

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479071#comment-16479071 ]

Yonik Seeley commented on SOLR-11277:
-------------------------------------

bq. is this what you'd suggested? 
Yes, that should handle the observed NPE.

Another thing I noticed:
It seems like under heavy indexing, many different threads will detect tlog sizes greater than the limit (and continue to until the part of the commit that rolls over the tlog happens).  All of those threads will call _scheduleCommitWithin(1ms) which will all call getDelay on the pending commit task to see if it needs to do it sooner.  More of a performance implication, but probably not significant compared to the cost of a commit.  docsSinceCommit will also be incorrectly zeroed, but given it's use, it shouldn't be a big deal if it can be off by a few.


> Add auto hard commit setting based on tlog size
> -----------------------------------------------
>
>                 Key: SOLR-11277
>                 URL: https://issues.apache.org/jira/browse/SOLR-11277
>             Project: Solr
>          Issue Type: New Feature
>      Security Level: Public(Default Security Level. Issues are Public)
>            Reporter: Rupa Shankar
>            Assignee: Anshum Gupta
>            Priority: Major
>             Fix For: 7.4, master (8.0)
>
>         Attachments: SOLR-11277.01.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, max_size_auto_commit.patch
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> When indexing documents of variable sizes and at variable schedules, it can be hard to estimate the optimal auto hard commit maxDocs or maxTime settings. We’ve had some occurrences of really huge tlogs, resulting in serious issues, so in an attempt to avoid this, it would be great to have a “maxSize” setting based on the tlog size on disk.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]