[jira] [Commented] (LUCENE-8253) ForceMergeDeletes does not merge soft-deleted segments

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (LUCENE-8253) ForceMergeDeletes does not merge soft-deleted segments

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/LUCENE-8253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440073#comment-16440073 ]

Steve Rowe commented on LUCENE-8253:
------------------------------------

{{git bisect}} blames commit {{c70ccea}} on this issue for reproducing Solr {{SegmentsInfoRequestHandlerTest}} failures, e.g. from [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/21840/]:

{noformat}
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SegmentsInfoRequestHandlerTest -Dtests.method=testSegmentInfosData -Dtests.seed=D8FA27F4CB25E126 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it -Dtests.timezone=Europe/Kaliningrad -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.00s J2 | SegmentsInfoRequestHandlerTest.testSegmentInfosData <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > at __randomizedtesting.SeedInfo.seed([D8FA27F4CB25E126:A45305AF7DCD56B9]:0)
   [junit4]    > at org.apache.lucene.index.IndexWriter$ReaderPool.noDups(IndexWriter.java:867)
   [junit4]    > at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:857)
   [junit4]    > at org.apache.lucene.index.IndexWriter.numDeletesToMerge(IndexWriter.java:5233)
   [junit4]    > at org.apache.lucene.index.LogMergePolicy.sizeDocs(LogMergePolicy.java:153)
   [junit4]    > at org.apache.lucene.index.LogDocMergePolicy.size(LogDocMergePolicy.java:44)
   [junit4]    > at org.apache.lucene.index.LogMergePolicy.findMerges(LogMergePolicy.java:469)
   [junit4]    > at org.apache.solr.handler.admin.SegmentsInfoRequestHandler.getMergeCandidatesNames(SegmentsInfoRequestHandler.java:100)
   [junit4]    > at org.apache.solr.handler.admin.SegmentsInfoRequestHandler.getSegmentsInfo(SegmentsInfoRequestHandler.java:59)
   [junit4]    > at org.apache.solr.handler.admin.SegmentsInfoRequestHandler.handleRequestBody(SegmentsInfoRequestHandler.java:48)
   [junit4]    > at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]    > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2508)
   [junit4]    > at org.apache.solr.util.TestHarness.query(TestHarness.java:337)
   [junit4]    > at org.apache.solr.util.TestHarness.query(TestHarness.java:319)
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:890)
   [junit4]    > at org.apache.solr.handler.admin.SegmentsInfoRequestHandlerTest.testSegmentInfosData(SegmentsInfoRequestHandlerTest.java:75)
   [junit4]    > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > at java.base/java.lang.Thread.run(Thread.java:844)
[...]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SegmentsInfoRequestHandlerTest -Dtests.method=testSegmentInfos -Dtests.seed=D8FA27F4CB25E126 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it -Dtests.timezone=Europe/Kaliningrad -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.00s J2 | SegmentsInfoRequestHandlerTest.testSegmentInfos <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > at __randomizedtesting.SeedInfo.seed([D8FA27F4CB25E126:94ADB04B01293976]:0)
   [junit4]    > at org.apache.lucene.index.IndexWriter$ReaderPool.noDups(IndexWriter.java:867)
   [junit4]    > at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:857)
   [junit4]    > at org.apache.lucene.index.IndexWriter.numDeletesToMerge(IndexWriter.java:5233)
   [junit4]    > at org.apache.lucene.index.LogMergePolicy.sizeDocs(LogMergePolicy.java:153)
   [junit4]    > at org.apache.lucene.index.LogDocMergePolicy.size(LogDocMergePolicy.java:44)
   [junit4]    > at org.apache.lucene.index.LogMergePolicy.findMerges(LogMergePolicy.java:469)
   [junit4]    > at org.apache.solr.handler.admin.SegmentsInfoRequestHandler.getMergeCandidatesNames(SegmentsInfoRequestHandler.java:100)
   [junit4]    > at org.apache.solr.handler.admin.SegmentsInfoRequestHandler.getSegmentsInfo(SegmentsInfoRequestHandler.java:59)
   [junit4]    > at org.apache.solr.handler.admin.SegmentsInfoRequestHandler.handleRequestBody(SegmentsInfoRequestHandler.java:48)
   [junit4]    > at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]    > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2508)
   [junit4]    > at org.apache.solr.util.TestHarness.query(TestHarness.java:337)
   [junit4]    > at org.apache.solr.util.TestHarness.query(TestHarness.java:319)
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:890)
   [junit4]    > at org.apache.solr.handler.admin.SegmentsInfoRequestHandlerTest.testSegmentInfos(SegmentsInfoRequestHandlerTest.java:61)
   [junit4]    > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > at java.base/java.lang.Thread.run(Thread.java:844)
[...]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SegmentsInfoRequestHandlerTest -Dtests.method=testSegmentInfosVersion -Dtests.seed=D8FA27F4CB25E126 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it -Dtests.timezone=Europe/Kaliningrad -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.00s J2 | SegmentsInfoRequestHandlerTest.testSegmentInfosVersion <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > at __randomizedtesting.SeedInfo.seed([D8FA27F4CB25E126:2024B21CB3493075]:0)
   [junit4]    > at org.apache.lucene.index.IndexWriter$ReaderPool.noDups(IndexWriter.java:867)
   [junit4]    > at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:857)
   [junit4]    > at org.apache.lucene.index.IndexWriter.numDeletesToMerge(IndexWriter.java:5233)
   [junit4]    > at org.apache.lucene.index.LogMergePolicy.sizeDocs(LogMergePolicy.java:153)
   [junit4]    > at org.apache.lucene.index.LogDocMergePolicy.size(LogDocMergePolicy.java:44)
   [junit4]    > at org.apache.lucene.index.LogMergePolicy.findMerges(LogMergePolicy.java:469)
   [junit4]    > at org.apache.solr.handler.admin.SegmentsInfoRequestHandler.getMergeCandidatesNames(SegmentsInfoRequestHandler.java:100)
   [junit4]    > at org.apache.solr.handler.admin.SegmentsInfoRequestHandler.getSegmentsInfo(SegmentsInfoRequestHandler.java:59)
   [junit4]    > at org.apache.solr.handler.admin.SegmentsInfoRequestHandler.handleRequestBody(SegmentsInfoRequestHandler.java:48)
   [junit4]    > at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
   [junit4]    > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2508)
   [junit4]    > at org.apache.solr.util.TestHarness.query(TestHarness.java:337)
   [junit4]    > at org.apache.solr.util.TestHarness.query(TestHarness.java:319)
   [junit4]    > at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:890)
   [junit4]    > at org.apache.solr.handler.admin.SegmentsInfoRequestHandlerTest.testSegmentInfosVersion(SegmentsInfoRequestHandlerTest.java:68)
   [junit4]    > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]    > at java.base/java.lang.Thread.run(Thread.java:844)
[...]
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name=PostingsFormat(name=Memory), id=PostingsFormat(name=Memory)}, docValues:{}, maxPointsInLeafNode=375, maxMBSortInHeap=6.1720643265572495, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@50ce333a), locale=it, timezone=Europe/Kaliningrad
   [junit4]   2> NOTE: Linux 4.13.0-37-generic amd64/Oracle Corporation 9.0.4 (64-bit)/cpus=8,threads=1,free=136326496,total=536870912
{noformat}

{{IndexWriter.ReaderPool.noDups()}} is:

{code:java|title:IndexWriter.java}
862:    // Make sure that every segment appears only once in the
863:    // pool:
864:    private boolean noDups() {
865:      Set<String> seen = new HashSet<>();
866:      for(SegmentCommitInfo info : readerMap.keySet()) {
867:        assert !seen.contains(info.info.name);
868:        seen.add(info.info.name);
869:      }
870:      return true;
871:    }
872:  }
{code}

> ForceMergeDeletes does not merge soft-deleted segments
> ------------------------------------------------------
>
>                 Key: LUCENE-8253
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8253
>             Project: Lucene - Core
>          Issue Type: Bug
>    Affects Versions: 7.4, master (8.0)
>            Reporter: Nhat Nguyen
>            Assignee: Simon Willnauer
>            Priority: Major
>         Attachments: LUCENE-8253.patch, test-merge.patch
>
>
> IndexWriter#forceMergeDeletes should merge segments having soft-deleted documents as hard-deleted documents if we configured "softDeletesField" in an IndexWriterConfig.
> Attached is a failed test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]