PLEASE READ! BadApple report. Last week was horrible!

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

PLEASE READ! BadApple report. Last week was horrible!

Erick Erickson
I don’t know whether we had some temporary glitch that broke lots of tests and they’ve been fixed or we had a major regression, but this needs to be addressed ASAP if they’re still failing. See everything below the line "ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail. I’ll raise a JIRA if we can’t get some traction quickly here.

Hey, stuff happens. there’s no problem with tests going totally weird for a while. If you can say “Oh, yeah, all those failures for class XYZ are probably fixed” that’s fine.

Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)….

Hoss’ rolllup for the last 24 hours is not encouraging in terms of the
problem already being fixed. There are lots of failures in some
classes, notably:

CloudHttp2SolrClientTest
CollectionsAPIDistributedZkTest
DeleteReplicaTest
TestDocCollectionWatcher

Unfortunately, the failure rate is not very high so reliably
reproducing is hard.

I’ve reproduced the last week’s failure in this e-mail, full
report attached.

Here’s Hoss’ rollup:
http://fucit.org/solr-jenkins-reports/failure-report.html

Usual synopsis:

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0  had  343 failures
Week: 1  had  86 failures
Week: 2  had  78 failures
Week: 3  had  117 failures


********Failures in Hoss' reports for the last 4 rollups.

There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix’d

Failures in the last 4 reports..
   Report   Pct     runs    fails           test
     0123   0.7     1617     11      ConnectionManagerTest.testReconnectWhenZkDisappeared
     0123   1.5     1606     12      ExecutePlanActionTest.testTaskTimeout
     0123   1.6     1320     19      MultiThreadedOCPTest.test
     0123   1.0     1620     13      RollingRestartTest.test
     0123   1.2     1617     12      SearchRateTriggerTest.testWaitForElapsed
     0123   3.8      119      7      ShardSplitTest.testSplitWithChaosMonkey
     0123   0.3     1519      7      TestInPlaceUpdatesDistrib.test
     0123   0.7     1629     14      TestIndexWriterDelete.testDeleteAllNoDeadLock
     0123   2.4     1548     18      TestPackages.testPluginLoading
     0123   0.3     1587      4      UnloadDistributedZkTest.test
********************************************

FAILURES IN THE LAST WEEK (343!)
Look particularly at the ones with only a zero in the “Report” column, those are
failures that were _not_ in the previous 3 week’s rollups.

   Report   Pct     runs    fails           test
     012    0.5     1165      4      CustomHighlightComponentTest.test
     012    1.0     1168      6      NodeMarkersRegistrationTest.testNodeMarkersRegistration
     012    1.0     1170      8      TestCryptoKeys.test
     01 3   0.7     1233     11      LeaderFailoverAfterPartitionTest.test
     01 3  63.2      102     39      StressHdfsTest.test
     01     0.3      709      2      ScheduledTriggerIntegrationTest.testScheduledTrigger
     01     0.2      768      2      ShardRoutingTest.test
     01     2.6      807     22      TestAllFilesHaveChecksumFooter.test
     01     2.6      808     22      TestAllFilesHaveCodecHeader.test
     01     0.2      769      2      TestCloudSchemaless.test
     01     0.2      769      2      TestDynamicLoading.testDynamicLoading
     01     0.3      707      2      TestDynamicLoadingUrl.testDynamicLoadingUrl
     01     0.5      767      4      TestPointFields.testFloatPointStats
     01    27.1       83     19      TestSQLHandler.doTest
     01     0.2      794     12      TestSameScoresWithThreads.test
     01     2.6      806     22      TestShardSearching.testSimple
     01     0.5      726      4      TestSimScenario.testSplitShard
     01     1.1      726      7      TestSimScenario.testSuggestions
     01     0.3      771      2      TestWithCollection.testAddReplicaSimple
     0 23   0.3     1223      4      CdcrVersionReplicationTest.testCdcrDocVersions
     0 23   0.8     1172      6      CloudHttp2SolrClientTest.testRetryUpdatesWhenClusterStateIsStale
     0 23   1.4     1202      8      CollectionsAPISolrJTest.testColStatus
     0 23   1.0     1249     11      HttpPartitionTest.test
     0 23   1.1     1210      8      HttpPartitionWithTlogReplicasTest.test
     0 23   0.5     1258      4      ShardSplitTest.testSplitShardWithRuleLink
     0 23   0.2     1231      4      TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
     0 23   0.2     1232      6      TestSolrConfigHandlerCloud.test
     0 2    0.3      767      2      DocValuesNotIndexedTest.testGroupingDVOnlySortLast
     0 2    0.3      750      2      TestLBHttp2SolrClient.testTwoServers
     0 2    0.3      794      2      TestSolrCloudSnapshots.testSnapshots
     0 2   40.7       51     12      TestXYMultiPolygonShapeQueries.testRandomBig
     0  3   0.2      842      2      ConnectionManagerTest.testLikelyExpired
     0  3   0.2      835      3      LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
     0  3   0.3      832      3      ReplaceNodeTest.test
     0  3   0.3      826      2      SolrCloudReportersTest.testExplicitConfiguration
     0  3   0.2      833      2      TestPullReplicaErrorHandling.testCantConnectToLeader
     0  3   2.4      867     21      TestRandomChains.testRandomChains
     0  3   0.5      830      3      TestWithCollection.testMoveReplicaMainCollection

ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!

     0      0.3      361      1      AutoAddReplicasIntegrationTest.testClusterPropOverridesCollecitonProp
     0      2.9      380     11      BasicDistributedZkTest.test
     0      0.2      404      1      BasicZkTest.testBasic
     0      0.2      414      1      ChaosMonkeySafeLeaderWithPullReplicasTest.test
     0     62.8      853    536      CheckHdfsIndexTest.doTest
     0      0.8      355      3      CloudHttp2SolrClientTest.checkCollectionParameters
     0      0.8      355      3      CloudHttp2SolrClientTest.customHttpClientTest
     0      1.1      355      4      CloudHttp2SolrClientTest.preferLocalShardsTest
     0      0.8      355      3      CloudHttp2SolrClientTest.preferReplicaTypesTest
     0      0.8      355      3      CloudHttp2SolrClientTest.singleShardedPreferenceRules
     0      0.8      355      3      CloudHttp2SolrClientTest.stateVersionParamTest
     0      1.1      355      4      CloudHttp2SolrClientTest.testAliasHandling
     0      0.8      355      3      CloudHttp2SolrClientTest.testCollectionDoesntExist
     0      0.8      355      3      CloudHttp2SolrClientTest.testInitializationWithSolrUrls
     0      0.8      355      3      CloudHttp2SolrClientTest.testNonRetryableRequests
     0      1.1      355      4      CloudHttp2SolrClientTest.testOverwriteOption
     0      0.8      355      3      CloudHttp2SolrClientTest.testParallelUpdateQTime
     0      0.6      355      2      CloudHttp2SolrClientTest.testRouting
     0      0.8      355      3      CloudHttp2SolrClientTest.testShutdown
     0      0.6      355      2      CloudHttp2SolrClientTest.testVersionsAreReturned
     0      0.8      355      3      CloudHttp2SolrClientTest.testWrongZkChrootTest
     0      0.5      392      2      CloudSolrClientTest.checkCollectionParameters
     0      0.3      392      1      CloudSolrClientTest.preferLocalShardsTest
     0      0.3      392      1      CloudSolrClientTest.preferReplicaTypesTest
     0      0.3      357      1      CloudSolrClientTest.singleShardedPreferenceRules
     0      0.5      392      2      CloudSolrClientTest.stateVersionParamTest
     0      0.5      392      2      CloudSolrClientTest.testAliasHandling
     0      0.3      392      1      CloudSolrClientTest.testInitializationWithSolrUrls
     0      0.3      392      1      CloudSolrClientTest.testNonRetryableRequests
     0      0.3      392      1      CloudSolrClientTest.testOverwriteOption
     0      0.3      392      1      CloudSolrClientTest.testParallelUpdateQTime
     0      0.3      357      1      CloudSolrClientTest.testPing
     0      0.5      392      2      CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale
     0      0.3      392      1      CloudSolrClientTest.testShutdown
     0      0.5      392      2      CloudSolrClientTest.testVersionsAreReturned
     0      0.3      392      1      CloudSolrClientTest.testWrongZkChrootTest
     0      0.3      399      1      CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls
     0      1.3      397      5      CollectionsAPIDistributedZkTest.addReplicaTest
     0      0.8      397      3      CollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk
     0      1.0      397      4      CollectionsAPIDistributedZkTest.deleteCollectionRemovesStaleZkCollectionsNode
     0      0.8      397      3      CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
     0      0.3      397      1      CollectionsAPIDistributedZkTest.testBadActionNames
     0      0.8      397      3      CollectionsAPIDistributedZkTest.testCollectionReload
     0      1.0      397      4      CollectionsAPIDistributedZkTest.testCollectionsAPI
     0      1.3      397      5      CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes
     0      1.0      397      4      CollectionsAPIDistributedZkTest.testCreateNodeSet
     0      1.3      397      5      CollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore
     0      0.3      397      1      CollectionsAPIDistributedZkTest.testCreationAndDeletion
     0      0.5      397      2      CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection
     0      0.8      397      3      CollectionsAPIDistributedZkTest.testMaxNodesPerShard
     0      0.8      397      3      CollectionsAPIDistributedZkTest.testMissingNumShards
     0      0.5      397      2      CollectionsAPIDistributedZkTest.testMissingRequiredParameters
     0      0.8      397      3      CollectionsAPIDistributedZkTest.testNoConfigSetExist
     0      0.8      397      3      CollectionsAPIDistributedZkTest.testSpecificConfigsets
     0      0.8      397      3      CollectionsAPIDistributedZkTest.testTooManyReplicas
     0      1.0      397      4      CollectionsAPIDistributedZkTest.testZeroNumShards
     0      0.7      405      3      CollectionsAPISolrJTest.testAddAndDeleteReplica
     0      0.2      405      1      CollectionsAPISolrJTest.testAddAndDeleteReplicaProp
     0      0.7      405      3      CollectionsAPISolrJTest.testBalanceShardUnique
     0      0.5      405      2      CollectionsAPISolrJTest.testCloudInfoInCoreStatus
     0      1.5      405      6      CollectionsAPISolrJTest.testClusterProp
     0      0.5      405      2      CollectionsAPISolrJTest.testCollectionProp
     0      0.5      405      2      CollectionsAPISolrJTest.testCreateAndDeleteAlias
     0      0.5      405      2      CollectionsAPISolrJTest.testCreateAndDeleteCollection
     0      0.7      405      3      CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesNewFormat
     0      1.7      405      7      CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesOldFormat
     0      0.7      405      3      CollectionsAPISolrJTest.testCreateCollectionWithPropertyParam
     0      1.5      405      6      CollectionsAPISolrJTest.testCreateWithDefaultConfigSet
     0      1.9      370      7      CollectionsAPISolrJTest.testDeleteAliasedCollection
     0      1.2      405      5      CollectionsAPISolrJTest.testList
     0      1.0      405      4      CollectionsAPISolrJTest.testModifyCollectionAttribute
     0      1.2      405      5      CollectionsAPISolrJTest.testOverseerStatus
     0      0.8      370      3      CollectionsAPISolrJTest.testReadOnlyCollection
     0      1.6      370      6      CollectionsAPISolrJTest.testRenameCollection
     0      0.7      405      3      CollectionsAPISolrJTest.testSplitShard
     0      0.5      366      2      ComputePlanActionTest.testNodeAdded
     0      1.1      376      4      DeleteReplicaTest.deleteLiveReplicaTest
     0      1.5      411      6      DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup
     0      0.5      411      2      DeleteReplicaTest.deleteReplicaByCount
     0      0.5      376      2      DeleteReplicaTest.deleteReplicaByCountForAllShards
     0      1.2      411      5      DeleteReplicaTest.deleteReplicaFromClusterState
     0      1.5      411      6      DeleteReplicaTest.deleteReplicaFromClusterStateLegacy
     0      1.7      411      7      DeleteReplicaTest.deleteReplicaOnIndexing
     0      0.8      376      3      DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
     0      1.3      376      5      DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy
     0      0.7      405      3      DeleteShardTest.test
     0      0.2      405      1      DeleteShardTest.testDirectoryCleanupAfterDeleteShard
     0      0.5      366      2      DistribDocExpirationUpdateProcessorTest.testBasicAuth
     0      0.5      366      2      DistribDocExpirationUpdateProcessorTest.testNoAuth
     0      0.2      402      1      DistributedQueueTest.testDistributedQueue
     0      0.7      402      3      DistributedQueueTest.testDistributedQueueBlocking
     0      0.7      402      3      DistributedQueueTest.testDistributedQueueCache
     0      0.2      402      1      DistributedQueueTest.testLeakChildWatcher
     0      0.5      402      2      DistributedQueueTest.testLocallyOffer
     0      0.7      402      3      DistributedQueueTest.testPeekElements
     0      0.2      403      1      ExecutePlanActionTest.testExecute
     0      0.5      403      2      ExecutePlanActionTest.testIntegration
     0      0.7      403      3      ExecutePlanActionTest.testTaskFail
     0      0.3      370      1      IndexSizeTriggerSizeEstimationTest.testEstimatedIndexSize
     0      0.3      366      1      IndexSizeTriggerTest.testInvalidSplitConfig
     0      0.3      366      1      IndexSizeTriggerTest.testMergeIntegration
     0      0.3      366      1      IndexSizeTriggerTest.testSplitConfig
     0      0.3      366      1      IndexSizeTriggerTest.testSplitIntegration
     0      0.3      366      1      IndexSizeTriggerTest.testTrigger
     0      0.3      366      1      IndexSizeTriggerTest.testValidProperties
     0      0.3      360      1      JWTAuthPluginIntegrationTest.infoRequestWithoutToken
     0      0.3      360      1      JWTAuthPluginIntegrationTest.testMetrics
     0      0.2      403      1      LeaderElectionTest.testBasic
     0      0.5      403      2      LeaderElectionTest.testCancelElection
     0      0.5      403      2      LeaderElectionTest.testElection
     0      0.5      403      2      LeaderElectionTest.testParallelElection
     0      0.5      403      2      LeaderElectionTest.testStressElection
     0      0.2      404      1      LeaderFailureAfterFreshStartTest.test
     0      0.3      367      1      MoveReplicaTest.testFailedMove
     0      0.3      362      1      NestedShardedAtomicUpdateTest.test
     0      0.5      400      2      NodeAddedTriggerIntegrationTest.testNodeAddedTriggerRestoreState
     0      0.3      398      1      NodeAddedTriggerTest.testListenerAcceptance
     0      0.3      398      1      NodeAddedTriggerTest.testRestoreState
     0      0.3      398      1      NodeAddedTriggerTest.testTrigger
     0      0.3      397      1      NodeLostTriggerIntegrationTest.testNodeLostTriggerRestoreState
     0      0.3      398      1      OutOfBoxZkACLAndCredentialsProvidersTest.testOpenACLUnsafeAllover
     0      0.3      398      1      OutOfBoxZkACLAndCredentialsProvidersTest.testOutOfBoxSolrZkClient
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testNoCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
     0      0.3      400      1      OverseerTaskQueueTest.testDistributedQueueCache
     0      0.3      400      1      OverseerTaskQueueTest.testLocallyOffer
     0      0.3      400      1      PingRequestHandlerTest.testPingInClusterWithNoHealthCheck
     0      0.5      398      2      RecoveryAfterSoftCommitTest.test
     0      0.5      396      2      ReplicationFactorTest.test
     0      0.7      412      3      ShardSplitTest.testSplitLocking
     0      0.8      377      3      ShardSplitTest.testSplitMixedReplicaTypes
     0      0.5      411      2      ShardSplitTest.testSplitStaticIndexReplication
     0      0.3      353      1      SolrZkClientTest.getConfig
     0      0.3      353      1      SolrZkClientTest.testWrappingWatches
     0      0.3      398      1      SpatialHeatmapFacetsTest.testClassicFacets
     0      0.3      398      1      SpatialHeatmapFacetsTest.testJsonFacets
     0      0.3     1194      3      SpatialHeatmapFacetsTest.testPng
     0      1.0      209      2      StreamDecoratorTest.testParallelTerminatingDaemonUpdateStream
     0      0.3      397      1      SystemLogListenerTest.test
     0      0.5      388      2      TestCloudCollectionsListeners.testCollectionDeletion
     0      0.3      388      1      TestCloudCollectionsListeners.testSimpleCloudCollectionsListener
     0      0.3      397      1      TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeaderAfterRestart
     0      0.3      396      1      TestCloudSearcherWarming.testPeersyncFailureReplicationSuccess
     0      0.8      354      3      TestCollectionStateWatchers.testCanWaitForNonexistantCollection
     0      0.8      354      3      TestCollectionStateWatchers.testCollectionWatchWithShutdownOfActiveNode
     0      0.8      354      3      TestCollectionStateWatchers.testCollectionWatchWithShutdownOfUnusedNode
     0      0.8      389      3      TestCollectionStateWatchers.testDeletionsTriggerWatches
     0      0.6      354      2      TestCollectionStateWatchers.testLiveNodeChangesTriggerWatches
     0      0.8      389      3      TestCollectionStateWatchers.testPredicateFailureTimesOut
     0      0.8      389      3      TestCollectionStateWatchers.testStateWatcherChecksCurrentStateOnRegister
     0      0.8      389      3      TestCollectionStateWatchers.testWaitForStateChecksCurrentState
     0      0.8      389      3      TestCollectionStateWatchers.testWaitForStateWatcherIsRetainedOnPredicateFailure
     0      0.5      389      2      TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout
     0      0.5      389      2      TestCollectionStateWatchers.testWatchesWorkForStateFormat1
     0      0.3      400      1      TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete
     0      0.3      399      1      TestConfigSetsAPI.testCreate
     0      0.3      399      1      TestConfigSetsAPI.testList
     0      0.3      399      1      TestConfigSetsAPI.testUpload
     0      0.3      399      1      TestConfigSetsAPI.testUploadDisabled
     0      0.3      399      1      TestConfigSetsAPI.testUploadErrors
     0      0.3      399      1      TestConfigSetsAPI.testUploadWithScriptUpdateProcessor
     0      0.3      399      1      TestConfigSetsAPI.testUserAndTestDefaultConfigsetsAreSame
     0      0.3      400      1      TestDistribDocBasedVersion.test
     0      0.3      354      1      TestDocCollectionWatcher.testCanWaitForNonexistantCollection
     0      0.6      354      2      TestDocCollectionWatcher.testDeletionsTriggerWatches
     0      0.6      354      2      TestDocCollectionWatcher.testPredicateFailureTimesOut
     0      0.6      354      2      TestDocCollectionWatcher.testStateWatcherChecksCurrentStateOnRegister
     0      0.6      354      2      TestDocCollectionWatcher.testWaitForStateChecksCurrentState
     0      0.6      354      2      TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFailure
     0      0.6      354      2      TestDocCollectionWatcher.testWatcherIsRemovedAfterTimeout
     0      0.6      354      2      TestDocCollectionWatcher.testWatchesWorkForStateFormat1
     0     14.3       21      3      TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit
     0      0.2      414      1      TestDuelingCodecs.testCrazyReaderEquals
     0      4.5       22      1      TestDuelingCodecsAtNight.testCrazyReaderEquals
     0      0.2      419      1      TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads
     0      0.3      352      1      TestJaegerConfigurator.testInjected
     0      0.3      387      1      TestLTROnSolrCloud.testSimpleQuery
     0      1.2      402      5      TestPointFields.testDoublePointStats
     0      0.2      402      1      TestPointFields.testIntPointFieldMultiValuedRangeFacet
     0      1.0      406      4      TestRandomDVFaceting.testRandomFaceting
     0      0.5      380      2      TestSimScenario.testAutoAddReplicas
     0      0.3      397      1      TestSimTriggerIntegration.testCooldown
     0      0.2      419      1      TestSimpleExplanationsWithFillerDocs.testMultiFieldBQofPQ5
     0      0.2      402      1      TestSolrConfigHandlerConcurrent.test
     0      0.5      366      2      TestWithCollection.testAddReplicaWithPolicy
     0      0.3      400      1      TestWithCollection.testCreateCollection
     0      0.3      400      1      TestWithCollection.testCreateCollectionNoWithCollection
     0      0.3      400      1      TestWithCollection.testDeleteWithCollection
     0      0.3      400      1      TestWithCollection.testMoveReplicaWithCollection
     0      0.3      366      1      TestWithCollection.testNodeAdded
     0      0.2      401      1      ZkCLITest.testInvalidZKAddress
     0      0.2      401      1      ZkCLITest.testPutFile
     0      0.3      394      1      ZkSolrClientTest.testMakeRootNode
     0      0.3      394      1      ZkSolrClientTest.testReconnect
     0      0.3      394      1      ZkSolrClientTest.testWatchChildren






---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

e-mail-2020-05-04.txt (38K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: PLEASE READ! BadApple report. Last week was horrible!

Michael McCandless-2
Hi Erick,

It's possible this was the root cause of many of the failures: https://issues.apache.org/jira/browse/LUCENE-9191

Do these transient failures look something like this?

   [junit4]    > Throwable #1: java.nio.charset.MalformedInputException: Input length = 1
   [junit4]    >        at __randomizedtesting.SeedInfo.seed([172C6414BE5E2A2C:E5829DFC005A1F0]:0)
   [junit4]    >        at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
   [junit4]    >        at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
   [junit4]    >        at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
   [junit4]    >        at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
   [junit4]    >        at java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
   [junit4]    >        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
   [junit4]    >        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
   [junit4]    >        at org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175)
   [junit4]    >        at org.apache.lucene.util.LineFileDocs.<init>(LineFileDocs.java:65)
   [junit4]    >        at org.apache.lucene.util.LineFileDocs.<init>(LineFileDocs.java:69)

If so, then it is likely the root cause ... I'm working on a fix.  Sorry!

On Mon, May 4, 2020 at 7:54 AM Erick Erickson <[hidden email]> wrote:
I don’t know whether we had some temporary glitch that broke lots of tests and they’ve been fixed or we had a major regression, but this needs to be addressed ASAP if they’re still failing. See everything below the line "ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail. I’ll raise a JIRA if we can’t get some traction quickly here.

Hey, stuff happens. there’s no problem with tests going totally weird for a while. If you can say “Oh, yeah, all those failures for class XYZ are probably fixed” that’s fine.

Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)….

Hoss’ rolllup for the last 24 hours is not encouraging in terms of the
problem already being fixed. There are lots of failures in some
classes, notably:

CloudHttp2SolrClientTest
CollectionsAPIDistributedZkTest
DeleteReplicaTest
TestDocCollectionWatcher

Unfortunately, the failure rate is not very high so reliably
reproducing is hard.

I’ve reproduced the last week’s failure in this e-mail, full
report attached.

Here’s Hoss’ rollup:
http://fucit.org/solr-jenkins-reports/failure-report.html

Usual synopsis:

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0  had  343 failures
Week: 1  had  86 failures
Week: 2  had  78 failures
Week: 3  had  117 failures


********Failures in Hoss' reports for the last 4 rollups.

There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix’d

Failures in the last 4 reports..
   Report   Pct     runs    fails           test
     0123   0.7     1617     11      ConnectionManagerTest.testReconnectWhenZkDisappeared
     0123   1.5     1606     12      ExecutePlanActionTest.testTaskTimeout
     0123   1.6     1320     19      MultiThreadedOCPTest.test
     0123   1.0     1620     13      RollingRestartTest.test
     0123   1.2     1617     12      SearchRateTriggerTest.testWaitForElapsed
     0123   3.8      119      7      ShardSplitTest.testSplitWithChaosMonkey
     0123   0.3     1519      7      TestInPlaceUpdatesDistrib.test
     0123   0.7     1629     14      TestIndexWriterDelete.testDeleteAllNoDeadLock
     0123   2.4     1548     18      TestPackages.testPluginLoading
     0123   0.3     1587      4      UnloadDistributedZkTest.test
********************************************

FAILURES IN THE LAST WEEK (343!)
Look particularly at the ones with only a zero in the “Report” column, those are
failures that were _not_ in the previous 3 week’s rollups.

   Report   Pct     runs    fails           test
     012    0.5     1165      4      CustomHighlightComponentTest.test
     012    1.0     1168      6      NodeMarkersRegistrationTest.testNodeMarkersRegistration
     012    1.0     1170      8      TestCryptoKeys.test
     01 3   0.7     1233     11      LeaderFailoverAfterPartitionTest.test
     01 3  63.2      102     39      StressHdfsTest.test
     01     0.3      709      2      ScheduledTriggerIntegrationTest.testScheduledTrigger
     01     0.2      768      2      ShardRoutingTest.test
     01     2.6      807     22      TestAllFilesHaveChecksumFooter.test
     01     2.6      808     22      TestAllFilesHaveCodecHeader.test
     01     0.2      769      2      TestCloudSchemaless.test
     01     0.2      769      2      TestDynamicLoading.testDynamicLoading
     01     0.3      707      2      TestDynamicLoadingUrl.testDynamicLoadingUrl
     01     0.5      767      4      TestPointFields.testFloatPointStats
     01    27.1       83     19      TestSQLHandler.doTest
     01     0.2      794     12      TestSameScoresWithThreads.test
     01     2.6      806     22      TestShardSearching.testSimple
     01     0.5      726      4      TestSimScenario.testSplitShard
     01     1.1      726      7      TestSimScenario.testSuggestions
     01     0.3      771      2      TestWithCollection.testAddReplicaSimple
     0 23   0.3     1223      4      CdcrVersionReplicationTest.testCdcrDocVersions
     0 23   0.8     1172      6      CloudHttp2SolrClientTest.testRetryUpdatesWhenClusterStateIsStale
     0 23   1.4     1202      8      CollectionsAPISolrJTest.testColStatus
     0 23   1.0     1249     11      HttpPartitionTest.test
     0 23   1.1     1210      8      HttpPartitionWithTlogReplicasTest.test
     0 23   0.5     1258      4      ShardSplitTest.testSplitShardWithRuleLink
     0 23   0.2     1231      4      TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
     0 23   0.2     1232      6      TestSolrConfigHandlerCloud.test
     0 2    0.3      767      2      DocValuesNotIndexedTest.testGroupingDVOnlySortLast
     0 2    0.3      750      2      TestLBHttp2SolrClient.testTwoServers
     0 2    0.3      794      2      TestSolrCloudSnapshots.testSnapshots
     0 2   40.7       51     12      TestXYMultiPolygonShapeQueries.testRandomBig
     0  3   0.2      842      2      ConnectionManagerTest.testLikelyExpired
     0  3   0.2      835      3      LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
     0  3   0.3      832      3      ReplaceNodeTest.test
     0  3   0.3      826      2      SolrCloudReportersTest.testExplicitConfiguration
     0  3   0.2      833      2      TestPullReplicaErrorHandling.testCantConnectToLeader
     0  3   2.4      867     21      TestRandomChains.testRandomChains
     0  3   0.5      830      3      TestWithCollection.testMoveReplicaMainCollection

ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!

     0      0.3      361      1      AutoAddReplicasIntegrationTest.testClusterPropOverridesCollecitonProp
     0      2.9      380     11      BasicDistributedZkTest.test
     0      0.2      404      1      BasicZkTest.testBasic
     0      0.2      414      1      ChaosMonkeySafeLeaderWithPullReplicasTest.test
     0     62.8      853    536      CheckHdfsIndexTest.doTest
     0      0.8      355      3      CloudHttp2SolrClientTest.checkCollectionParameters
     0      0.8      355      3      CloudHttp2SolrClientTest.customHttpClientTest
     0      1.1      355      4      CloudHttp2SolrClientTest.preferLocalShardsTest
     0      0.8      355      3      CloudHttp2SolrClientTest.preferReplicaTypesTest
     0      0.8      355      3      CloudHttp2SolrClientTest.singleShardedPreferenceRules
     0      0.8      355      3      CloudHttp2SolrClientTest.stateVersionParamTest
     0      1.1      355      4      CloudHttp2SolrClientTest.testAliasHandling
     0      0.8      355      3      CloudHttp2SolrClientTest.testCollectionDoesntExist
     0      0.8      355      3      CloudHttp2SolrClientTest.testInitializationWithSolrUrls
     0      0.8      355      3      CloudHttp2SolrClientTest.testNonRetryableRequests
     0      1.1      355      4      CloudHttp2SolrClientTest.testOverwriteOption
     0      0.8      355      3      CloudHttp2SolrClientTest.testParallelUpdateQTime
     0      0.6      355      2      CloudHttp2SolrClientTest.testRouting
     0      0.8      355      3      CloudHttp2SolrClientTest.testShutdown
     0      0.6      355      2      CloudHttp2SolrClientTest.testVersionsAreReturned
     0      0.8      355      3      CloudHttp2SolrClientTest.testWrongZkChrootTest
     0      0.5      392      2      CloudSolrClientTest.checkCollectionParameters
     0      0.3      392      1      CloudSolrClientTest.preferLocalShardsTest
     0      0.3      392      1      CloudSolrClientTest.preferReplicaTypesTest
     0      0.3      357      1      CloudSolrClientTest.singleShardedPreferenceRules
     0      0.5      392      2      CloudSolrClientTest.stateVersionParamTest
     0      0.5      392      2      CloudSolrClientTest.testAliasHandling
     0      0.3      392      1      CloudSolrClientTest.testInitializationWithSolrUrls
     0      0.3      392      1      CloudSolrClientTest.testNonRetryableRequests
     0      0.3      392      1      CloudSolrClientTest.testOverwriteOption
     0      0.3      392      1      CloudSolrClientTest.testParallelUpdateQTime
     0      0.3      357      1      CloudSolrClientTest.testPing
     0      0.5      392      2      CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale
     0      0.3      392      1      CloudSolrClientTest.testShutdown
     0      0.5      392      2      CloudSolrClientTest.testVersionsAreReturned
     0      0.3      392      1      CloudSolrClientTest.testWrongZkChrootTest
     0      0.3      399      1      CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls
     0      1.3      397      5      CollectionsAPIDistributedZkTest.addReplicaTest
     0      0.8      397      3      CollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk
     0      1.0      397      4      CollectionsAPIDistributedZkTest.deleteCollectionRemovesStaleZkCollectionsNode
     0      0.8      397      3      CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
     0      0.3      397      1      CollectionsAPIDistributedZkTest.testBadActionNames
     0      0.8      397      3      CollectionsAPIDistributedZkTest.testCollectionReload
     0      1.0      397      4      CollectionsAPIDistributedZkTest.testCollectionsAPI
     0      1.3      397      5      CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes
     0      1.0      397      4      CollectionsAPIDistributedZkTest.testCreateNodeSet
     0      1.3      397      5      CollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore
     0      0.3      397      1      CollectionsAPIDistributedZkTest.testCreationAndDeletion
     0      0.5      397      2      CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection
     0      0.8      397      3      CollectionsAPIDistributedZkTest.testMaxNodesPerShard
     0      0.8      397      3      CollectionsAPIDistributedZkTest.testMissingNumShards
     0      0.5      397      2      CollectionsAPIDistributedZkTest.testMissingRequiredParameters
     0      0.8      397      3      CollectionsAPIDistributedZkTest.testNoConfigSetExist
     0      0.8      397      3      CollectionsAPIDistributedZkTest.testSpecificConfigsets
     0      0.8      397      3      CollectionsAPIDistributedZkTest.testTooManyReplicas
     0      1.0      397      4      CollectionsAPIDistributedZkTest.testZeroNumShards
     0      0.7      405      3      CollectionsAPISolrJTest.testAddAndDeleteReplica
     0      0.2      405      1      CollectionsAPISolrJTest.testAddAndDeleteReplicaProp
     0      0.7      405      3      CollectionsAPISolrJTest.testBalanceShardUnique
     0      0.5      405      2      CollectionsAPISolrJTest.testCloudInfoInCoreStatus
     0      1.5      405      6      CollectionsAPISolrJTest.testClusterProp
     0      0.5      405      2      CollectionsAPISolrJTest.testCollectionProp
     0      0.5      405      2      CollectionsAPISolrJTest.testCreateAndDeleteAlias
     0      0.5      405      2      CollectionsAPISolrJTest.testCreateAndDeleteCollection
     0      0.7      405      3      CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesNewFormat
     0      1.7      405      7      CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesOldFormat
     0      0.7      405      3      CollectionsAPISolrJTest.testCreateCollectionWithPropertyParam
     0      1.5      405      6      CollectionsAPISolrJTest.testCreateWithDefaultConfigSet
     0      1.9      370      7      CollectionsAPISolrJTest.testDeleteAliasedCollection
     0      1.2      405      5      CollectionsAPISolrJTest.testList
     0      1.0      405      4      CollectionsAPISolrJTest.testModifyCollectionAttribute
     0      1.2      405      5      CollectionsAPISolrJTest.testOverseerStatus
     0      0.8      370      3      CollectionsAPISolrJTest.testReadOnlyCollection
     0      1.6      370      6      CollectionsAPISolrJTest.testRenameCollection
     0      0.7      405      3      CollectionsAPISolrJTest.testSplitShard
     0      0.5      366      2      ComputePlanActionTest.testNodeAdded
     0      1.1      376      4      DeleteReplicaTest.deleteLiveReplicaTest
     0      1.5      411      6      DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup
     0      0.5      411      2      DeleteReplicaTest.deleteReplicaByCount
     0      0.5      376      2      DeleteReplicaTest.deleteReplicaByCountForAllShards
     0      1.2      411      5      DeleteReplicaTest.deleteReplicaFromClusterState
     0      1.5      411      6      DeleteReplicaTest.deleteReplicaFromClusterStateLegacy
     0      1.7      411      7      DeleteReplicaTest.deleteReplicaOnIndexing
     0      0.8      376      3      DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
     0      1.3      376      5      DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy
     0      0.7      405      3      DeleteShardTest.test
     0      0.2      405      1      DeleteShardTest.testDirectoryCleanupAfterDeleteShard
     0      0.5      366      2      DistribDocExpirationUpdateProcessorTest.testBasicAuth
     0      0.5      366      2      DistribDocExpirationUpdateProcessorTest.testNoAuth
     0      0.2      402      1      DistributedQueueTest.testDistributedQueue
     0      0.7      402      3      DistributedQueueTest.testDistributedQueueBlocking
     0      0.7      402      3      DistributedQueueTest.testDistributedQueueCache
     0      0.2      402      1      DistributedQueueTest.testLeakChildWatcher
     0      0.5      402      2      DistributedQueueTest.testLocallyOffer
     0      0.7      402      3      DistributedQueueTest.testPeekElements
     0      0.2      403      1      ExecutePlanActionTest.testExecute
     0      0.5      403      2      ExecutePlanActionTest.testIntegration
     0      0.7      403      3      ExecutePlanActionTest.testTaskFail
     0      0.3      370      1      IndexSizeTriggerSizeEstimationTest.testEstimatedIndexSize
     0      0.3      366      1      IndexSizeTriggerTest.testInvalidSplitConfig
     0      0.3      366      1      IndexSizeTriggerTest.testMergeIntegration
     0      0.3      366      1      IndexSizeTriggerTest.testSplitConfig
     0      0.3      366      1      IndexSizeTriggerTest.testSplitIntegration
     0      0.3      366      1      IndexSizeTriggerTest.testTrigger
     0      0.3      366      1      IndexSizeTriggerTest.testValidProperties
     0      0.3      360      1      JWTAuthPluginIntegrationTest.infoRequestWithoutToken
     0      0.3      360      1      JWTAuthPluginIntegrationTest.testMetrics
     0      0.2      403      1      LeaderElectionTest.testBasic
     0      0.5      403      2      LeaderElectionTest.testCancelElection
     0      0.5      403      2      LeaderElectionTest.testElection
     0      0.5      403      2      LeaderElectionTest.testParallelElection
     0      0.5      403      2      LeaderElectionTest.testStressElection
     0      0.2      404      1      LeaderFailureAfterFreshStartTest.test
     0      0.3      367      1      MoveReplicaTest.testFailedMove
     0      0.3      362      1      NestedShardedAtomicUpdateTest.test
     0      0.5      400      2      NodeAddedTriggerIntegrationTest.testNodeAddedTriggerRestoreState
     0      0.3      398      1      NodeAddedTriggerTest.testListenerAcceptance
     0      0.3      398      1      NodeAddedTriggerTest.testRestoreState
     0      0.3      398      1      NodeAddedTriggerTest.testTrigger
     0      0.3      397      1      NodeLostTriggerIntegrationTest.testNodeLostTriggerRestoreState
     0      0.3      398      1      OutOfBoxZkACLAndCredentialsProvidersTest.testOpenACLUnsafeAllover
     0      0.3      398      1      OutOfBoxZkACLAndCredentialsProvidersTest.testOutOfBoxSolrZkClient
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testNoCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
     0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
     0      0.3      400      1      OverseerTaskQueueTest.testDistributedQueueCache
     0      0.3      400      1      OverseerTaskQueueTest.testLocallyOffer
     0      0.3      400      1      PingRequestHandlerTest.testPingInClusterWithNoHealthCheck
     0      0.5      398      2      RecoveryAfterSoftCommitTest.test
     0      0.5      396      2      ReplicationFactorTest.test
     0      0.7      412      3      ShardSplitTest.testSplitLocking
     0      0.8      377      3      ShardSplitTest.testSplitMixedReplicaTypes
     0      0.5      411      2      ShardSplitTest.testSplitStaticIndexReplication
     0      0.3      353      1      SolrZkClientTest.getConfig
     0      0.3      353      1      SolrZkClientTest.testWrappingWatches
     0      0.3      398      1      SpatialHeatmapFacetsTest.testClassicFacets
     0      0.3      398      1      SpatialHeatmapFacetsTest.testJsonFacets
     0      0.3     1194      3      SpatialHeatmapFacetsTest.testPng
     0      1.0      209      2      StreamDecoratorTest.testParallelTerminatingDaemonUpdateStream
     0      0.3      397      1      SystemLogListenerTest.test
     0      0.5      388      2      TestCloudCollectionsListeners.testCollectionDeletion
     0      0.3      388      1      TestCloudCollectionsListeners.testSimpleCloudCollectionsListener
     0      0.3      397      1      TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeaderAfterRestart
     0      0.3      396      1      TestCloudSearcherWarming.testPeersyncFailureReplicationSuccess
     0      0.8      354      3      TestCollectionStateWatchers.testCanWaitForNonexistantCollection
     0      0.8      354      3      TestCollectionStateWatchers.testCollectionWatchWithShutdownOfActiveNode
     0      0.8      354      3      TestCollectionStateWatchers.testCollectionWatchWithShutdownOfUnusedNode
     0      0.8      389      3      TestCollectionStateWatchers.testDeletionsTriggerWatches
     0      0.6      354      2      TestCollectionStateWatchers.testLiveNodeChangesTriggerWatches
     0      0.8      389      3      TestCollectionStateWatchers.testPredicateFailureTimesOut
     0      0.8      389      3      TestCollectionStateWatchers.testStateWatcherChecksCurrentStateOnRegister
     0      0.8      389      3      TestCollectionStateWatchers.testWaitForStateChecksCurrentState
     0      0.8      389      3      TestCollectionStateWatchers.testWaitForStateWatcherIsRetainedOnPredicateFailure
     0      0.5      389      2      TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout
     0      0.5      389      2      TestCollectionStateWatchers.testWatchesWorkForStateFormat1
     0      0.3      400      1      TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete
     0      0.3      399      1      TestConfigSetsAPI.testCreate
     0      0.3      399      1      TestConfigSetsAPI.testList
     0      0.3      399      1      TestConfigSetsAPI.testUpload
     0      0.3      399      1      TestConfigSetsAPI.testUploadDisabled
     0      0.3      399      1      TestConfigSetsAPI.testUploadErrors
     0      0.3      399      1      TestConfigSetsAPI.testUploadWithScriptUpdateProcessor
     0      0.3      399      1      TestConfigSetsAPI.testUserAndTestDefaultConfigsetsAreSame
     0      0.3      400      1      TestDistribDocBasedVersion.test
     0      0.3      354      1      TestDocCollectionWatcher.testCanWaitForNonexistantCollection
     0      0.6      354      2      TestDocCollectionWatcher.testDeletionsTriggerWatches
     0      0.6      354      2      TestDocCollectionWatcher.testPredicateFailureTimesOut
     0      0.6      354      2      TestDocCollectionWatcher.testStateWatcherChecksCurrentStateOnRegister
     0      0.6      354      2      TestDocCollectionWatcher.testWaitForStateChecksCurrentState
     0      0.6      354      2      TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFailure
     0      0.6      354      2      TestDocCollectionWatcher.testWatcherIsRemovedAfterTimeout
     0      0.6      354      2      TestDocCollectionWatcher.testWatchesWorkForStateFormat1
     0     14.3       21      3      TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit
     0      0.2      414      1      TestDuelingCodecs.testCrazyReaderEquals
     0      4.5       22      1      TestDuelingCodecsAtNight.testCrazyReaderEquals
     0      0.2      419      1      TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads
     0      0.3      352      1      TestJaegerConfigurator.testInjected
     0      0.3      387      1      TestLTROnSolrCloud.testSimpleQuery
     0      1.2      402      5      TestPointFields.testDoublePointStats
     0      0.2      402      1      TestPointFields.testIntPointFieldMultiValuedRangeFacet
     0      1.0      406      4      TestRandomDVFaceting.testRandomFaceting
     0      0.5      380      2      TestSimScenario.testAutoAddReplicas
     0      0.3      397      1      TestSimTriggerIntegration.testCooldown
     0      0.2      419      1      TestSimpleExplanationsWithFillerDocs.testMultiFieldBQofPQ5
     0      0.2      402      1      TestSolrConfigHandlerConcurrent.test
     0      0.5      366      2      TestWithCollection.testAddReplicaWithPolicy
     0      0.3      400      1      TestWithCollection.testCreateCollection
     0      0.3      400      1      TestWithCollection.testCreateCollectionNoWithCollection
     0      0.3      400      1      TestWithCollection.testDeleteWithCollection
     0      0.3      400      1      TestWithCollection.testMoveReplicaWithCollection
     0      0.3      366      1      TestWithCollection.testNodeAdded
     0      0.2      401      1      ZkCLITest.testInvalidZKAddress
     0      0.2      401      1      ZkCLITest.testPutFile
     0      0.3      394      1      ZkSolrClientTest.testMakeRootNode
     0      0.3      394      1      ZkSolrClientTest.testReconnect
     0      0.3      394      1      ZkSolrClientTest.testWatchChildren





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: PLEASE READ! BadApple report. Last week was horrible!

Erick Erickson
Mike:

I have no idea. Hoss’ rollups don’t link back to builds, they
just aggregate the results.

Not a huge deal if it’s something like this of course. Let’s just
say I’ve had my share or “moments” ;).

And unfortunately, the test failures are pretty rare on a
percentage basis, so it’s hard to tell.

I’m watching LUCENE-9191 and I’ll look back at Hoss’ rollups
a day after you push it and see if the failures disappear.

It’ll take a while for the fixes to roll through all the reporting.

Tell you what. I’ll try beasting one of the classes that fails a lot and then
try it again after you push LUCENE-9191 and we’ll go from there.

Thanks for getting into this so promptly!

Erick

> On May 4, 2020, at 9:10 AM, Michael McCandless <[hidden email]> wrote:
>
> Hi Erick,
>
> It's possible this was the root cause of many of the failures: https://issues.apache.org/jira/browse/LUCENE-9191
>
> Do these transient failures look something like this?
>
>    [junit4]    > Throwable #1: java.nio.charset.MalformedInputException: Input length = 1
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([172C6414BE5E2A2C:E5829DFC005A1F0]:0)
>    [junit4]    >        at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
>    [junit4]    >        at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
>    [junit4]    >        at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
>    [junit4]    >        at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
>    [junit4]    >        at java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
>    [junit4]    >        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
>    [junit4]    >        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
>    [junit4]    >        at org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175)
>    [junit4]    >        at org.apache.lucene.util.LineFileDocs.<init>(LineFileDocs.java:65)
>    [junit4]    >        at org.apache.lucene.util.LineFileDocs.<init>(LineFileDocs.java:69)
>
>
> If so, then it is likely the root cause ... I'm working on a fix.  Sorry!
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Mon, May 4, 2020 at 7:54 AM Erick Erickson <[hidden email]> wrote:
> I don’t know whether we had some temporary glitch that broke lots of tests and they’ve been fixed or we had a major regression, but this needs to be addressed ASAP if they’re still failing. See everything below the line "ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail. I’ll raise a JIRA if we can’t get some traction quickly here.
>
> Hey, stuff happens. there’s no problem with tests going totally weird for a while. If you can say “Oh, yeah, all those failures for class XYZ are probably fixed” that’s fine.
>
> Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)….
>
> Hoss’ rolllup for the last 24 hours is not encouraging in terms of the
> problem already being fixed. There are lots of failures in some
> classes, notably:
>
> CloudHttp2SolrClientTest
> CollectionsAPIDistributedZkTest
> DeleteReplicaTest
> TestDocCollectionWatcher
>
> Unfortunately, the failure rate is not very high so reliably
> reproducing is hard.
>
> I’ve reproduced the last week’s failure in this e-mail, full
> report attached.
>
> Here’s Hoss’ rollup:
> http://fucit.org/solr-jenkins-reports/failure-report.html
>
> Usual synopsis:
>
> Raw fail count by week totals, most recent week first (corresponds to bits):
> Week: 0  had  343 failures
> Week: 1  had  86 failures
> Week: 2  had  78 failures
> Week: 3  had  117 failures
>
>
> ********Failures in Hoss' reports for the last 4 rollups.
>
> There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
> These tests were NOT BadApple'd or AwaitsFix’d
>
> Failures in the last 4 reports..
>    Report   Pct     runs    fails           test
>      0123   0.7     1617     11      ConnectionManagerTest.testReconnectWhenZkDisappeared
>      0123   1.5     1606     12      ExecutePlanActionTest.testTaskTimeout
>      0123   1.6     1320     19      MultiThreadedOCPTest.test
>      0123   1.0     1620     13      RollingRestartTest.test
>      0123   1.2     1617     12      SearchRateTriggerTest.testWaitForElapsed
>      0123   3.8      119      7      ShardSplitTest.testSplitWithChaosMonkey
>      0123   0.3     1519      7      TestInPlaceUpdatesDistrib.test
>      0123   0.7     1629     14      TestIndexWriterDelete.testDeleteAllNoDeadLock
>      0123   2.4     1548     18      TestPackages.testPluginLoading
>      0123   0.3     1587      4      UnloadDistributedZkTest.test
> ********************************************
>
> FAILURES IN THE LAST WEEK (343!)
> Look particularly at the ones with only a zero in the “Report” column, those are
> failures that were _not_ in the previous 3 week’s rollups.
>
>    Report   Pct     runs    fails           test
>      012    0.5     1165      4      CustomHighlightComponentTest.test
>      012    1.0     1168      6      NodeMarkersRegistrationTest.testNodeMarkersRegistration
>      012    1.0     1170      8      TestCryptoKeys.test
>      01 3   0.7     1233     11      LeaderFailoverAfterPartitionTest.test
>      01 3  63.2      102     39      StressHdfsTest.test
>      01     0.3      709      2      ScheduledTriggerIntegrationTest.testScheduledTrigger
>      01     0.2      768      2      ShardRoutingTest.test
>      01     2.6      807     22      TestAllFilesHaveChecksumFooter.test
>      01     2.6      808     22      TestAllFilesHaveCodecHeader.test
>      01     0.2      769      2      TestCloudSchemaless.test
>      01     0.2      769      2      TestDynamicLoading.testDynamicLoading
>      01     0.3      707      2      TestDynamicLoadingUrl.testDynamicLoadingUrl
>      01     0.5      767      4      TestPointFields.testFloatPointStats
>      01    27.1       83     19      TestSQLHandler.doTest
>      01     0.2      794     12      TestSameScoresWithThreads.test
>      01     2.6      806     22      TestShardSearching.testSimple
>      01     0.5      726      4      TestSimScenario.testSplitShard
>      01     1.1      726      7      TestSimScenario.testSuggestions
>      01     0.3      771      2      TestWithCollection.testAddReplicaSimple
>      0 23   0.3     1223      4      CdcrVersionReplicationTest.testCdcrDocVersions
>      0 23   0.8     1172      6      CloudHttp2SolrClientTest.testRetryUpdatesWhenClusterStateIsStale
>      0 23   1.4     1202      8      CollectionsAPISolrJTest.testColStatus
>      0 23   1.0     1249     11      HttpPartitionTest.test
>      0 23   1.1     1210      8      HttpPartitionWithTlogReplicasTest.test
>      0 23   0.5     1258      4      ShardSplitTest.testSplitShardWithRuleLink
>      0 23   0.2     1231      4      TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
>      0 23   0.2     1232      6      TestSolrConfigHandlerCloud.test
>      0 2    0.3      767      2      DocValuesNotIndexedTest.testGroupingDVOnlySortLast
>      0 2    0.3      750      2      TestLBHttp2SolrClient.testTwoServers
>      0 2    0.3      794      2      TestSolrCloudSnapshots.testSnapshots
>      0 2   40.7       51     12      TestXYMultiPolygonShapeQueries.testRandomBig
>      0  3   0.2      842      2      ConnectionManagerTest.testLikelyExpired
>      0  3   0.2      835      3      LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
>      0  3   0.3      832      3      ReplaceNodeTest.test
>      0  3   0.3      826      2      SolrCloudReportersTest.testExplicitConfiguration
>      0  3   0.2      833      2      TestPullReplicaErrorHandling.testCantConnectToLeader
>      0  3   2.4      867     21      TestRandomChains.testRandomChains
>      0  3   0.5      830      3      TestWithCollection.testMoveReplicaMainCollection
>
> ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!
>
>      0      0.3      361      1      AutoAddReplicasIntegrationTest.testClusterPropOverridesCollecitonProp
>      0      2.9      380     11      BasicDistributedZkTest.test
>      0      0.2      404      1      BasicZkTest.testBasic
>      0      0.2      414      1      ChaosMonkeySafeLeaderWithPullReplicasTest.test
>      0     62.8      853    536      CheckHdfsIndexTest.doTest
>      0      0.8      355      3      CloudHttp2SolrClientTest.checkCollectionParameters
>      0      0.8      355      3      CloudHttp2SolrClientTest.customHttpClientTest
>      0      1.1      355      4      CloudHttp2SolrClientTest.preferLocalShardsTest
>      0      0.8      355      3      CloudHttp2SolrClientTest.preferReplicaTypesTest
>      0      0.8      355      3      CloudHttp2SolrClientTest.singleShardedPreferenceRules
>      0      0.8      355      3      CloudHttp2SolrClientTest.stateVersionParamTest
>      0      1.1      355      4      CloudHttp2SolrClientTest.testAliasHandling
>      0      0.8      355      3      CloudHttp2SolrClientTest.testCollectionDoesntExist
>      0      0.8      355      3      CloudHttp2SolrClientTest.testInitializationWithSolrUrls
>      0      0.8      355      3      CloudHttp2SolrClientTest.testNonRetryableRequests
>      0      1.1      355      4      CloudHttp2SolrClientTest.testOverwriteOption
>      0      0.8      355      3      CloudHttp2SolrClientTest.testParallelUpdateQTime
>      0      0.6      355      2      CloudHttp2SolrClientTest.testRouting
>      0      0.8      355      3      CloudHttp2SolrClientTest.testShutdown
>      0      0.6      355      2      CloudHttp2SolrClientTest.testVersionsAreReturned
>      0      0.8      355      3      CloudHttp2SolrClientTest.testWrongZkChrootTest
>      0      0.5      392      2      CloudSolrClientTest.checkCollectionParameters
>      0      0.3      392      1      CloudSolrClientTest.preferLocalShardsTest
>      0      0.3      392      1      CloudSolrClientTest.preferReplicaTypesTest
>      0      0.3      357      1      CloudSolrClientTest.singleShardedPreferenceRules
>      0      0.5      392      2      CloudSolrClientTest.stateVersionParamTest
>      0      0.5      392      2      CloudSolrClientTest.testAliasHandling
>      0      0.3      392      1      CloudSolrClientTest.testInitializationWithSolrUrls
>      0      0.3      392      1      CloudSolrClientTest.testNonRetryableRequests
>      0      0.3      392      1      CloudSolrClientTest.testOverwriteOption
>      0      0.3      392      1      CloudSolrClientTest.testParallelUpdateQTime
>      0      0.3      357      1      CloudSolrClientTest.testPing
>      0      0.5      392      2      CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale
>      0      0.3      392      1      CloudSolrClientTest.testShutdown
>      0      0.5      392      2      CloudSolrClientTest.testVersionsAreReturned
>      0      0.3      392      1      CloudSolrClientTest.testWrongZkChrootTest
>      0      0.3      399      1      CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls
>      0      1.3      397      5      CollectionsAPIDistributedZkTest.addReplicaTest
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk
>      0      1.0      397      4      CollectionsAPIDistributedZkTest.deleteCollectionRemovesStaleZkCollectionsNode
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
>      0      0.3      397      1      CollectionsAPIDistributedZkTest.testBadActionNames
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.testCollectionReload
>      0      1.0      397      4      CollectionsAPIDistributedZkTest.testCollectionsAPI
>      0      1.3      397      5      CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes
>      0      1.0      397      4      CollectionsAPIDistributedZkTest.testCreateNodeSet
>      0      1.3      397      5      CollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore
>      0      0.3      397      1      CollectionsAPIDistributedZkTest.testCreationAndDeletion
>      0      0.5      397      2      CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.testMaxNodesPerShard
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.testMissingNumShards
>      0      0.5      397      2      CollectionsAPIDistributedZkTest.testMissingRequiredParameters
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.testNoConfigSetExist
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.testSpecificConfigsets
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.testTooManyReplicas
>      0      1.0      397      4      CollectionsAPIDistributedZkTest.testZeroNumShards
>      0      0.7      405      3      CollectionsAPISolrJTest.testAddAndDeleteReplica
>      0      0.2      405      1      CollectionsAPISolrJTest.testAddAndDeleteReplicaProp
>      0      0.7      405      3      CollectionsAPISolrJTest.testBalanceShardUnique
>      0      0.5      405      2      CollectionsAPISolrJTest.testCloudInfoInCoreStatus
>      0      1.5      405      6      CollectionsAPISolrJTest.testClusterProp
>      0      0.5      405      2      CollectionsAPISolrJTest.testCollectionProp
>      0      0.5      405      2      CollectionsAPISolrJTest.testCreateAndDeleteAlias
>      0      0.5      405      2      CollectionsAPISolrJTest.testCreateAndDeleteCollection
>      0      0.7      405      3      CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesNewFormat
>      0      1.7      405      7      CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesOldFormat
>      0      0.7      405      3      CollectionsAPISolrJTest.testCreateCollectionWithPropertyParam
>      0      1.5      405      6      CollectionsAPISolrJTest.testCreateWithDefaultConfigSet
>      0      1.9      370      7      CollectionsAPISolrJTest.testDeleteAliasedCollection
>      0      1.2      405      5      CollectionsAPISolrJTest.testList
>      0      1.0      405      4      CollectionsAPISolrJTest.testModifyCollectionAttribute
>      0      1.2      405      5      CollectionsAPISolrJTest.testOverseerStatus
>      0      0.8      370      3      CollectionsAPISolrJTest.testReadOnlyCollection
>      0      1.6      370      6      CollectionsAPISolrJTest.testRenameCollection
>      0      0.7      405      3      CollectionsAPISolrJTest.testSplitShard
>      0      0.5      366      2      ComputePlanActionTest.testNodeAdded
>      0      1.1      376      4      DeleteReplicaTest.deleteLiveReplicaTest
>      0      1.5      411      6      DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup
>      0      0.5      411      2      DeleteReplicaTest.deleteReplicaByCount
>      0      0.5      376      2      DeleteReplicaTest.deleteReplicaByCountForAllShards
>      0      1.2      411      5      DeleteReplicaTest.deleteReplicaFromClusterState
>      0      1.5      411      6      DeleteReplicaTest.deleteReplicaFromClusterStateLegacy
>      0      1.7      411      7      DeleteReplicaTest.deleteReplicaOnIndexing
>      0      0.8      376      3      DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
>      0      1.3      376      5      DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy
>      0      0.7      405      3      DeleteShardTest.test
>      0      0.2      405      1      DeleteShardTest.testDirectoryCleanupAfterDeleteShard
>      0      0.5      366      2      DistribDocExpirationUpdateProcessorTest.testBasicAuth
>      0      0.5      366      2      DistribDocExpirationUpdateProcessorTest.testNoAuth
>      0      0.2      402      1      DistributedQueueTest.testDistributedQueue
>      0      0.7      402      3      DistributedQueueTest.testDistributedQueueBlocking
>      0      0.7      402      3      DistributedQueueTest.testDistributedQueueCache
>      0      0.2      402      1      DistributedQueueTest.testLeakChildWatcher
>      0      0.5      402      2      DistributedQueueTest.testLocallyOffer
>      0      0.7      402      3      DistributedQueueTest.testPeekElements
>      0      0.2      403      1      ExecutePlanActionTest.testExecute
>      0      0.5      403      2      ExecutePlanActionTest.testIntegration
>      0      0.7      403      3      ExecutePlanActionTest.testTaskFail
>      0      0.3      370      1      IndexSizeTriggerSizeEstimationTest.testEstimatedIndexSize
>      0      0.3      366      1      IndexSizeTriggerTest.testInvalidSplitConfig
>      0      0.3      366      1      IndexSizeTriggerTest.testMergeIntegration
>      0      0.3      366      1      IndexSizeTriggerTest.testSplitConfig
>      0      0.3      366      1      IndexSizeTriggerTest.testSplitIntegration
>      0      0.3      366      1      IndexSizeTriggerTest.testTrigger
>      0      0.3      366      1      IndexSizeTriggerTest.testValidProperties
>      0      0.3      360      1      JWTAuthPluginIntegrationTest.infoRequestWithoutToken
>      0      0.3      360      1      JWTAuthPluginIntegrationTest.testMetrics
>      0      0.2      403      1      LeaderElectionTest.testBasic
>      0      0.5      403      2      LeaderElectionTest.testCancelElection
>      0      0.5      403      2      LeaderElectionTest.testElection
>      0      0.5      403      2      LeaderElectionTest.testParallelElection
>      0      0.5      403      2      LeaderElectionTest.testStressElection
>      0      0.2      404      1      LeaderFailureAfterFreshStartTest.test
>      0      0.3      367      1      MoveReplicaTest.testFailedMove
>      0      0.3      362      1      NestedShardedAtomicUpdateTest.test
>      0      0.5      400      2      NodeAddedTriggerIntegrationTest.testNodeAddedTriggerRestoreState
>      0      0.3      398      1      NodeAddedTriggerTest.testListenerAcceptance
>      0      0.3      398      1      NodeAddedTriggerTest.testRestoreState
>      0      0.3      398      1      NodeAddedTriggerTest.testTrigger
>      0      0.3      397      1      NodeLostTriggerIntegrationTest.testNodeLostTriggerRestoreState
>      0      0.3      398      1      OutOfBoxZkACLAndCredentialsProvidersTest.testOpenACLUnsafeAllover
>      0      0.3      398      1      OutOfBoxZkACLAndCredentialsProvidersTest.testOutOfBoxSolrZkClient
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testNoCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
>      0      0.3      400      1      OverseerTaskQueueTest.testDistributedQueueCache
>      0      0.3      400      1      OverseerTaskQueueTest.testLocallyOffer
>      0      0.3      400      1      PingRequestHandlerTest.testPingInClusterWithNoHealthCheck
>      0      0.5      398      2      RecoveryAfterSoftCommitTest.test
>      0      0.5      396      2      ReplicationFactorTest.test
>      0      0.7      412      3      ShardSplitTest.testSplitLocking
>      0      0.8      377      3      ShardSplitTest.testSplitMixedReplicaTypes
>      0      0.5      411      2      ShardSplitTest.testSplitStaticIndexReplication
>      0      0.3      353      1      SolrZkClientTest.getConfig
>      0      0.3      353      1      SolrZkClientTest.testWrappingWatches
>      0      0.3      398      1      SpatialHeatmapFacetsTest.testClassicFacets
>      0      0.3      398      1      SpatialHeatmapFacetsTest.testJsonFacets
>      0      0.3     1194      3      SpatialHeatmapFacetsTest.testPng
>      0      1.0      209      2      StreamDecoratorTest.testParallelTerminatingDaemonUpdateStream
>      0      0.3      397      1      SystemLogListenerTest.test
>      0      0.5      388      2      TestCloudCollectionsListeners.testCollectionDeletion
>      0      0.3      388      1      TestCloudCollectionsListeners.testSimpleCloudCollectionsListener
>      0      0.3      397      1      TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeaderAfterRestart
>      0      0.3      396      1      TestCloudSearcherWarming.testPeersyncFailureReplicationSuccess
>      0      0.8      354      3      TestCollectionStateWatchers.testCanWaitForNonexistantCollection
>      0      0.8      354      3      TestCollectionStateWatchers.testCollectionWatchWithShutdownOfActiveNode
>      0      0.8      354      3      TestCollectionStateWatchers.testCollectionWatchWithShutdownOfUnusedNode
>      0      0.8      389      3      TestCollectionStateWatchers.testDeletionsTriggerWatches
>      0      0.6      354      2      TestCollectionStateWatchers.testLiveNodeChangesTriggerWatches
>      0      0.8      389      3      TestCollectionStateWatchers.testPredicateFailureTimesOut
>      0      0.8      389      3      TestCollectionStateWatchers.testStateWatcherChecksCurrentStateOnRegister
>      0      0.8      389      3      TestCollectionStateWatchers.testWaitForStateChecksCurrentState
>      0      0.8      389      3      TestCollectionStateWatchers.testWaitForStateWatcherIsRetainedOnPredicateFailure
>      0      0.5      389      2      TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout
>      0      0.5      389      2      TestCollectionStateWatchers.testWatchesWorkForStateFormat1
>      0      0.3      400      1      TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete
>      0      0.3      399      1      TestConfigSetsAPI.testCreate
>      0      0.3      399      1      TestConfigSetsAPI.testList
>      0      0.3      399      1      TestConfigSetsAPI.testUpload
>      0      0.3      399      1      TestConfigSetsAPI.testUploadDisabled
>      0      0.3      399      1      TestConfigSetsAPI.testUploadErrors
>      0      0.3      399      1      TestConfigSetsAPI.testUploadWithScriptUpdateProcessor
>      0      0.3      399      1      TestConfigSetsAPI.testUserAndTestDefaultConfigsetsAreSame
>      0      0.3      400      1      TestDistribDocBasedVersion.test
>      0      0.3      354      1      TestDocCollectionWatcher.testCanWaitForNonexistantCollection
>      0      0.6      354      2      TestDocCollectionWatcher.testDeletionsTriggerWatches
>      0      0.6      354      2      TestDocCollectionWatcher.testPredicateFailureTimesOut
>      0      0.6      354      2      TestDocCollectionWatcher.testStateWatcherChecksCurrentStateOnRegister
>      0      0.6      354      2      TestDocCollectionWatcher.testWaitForStateChecksCurrentState
>      0      0.6      354      2      TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFailure
>      0      0.6      354      2      TestDocCollectionWatcher.testWatcherIsRemovedAfterTimeout
>      0      0.6      354      2      TestDocCollectionWatcher.testWatchesWorkForStateFormat1
>      0     14.3       21      3      TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit
>      0      0.2      414      1      TestDuelingCodecs.testCrazyReaderEquals
>      0      4.5       22      1      TestDuelingCodecsAtNight.testCrazyReaderEquals
>      0      0.2      419      1      TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads
>      0      0.3      352      1      TestJaegerConfigurator.testInjected
>      0      0.3      387      1      TestLTROnSolrCloud.testSimpleQuery
>      0      1.2      402      5      TestPointFields.testDoublePointStats
>      0      0.2      402      1      TestPointFields.testIntPointFieldMultiValuedRangeFacet
>      0      1.0      406      4      TestRandomDVFaceting.testRandomFaceting
>      0      0.5      380      2      TestSimScenario.testAutoAddReplicas
>      0      0.3      397      1      TestSimTriggerIntegration.testCooldown
>      0      0.2      419      1      TestSimpleExplanationsWithFillerDocs.testMultiFieldBQofPQ5
>      0      0.2      402      1      TestSolrConfigHandlerConcurrent.test
>      0      0.5      366      2      TestWithCollection.testAddReplicaWithPolicy
>      0      0.3      400      1      TestWithCollection.testCreateCollection
>      0      0.3      400      1      TestWithCollection.testCreateCollectionNoWithCollection
>      0      0.3      400      1      TestWithCollection.testDeleteWithCollection
>      0      0.3      400      1      TestWithCollection.testMoveReplicaWithCollection
>      0      0.3      366      1      TestWithCollection.testNodeAdded
>      0      0.2      401      1      ZkCLITest.testInvalidZKAddress
>      0      0.2      401      1      ZkCLITest.testPutFile
>      0      0.3      394      1      ZkSolrClientTest.testMakeRootNode
>      0      0.3      394      1      ZkSolrClientTest.testReconnect
>      0      0.3      394      1      ZkSolrClientTest.testWatchChildren
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: PLEASE READ! BadApple report. Last week was horrible!

Michael McCandless-2
Hi Erick,

OK I pushed a fix!  See if it decreases the failure rate for those newly bad apples?

Sorry and thanks :)

On Mon, May 4, 2020 at 1:06 PM Erick Erickson <[hidden email]> wrote:
Mike:

I have no idea. Hoss’ rollups don’t link back to builds, they
just aggregate the results.

Not a huge deal if it’s something like this of course. Let’s just
say I’ve had my share or “moments” ;).

And unfortunately, the test failures are pretty rare on a
percentage basis, so it’s hard to tell.

I’m watching LUCENE-9191 and I’ll look back at Hoss’ rollups
a day after you push it and see if the failures disappear.

It’ll take a while for the fixes to roll through all the reporting.

Tell you what. I’ll try beasting one of the classes that fails a lot and then
try it again after you push LUCENE-9191 and we’ll go from there.

Thanks for getting into this so promptly!

Erick

> On May 4, 2020, at 9:10 AM, Michael McCandless <[hidden email]> wrote:
>
> Hi Erick,
>
> It's possible this was the root cause of many of the failures: https://issues.apache.org/jira/browse/LUCENE-9191
>
> Do these transient failures look something like this?
>
>    [junit4]    > Throwable #1: java.nio.charset.MalformedInputException: Input length = 1
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([172C6414BE5E2A2C:E5829DFC005A1F0]:0)
>    [junit4]    >        at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
>    [junit4]    >        at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
>    [junit4]    >        at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
>    [junit4]    >        at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
>    [junit4]    >        at java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
>    [junit4]    >        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
>    [junit4]    >        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
>    [junit4]    >        at org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175)
>    [junit4]    >        at org.apache.lucene.util.LineFileDocs.<init>(LineFileDocs.java:65)
>    [junit4]    >        at org.apache.lucene.util.LineFileDocs.<init>(LineFileDocs.java:69)
>
>
> If so, then it is likely the root cause ... I'm working on a fix.  Sorry!
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Mon, May 4, 2020 at 7:54 AM Erick Erickson <[hidden email]> wrote:
> I don’t know whether we had some temporary glitch that broke lots of tests and they’ve been fixed or we had a major regression, but this needs to be addressed ASAP if they’re still failing. See everything below the line "ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail. I’ll raise a JIRA if we can’t get some traction quickly here.
>
> Hey, stuff happens. there’s no problem with tests going totally weird for a while. If you can say “Oh, yeah, all those failures for class XYZ are probably fixed” that’s fine.
>
> Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)….
>
> Hoss’ rolllup for the last 24 hours is not encouraging in terms of the
> problem already being fixed. There are lots of failures in some
> classes, notably:
>
> CloudHttp2SolrClientTest
> CollectionsAPIDistributedZkTest
> DeleteReplicaTest
> TestDocCollectionWatcher
>
> Unfortunately, the failure rate is not very high so reliably
> reproducing is hard.
>
> I’ve reproduced the last week’s failure in this e-mail, full
> report attached.
>
> Here’s Hoss’ rollup:
> http://fucit.org/solr-jenkins-reports/failure-report.html
>
> Usual synopsis:
>
> Raw fail count by week totals, most recent week first (corresponds to bits):
> Week: 0  had  343 failures
> Week: 1  had  86 failures
> Week: 2  had  78 failures
> Week: 3  had  117 failures
>
>
> ********Failures in Hoss' reports for the last 4 rollups.
>
> There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
> These tests were NOT BadApple'd or AwaitsFix’d
>
> Failures in the last 4 reports..
>    Report   Pct     runs    fails           test
>      0123   0.7     1617     11      ConnectionManagerTest.testReconnectWhenZkDisappeared
>      0123   1.5     1606     12      ExecutePlanActionTest.testTaskTimeout
>      0123   1.6     1320     19      MultiThreadedOCPTest.test
>      0123   1.0     1620     13      RollingRestartTest.test
>      0123   1.2     1617     12      SearchRateTriggerTest.testWaitForElapsed
>      0123   3.8      119      7      ShardSplitTest.testSplitWithChaosMonkey
>      0123   0.3     1519      7      TestInPlaceUpdatesDistrib.test
>      0123   0.7     1629     14      TestIndexWriterDelete.testDeleteAllNoDeadLock
>      0123   2.4     1548     18      TestPackages.testPluginLoading
>      0123   0.3     1587      4      UnloadDistributedZkTest.test
> ********************************************
>
> FAILURES IN THE LAST WEEK (343!)
> Look particularly at the ones with only a zero in the “Report” column, those are
> failures that were _not_ in the previous 3 week’s rollups.
>
>    Report   Pct     runs    fails           test
>      012    0.5     1165      4      CustomHighlightComponentTest.test
>      012    1.0     1168      6      NodeMarkersRegistrationTest.testNodeMarkersRegistration
>      012    1.0     1170      8      TestCryptoKeys.test
>      01 3   0.7     1233     11      LeaderFailoverAfterPartitionTest.test
>      01 3  63.2      102     39      StressHdfsTest.test
>      01     0.3      709      2      ScheduledTriggerIntegrationTest.testScheduledTrigger
>      01     0.2      768      2      ShardRoutingTest.test
>      01     2.6      807     22      TestAllFilesHaveChecksumFooter.test
>      01     2.6      808     22      TestAllFilesHaveCodecHeader.test
>      01     0.2      769      2      TestCloudSchemaless.test
>      01     0.2      769      2      TestDynamicLoading.testDynamicLoading
>      01     0.3      707      2      TestDynamicLoadingUrl.testDynamicLoadingUrl
>      01     0.5      767      4      TestPointFields.testFloatPointStats
>      01    27.1       83     19      TestSQLHandler.doTest
>      01     0.2      794     12      TestSameScoresWithThreads.test
>      01     2.6      806     22      TestShardSearching.testSimple
>      01     0.5      726      4      TestSimScenario.testSplitShard
>      01     1.1      726      7      TestSimScenario.testSuggestions
>      01     0.3      771      2      TestWithCollection.testAddReplicaSimple
>      0 23   0.3     1223      4      CdcrVersionReplicationTest.testCdcrDocVersions
>      0 23   0.8     1172      6      CloudHttp2SolrClientTest.testRetryUpdatesWhenClusterStateIsStale
>      0 23   1.4     1202      8      CollectionsAPISolrJTest.testColStatus
>      0 23   1.0     1249     11      HttpPartitionTest.test
>      0 23   1.1     1210      8      HttpPartitionWithTlogReplicasTest.test
>      0 23   0.5     1258      4      ShardSplitTest.testSplitShardWithRuleLink
>      0 23   0.2     1231      4      TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
>      0 23   0.2     1232      6      TestSolrConfigHandlerCloud.test
>      0 2    0.3      767      2      DocValuesNotIndexedTest.testGroupingDVOnlySortLast
>      0 2    0.3      750      2      TestLBHttp2SolrClient.testTwoServers
>      0 2    0.3      794      2      TestSolrCloudSnapshots.testSnapshots
>      0 2   40.7       51     12      TestXYMultiPolygonShapeQueries.testRandomBig
>      0  3   0.2      842      2      ConnectionManagerTest.testLikelyExpired
>      0  3   0.2      835      3      LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
>      0  3   0.3      832      3      ReplaceNodeTest.test
>      0  3   0.3      826      2      SolrCloudReportersTest.testExplicitConfiguration
>      0  3   0.2      833      2      TestPullReplicaErrorHandling.testCantConnectToLeader
>      0  3   2.4      867     21      TestRandomChains.testRandomChains
>      0  3   0.5      830      3      TestWithCollection.testMoveReplicaMainCollection
>
> ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!
>
>      0      0.3      361      1      AutoAddReplicasIntegrationTest.testClusterPropOverridesCollecitonProp
>      0      2.9      380     11      BasicDistributedZkTest.test
>      0      0.2      404      1      BasicZkTest.testBasic
>      0      0.2      414      1      ChaosMonkeySafeLeaderWithPullReplicasTest.test
>      0     62.8      853    536      CheckHdfsIndexTest.doTest
>      0      0.8      355      3      CloudHttp2SolrClientTest.checkCollectionParameters
>      0      0.8      355      3      CloudHttp2SolrClientTest.customHttpClientTest
>      0      1.1      355      4      CloudHttp2SolrClientTest.preferLocalShardsTest
>      0      0.8      355      3      CloudHttp2SolrClientTest.preferReplicaTypesTest
>      0      0.8      355      3      CloudHttp2SolrClientTest.singleShardedPreferenceRules
>      0      0.8      355      3      CloudHttp2SolrClientTest.stateVersionParamTest
>      0      1.1      355      4      CloudHttp2SolrClientTest.testAliasHandling
>      0      0.8      355      3      CloudHttp2SolrClientTest.testCollectionDoesntExist
>      0      0.8      355      3      CloudHttp2SolrClientTest.testInitializationWithSolrUrls
>      0      0.8      355      3      CloudHttp2SolrClientTest.testNonRetryableRequests
>      0      1.1      355      4      CloudHttp2SolrClientTest.testOverwriteOption
>      0      0.8      355      3      CloudHttp2SolrClientTest.testParallelUpdateQTime
>      0      0.6      355      2      CloudHttp2SolrClientTest.testRouting
>      0      0.8      355      3      CloudHttp2SolrClientTest.testShutdown
>      0      0.6      355      2      CloudHttp2SolrClientTest.testVersionsAreReturned
>      0      0.8      355      3      CloudHttp2SolrClientTest.testWrongZkChrootTest
>      0      0.5      392      2      CloudSolrClientTest.checkCollectionParameters
>      0      0.3      392      1      CloudSolrClientTest.preferLocalShardsTest
>      0      0.3      392      1      CloudSolrClientTest.preferReplicaTypesTest
>      0      0.3      357      1      CloudSolrClientTest.singleShardedPreferenceRules
>      0      0.5      392      2      CloudSolrClientTest.stateVersionParamTest
>      0      0.5      392      2      CloudSolrClientTest.testAliasHandling
>      0      0.3      392      1      CloudSolrClientTest.testInitializationWithSolrUrls
>      0      0.3      392      1      CloudSolrClientTest.testNonRetryableRequests
>      0      0.3      392      1      CloudSolrClientTest.testOverwriteOption
>      0      0.3      392      1      CloudSolrClientTest.testParallelUpdateQTime
>      0      0.3      357      1      CloudSolrClientTest.testPing
>      0      0.5      392      2      CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale
>      0      0.3      392      1      CloudSolrClientTest.testShutdown
>      0      0.5      392      2      CloudSolrClientTest.testVersionsAreReturned
>      0      0.3      392      1      CloudSolrClientTest.testWrongZkChrootTest
>      0      0.3      399      1      CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls
>      0      1.3      397      5      CollectionsAPIDistributedZkTest.addReplicaTest
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk
>      0      1.0      397      4      CollectionsAPIDistributedZkTest.deleteCollectionRemovesStaleZkCollectionsNode
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
>      0      0.3      397      1      CollectionsAPIDistributedZkTest.testBadActionNames
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.testCollectionReload
>      0      1.0      397      4      CollectionsAPIDistributedZkTest.testCollectionsAPI
>      0      1.3      397      5      CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes
>      0      1.0      397      4      CollectionsAPIDistributedZkTest.testCreateNodeSet
>      0      1.3      397      5      CollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore
>      0      0.3      397      1      CollectionsAPIDistributedZkTest.testCreationAndDeletion
>      0      0.5      397      2      CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.testMaxNodesPerShard
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.testMissingNumShards
>      0      0.5      397      2      CollectionsAPIDistributedZkTest.testMissingRequiredParameters
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.testNoConfigSetExist
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.testSpecificConfigsets
>      0      0.8      397      3      CollectionsAPIDistributedZkTest.testTooManyReplicas
>      0      1.0      397      4      CollectionsAPIDistributedZkTest.testZeroNumShards
>      0      0.7      405      3      CollectionsAPISolrJTest.testAddAndDeleteReplica
>      0      0.2      405      1      CollectionsAPISolrJTest.testAddAndDeleteReplicaProp
>      0      0.7      405      3      CollectionsAPISolrJTest.testBalanceShardUnique
>      0      0.5      405      2      CollectionsAPISolrJTest.testCloudInfoInCoreStatus
>      0      1.5      405      6      CollectionsAPISolrJTest.testClusterProp
>      0      0.5      405      2      CollectionsAPISolrJTest.testCollectionProp
>      0      0.5      405      2      CollectionsAPISolrJTest.testCreateAndDeleteAlias
>      0      0.5      405      2      CollectionsAPISolrJTest.testCreateAndDeleteCollection
>      0      0.7      405      3      CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesNewFormat
>      0      1.7      405      7      CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesOldFormat
>      0      0.7      405      3      CollectionsAPISolrJTest.testCreateCollectionWithPropertyParam
>      0      1.5      405      6      CollectionsAPISolrJTest.testCreateWithDefaultConfigSet
>      0      1.9      370      7      CollectionsAPISolrJTest.testDeleteAliasedCollection
>      0      1.2      405      5      CollectionsAPISolrJTest.testList
>      0      1.0      405      4      CollectionsAPISolrJTest.testModifyCollectionAttribute
>      0      1.2      405      5      CollectionsAPISolrJTest.testOverseerStatus
>      0      0.8      370      3      CollectionsAPISolrJTest.testReadOnlyCollection
>      0      1.6      370      6      CollectionsAPISolrJTest.testRenameCollection
>      0      0.7      405      3      CollectionsAPISolrJTest.testSplitShard
>      0      0.5      366      2      ComputePlanActionTest.testNodeAdded
>      0      1.1      376      4      DeleteReplicaTest.deleteLiveReplicaTest
>      0      1.5      411      6      DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup
>      0      0.5      411      2      DeleteReplicaTest.deleteReplicaByCount
>      0      0.5      376      2      DeleteReplicaTest.deleteReplicaByCountForAllShards
>      0      1.2      411      5      DeleteReplicaTest.deleteReplicaFromClusterState
>      0      1.5      411      6      DeleteReplicaTest.deleteReplicaFromClusterStateLegacy
>      0      1.7      411      7      DeleteReplicaTest.deleteReplicaOnIndexing
>      0      0.8      376      3      DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
>      0      1.3      376      5      DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy
>      0      0.7      405      3      DeleteShardTest.test
>      0      0.2      405      1      DeleteShardTest.testDirectoryCleanupAfterDeleteShard
>      0      0.5      366      2      DistribDocExpirationUpdateProcessorTest.testBasicAuth
>      0      0.5      366      2      DistribDocExpirationUpdateProcessorTest.testNoAuth
>      0      0.2      402      1      DistributedQueueTest.testDistributedQueue
>      0      0.7      402      3      DistributedQueueTest.testDistributedQueueBlocking
>      0      0.7      402      3      DistributedQueueTest.testDistributedQueueCache
>      0      0.2      402      1      DistributedQueueTest.testLeakChildWatcher
>      0      0.5      402      2      DistributedQueueTest.testLocallyOffer
>      0      0.7      402      3      DistributedQueueTest.testPeekElements
>      0      0.2      403      1      ExecutePlanActionTest.testExecute
>      0      0.5      403      2      ExecutePlanActionTest.testIntegration
>      0      0.7      403      3      ExecutePlanActionTest.testTaskFail
>      0      0.3      370      1      IndexSizeTriggerSizeEstimationTest.testEstimatedIndexSize
>      0      0.3      366      1      IndexSizeTriggerTest.testInvalidSplitConfig
>      0      0.3      366      1      IndexSizeTriggerTest.testMergeIntegration
>      0      0.3      366      1      IndexSizeTriggerTest.testSplitConfig
>      0      0.3      366      1      IndexSizeTriggerTest.testSplitIntegration
>      0      0.3      366      1      IndexSizeTriggerTest.testTrigger
>      0      0.3      366      1      IndexSizeTriggerTest.testValidProperties
>      0      0.3      360      1      JWTAuthPluginIntegrationTest.infoRequestWithoutToken
>      0      0.3      360      1      JWTAuthPluginIntegrationTest.testMetrics
>      0      0.2      403      1      LeaderElectionTest.testBasic
>      0      0.5      403      2      LeaderElectionTest.testCancelElection
>      0      0.5      403      2      LeaderElectionTest.testElection
>      0      0.5      403      2      LeaderElectionTest.testParallelElection
>      0      0.5      403      2      LeaderElectionTest.testStressElection
>      0      0.2      404      1      LeaderFailureAfterFreshStartTest.test
>      0      0.3      367      1      MoveReplicaTest.testFailedMove
>      0      0.3      362      1      NestedShardedAtomicUpdateTest.test
>      0      0.5      400      2      NodeAddedTriggerIntegrationTest.testNodeAddedTriggerRestoreState
>      0      0.3      398      1      NodeAddedTriggerTest.testListenerAcceptance
>      0      0.3      398      1      NodeAddedTriggerTest.testRestoreState
>      0      0.3      398      1      NodeAddedTriggerTest.testTrigger
>      0      0.3      397      1      NodeLostTriggerIntegrationTest.testNodeLostTriggerRestoreState
>      0      0.3      398      1      OutOfBoxZkACLAndCredentialsProvidersTest.testOpenACLUnsafeAllover
>      0      0.3      398      1      OutOfBoxZkACLAndCredentialsProvidersTest.testOutOfBoxSolrZkClient
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testNoCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
>      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
>      0      0.3      400      1      OverseerTaskQueueTest.testDistributedQueueCache
>      0      0.3      400      1      OverseerTaskQueueTest.testLocallyOffer
>      0      0.3      400      1      PingRequestHandlerTest.testPingInClusterWithNoHealthCheck
>      0      0.5      398      2      RecoveryAfterSoftCommitTest.test
>      0      0.5      396      2      ReplicationFactorTest.test
>      0      0.7      412      3      ShardSplitTest.testSplitLocking
>      0      0.8      377      3      ShardSplitTest.testSplitMixedReplicaTypes
>      0      0.5      411      2      ShardSplitTest.testSplitStaticIndexReplication
>      0      0.3      353      1      SolrZkClientTest.getConfig
>      0      0.3      353      1      SolrZkClientTest.testWrappingWatches
>      0      0.3      398      1      SpatialHeatmapFacetsTest.testClassicFacets
>      0      0.3      398      1      SpatialHeatmapFacetsTest.testJsonFacets
>      0      0.3     1194      3      SpatialHeatmapFacetsTest.testPng
>      0      1.0      209      2      StreamDecoratorTest.testParallelTerminatingDaemonUpdateStream
>      0      0.3      397      1      SystemLogListenerTest.test
>      0      0.5      388      2      TestCloudCollectionsListeners.testCollectionDeletion
>      0      0.3      388      1      TestCloudCollectionsListeners.testSimpleCloudCollectionsListener
>      0      0.3      397      1      TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeaderAfterRestart
>      0      0.3      396      1      TestCloudSearcherWarming.testPeersyncFailureReplicationSuccess
>      0      0.8      354      3      TestCollectionStateWatchers.testCanWaitForNonexistantCollection
>      0      0.8      354      3      TestCollectionStateWatchers.testCollectionWatchWithShutdownOfActiveNode
>      0      0.8      354      3      TestCollectionStateWatchers.testCollectionWatchWithShutdownOfUnusedNode
>      0      0.8      389      3      TestCollectionStateWatchers.testDeletionsTriggerWatches
>      0      0.6      354      2      TestCollectionStateWatchers.testLiveNodeChangesTriggerWatches
>      0      0.8      389      3      TestCollectionStateWatchers.testPredicateFailureTimesOut
>      0      0.8      389      3      TestCollectionStateWatchers.testStateWatcherChecksCurrentStateOnRegister
>      0      0.8      389      3      TestCollectionStateWatchers.testWaitForStateChecksCurrentState
>      0      0.8      389      3      TestCollectionStateWatchers.testWaitForStateWatcherIsRetainedOnPredicateFailure
>      0      0.5      389      2      TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout
>      0      0.5      389      2      TestCollectionStateWatchers.testWatchesWorkForStateFormat1
>      0      0.3      400      1      TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete
>      0      0.3      399      1      TestConfigSetsAPI.testCreate
>      0      0.3      399      1      TestConfigSetsAPI.testList
>      0      0.3      399      1      TestConfigSetsAPI.testUpload
>      0      0.3      399      1      TestConfigSetsAPI.testUploadDisabled
>      0      0.3      399      1      TestConfigSetsAPI.testUploadErrors
>      0      0.3      399      1      TestConfigSetsAPI.testUploadWithScriptUpdateProcessor
>      0      0.3      399      1      TestConfigSetsAPI.testUserAndTestDefaultConfigsetsAreSame
>      0      0.3      400      1      TestDistribDocBasedVersion.test
>      0      0.3      354      1      TestDocCollectionWatcher.testCanWaitForNonexistantCollection
>      0      0.6      354      2      TestDocCollectionWatcher.testDeletionsTriggerWatches
>      0      0.6      354      2      TestDocCollectionWatcher.testPredicateFailureTimesOut
>      0      0.6      354      2      TestDocCollectionWatcher.testStateWatcherChecksCurrentStateOnRegister
>      0      0.6      354      2      TestDocCollectionWatcher.testWaitForStateChecksCurrentState
>      0      0.6      354      2      TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFailure
>      0      0.6      354      2      TestDocCollectionWatcher.testWatcherIsRemovedAfterTimeout
>      0      0.6      354      2      TestDocCollectionWatcher.testWatchesWorkForStateFormat1
>      0     14.3       21      3      TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit
>      0      0.2      414      1      TestDuelingCodecs.testCrazyReaderEquals
>      0      4.5       22      1      TestDuelingCodecsAtNight.testCrazyReaderEquals
>      0      0.2      419      1      TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads
>      0      0.3      352      1      TestJaegerConfigurator.testInjected
>      0      0.3      387      1      TestLTROnSolrCloud.testSimpleQuery
>      0      1.2      402      5      TestPointFields.testDoublePointStats
>      0      0.2      402      1      TestPointFields.testIntPointFieldMultiValuedRangeFacet
>      0      1.0      406      4      TestRandomDVFaceting.testRandomFaceting
>      0      0.5      380      2      TestSimScenario.testAutoAddReplicas
>      0      0.3      397      1      TestSimTriggerIntegration.testCooldown
>      0      0.2      419      1      TestSimpleExplanationsWithFillerDocs.testMultiFieldBQofPQ5
>      0      0.2      402      1      TestSolrConfigHandlerConcurrent.test
>      0      0.5      366      2      TestWithCollection.testAddReplicaWithPolicy
>      0      0.3      400      1      TestWithCollection.testCreateCollection
>      0      0.3      400      1      TestWithCollection.testCreateCollectionNoWithCollection
>      0      0.3      400      1      TestWithCollection.testDeleteWithCollection
>      0      0.3      400      1      TestWithCollection.testMoveReplicaWithCollection
>      0      0.3      366      1      TestWithCollection.testNodeAdded
>      0      0.2      401      1      ZkCLITest.testInvalidZKAddress
>      0      0.2      401      1      ZkCLITest.testPutFile
>      0      0.3      394      1      ZkSolrClientTest.testMakeRootNode
>      0      0.3      394      1      ZkSolrClientTest.testReconnect
>      0      0.3      394      1      ZkSolrClientTest.testWatchChildren
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: PLEASE READ! BadApple report. Last week was horrible!

Erick Erickson
Mike:

I saw the push. Hoss’ rollups go for “the last 24 hours”, so it’ll be Tuesday evening before things have had a chance to work their way through, I’ll look tomorrow.

Meanwhile I’m beasting one of the failing test suites (without the change) and 280 iterations so far and no failures. That said, the failure rate was < 1% so it’s not conclusive. Only another 720 runs to go before I pull the latest changes and try again… ;)



> On May 4, 2020, at 1:33 PM, Michael McCandless <[hidden email]> wrote:
>
> Hi Erick,
>
> OK I pushed a fix!  See if it decreases the failure rate for those newly bad apples?
>
> Sorry and thanks :)
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Mon, May 4, 2020 at 1:06 PM Erick Erickson <[hidden email]> wrote:
> Mike:
>
> I have no idea. Hoss’ rollups don’t link back to builds, they
> just aggregate the results.
>
> Not a huge deal if it’s something like this of course. Let’s just
> say I’ve had my share or “moments” ;).
>
> And unfortunately, the test failures are pretty rare on a
> percentage basis, so it’s hard to tell.
>
> I’m watching LUCENE-9191 and I’ll look back at Hoss’ rollups
> a day after you push it and see if the failures disappear.
>
> It’ll take a while for the fixes to roll through all the reporting.
>
> Tell you what. I’ll try beasting one of the classes that fails a lot and then
> try it again after you push LUCENE-9191 and we’ll go from there.
>
> Thanks for getting into this so promptly!
>
> Erick
>
> > On May 4, 2020, at 9:10 AM, Michael McCandless <[hidden email]> wrote:
> >
> > Hi Erick,
> >
> > It's possible this was the root cause of many of the failures: https://issues.apache.org/jira/browse/LUCENE-9191
> >
> > Do these transient failures look something like this?
> >
> >    [junit4]    > Throwable #1: java.nio.charset.MalformedInputException: Input length = 1
> >    [junit4]    >        at __randomizedtesting.SeedInfo.seed([172C6414BE5E2A2C:E5829DFC005A1F0]:0)
> >    [junit4]    >        at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
> >    [junit4]    >        at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
> >    [junit4]    >        at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> >    [junit4]    >        at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
> >    [junit4]    >        at java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
> >    [junit4]    >        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
> >    [junit4]    >        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
> >    [junit4]    >        at org.apache.lucene.util.LineFileDocs.open(LineFileDocs.java:175)
> >    [junit4]    >        at org.apache.lucene.util.LineFileDocs.<init>(LineFileDocs.java:65)
> >    [junit4]    >        at org.apache.lucene.util.LineFileDocs.<init>(LineFileDocs.java:69)
> >
> >
> > If so, then it is likely the root cause ... I'm working on a fix.  Sorry!
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> >
> > On Mon, May 4, 2020 at 7:54 AM Erick Erickson <[hidden email]> wrote:
> > I don’t know whether we had some temporary glitch that broke lots of tests and they’ve been fixed or we had a major regression, but this needs to be addressed ASAP if they’re still failing. See everything below the line "ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail. I’ll raise a JIRA if we can’t get some traction quickly here.
> >
> > Hey, stuff happens. there’s no problem with tests going totally weird for a while. If you can say “Oh, yeah, all those failures for class XYZ are probably fixed” that’s fine.
> >
> > Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)….
> >
> > Hoss’ rolllup for the last 24 hours is not encouraging in terms of the
> > problem already being fixed. There are lots of failures in some
> > classes, notably:
> >
> > CloudHttp2SolrClientTest
> > CollectionsAPIDistributedZkTest
> > DeleteReplicaTest
> > TestDocCollectionWatcher
> >
> > Unfortunately, the failure rate is not very high so reliably
> > reproducing is hard.
> >
> > I’ve reproduced the last week’s failure in this e-mail, full
> > report attached.
> >
> > Here’s Hoss’ rollup:
> > http://fucit.org/solr-jenkins-reports/failure-report.html
> >
> > Usual synopsis:
> >
> > Raw fail count by week totals, most recent week first (corresponds to bits):
> > Week: 0  had  343 failures
> > Week: 1  had  86 failures
> > Week: 2  had  78 failures
> > Week: 3  had  117 failures
> >
> >
> > ********Failures in Hoss' reports for the last 4 rollups.
> >
> > There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
> > These tests were NOT BadApple'd or AwaitsFix’d
> >
> > Failures in the last 4 reports..
> >    Report   Pct     runs    fails           test
> >      0123   0.7     1617     11      ConnectionManagerTest.testReconnectWhenZkDisappeared
> >      0123   1.5     1606     12      ExecutePlanActionTest.testTaskTimeout
> >      0123   1.6     1320     19      MultiThreadedOCPTest.test
> >      0123   1.0     1620     13      RollingRestartTest.test
> >      0123   1.2     1617     12      SearchRateTriggerTest.testWaitForElapsed
> >      0123   3.8      119      7      ShardSplitTest.testSplitWithChaosMonkey
> >      0123   0.3     1519      7      TestInPlaceUpdatesDistrib.test
> >      0123   0.7     1629     14      TestIndexWriterDelete.testDeleteAllNoDeadLock
> >      0123   2.4     1548     18      TestPackages.testPluginLoading
> >      0123   0.3     1587      4      UnloadDistributedZkTest.test
> > ********************************************
> >
> > FAILURES IN THE LAST WEEK (343!)
> > Look particularly at the ones with only a zero in the “Report” column, those are
> > failures that were _not_ in the previous 3 week’s rollups.
> >
> >    Report   Pct     runs    fails           test
> >      012    0.5     1165      4      CustomHighlightComponentTest.test
> >      012    1.0     1168      6      NodeMarkersRegistrationTest.testNodeMarkersRegistration
> >      012    1.0     1170      8      TestCryptoKeys.test
> >      01 3   0.7     1233     11      LeaderFailoverAfterPartitionTest.test
> >      01 3  63.2      102     39      StressHdfsTest.test
> >      01     0.3      709      2      ScheduledTriggerIntegrationTest.testScheduledTrigger
> >      01     0.2      768      2      ShardRoutingTest.test
> >      01     2.6      807     22      TestAllFilesHaveChecksumFooter.test
> >      01     2.6      808     22      TestAllFilesHaveCodecHeader.test
> >      01     0.2      769      2      TestCloudSchemaless.test
> >      01     0.2      769      2      TestDynamicLoading.testDynamicLoading
> >      01     0.3      707      2      TestDynamicLoadingUrl.testDynamicLoadingUrl
> >      01     0.5      767      4      TestPointFields.testFloatPointStats
> >      01    27.1       83     19      TestSQLHandler.doTest
> >      01     0.2      794     12      TestSameScoresWithThreads.test
> >      01     2.6      806     22      TestShardSearching.testSimple
> >      01     0.5      726      4      TestSimScenario.testSplitShard
> >      01     1.1      726      7      TestSimScenario.testSuggestions
> >      01     0.3      771      2      TestWithCollection.testAddReplicaSimple
> >      0 23   0.3     1223      4      CdcrVersionReplicationTest.testCdcrDocVersions
> >      0 23   0.8     1172      6      CloudHttp2SolrClientTest.testRetryUpdatesWhenClusterStateIsStale
> >      0 23   1.4     1202      8      CollectionsAPISolrJTest.testColStatus
> >      0 23   1.0     1249     11      HttpPartitionTest.test
> >      0 23   1.1     1210      8      HttpPartitionWithTlogReplicasTest.test
> >      0 23   0.5     1258      4      ShardSplitTest.testSplitShardWithRuleLink
> >      0 23   0.2     1231      4      TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
> >      0 23   0.2     1232      6      TestSolrConfigHandlerCloud.test
> >      0 2    0.3      767      2      DocValuesNotIndexedTest.testGroupingDVOnlySortLast
> >      0 2    0.3      750      2      TestLBHttp2SolrClient.testTwoServers
> >      0 2    0.3      794      2      TestSolrCloudSnapshots.testSnapshots
> >      0 2   40.7       51     12      TestXYMultiPolygonShapeQueries.testRandomBig
> >      0  3   0.2      842      2      ConnectionManagerTest.testLikelyExpired
> >      0  3   0.2      835      3      LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
> >      0  3   0.3      832      3      ReplaceNodeTest.test
> >      0  3   0.3      826      2      SolrCloudReportersTest.testExplicitConfiguration
> >      0  3   0.2      833      2      TestPullReplicaErrorHandling.testCantConnectToLeader
> >      0  3   2.4      867     21      TestRandomChains.testRandomChains
> >      0  3   0.5      830      3      TestWithCollection.testMoveReplicaMainCollection
> >
> > ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!
> >
> >      0      0.3      361      1      AutoAddReplicasIntegrationTest.testClusterPropOverridesCollecitonProp
> >      0      2.9      380     11      BasicDistributedZkTest.test
> >      0      0.2      404      1      BasicZkTest.testBasic
> >      0      0.2      414      1      ChaosMonkeySafeLeaderWithPullReplicasTest.test
> >      0     62.8      853    536      CheckHdfsIndexTest.doTest
> >      0      0.8      355      3      CloudHttp2SolrClientTest.checkCollectionParameters
> >      0      0.8      355      3      CloudHttp2SolrClientTest.customHttpClientTest
> >      0      1.1      355      4      CloudHttp2SolrClientTest.preferLocalShardsTest
> >      0      0.8      355      3      CloudHttp2SolrClientTest.preferReplicaTypesTest
> >      0      0.8      355      3      CloudHttp2SolrClientTest.singleShardedPreferenceRules
> >      0      0.8      355      3      CloudHttp2SolrClientTest.stateVersionParamTest
> >      0      1.1      355      4      CloudHttp2SolrClientTest.testAliasHandling
> >      0      0.8      355      3      CloudHttp2SolrClientTest.testCollectionDoesntExist
> >      0      0.8      355      3      CloudHttp2SolrClientTest.testInitializationWithSolrUrls
> >      0      0.8      355      3      CloudHttp2SolrClientTest.testNonRetryableRequests
> >      0      1.1      355      4      CloudHttp2SolrClientTest.testOverwriteOption
> >      0      0.8      355      3      CloudHttp2SolrClientTest.testParallelUpdateQTime
> >      0      0.6      355      2      CloudHttp2SolrClientTest.testRouting
> >      0      0.8      355      3      CloudHttp2SolrClientTest.testShutdown
> >      0      0.6      355      2      CloudHttp2SolrClientTest.testVersionsAreReturned
> >      0      0.8      355      3      CloudHttp2SolrClientTest.testWrongZkChrootTest
> >      0      0.5      392      2      CloudSolrClientTest.checkCollectionParameters
> >      0      0.3      392      1      CloudSolrClientTest.preferLocalShardsTest
> >      0      0.3      392      1      CloudSolrClientTest.preferReplicaTypesTest
> >      0      0.3      357      1      CloudSolrClientTest.singleShardedPreferenceRules
> >      0      0.5      392      2      CloudSolrClientTest.stateVersionParamTest
> >      0      0.5      392      2      CloudSolrClientTest.testAliasHandling
> >      0      0.3      392      1      CloudSolrClientTest.testInitializationWithSolrUrls
> >      0      0.3      392      1      CloudSolrClientTest.testNonRetryableRequests
> >      0      0.3      392      1      CloudSolrClientTest.testOverwriteOption
> >      0      0.3      392      1      CloudSolrClientTest.testParallelUpdateQTime
> >      0      0.3      357      1      CloudSolrClientTest.testPing
> >      0      0.5      392      2      CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale
> >      0      0.3      392      1      CloudSolrClientTest.testShutdown
> >      0      0.5      392      2      CloudSolrClientTest.testVersionsAreReturned
> >      0      0.3      392      1      CloudSolrClientTest.testWrongZkChrootTest
> >      0      0.3      399      1      CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls
> >      0      1.3      397      5      CollectionsAPIDistributedZkTest.addReplicaTest
> >      0      0.8      397      3      CollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk
> >      0      1.0      397      4      CollectionsAPIDistributedZkTest.deleteCollectionRemovesStaleZkCollectionsNode
> >      0      0.8      397      3      CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
> >      0      0.3      397      1      CollectionsAPIDistributedZkTest.testBadActionNames
> >      0      0.8      397      3      CollectionsAPIDistributedZkTest.testCollectionReload
> >      0      1.0      397      4      CollectionsAPIDistributedZkTest.testCollectionsAPI
> >      0      1.3      397      5      CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes
> >      0      1.0      397      4      CollectionsAPIDistributedZkTest.testCreateNodeSet
> >      0      1.3      397      5      CollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore
> >      0      0.3      397      1      CollectionsAPIDistributedZkTest.testCreationAndDeletion
> >      0      0.5      397      2      CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection
> >      0      0.8      397      3      CollectionsAPIDistributedZkTest.testMaxNodesPerShard
> >      0      0.8      397      3      CollectionsAPIDistributedZkTest.testMissingNumShards
> >      0      0.5      397      2      CollectionsAPIDistributedZkTest.testMissingRequiredParameters
> >      0      0.8      397      3      CollectionsAPIDistributedZkTest.testNoConfigSetExist
> >      0      0.8      397      3      CollectionsAPIDistributedZkTest.testSpecificConfigsets
> >      0      0.8      397      3      CollectionsAPIDistributedZkTest.testTooManyReplicas
> >      0      1.0      397      4      CollectionsAPIDistributedZkTest.testZeroNumShards
> >      0      0.7      405      3      CollectionsAPISolrJTest.testAddAndDeleteReplica
> >      0      0.2      405      1      CollectionsAPISolrJTest.testAddAndDeleteReplicaProp
> >      0      0.7      405      3      CollectionsAPISolrJTest.testBalanceShardUnique
> >      0      0.5      405      2      CollectionsAPISolrJTest.testCloudInfoInCoreStatus
> >      0      1.5      405      6      CollectionsAPISolrJTest.testClusterProp
> >      0      0.5      405      2      CollectionsAPISolrJTest.testCollectionProp
> >      0      0.5      405      2      CollectionsAPISolrJTest.testCreateAndDeleteAlias
> >      0      0.5      405      2      CollectionsAPISolrJTest.testCreateAndDeleteCollection
> >      0      0.7      405      3      CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesNewFormat
> >      0      1.7      405      7      CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesOldFormat
> >      0      0.7      405      3      CollectionsAPISolrJTest.testCreateCollectionWithPropertyParam
> >      0      1.5      405      6      CollectionsAPISolrJTest.testCreateWithDefaultConfigSet
> >      0      1.9      370      7      CollectionsAPISolrJTest.testDeleteAliasedCollection
> >      0      1.2      405      5      CollectionsAPISolrJTest.testList
> >      0      1.0      405      4      CollectionsAPISolrJTest.testModifyCollectionAttribute
> >      0      1.2      405      5      CollectionsAPISolrJTest.testOverseerStatus
> >      0      0.8      370      3      CollectionsAPISolrJTest.testReadOnlyCollection
> >      0      1.6      370      6      CollectionsAPISolrJTest.testRenameCollection
> >      0      0.7      405      3      CollectionsAPISolrJTest.testSplitShard
> >      0      0.5      366      2      ComputePlanActionTest.testNodeAdded
> >      0      1.1      376      4      DeleteReplicaTest.deleteLiveReplicaTest
> >      0      1.5      411      6      DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup
> >      0      0.5      411      2      DeleteReplicaTest.deleteReplicaByCount
> >      0      0.5      376      2      DeleteReplicaTest.deleteReplicaByCountForAllShards
> >      0      1.2      411      5      DeleteReplicaTest.deleteReplicaFromClusterState
> >      0      1.5      411      6      DeleteReplicaTest.deleteReplicaFromClusterStateLegacy
> >      0      1.7      411      7      DeleteReplicaTest.deleteReplicaOnIndexing
> >      0      0.8      376      3      DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
> >      0      1.3      376      5      DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy
> >      0      0.7      405      3      DeleteShardTest.test
> >      0      0.2      405      1      DeleteShardTest.testDirectoryCleanupAfterDeleteShard
> >      0      0.5      366      2      DistribDocExpirationUpdateProcessorTest.testBasicAuth
> >      0      0.5      366      2      DistribDocExpirationUpdateProcessorTest.testNoAuth
> >      0      0.2      402      1      DistributedQueueTest.testDistributedQueue
> >      0      0.7      402      3      DistributedQueueTest.testDistributedQueueBlocking
> >      0      0.7      402      3      DistributedQueueTest.testDistributedQueueCache
> >      0      0.2      402      1      DistributedQueueTest.testLeakChildWatcher
> >      0      0.5      402      2      DistributedQueueTest.testLocallyOffer
> >      0      0.7      402      3      DistributedQueueTest.testPeekElements
> >      0      0.2      403      1      ExecutePlanActionTest.testExecute
> >      0      0.5      403      2      ExecutePlanActionTest.testIntegration
> >      0      0.7      403      3      ExecutePlanActionTest.testTaskFail
> >      0      0.3      370      1      IndexSizeTriggerSizeEstimationTest.testEstimatedIndexSize
> >      0      0.3      366      1      IndexSizeTriggerTest.testInvalidSplitConfig
> >      0      0.3      366      1      IndexSizeTriggerTest.testMergeIntegration
> >      0      0.3      366      1      IndexSizeTriggerTest.testSplitConfig
> >      0      0.3      366      1      IndexSizeTriggerTest.testSplitIntegration
> >      0      0.3      366      1      IndexSizeTriggerTest.testTrigger
> >      0      0.3      366      1      IndexSizeTriggerTest.testValidProperties
> >      0      0.3      360      1      JWTAuthPluginIntegrationTest.infoRequestWithoutToken
> >      0      0.3      360      1      JWTAuthPluginIntegrationTest.testMetrics
> >      0      0.2      403      1      LeaderElectionTest.testBasic
> >      0      0.5      403      2      LeaderElectionTest.testCancelElection
> >      0      0.5      403      2      LeaderElectionTest.testElection
> >      0      0.5      403      2      LeaderElectionTest.testParallelElection
> >      0      0.5      403      2      LeaderElectionTest.testStressElection
> >      0      0.2      404      1      LeaderFailureAfterFreshStartTest.test
> >      0      0.3      367      1      MoveReplicaTest.testFailedMove
> >      0      0.3      362      1      NestedShardedAtomicUpdateTest.test
> >      0      0.5      400      2      NodeAddedTriggerIntegrationTest.testNodeAddedTriggerRestoreState
> >      0      0.3      398      1      NodeAddedTriggerTest.testListenerAcceptance
> >      0      0.3      398      1      NodeAddedTriggerTest.testRestoreState
> >      0      0.3      398      1      NodeAddedTriggerTest.testTrigger
> >      0      0.3      397      1      NodeLostTriggerIntegrationTest.testNodeLostTriggerRestoreState
> >      0      0.3      398      1      OutOfBoxZkACLAndCredentialsProvidersTest.testOpenACLUnsafeAllover
> >      0      0.3      398      1      OutOfBoxZkACLAndCredentialsProvidersTest.testOutOfBoxSolrZkClient
> >      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
> >      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
> >      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testNoCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
> >      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
> >      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
> >      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingCompletelyNewProviders
> >      0      0.3      400      1      OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClientFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
> >      0      0.3      400      1      OverseerTaskQueueTest.testDistributedQueueCache
> >      0      0.3      400      1      OverseerTaskQueueTest.testLocallyOffer
> >      0      0.3      400      1      PingRequestHandlerTest.testPingInClusterWithNoHealthCheck
> >      0      0.5      398      2      RecoveryAfterSoftCommitTest.test
> >      0      0.5      396      2      ReplicationFactorTest.test
> >      0      0.7      412      3      ShardSplitTest.testSplitLocking
> >      0      0.8      377      3      ShardSplitTest.testSplitMixedReplicaTypes
> >      0      0.5      411      2      ShardSplitTest.testSplitStaticIndexReplication
> >      0      0.3      353      1      SolrZkClientTest.getConfig
> >      0      0.3      353      1      SolrZkClientTest.testWrappingWatches
> >      0      0.3      398      1      SpatialHeatmapFacetsTest.testClassicFacets
> >      0      0.3      398      1      SpatialHeatmapFacetsTest.testJsonFacets
> >      0      0.3     1194      3      SpatialHeatmapFacetsTest.testPng
> >      0      1.0      209      2      StreamDecoratorTest.testParallelTerminatingDaemonUpdateStream
> >      0      0.3      397      1      SystemLogListenerTest.test
> >      0      0.5      388      2      TestCloudCollectionsListeners.testCollectionDeletion
> >      0      0.3      388      1      TestCloudCollectionsListeners.testSimpleCloudCollectionsListener
> >      0      0.3      397      1      TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeaderAfterRestart
> >      0      0.3      396      1      TestCloudSearcherWarming.testPeersyncFailureReplicationSuccess
> >      0      0.8      354      3      TestCollectionStateWatchers.testCanWaitForNonexistantCollection
> >      0      0.8      354      3      TestCollectionStateWatchers.testCollectionWatchWithShutdownOfActiveNode
> >      0      0.8      354      3      TestCollectionStateWatchers.testCollectionWatchWithShutdownOfUnusedNode
> >      0      0.8      389      3      TestCollectionStateWatchers.testDeletionsTriggerWatches
> >      0      0.6      354      2      TestCollectionStateWatchers.testLiveNodeChangesTriggerWatches
> >      0      0.8      389      3      TestCollectionStateWatchers.testPredicateFailureTimesOut
> >      0      0.8      389      3      TestCollectionStateWatchers.testStateWatcherChecksCurrentStateOnRegister
> >      0      0.8      389      3      TestCollectionStateWatchers.testWaitForStateChecksCurrentState
> >      0      0.8      389      3      TestCollectionStateWatchers.testWaitForStateWatcherIsRetainedOnPredicateFailure
> >      0      0.5      389      2      TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout
> >      0      0.5      389      2      TestCollectionStateWatchers.testWatchesWorkForStateFormat1
> >      0      0.3      400      1      TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete
> >      0      0.3      399      1      TestConfigSetsAPI.testCreate
> >      0      0.3      399      1      TestConfigSetsAPI.testList
> >      0      0.3      399      1      TestConfigSetsAPI.testUpload
> >      0      0.3      399      1      TestConfigSetsAPI.testUploadDisabled
> >      0      0.3      399      1      TestConfigSetsAPI.testUploadErrors
> >      0      0.3      399      1      TestConfigSetsAPI.testUploadWithScriptUpdateProcessor
> >      0      0.3      399      1      TestConfigSetsAPI.testUserAndTestDefaultConfigsetsAreSame
> >      0      0.3      400      1      TestDistribDocBasedVersion.test
> >      0      0.3      354      1      TestDocCollectionWatcher.testCanWaitForNonexistantCollection
> >      0      0.6      354      2      TestDocCollectionWatcher.testDeletionsTriggerWatches
> >      0      0.6      354      2      TestDocCollectionWatcher.testPredicateFailureTimesOut
> >      0      0.6      354      2      TestDocCollectionWatcher.testStateWatcherChecksCurrentStateOnRegister
> >      0      0.6      354      2      TestDocCollectionWatcher.testWaitForStateChecksCurrentState
> >      0      0.6      354      2      TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFailure
> >      0      0.6      354      2      TestDocCollectionWatcher.testWatcherIsRemovedAfterTimeout
> >      0      0.6      354      2      TestDocCollectionWatcher.testWatchesWorkForStateFormat1
> >      0     14.3       21      3      TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit
> >      0      0.2      414      1      TestDuelingCodecs.testCrazyReaderEquals
> >      0      4.5       22      1      TestDuelingCodecsAtNight.testCrazyReaderEquals
> >      0      0.2      419      1      TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads
> >      0      0.3      352      1      TestJaegerConfigurator.testInjected
> >      0      0.3      387      1      TestLTROnSolrCloud.testSimpleQuery
> >      0      1.2      402      5      TestPointFields.testDoublePointStats
> >      0      0.2      402      1      TestPointFields.testIntPointFieldMultiValuedRangeFacet
> >      0      1.0      406      4      TestRandomDVFaceting.testRandomFaceting
> >      0      0.5      380      2      TestSimScenario.testAutoAddReplicas
> >      0      0.3      397      1      TestSimTriggerIntegration.testCooldown
> >      0      0.2      419      1      TestSimpleExplanationsWithFillerDocs.testMultiFieldBQofPQ5
> >      0      0.2      402      1      TestSolrConfigHandlerConcurrent.test
> >      0      0.5      366      2      TestWithCollection.testAddReplicaWithPolicy
> >      0      0.3      400      1      TestWithCollection.testCreateCollection
> >      0      0.3      400      1      TestWithCollection.testCreateCollectionNoWithCollection
> >      0      0.3      400      1      TestWithCollection.testDeleteWithCollection
> >      0      0.3      400      1      TestWithCollection.testMoveReplicaWithCollection
> >      0      0.3      366      1      TestWithCollection.testNodeAdded
> >      0      0.2      401      1      ZkCLITest.testInvalidZKAddress
> >      0      0.2      401      1      ZkCLITest.testPutFile
> >      0      0.3      394      1      ZkSolrClientTest.testMakeRootNode
> >      0      0.3      394      1      ZkSolrClientTest.testReconnect
> >      0      0.3      394      1      ZkSolrClientTest.testWatchChildren
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: PLEASE READ! BadApple report. Last week was horrible!

Uwe Schindler
In reply to this post by Erick Erickson
Hi,

there was also a problem with the Windows Node. It ran out of disk space, because some test seem to have filled up all of the disk. All followup builds failed. I cleaned all Workspaces (8.x, master) and it freed 20 Gigabytes!

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: [hidden email]

> -----Original Message-----
> From: Erick Erickson <[hidden email]>
> Sent: Monday, May 4, 2020 1:54 PM
> To: [hidden email]
> Subject: PLEASE READ! BadApple report. Last week was horrible!
>
> I don’t know whether we had some temporary glitch that broke lots of tests
> and they’ve been fixed or we had a major regression, but this needs to be
> addressed ASAP if they’re still failing. See everything below the line "ALL OF
> THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail.
> I’ll raise a JIRA if we can’t get some traction quickly here.
>
> Hey, stuff happens. there’s no problem with tests going totally weird for a
> while. If you can say “Oh, yeah, all those failures for class XYZ are probably
> fixed” that’s fine.
>
> Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)….
>
> Hoss’ rolllup for the last 24 hours is not encouraging in terms of the
> problem already being fixed. There are lots of failures in some
> classes, notably:
>
> CloudHttp2SolrClientTest
> CollectionsAPIDistributedZkTest
> DeleteReplicaTest
> TestDocCollectionWatcher
>
> Unfortunately, the failure rate is not very high so reliably
> reproducing is hard.
>
> I’ve reproduced the last week’s failure in this e-mail, full
> report attached.
>
> Here’s Hoss’ rollup:
> http://fucit.org/solr-jenkins-reports/failure-report.html
>
> Usual synopsis:
>
> Raw fail count by week totals, most recent week first (corresponds to bits):
> Week: 0  had  343 failures
> Week: 1  had  86 failures
> Week: 2  had  78 failures
> Week: 3  had  117 failures
>
>
> ********Failures in Hoss' reports for the last 4 rollups.
>
> There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the
> date I downloaded the rollup file, newest->oldest. See above for the dates the
> files were collected
> These tests were NOT BadApple'd or AwaitsFix’d
>
> Failures in the last 4 reports..
>    Report   Pct     runs    fails           test
>      0123   0.7     1617     11
> ConnectionManagerTest.testReconnectWhenZkDisappeared
>      0123   1.5     1606     12      ExecutePlanActionTest.testTaskTimeout
>      0123   1.6     1320     19      MultiThreadedOCPTest.test
>      0123   1.0     1620     13      RollingRestartTest.test
>      0123   1.2     1617     12      SearchRateTriggerTest.testWaitForElapsed
>      0123   3.8      119      7      ShardSplitTest.testSplitWithChaosMonkey
>      0123   0.3     1519      7      TestInPlaceUpdatesDistrib.test
>      0123   0.7     1629     14      TestIndexWriterDelete.testDeleteAllNoDeadLock
>      0123   2.4     1548     18      TestPackages.testPluginLoading
>      0123   0.3     1587      4      UnloadDistributedZkTest.test
> ********************************************
>
> FAILURES IN THE LAST WEEK (343!)
> Look particularly at the ones with only a zero in the “Report” column, those are
> failures that were _not_ in the previous 3 week’s rollups.
>
>    Report   Pct     runs    fails           test
>      012    0.5     1165      4      CustomHighlightComponentTest.test
>      012    1.0     1168      6
> NodeMarkersRegistrationTest.testNodeMarkersRegistration
>      012    1.0     1170      8      TestCryptoKeys.test
>      01 3   0.7     1233     11      LeaderFailoverAfterPartitionTest.test
>      01 3  63.2      102     39      StressHdfsTest.test
>      01     0.3      709      2
> ScheduledTriggerIntegrationTest.testScheduledTrigger
>      01     0.2      768      2      ShardRoutingTest.test
>      01     2.6      807     22      TestAllFilesHaveChecksumFooter.test
>      01     2.6      808     22      TestAllFilesHaveCodecHeader.test
>      01     0.2      769      2      TestCloudSchemaless.test
>      01     0.2      769      2      TestDynamicLoading.testDynamicLoading
>      01     0.3      707      2      TestDynamicLoadingUrl.testDynamicLoadingUrl
>      01     0.5      767      4      TestPointFields.testFloatPointStats
>      01    27.1       83     19      TestSQLHandler.doTest
>      01     0.2      794     12      TestSameScoresWithThreads.test
>      01     2.6      806     22      TestShardSearching.testSimple
>      01     0.5      726      4      TestSimScenario.testSplitShard
>      01     1.1      726      7      TestSimScenario.testSuggestions
>      01     0.3      771      2      TestWithCollection.testAddReplicaSimple
>      0 23   0.3     1223      4      CdcrVersionReplicationTest.testCdcrDocVersions
>      0 23   0.8     1172      6
> CloudHttp2SolrClientTest.testRetryUpdatesWhenClusterStateIsStale
>      0 23   1.4     1202      8      CollectionsAPISolrJTest.testColStatus
>      0 23   1.0     1249     11      HttpPartitionTest.test
>      0 23   1.1     1210      8      HttpPartitionWithTlogReplicasTest.test
>      0 23   0.5     1258      4      ShardSplitTest.testSplitShardWithRuleLink
>      0 23   0.2     1231      4
> TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
>      0 23   0.2     1232      6      TestSolrConfigHandlerCloud.test
>      0 2    0.3      767      2
> DocValuesNotIndexedTest.testGroupingDVOnlySortLast
>      0 2    0.3      750      2      TestLBHttp2SolrClient.testTwoServers
>      0 2    0.3      794      2      TestSolrCloudSnapshots.testSnapshots
>      0 2   40.7       51     12      TestXYMultiPolygonShapeQueries.testRandomBig
>      0  3   0.2      842      2      ConnectionManagerTest.testLikelyExpired
>      0  3   0.2      835      3
> LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
>      0  3   0.3      832      3      ReplaceNodeTest.test
>      0  3   0.3      826      2      SolrCloudReportersTest.testExplicitConfiguration
>      0  3   0.2      833      2
> TestPullReplicaErrorHandling.testCantConnectToLeader
>      0  3   2.4      867     21      TestRandomChains.testRandomChains
>      0  3   0.5      830      3      TestWithCollection.testMoveReplicaMainCollection
>
> ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!
>
>      0      0.3      361      1
> AutoAddReplicasIntegrationTest.testClusterPropOverridesCollecitonProp
>      0      2.9      380     11      BasicDistributedZkTest.test
>      0      0.2      404      1      BasicZkTest.testBasic
>      0      0.2      414      1      ChaosMonkeySafeLeaderWithPullReplicasTest.test
>      0     62.8      853    536      CheckHdfsIndexTest.doTest
>      0      0.8      355      3      CloudHttp2SolrClientTest.checkCollectionParameters
>      0      0.8      355      3      CloudHttp2SolrClientTest.customHttpClientTest
>      0      1.1      355      4      CloudHttp2SolrClientTest.preferLocalShardsTest
>      0      0.8      355      3      CloudHttp2SolrClientTest.preferReplicaTypesTest
>      0      0.8      355      3
> CloudHttp2SolrClientTest.singleShardedPreferenceRules
>      0      0.8      355      3      CloudHttp2SolrClientTest.stateVersionParamTest
>      0      1.1      355      4      CloudHttp2SolrClientTest.testAliasHandling
>      0      0.8      355      3      CloudHttp2SolrClientTest.testCollectionDoesntExist
>      0      0.8      355      3
> CloudHttp2SolrClientTest.testInitializationWithSolrUrls
>      0      0.8      355      3      CloudHttp2SolrClientTest.testNonRetryableRequests
>      0      1.1      355      4      CloudHttp2SolrClientTest.testOverwriteOption
>      0      0.8      355      3      CloudHttp2SolrClientTest.testParallelUpdateQTime
>      0      0.6      355      2      CloudHttp2SolrClientTest.testRouting
>      0      0.8      355      3      CloudHttp2SolrClientTest.testShutdown
>      0      0.6      355      2      CloudHttp2SolrClientTest.testVersionsAreReturned
>      0      0.8      355      3      CloudHttp2SolrClientTest.testWrongZkChrootTest
>      0      0.5      392      2      CloudSolrClientTest.checkCollectionParameters
>      0      0.3      392      1      CloudSolrClientTest.preferLocalShardsTest
>      0      0.3      392      1      CloudSolrClientTest.preferReplicaTypesTest
>      0      0.3      357      1      CloudSolrClientTest.singleShardedPreferenceRules
>      0      0.5      392      2      CloudSolrClientTest.stateVersionParamTest
>      0      0.5      392      2      CloudSolrClientTest.testAliasHandling
>      0      0.3      392      1      CloudSolrClientTest.testInitializationWithSolrUrls
>      0      0.3      392      1      CloudSolrClientTest.testNonRetryableRequests
>      0      0.3      392      1      CloudSolrClientTest.testOverwriteOption
>      0      0.3      392      1      CloudSolrClientTest.testParallelUpdateQTime
>      0      0.3      357      1      CloudSolrClientTest.testPing
>      0      0.5      392      2
> CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale
>      0      0.3      392      1      CloudSolrClientTest.testShutdown
>      0      0.5      392      2      CloudSolrClientTest.testVersionsAreReturned
>      0      0.3      392      1      CloudSolrClientTest.testWrongZkChrootTest
>      0      0.3      399      1
> CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls
>      0      1.3      397      5      CollectionsAPIDistributedZkTest.addReplicaTest
>      0      0.8      397      3
> CollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk
>      0      1.0      397      4
> CollectionsAPIDistributedZkTest.deleteCollectionRemovesStaleZkCollectionsNo
> de
>      0      0.8      397      3
> CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
>      0      0.3      397      1
> CollectionsAPIDistributedZkTest.testBadActionNames
>      0      0.8      397      3
> CollectionsAPIDistributedZkTest.testCollectionReload
>      0      1.0      397      4      CollectionsAPIDistributedZkTest.testCollectionsAPI
>      0      1.3      397      5
> CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes
>      0      1.0      397      4      CollectionsAPIDistributedZkTest.testCreateNodeSet
>      0      1.3      397      5
> CollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore
>      0      0.3      397      1
> CollectionsAPIDistributedZkTest.testCreationAndDeletion
>      0      0.5      397      2
> CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection
>      0      0.8      397      3
> CollectionsAPIDistributedZkTest.testMaxNodesPerShard
>      0      0.8      397      3
> CollectionsAPIDistributedZkTest.testMissingNumShards
>      0      0.5      397      2
> CollectionsAPIDistributedZkTest.testMissingRequiredParameters
>      0      0.8      397      3
> CollectionsAPIDistributedZkTest.testNoConfigSetExist
>      0      0.8      397      3
> CollectionsAPIDistributedZkTest.testSpecificConfigsets
>      0      0.8      397      3
> CollectionsAPIDistributedZkTest.testTooManyReplicas
>      0      1.0      397      4      CollectionsAPIDistributedZkTest.testZeroNumShards
>      0      0.7      405      3      CollectionsAPISolrJTest.testAddAndDeleteReplica
>      0      0.2      405      1
> CollectionsAPISolrJTest.testAddAndDeleteReplicaProp
>      0      0.7      405      3      CollectionsAPISolrJTest.testBalanceShardUnique
>      0      0.5      405      2      CollectionsAPISolrJTest.testCloudInfoInCoreStatus
>      0      1.5      405      6      CollectionsAPISolrJTest.testClusterProp
>      0      0.5      405      2      CollectionsAPISolrJTest.testCollectionProp
>      0      0.5      405      2      CollectionsAPISolrJTest.testCreateAndDeleteAlias
>      0      0.5      405      2
> CollectionsAPISolrJTest.testCreateAndDeleteCollection
>      0      0.7      405      3
> CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesNewFormat
>      0      1.7      405      7
> CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesOldFormat
>      0      0.7      405      3
> CollectionsAPISolrJTest.testCreateCollectionWithPropertyParam
>      0      1.5      405      6
> CollectionsAPISolrJTest.testCreateWithDefaultConfigSet
>      0      1.9      370      7      CollectionsAPISolrJTest.testDeleteAliasedCollection
>      0      1.2      405      5      CollectionsAPISolrJTest.testList
>      0      1.0      405      4
> CollectionsAPISolrJTest.testModifyCollectionAttribute
>      0      1.2      405      5      CollectionsAPISolrJTest.testOverseerStatus
>      0      0.8      370      3      CollectionsAPISolrJTest.testReadOnlyCollection
>      0      1.6      370      6      CollectionsAPISolrJTest.testRenameCollection
>      0      0.7      405      3      CollectionsAPISolrJTest.testSplitShard
>      0      0.5      366      2      ComputePlanActionTest.testNodeAdded
>      0      1.1      376      4      DeleteReplicaTest.deleteLiveReplicaTest
>      0      1.5      411      6
> DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup
>      0      0.5      411      2      DeleteReplicaTest.deleteReplicaByCount
>      0      0.5      376      2      DeleteReplicaTest.deleteReplicaByCountForAllShards
>      0      1.2      411      5      DeleteReplicaTest.deleteReplicaFromClusterState
>      0      1.5      411      6
> DeleteReplicaTest.deleteReplicaFromClusterStateLegacy
>      0      1.7      411      7      DeleteReplicaTest.deleteReplicaOnIndexing
>      0      0.8      376      3
> DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
>      0      1.3      376      5
> DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy
>      0      0.7      405      3      DeleteShardTest.test
>      0      0.2      405      1
> DeleteShardTest.testDirectoryCleanupAfterDeleteShard
>      0      0.5      366      2
> DistribDocExpirationUpdateProcessorTest.testBasicAuth
>      0      0.5      366      2
> DistribDocExpirationUpdateProcessorTest.testNoAuth
>      0      0.2      402      1      DistributedQueueTest.testDistributedQueue
>      0      0.7      402      3      DistributedQueueTest.testDistributedQueueBlocking
>      0      0.7      402      3      DistributedQueueTest.testDistributedQueueCache
>      0      0.2      402      1      DistributedQueueTest.testLeakChildWatcher
>      0      0.5      402      2      DistributedQueueTest.testLocallyOffer
>      0      0.7      402      3      DistributedQueueTest.testPeekElements
>      0      0.2      403      1      ExecutePlanActionTest.testExecute
>      0      0.5      403      2      ExecutePlanActionTest.testIntegration
>      0      0.7      403      3      ExecutePlanActionTest.testTaskFail
>      0      0.3      370      1
> IndexSizeTriggerSizeEstimationTest.testEstimatedIndexSize
>      0      0.3      366      1      IndexSizeTriggerTest.testInvalidSplitConfig
>      0      0.3      366      1      IndexSizeTriggerTest.testMergeIntegration
>      0      0.3      366      1      IndexSizeTriggerTest.testSplitConfig
>      0      0.3      366      1      IndexSizeTriggerTest.testSplitIntegration
>      0      0.3      366      1      IndexSizeTriggerTest.testTrigger
>      0      0.3      366      1      IndexSizeTriggerTest.testValidProperties
>      0      0.3      360      1
> JWTAuthPluginIntegrationTest.infoRequestWithoutToken
>      0      0.3      360      1      JWTAuthPluginIntegrationTest.testMetrics
>      0      0.2      403      1      LeaderElectionTest.testBasic
>      0      0.5      403      2      LeaderElectionTest.testCancelElection
>      0      0.5      403      2      LeaderElectionTest.testElection
>      0      0.5      403      2      LeaderElectionTest.testParallelElection
>      0      0.5      403      2      LeaderElectionTest.testStressElection
>      0      0.2      404      1      LeaderFailureAfterFreshStartTest.test
>      0      0.3      367      1      MoveReplicaTest.testFailedMove
>      0      0.3      362      1      NestedShardedAtomicUpdateTest.test
>      0      0.5      400      2
> NodeAddedTriggerIntegrationTest.testNodeAddedTriggerRestoreState
>      0      0.3      398      1      NodeAddedTriggerTest.testListenerAcceptance
>      0      0.3      398      1      NodeAddedTriggerTest.testRestoreState
>      0      0.3      398      1      NodeAddedTriggerTest.testTrigger
>      0      0.3      397      1
> NodeLostTriggerIntegrationTest.testNodeLostTriggerRestoreState
>      0      0.3      398      1
> OutOfBoxZkACLAndCredentialsProvidersTest.testOpenACLUnsafeAllover
>      0      0.3      398      1
> OutOfBoxZkACLAndCredentialsProvidersTest.testOutOfBoxSolrZkClient
>      0      0.3      400      1
> OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFac
> toryUsingCompletelyNewProviders
>      0      0.3      400      1
> OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFac
> toryUsingVMParamsProvidersButWithDifferentVMParamsNames
>      0      0.3      400      1
> OverriddenZkACLAndCredentialsProvidersTest.testNoCredentialsSolrZkClientFa
> ctoryUsingCompletelyNewProviders
>      0      0.3      400      1
> OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkCli
> entFactoryUsingCompletelyNewProviders
>      0      0.3      400      1
> OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkCli
> entFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
>      0      0.3      400      1
> OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClien
> tFactoryUsingCompletelyNewProviders
>      0      0.3      400      1
> OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClien
> tFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
>      0      0.3      400      1      OverseerTaskQueueTest.testDistributedQueueCache
>      0      0.3      400      1      OverseerTaskQueueTest.testLocallyOffer
>      0      0.3      400      1
> PingRequestHandlerTest.testPingInClusterWithNoHealthCheck
>      0      0.5      398      2      RecoveryAfterSoftCommitTest.test
>      0      0.5      396      2      ReplicationFactorTest.test
>      0      0.7      412      3      ShardSplitTest.testSplitLocking
>      0      0.8      377      3      ShardSplitTest.testSplitMixedReplicaTypes
>      0      0.5      411      2      ShardSplitTest.testSplitStaticIndexReplication
>      0      0.3      353      1      SolrZkClientTest.getConfig
>      0      0.3      353      1      SolrZkClientTest.testWrappingWatches
>      0      0.3      398      1      SpatialHeatmapFacetsTest.testClassicFacets
>      0      0.3      398      1      SpatialHeatmapFacetsTest.testJsonFacets
>      0      0.3     1194      3      SpatialHeatmapFacetsTest.testPng
>      0      1.0      209      2
> StreamDecoratorTest.testParallelTerminatingDaemonUpdateStream
>      0      0.3      397      1      SystemLogListenerTest.test
>      0      0.5      388      2      TestCloudCollectionsListeners.testCollectionDeletion
>      0      0.3      388      1
> TestCloudCollectionsListeners.testSimpleCloudCollectionsListener
>      0      0.3      397      1
> TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeaderAfterRestart
>      0      0.3      396      1
> TestCloudSearcherWarming.testPeersyncFailureReplicationSuccess
>      0      0.8      354      3
> TestCollectionStateWatchers.testCanWaitForNonexistantCollection
>      0      0.8      354      3
> TestCollectionStateWatchers.testCollectionWatchWithShutdownOfActiveNode
>      0      0.8      354      3
> TestCollectionStateWatchers.testCollectionWatchWithShutdownOfUnusedNod
> e
>      0      0.8      389      3
> TestCollectionStateWatchers.testDeletionsTriggerWatches
>      0      0.6      354      2
> TestCollectionStateWatchers.testLiveNodeChangesTriggerWatches
>      0      0.8      389      3
> TestCollectionStateWatchers.testPredicateFailureTimesOut
>      0      0.8      389      3
> TestCollectionStateWatchers.testStateWatcherChecksCurrentStateOnRegister
>      0      0.8      389      3
> TestCollectionStateWatchers.testWaitForStateChecksCurrentState
>      0      0.8      389      3
> TestCollectionStateWatchers.testWaitForStateWatcherIsRetainedOnPredicateF
> ailure
>      0      0.5      389      2
> TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout
>      0      0.5      389      2
> TestCollectionStateWatchers.testWatchesWorkForStateFormat1
>      0      0.3      400      1
> TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete
>      0      0.3      399      1      TestConfigSetsAPI.testCreate
>      0      0.3      399      1      TestConfigSetsAPI.testList
>      0      0.3      399      1      TestConfigSetsAPI.testUpload
>      0      0.3      399      1      TestConfigSetsAPI.testUploadDisabled
>      0      0.3      399      1      TestConfigSetsAPI.testUploadErrors
>      0      0.3      399      1
> TestConfigSetsAPI.testUploadWithScriptUpdateProcessor
>      0      0.3      399      1
> TestConfigSetsAPI.testUserAndTestDefaultConfigsetsAreSame
>      0      0.3      400      1      TestDistribDocBasedVersion.test
>      0      0.3      354      1
> TestDocCollectionWatcher.testCanWaitForNonexistantCollection
>      0      0.6      354      2
> TestDocCollectionWatcher.testDeletionsTriggerWatches
>      0      0.6      354      2
> TestDocCollectionWatcher.testPredicateFailureTimesOut
>      0      0.6      354      2
> TestDocCollectionWatcher.testStateWatcherChecksCurrentStateOnRegister
>      0      0.6      354      2
> TestDocCollectionWatcher.testWaitForStateChecksCurrentState
>      0      0.6      354      2
> TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFail
> ure
>      0      0.6      354      2
> TestDocCollectionWatcher.testWatcherIsRemovedAfterTimeout
>      0      0.6      354      2
> TestDocCollectionWatcher.testWatchesWorkForStateFormat1
>      0     14.3       21      3
> TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit
>      0      0.2      414      1      TestDuelingCodecs.testCrazyReaderEquals
>      0      4.5       22      1      TestDuelingCodecsAtNight.testCrazyReaderEquals
>      0      0.2      419      1
> TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads
>      0      0.3      352      1      TestJaegerConfigurator.testInjected
>      0      0.3      387      1      TestLTROnSolrCloud.testSimpleQuery
>      0      1.2      402      5      TestPointFields.testDoublePointStats
>      0      0.2      402      1
> TestPointFields.testIntPointFieldMultiValuedRangeFacet
>      0      1.0      406      4      TestRandomDVFaceting.testRandomFaceting
>      0      0.5      380      2      TestSimScenario.testAutoAddReplicas
>      0      0.3      397      1      TestSimTriggerIntegration.testCooldown
>      0      0.2      419      1
> TestSimpleExplanationsWithFillerDocs.testMultiFieldBQofPQ5
>      0      0.2      402      1      TestSolrConfigHandlerConcurrent.test
>      0      0.5      366      2      TestWithCollection.testAddReplicaWithPolicy
>      0      0.3      400      1      TestWithCollection.testCreateCollection
>      0      0.3      400      1
> TestWithCollection.testCreateCollectionNoWithCollection
>      0      0.3      400      1      TestWithCollection.testDeleteWithCollection
>      0      0.3      400      1      TestWithCollection.testMoveReplicaWithCollection
>      0      0.3      366      1      TestWithCollection.testNodeAdded
>      0      0.2      401      1      ZkCLITest.testInvalidZKAddress
>      0      0.2      401      1      ZkCLITest.testPutFile
>      0      0.3      394      1      ZkSolrClientTest.testMakeRootNode
>      0      0.3      394      1      ZkSolrClientTest.testReconnect
>      0      0.3      394      1      ZkSolrClientTest.testWatchChildren
>
>
>



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: PLEASE READ! BadApple report. Last week was horrible!

Erick Erickson
Thanks Uwe and Mike. I’ll check Hoss’ rollups regularly this week, it’ll be tomorrow (Wednesday) before Uwe’s changes get a chance to be reflected there.

And FWIW, I beasted one of the failing suites last night _without_ Mike’s changes and didn’t get any failures so I can’t say anything about whether Mike’s changes helped or not.

Side note: I’m seeing about 40% of the tests throw an NPE in both before and after but the tests succeed, so I’d guess it’s totally unrelated, at  glance, the replica is being unloaded and the replication handler hasn’t stopped yet:

   [junit4]   2> 33676 INFO  (qtp1195412159-1051) [n:127.0.0.1:49614_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=9929b23d-8505-48a5-841a-0992da1af4752119758844417043&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 33746 INFO  (parallelCoreAdminExecutor-541-thread-6-processing-n:127.0.0.1:49614_solr x:replicaTypesTestColl_shard3_replica_n12 9929b23d-8505-48a5-841a-0992da1af4752119758845055723 UNLOAD) [n:127.0.0.1:49614_solr    x:replicaTypesTestColl_shard3_replica_n12 ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.replicaTypesTestColl.shard3.replica_n12 tag=null
   [junit4]   2> 33746 INFO  (parallelCoreAdminExecutor-541-thread-6-processing-n:127.0.0.1:49614_solr x:replicaTypesTestColl_shard3_replica_n12 9929b23d-8505-48a5-841a-0992da1af4752119758845055723 UNLOAD) [n:127.0.0.1:49614_solr    x:replicaTypesTestColl_shard3_replica_n12 ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@57b06a4b: rootName = solr_49614, domain = solr.core.replicaTypesTestColl.shard3.replica_n12, service url = null, agent id = null] for registry solr.core.replicaTypesTestColl.shard3.replica_n12/com.codahale.metrics.MetricRegistry@6cceb313
   [junit4]   2> 33746 INFO  (parallelCoreAdminExecutor-541-thread-4-processing-n:127.0.0.1:49614_solr x:replicaTypesTestColl_shard1_replica_n1 9929b23d-8505-48a5-841a-0992da1af4752119758844417043 UNLOAD) [n:127.0.0.1:49614_solr     ] o.a.s.c.SolrCore [replicaTypesTestColl_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@55431ea6
   [junit4]   2> 33748 INFO  (parallelCoreAdminExecutor-523-thread-6-processing-n:127.0.0.1:49612_solr x:replicaTypesTestColl_shard2_replica_t8 9929b23d-8505-48a5-841a-0992da1af4752119758844946997 UNLOAD) [n:127.0.0.1:49612_solr    x:replicaTypesTestColl_shard2_replica_t8 ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.replicaTypesTestColl.shard2.replica_t8 tag=null
   [junit4]   2> 33748 INFO  (parallelCoreAdminExecutor-523-thread-4-processing-n:127.0.0.1:49612_solr x:replicaTypesTestColl_shard1_replica_t2 9929b23d-8505-48a5-841a-0992da1af4752119758844761938 UNLOAD) [n:127.0.0.1:49612_solr    x:replicaTypesTestColl_shard1_replica_t2 ] o.a.s.c.ZkController replicaTypesTestColl_shard1_replica_t2 stopping background replication from leader
   [junit4]   2> 33749 INFO  (parallelCoreAdminExecutor-523-thread-6-processing-n:127.0.0.1:49612_solr x:replicaTypesTestColl_shard2_replica_t8 9929b23d-8505-48a5-841a-0992da1af4752119758844946997 UNLOAD) [n:127.0.0.1:49612_solr    x:replicaTypesTestColl_shard2_replica_t8 ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@534a73a: rootName = solr_49612, domain = solr.core.replicaTypesTestColl.shard2.replica_t8, service url = null, agent id = null] for registry solr.core.replicaTypesTestColl.shard2.replica_t8/com.codahale.metrics.MetricRegistry@33c32317
   [junit4]   2> 33749 INFO  (parallelCoreAdminExecutor-523-thread-4-processing-n:127.0.0.1:49612_solr x:replicaTypesTestColl_shard1_replica_t2 9929b23d-8505-48a5-841a-0992da1af4752119758844761938 UNLOAD) [n:127.0.0.1:49612_solr     ] o.a.s.c.SolrCore [replicaTypesTestColl_shard1_replica_t2]  CLOSING SolrCore org.apache.solr.core.SolrCore@1ce79412
   [junit4]   2> 33750 INFO  (parallelCoreAdminExecutor-546-thread-5-processing-n:127.0.0.1:49613_solr x:replicaTypesTestColl_shard1_replica_p4 9929b23d-8505-48a5-841a-0992da1af4752119758844836987 UNLOAD) [n:127.0.0.1:49613_solr    x:replicaTypesTestColl_shard1_replica_p4 ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.replicaTypesTestColl.shard1.replica_p4 tag=null
   [junit4]   2> 33750 INFO  (parallelCoreAdminExecutor-546-thread-4-processing-n:127.0.0.1:49613_solr x:replicaTypesTestColl_shard2_replica_n6 9929b23d-8505-48a5-841a-0992da1af4752119758844903977 UNLOAD) [n:127.0.0.1:49613_solr     ] o.a.s.c.SolrCore [replicaTypesTestColl_shard2_replica_n6]  CLOSING SolrCore org.apache.solr.core.SolrCore@36707173
   [junit4]   2> 33750 INFO  (parallelCoreAdminExecutor-546-thread-5-processing-n:127.0.0.1:49613_solr x:replicaTypesTestColl_shard1_replica_p4 9929b23d-8505-48a5-841a-0992da1af4752119758844836987 UNLOAD) [n:127.0.0.1:49613_solr    x:replicaTypesTestColl_shard1_replica_p4 ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1f2a6e95: rootName = solr_49613, domain = solr.core.replicaTypesTestColl.shard1.replica_p4, service url = null, agent id = null] for registry solr.core.replicaTypesTestColl.shard1.replica_p4/com.codahale.metrics.MetricRegistry@2edb03e2
 [junit4]   2> 33770 ERROR (indexFetcher-621-thread-1) [n:127.0.0.1:49612_solr     ] o.a.s.h.ReplicationHandler Index fetch failed :java.lang.NullPointerException
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.getLeaderReplica(IndexFetcher.java:709)
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:387)
   [junit4]   2> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351)
   [junit4]   2> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:422)
   [junit4]   2> at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1208)
   [junit4]   2> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
   [junit4]   2> at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
   [junit4]   2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
   [junit4]   2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   [junit4]   2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   [junit4]   2> at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2>

> On May 5, 2020, at 4:33 AM, Uwe Schindler <[hidden email]> wrote:
>
> Hi,
>
> there was also a problem with the Windows Node. It ran out of disk space, because some test seem to have filled up all of the disk. All followup builds failed. I cleaned all Workspaces (8.x, master) and it freed 20 Gigabytes!
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: [hidden email]
>
>> -----Original Message-----
>> From: Erick Erickson <[hidden email]>
>> Sent: Monday, May 4, 2020 1:54 PM
>> To: [hidden email]
>> Subject: PLEASE READ! BadApple report. Last week was horrible!
>>
>> I don’t know whether we had some temporary glitch that broke lots of tests
>> and they’ve been fixed or we had a major regression, but this needs to be
>> addressed ASAP if they’re still failing. See everything below the line "ALL OF
>> THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!” in this e-mail.
>> I’ll raise a JIRA if we can’t get some traction quickly here.
>>
>> Hey, stuff happens. there’s no problem with tests going totally weird for a
>> while. If you can say “Oh, yeah, all those failures for class XYZ are probably
>> fixed” that’s fine.
>>
>> Gosh-a-rooni, I hope my logging changes aren’t the culprit (gulp)….
>>
>> Hoss’ rolllup for the last 24 hours is not encouraging in terms of the
>> problem already being fixed. There are lots of failures in some
>> classes, notably:
>>
>> CloudHttp2SolrClientTest
>> CollectionsAPIDistributedZkTest
>> DeleteReplicaTest
>> TestDocCollectionWatcher
>>
>> Unfortunately, the failure rate is not very high so reliably
>> reproducing is hard.
>>
>> I’ve reproduced the last week’s failure in this e-mail, full
>> report attached.
>>
>> Here’s Hoss’ rollup:
>> http://fucit.org/solr-jenkins-reports/failure-report.html
>>
>> Usual synopsis:
>>
>> Raw fail count by week totals, most recent week first (corresponds to bits):
>> Week: 0  had  343 failures
>> Week: 1  had  86 failures
>> Week: 2  had  78 failures
>> Week: 3  had  117 failures
>>
>>
>> ********Failures in Hoss' reports for the last 4 rollups.
>>
>> There were 497 unannotated tests that failed in Hoss' rollups. Ordered by the
>> date I downloaded the rollup file, newest->oldest. See above for the dates the
>> files were collected
>> These tests were NOT BadApple'd or AwaitsFix’d
>>
>> Failures in the last 4 reports..
>>   Report   Pct     runs    fails           test
>>     0123   0.7     1617     11
>> ConnectionManagerTest.testReconnectWhenZkDisappeared
>>     0123   1.5     1606     12      ExecutePlanActionTest.testTaskTimeout
>>     0123   1.6     1320     19      MultiThreadedOCPTest.test
>>     0123   1.0     1620     13      RollingRestartTest.test
>>     0123   1.2     1617     12      SearchRateTriggerTest.testWaitForElapsed
>>     0123   3.8      119      7      ShardSplitTest.testSplitWithChaosMonkey
>>     0123   0.3     1519      7      TestInPlaceUpdatesDistrib.test
>>     0123   0.7     1629     14      TestIndexWriterDelete.testDeleteAllNoDeadLock
>>     0123   2.4     1548     18      TestPackages.testPluginLoading
>>     0123   0.3     1587      4      UnloadDistributedZkTest.test
>> ********************************************
>>
>> FAILURES IN THE LAST WEEK (343!)
>> Look particularly at the ones with only a zero in the “Report” column, those are
>> failures that were _not_ in the previous 3 week’s rollups.
>>
>>   Report   Pct     runs    fails           test
>>     012    0.5     1165      4      CustomHighlightComponentTest.test
>>     012    1.0     1168      6
>> NodeMarkersRegistrationTest.testNodeMarkersRegistration
>>     012    1.0     1170      8      TestCryptoKeys.test
>>     01 3   0.7     1233     11      LeaderFailoverAfterPartitionTest.test
>>     01 3  63.2      102     39      StressHdfsTest.test
>>     01     0.3      709      2
>> ScheduledTriggerIntegrationTest.testScheduledTrigger
>>     01     0.2      768      2      ShardRoutingTest.test
>>     01     2.6      807     22      TestAllFilesHaveChecksumFooter.test
>>     01     2.6      808     22      TestAllFilesHaveCodecHeader.test
>>     01     0.2      769      2      TestCloudSchemaless.test
>>     01     0.2      769      2      TestDynamicLoading.testDynamicLoading
>>     01     0.3      707      2      TestDynamicLoadingUrl.testDynamicLoadingUrl
>>     01     0.5      767      4      TestPointFields.testFloatPointStats
>>     01    27.1       83     19      TestSQLHandler.doTest
>>     01     0.2      794     12      TestSameScoresWithThreads.test
>>     01     2.6      806     22      TestShardSearching.testSimple
>>     01     0.5      726      4      TestSimScenario.testSplitShard
>>     01     1.1      726      7      TestSimScenario.testSuggestions
>>     01     0.3      771      2      TestWithCollection.testAddReplicaSimple
>>     0 23   0.3     1223      4      CdcrVersionReplicationTest.testCdcrDocVersions
>>     0 23   0.8     1172      6
>> CloudHttp2SolrClientTest.testRetryUpdatesWhenClusterStateIsStale
>>     0 23   1.4     1202      8      CollectionsAPISolrJTest.testColStatus
>>     0 23   1.0     1249     11      HttpPartitionTest.test
>>     0 23   1.1     1210      8      HttpPartitionWithTlogReplicasTest.test
>>     0 23   0.5     1258      4      ShardSplitTest.testSplitShardWithRuleLink
>>     0 23   0.2     1231      4
>> TestQueryingOnDownCollection.testQueryToDownCollectionShouldFailFast
>>     0 23   0.2     1232      6      TestSolrConfigHandlerCloud.test
>>     0 2    0.3      767      2
>> DocValuesNotIndexedTest.testGroupingDVOnlySortLast
>>     0 2    0.3      750      2      TestLBHttp2SolrClient.testTwoServers
>>     0 2    0.3      794      2      TestSolrCloudSnapshots.testSnapshots
>>     0 2   40.7       51     12      TestXYMultiPolygonShapeQueries.testRandomBig
>>     0  3   0.2      842      2      ConnectionManagerTest.testLikelyExpired
>>     0  3   0.2      835      3
>> LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
>>     0  3   0.3      832      3      ReplaceNodeTest.test
>>     0  3   0.3      826      2      SolrCloudReportersTest.testExplicitConfiguration
>>     0  3   0.2      833      2
>> TestPullReplicaErrorHandling.testCantConnectToLeader
>>     0  3   2.4      867     21      TestRandomChains.testRandomChains
>>     0  3   0.5      830      3      TestWithCollection.testMoveReplicaMainCollection
>>
>> ALL OF THE TESTS BELOW HERE HAVE ONLY FAILED IN THE LAST WEEK!
>>
>>     0      0.3      361      1
>> AutoAddReplicasIntegrationTest.testClusterPropOverridesCollecitonProp
>>     0      2.9      380     11      BasicDistributedZkTest.test
>>     0      0.2      404      1      BasicZkTest.testBasic
>>     0      0.2      414      1      ChaosMonkeySafeLeaderWithPullReplicasTest.test
>>     0     62.8      853    536      CheckHdfsIndexTest.doTest
>>     0      0.8      355      3      CloudHttp2SolrClientTest.checkCollectionParameters
>>     0      0.8      355      3      CloudHttp2SolrClientTest.customHttpClientTest
>>     0      1.1      355      4      CloudHttp2SolrClientTest.preferLocalShardsTest
>>     0      0.8      355      3      CloudHttp2SolrClientTest.preferReplicaTypesTest
>>     0      0.8      355      3
>> CloudHttp2SolrClientTest.singleShardedPreferenceRules
>>     0      0.8      355      3      CloudHttp2SolrClientTest.stateVersionParamTest
>>     0      1.1      355      4      CloudHttp2SolrClientTest.testAliasHandling
>>     0      0.8      355      3      CloudHttp2SolrClientTest.testCollectionDoesntExist
>>     0      0.8      355      3
>> CloudHttp2SolrClientTest.testInitializationWithSolrUrls
>>     0      0.8      355      3      CloudHttp2SolrClientTest.testNonRetryableRequests
>>     0      1.1      355      4      CloudHttp2SolrClientTest.testOverwriteOption
>>     0      0.8      355      3      CloudHttp2SolrClientTest.testParallelUpdateQTime
>>     0      0.6      355      2      CloudHttp2SolrClientTest.testRouting
>>     0      0.8      355      3      CloudHttp2SolrClientTest.testShutdown
>>     0      0.6      355      2      CloudHttp2SolrClientTest.testVersionsAreReturned
>>     0      0.8      355      3      CloudHttp2SolrClientTest.testWrongZkChrootTest
>>     0      0.5      392      2      CloudSolrClientTest.checkCollectionParameters
>>     0      0.3      392      1      CloudSolrClientTest.preferLocalShardsTest
>>     0      0.3      392      1      CloudSolrClientTest.preferReplicaTypesTest
>>     0      0.3      357      1      CloudSolrClientTest.singleShardedPreferenceRules
>>     0      0.5      392      2      CloudSolrClientTest.stateVersionParamTest
>>     0      0.5      392      2      CloudSolrClientTest.testAliasHandling
>>     0      0.3      392      1      CloudSolrClientTest.testInitializationWithSolrUrls
>>     0      0.3      392      1      CloudSolrClientTest.testNonRetryableRequests
>>     0      0.3      392      1      CloudSolrClientTest.testOverwriteOption
>>     0      0.3      392      1      CloudSolrClientTest.testParallelUpdateQTime
>>     0      0.3      357      1      CloudSolrClientTest.testPing
>>     0      0.5      392      2
>> CloudSolrClientTest.testRetryUpdatesWhenClusterStateIsStale
>>     0      0.3      392      1      CloudSolrClientTest.testShutdown
>>     0      0.5      392      2      CloudSolrClientTest.testVersionsAreReturned
>>     0      0.3      392      1      CloudSolrClientTest.testWrongZkChrootTest
>>     0      0.3      399      1
>> CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls
>>     0      1.3      397      5      CollectionsAPIDistributedZkTest.addReplicaTest
>>     0      0.8      397      3
>> CollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk
>>     0      1.0      397      4
>> CollectionsAPIDistributedZkTest.deleteCollectionRemovesStaleZkCollectionsNo
>> de
>>     0      0.8      397      3
>> CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
>>     0      0.3      397      1
>> CollectionsAPIDistributedZkTest.testBadActionNames
>>     0      0.8      397      3
>> CollectionsAPIDistributedZkTest.testCollectionReload
>>     0      1.0      397      4      CollectionsAPIDistributedZkTest.testCollectionsAPI
>>     0      1.3      397      5
>> CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes
>>     0      1.0      397      4      CollectionsAPIDistributedZkTest.testCreateNodeSet
>>     0      1.3      397      5
>> CollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore
>>     0      0.3      397      1
>> CollectionsAPIDistributedZkTest.testCreationAndDeletion
>>     0      0.5      397      2
>> CollectionsAPIDistributedZkTest.testDeleteNonExistentCollection
>>     0      0.8      397      3
>> CollectionsAPIDistributedZkTest.testMaxNodesPerShard
>>     0      0.8      397      3
>> CollectionsAPIDistributedZkTest.testMissingNumShards
>>     0      0.5      397      2
>> CollectionsAPIDistributedZkTest.testMissingRequiredParameters
>>     0      0.8      397      3
>> CollectionsAPIDistributedZkTest.testNoConfigSetExist
>>     0      0.8      397      3
>> CollectionsAPIDistributedZkTest.testSpecificConfigsets
>>     0      0.8      397      3
>> CollectionsAPIDistributedZkTest.testTooManyReplicas
>>     0      1.0      397      4      CollectionsAPIDistributedZkTest.testZeroNumShards
>>     0      0.7      405      3      CollectionsAPISolrJTest.testAddAndDeleteReplica
>>     0      0.2      405      1
>> CollectionsAPISolrJTest.testAddAndDeleteReplicaProp
>>     0      0.7      405      3      CollectionsAPISolrJTest.testBalanceShardUnique
>>     0      0.5      405      2      CollectionsAPISolrJTest.testCloudInfoInCoreStatus
>>     0      1.5      405      6      CollectionsAPISolrJTest.testClusterProp
>>     0      0.5      405      2      CollectionsAPISolrJTest.testCollectionProp
>>     0      0.5      405      2      CollectionsAPISolrJTest.testCreateAndDeleteAlias
>>     0      0.5      405      2
>> CollectionsAPISolrJTest.testCreateAndDeleteCollection
>>     0      0.7      405      3
>> CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesNewFormat
>>     0      1.7      405      7
>> CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesOldFormat
>>     0      0.7      405      3
>> CollectionsAPISolrJTest.testCreateCollectionWithPropertyParam
>>     0      1.5      405      6
>> CollectionsAPISolrJTest.testCreateWithDefaultConfigSet
>>     0      1.9      370      7      CollectionsAPISolrJTest.testDeleteAliasedCollection
>>     0      1.2      405      5      CollectionsAPISolrJTest.testList
>>     0      1.0      405      4
>> CollectionsAPISolrJTest.testModifyCollectionAttribute
>>     0      1.2      405      5      CollectionsAPISolrJTest.testOverseerStatus
>>     0      0.8      370      3      CollectionsAPISolrJTest.testReadOnlyCollection
>>     0      1.6      370      6      CollectionsAPISolrJTest.testRenameCollection
>>     0      0.7      405      3      CollectionsAPISolrJTest.testSplitShard
>>     0      0.5      366      2      ComputePlanActionTest.testNodeAdded
>>     0      1.1      376      4      DeleteReplicaTest.deleteLiveReplicaTest
>>     0      1.5      411      6
>> DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup
>>     0      0.5      411      2      DeleteReplicaTest.deleteReplicaByCount
>>     0      0.5      376      2      DeleteReplicaTest.deleteReplicaByCountForAllShards
>>     0      1.2      411      5      DeleteReplicaTest.deleteReplicaFromClusterState
>>     0      1.5      411      6
>> DeleteReplicaTest.deleteReplicaFromClusterStateLegacy
>>     0      1.7      411      7      DeleteReplicaTest.deleteReplicaOnIndexing
>>     0      0.8      376      3
>> DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
>>     0      1.3      376      5
>> DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy
>>     0      0.7      405      3      DeleteShardTest.test
>>     0      0.2      405      1
>> DeleteShardTest.testDirectoryCleanupAfterDeleteShard
>>     0      0.5      366      2
>> DistribDocExpirationUpdateProcessorTest.testBasicAuth
>>     0      0.5      366      2
>> DistribDocExpirationUpdateProcessorTest.testNoAuth
>>     0      0.2      402      1      DistributedQueueTest.testDistributedQueue
>>     0      0.7      402      3      DistributedQueueTest.testDistributedQueueBlocking
>>     0      0.7      402      3      DistributedQueueTest.testDistributedQueueCache
>>     0      0.2      402      1      DistributedQueueTest.testLeakChildWatcher
>>     0      0.5      402      2      DistributedQueueTest.testLocallyOffer
>>     0      0.7      402      3      DistributedQueueTest.testPeekElements
>>     0      0.2      403      1      ExecutePlanActionTest.testExecute
>>     0      0.5      403      2      ExecutePlanActionTest.testIntegration
>>     0      0.7      403      3      ExecutePlanActionTest.testTaskFail
>>     0      0.3      370      1
>> IndexSizeTriggerSizeEstimationTest.testEstimatedIndexSize
>>     0      0.3      366      1      IndexSizeTriggerTest.testInvalidSplitConfig
>>     0      0.3      366      1      IndexSizeTriggerTest.testMergeIntegration
>>     0      0.3      366      1      IndexSizeTriggerTest.testSplitConfig
>>     0      0.3      366      1      IndexSizeTriggerTest.testSplitIntegration
>>     0      0.3      366      1      IndexSizeTriggerTest.testTrigger
>>     0      0.3      366      1      IndexSizeTriggerTest.testValidProperties
>>     0      0.3      360      1
>> JWTAuthPluginIntegrationTest.infoRequestWithoutToken
>>     0      0.3      360      1      JWTAuthPluginIntegrationTest.testMetrics
>>     0      0.2      403      1      LeaderElectionTest.testBasic
>>     0      0.5      403      2      LeaderElectionTest.testCancelElection
>>     0      0.5      403      2      LeaderElectionTest.testElection
>>     0      0.5      403      2      LeaderElectionTest.testParallelElection
>>     0      0.5      403      2      LeaderElectionTest.testStressElection
>>     0      0.2      404      1      LeaderFailureAfterFreshStartTest.test
>>     0      0.3      367      1      MoveReplicaTest.testFailedMove
>>     0      0.3      362      1      NestedShardedAtomicUpdateTest.test
>>     0      0.5      400      2
>> NodeAddedTriggerIntegrationTest.testNodeAddedTriggerRestoreState
>>     0      0.3      398      1      NodeAddedTriggerTest.testListenerAcceptance
>>     0      0.3      398      1      NodeAddedTriggerTest.testRestoreState
>>     0      0.3      398      1      NodeAddedTriggerTest.testTrigger
>>     0      0.3      397      1
>> NodeLostTriggerIntegrationTest.testNodeLostTriggerRestoreState
>>     0      0.3      398      1
>> OutOfBoxZkACLAndCredentialsProvidersTest.testOpenACLUnsafeAllover
>>     0      0.3      398      1
>> OutOfBoxZkACLAndCredentialsProvidersTest.testOutOfBoxSolrZkClient
>>     0      0.3      400      1
>> OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFac
>> toryUsingCompletelyNewProviders
>>     0      0.3      400      1
>> OverriddenZkACLAndCredentialsProvidersTest.testAllCredentialsSolrZkClientFac
>> toryUsingVMParamsProvidersButWithDifferentVMParamsNames
>>     0      0.3      400      1
>> OverriddenZkACLAndCredentialsProvidersTest.testNoCredentialsSolrZkClientFa
>> ctoryUsingCompletelyNewProviders
>>     0      0.3      400      1
>> OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkCli
>> entFactoryUsingCompletelyNewProviders
>>     0      0.3      400      1
>> OverriddenZkACLAndCredentialsProvidersTest.testReadonlyCredentialsSolrZkCli
>> entFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
>>     0      0.3      400      1
>> OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClien
>> tFactoryUsingCompletelyNewProviders
>>     0      0.3      400      1
>> OverriddenZkACLAndCredentialsProvidersTest.testWrongCredentialsSolrZkClien
>> tFactoryUsingVMParamsProvidersButWithDifferentVMParamsNames
>>     0      0.3      400      1      OverseerTaskQueueTest.testDistributedQueueCache
>>     0      0.3      400      1      OverseerTaskQueueTest.testLocallyOffer
>>     0      0.3      400      1
>> PingRequestHandlerTest.testPingInClusterWithNoHealthCheck
>>     0      0.5      398      2      RecoveryAfterSoftCommitTest.test
>>     0      0.5      396      2      ReplicationFactorTest.test
>>     0      0.7      412      3      ShardSplitTest.testSplitLocking
>>     0      0.8      377      3      ShardSplitTest.testSplitMixedReplicaTypes
>>     0      0.5      411      2      ShardSplitTest.testSplitStaticIndexReplication
>>     0      0.3      353      1      SolrZkClientTest.getConfig
>>     0      0.3      353      1      SolrZkClientTest.testWrappingWatches
>>     0      0.3      398      1      SpatialHeatmapFacetsTest.testClassicFacets
>>     0      0.3      398      1      SpatialHeatmapFacetsTest.testJsonFacets
>>     0      0.3     1194      3      SpatialHeatmapFacetsTest.testPng
>>     0      1.0      209      2
>> StreamDecoratorTest.testParallelTerminatingDaemonUpdateStream
>>     0      0.3      397      1      SystemLogListenerTest.test
>>     0      0.5      388      2      TestCloudCollectionsListeners.testCollectionDeletion
>>     0      0.3      388      1
>> TestCloudCollectionsListeners.testSimpleCloudCollectionsListener
>>     0      0.3      397      1
>> TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeaderAfterRestart
>>     0      0.3      396      1
>> TestCloudSearcherWarming.testPeersyncFailureReplicationSuccess
>>     0      0.8      354      3
>> TestCollectionStateWatchers.testCanWaitForNonexistantCollection
>>     0      0.8      354      3
>> TestCollectionStateWatchers.testCollectionWatchWithShutdownOfActiveNode
>>     0      0.8      354      3
>> TestCollectionStateWatchers.testCollectionWatchWithShutdownOfUnusedNod
>> e
>>     0      0.8      389      3
>> TestCollectionStateWatchers.testDeletionsTriggerWatches
>>     0      0.6      354      2
>> TestCollectionStateWatchers.testLiveNodeChangesTriggerWatches
>>     0      0.8      389      3
>> TestCollectionStateWatchers.testPredicateFailureTimesOut
>>     0      0.8      389      3
>> TestCollectionStateWatchers.testStateWatcherChecksCurrentStateOnRegister
>>     0      0.8      389      3
>> TestCollectionStateWatchers.testWaitForStateChecksCurrentState
>>     0      0.8      389      3
>> TestCollectionStateWatchers.testWaitForStateWatcherIsRetainedOnPredicateF
>> ailure
>>     0      0.5      389      2
>> TestCollectionStateWatchers.testWatcherIsRemovedAfterTimeout
>>     0      0.5      389      2
>> TestCollectionStateWatchers.testWatchesWorkForStateFormat1
>>     0      0.3      400      1
>> TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete
>>     0      0.3      399      1      TestConfigSetsAPI.testCreate
>>     0      0.3      399      1      TestConfigSetsAPI.testList
>>     0      0.3      399      1      TestConfigSetsAPI.testUpload
>>     0      0.3      399      1      TestConfigSetsAPI.testUploadDisabled
>>     0      0.3      399      1      TestConfigSetsAPI.testUploadErrors
>>     0      0.3      399      1
>> TestConfigSetsAPI.testUploadWithScriptUpdateProcessor
>>     0      0.3      399      1
>> TestConfigSetsAPI.testUserAndTestDefaultConfigsetsAreSame
>>     0      0.3      400      1      TestDistribDocBasedVersion.test
>>     0      0.3      354      1
>> TestDocCollectionWatcher.testCanWaitForNonexistantCollection
>>     0      0.6      354      2
>> TestDocCollectionWatcher.testDeletionsTriggerWatches
>>     0      0.6      354      2
>> TestDocCollectionWatcher.testPredicateFailureTimesOut
>>     0      0.6      354      2
>> TestDocCollectionWatcher.testStateWatcherChecksCurrentStateOnRegister
>>     0      0.6      354      2
>> TestDocCollectionWatcher.testWaitForStateChecksCurrentState
>>     0      0.6      354      2
>> TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFail
>> ure
>>     0      0.6      354      2
>> TestDocCollectionWatcher.testWatcherIsRemovedAfterTimeout
>>     0      0.6      354      2
>> TestDocCollectionWatcher.testWatchesWorkForStateFormat1
>>     0     14.3       21      3
>> TestDocTermOrdsUninvertLimit.testTriggerUnInvertLimit
>>     0      0.2      414      1      TestDuelingCodecs.testCrazyReaderEquals
>>     0      4.5       22      1      TestDuelingCodecsAtNight.testCrazyReaderEquals
>>     0      0.2      419      1
>> TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads
>>     0      0.3      352      1      TestJaegerConfigurator.testInjected
>>     0      0.3      387      1      TestLTROnSolrCloud.testSimpleQuery
>>     0      1.2      402      5      TestPointFields.testDoublePointStats
>>     0      0.2      402      1
>> TestPointFields.testIntPointFieldMultiValuedRangeFacet
>>     0      1.0      406      4      TestRandomDVFaceting.testRandomFaceting
>>     0      0.5      380      2      TestSimScenario.testAutoAddReplicas
>>     0      0.3      397      1      TestSimTriggerIntegration.testCooldown
>>     0      0.2      419      1
>> TestSimpleExplanationsWithFillerDocs.testMultiFieldBQofPQ5
>>     0      0.2      402      1      TestSolrConfigHandlerConcurrent.test
>>     0      0.5      366      2      TestWithCollection.testAddReplicaWithPolicy
>>     0      0.3      400      1      TestWithCollection.testCreateCollection
>>     0      0.3      400      1
>> TestWithCollection.testCreateCollectionNoWithCollection
>>     0      0.3      400      1      TestWithCollection.testDeleteWithCollection
>>     0      0.3      400      1      TestWithCollection.testMoveReplicaWithCollection
>>     0      0.3      366      1      TestWithCollection.testNodeAdded
>>     0      0.2      401      1      ZkCLITest.testInvalidZKAddress
>>     0      0.2      401      1      ZkCLITest.testPutFile
>>     0      0.3      394      1      ZkSolrClientTest.testMakeRootNode
>>     0      0.3      394      1      ZkSolrClientTest.testReconnect
>>     0      0.3      394      1      ZkSolrClientTest.testWatchChildren
>>
>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: PLEASE READ! BadApple report. Last week was horrible!

Chris Hostetter-3

: And FWIW, I beasted one of the failing suites last night _without_
: Mike’s changes and didn’t get any failures so I can’t say anything about
: whether Mike’s changes helped or not.

IIUC McCandless's failure only affects you if you use the "jenkins" test
data file (the really big wikipedia dump) ... see the jira he mentioned
for details.



-Hoss
http://www.lucidworks.com/


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: PLEASE READ! BadApple report. Last week was horrible!

Erick Erickson
OK, thanks Chris.

The 24 hour rollup still shows many failures in the several classes, I’ll check tomorrow
to see if that’s a consequence of the disk full problem.

> On May 5, 2020, at 1:41 PM, Chris Hostetter <[hidden email]> wrote:
>
>
> : And FWIW, I beasted one of the failing suites last night _without_
> : Mike’s changes and didn’t get any failures so I can’t say anything about
> : whether Mike’s changes helped or not.
>
> IIUC McCandless's failure only affects you if you use the "jenkins" test
> data file (the really big wikipedia dump) ... see the jira he mentioned
> for details.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: PLEASE READ! BadApple report. Last week was horrible!

Erick Erickson
In reply to this post by Chris Hostetter-3
OK, this morning things are back to normal. I think the disk space issue
was to blame because checking after Mike’s fix didn’t look like it
cured the problem.

Thanks all!

> On May 5, 2020, at 1:41 PM, Chris Hostetter <[hidden email]> wrote:
>
>
> : And FWIW, I beasted one of the failing suites last night _without_
> : Mike’s changes and didn’t get any failures so I can’t say anything about
> : whether Mike’s changes helped or not.
>
> IIUC McCandless's failure only affects you if you use the "jenkins" test
> data file (the really big wikipedia dump) ... see the jira he mentioned
> for details.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: PLEASE READ! BadApple report. Last week was horrible!

Michael McCandless-2
Phew!  Thanks for digging Erick, and for producing these BadApple reports.

On Wed, May 6, 2020 at 7:59 AM Erick Erickson <[hidden email]> wrote:
OK, this morning things are back to normal. I think the disk space issue
was to blame because checking after Mike’s fix didn’t look like it
cured the problem.

Thanks all!

> On May 5, 2020, at 1:41 PM, Chris Hostetter <[hidden email]> wrote:
>
>
> : And FWIW, I beasted one of the failing suites last night _without_
> : Mike’s changes and didn’t get any failures so I can’t say anything about
> : whether Mike’s changes helped or not.
>
> IIUC McCandless's failure only affects you if you use the "jenkins" test
> data file (the really big wikipedia dump) ... see the jira he mentioned
> for details.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]