BadApple report, but please read the first bit

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

BadApple report, but please read the first bit

Erick Erickson
There are several tests that are causing a lot of noise:

SharedFSAutoReplicaFailoverTest is failing 90%+ of the time.
TestBulkSchemaConcurrent 31%
StressHdfsTest  16%
SchemaApiFailureTest 13.88%

I encourage people to look at: http://fucit.org/solr-jenkins-reports/failure-report.html and see if anything looks like it is affected by recent work. TestBulkSchemaConcurrent has been failing off and on for a long time, but the failure rate picked up dramatically in the last couple of weeks. Ditto SchemaApiFailureTest.

Do we even care about Hdfs? Are we deprecating it or not?

Holding relatively steady otherwise:

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0  had  82 failures
Week: 1  had  94 failures
Week: 2  had  502 failures
Week: 3  had  19 failures


********Failures in Hoss' reports for the last 4 rollups.

There were 562 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
   Report   Pct     runs    fails           test
     0123   0.3     1271      8      RollingRestartTest.test
     0123  93.3       41     36      SharedFSAutoReplicaFailoverTest.test
     0123   3.5      627     16      TestCircuitBreaker.testBuildingMemoryPressure
     0123   1.0      627      8      TestCircuitBreaker.testResponseWithCBTiming
     0123   5.8     1483     79      TestContainerPlugin.testApiFromPackage
     0123   2.3     1335     23      TestDistributedGrouping.test
********************************************


Full report:


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

e-mail-2020-08-03.txt (43K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: BadApple report, but please read the first bit

David Smiley
Do we have any long term (aka "longitudinal") pass/fail rates for tests?

SharedFSAutoReplicaFailoverTest in particular is kinda-sorta tied to HDFS, and that's going away to a plug-in for 9.0.  The shared file system notion isn't well supported in SolrCloud, I think.

~ David Smiley
Apache Lucene/Solr Search Developer


On Mon, Aug 3, 2020 at 7:26 AM Erick Erickson <[hidden email]> wrote:
There are several tests that are causing a lot of noise:

SharedFSAutoReplicaFailoverTest is failing 90%+ of the time.
TestBulkSchemaConcurrent 31%
StressHdfsTest  16%
SchemaApiFailureTest 13.88%

I encourage people to look at: http://fucit.org/solr-jenkins-reports/failure-report.html and see if anything looks like it is affected by recent work. TestBulkSchemaConcurrent has been failing off and on for a long time, but the failure rate picked up dramatically in the last couple of weeks. Ditto SchemaApiFailureTest.

Do we even care about Hdfs? Are we deprecating it or not?

Holding relatively steady otherwise:

Raw fail count by week totals, most recent week first (corresponds to bits):
Week: 0  had  82 failures
Week: 1  had  94 failures
Week: 2  had  502 failures
Week: 3  had  19 failures


********Failures in Hoss' reports for the last 4 rollups.

There were 562 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
These tests were NOT BadApple'd or AwaitsFix'd

Failures in the last 4 reports..
   Report   Pct     runs    fails           test
     0123   0.3     1271      8      RollingRestartTest.test
     0123  93.3       41     36      SharedFSAutoReplicaFailoverTest.test
     0123   3.5      627     16      TestCircuitBreaker.testBuildingMemoryPressure
     0123   1.0      627      8      TestCircuitBreaker.testResponseWithCBTiming
     0123   5.8     1483     79      TestContainerPlugin.testApiFromPackage
     0123   2.3     1335     23      TestDistributedGrouping.test
********************************************


Full report:

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: BadApple report, but please read the first bit

Erick Erickson
I have the weekly rollups (with a few gaps) going back to about April 2018, but nothing’s been done to try to make them generally available. Each BadApple report has rates for the last 4 weeks in the attached file, just below "Failures over the last 4 weeks, but not every week. Ordered most-recent first:”



> On Aug 12, 2020, at 2:06 AM, David Smiley <[hidden email]> wrote:
>
> Do we have any long term (aka "longitudinal") pass/fail rates for tests?
>
> SharedFSAutoReplicaFailoverTest in particular is kinda-sorta tied to HDFS, and that's going away to a plug-in for 9.0.  The shared file system notion isn't well supported in SolrCloud, I think.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Aug 3, 2020 at 7:26 AM Erick Erickson <[hidden email]> wrote:
> There are several tests that are causing a lot of noise:
>
> SharedFSAutoReplicaFailoverTest is failing 90%+ of the time.
> TestBulkSchemaConcurrent 31%
> StressHdfsTest  16%
> SchemaApiFailureTest 13.88%
>
> I encourage people to look at: http://fucit.org/solr-jenkins-reports/failure-report.html and see if anything looks like it is affected by recent work. TestBulkSchemaConcurrent has been failing off and on for a long time, but the failure rate picked up dramatically in the last couple of weeks. Ditto SchemaApiFailureTest.
>
> Do we even care about Hdfs? Are we deprecating it or not?
>
> Holding relatively steady otherwise:
>
> Raw fail count by week totals, most recent week first (corresponds to bits):
> Week: 0  had  82 failures
> Week: 1  had  94 failures
> Week: 2  had  502 failures
> Week: 3  had  19 failures
>
>
> ********Failures in Hoss' reports for the last 4 rollups.
>
> There were 562 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
> These tests were NOT BadApple'd or AwaitsFix'd
>
> Failures in the last 4 reports..
>    Report   Pct     runs    fails           test
>      0123   0.3     1271      8      RollingRestartTest.test
>      0123  93.3       41     36      SharedFSAutoReplicaFailoverTest.test
>      0123   3.5      627     16      TestCircuitBreaker.testBuildingMemoryPressure
>      0123   1.0      627      8      TestCircuitBreaker.testResponseWithCBTiming
>      0123   5.8     1483     79      TestContainerPlugin.testApiFromPackage
>      0123   2.3     1335     23      TestDistributedGrouping.test
> ********************************************
>
>
> Full report:
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: BadApple report, but please read the first bit

Kevin Risden-3
http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

David for that specific test you asked the failures are recent with as far as I know no change to HDFS stuff. Starting June/July failing regularly.

Kevin Risden



On Wed, Aug 12, 2020 at 9:03 AM Erick Erickson <[hidden email]> wrote:
I have the weekly rollups (with a few gaps) going back to about April 2018, but nothing’s been done to try to make them generally available. Each BadApple report has rates for the last 4 weeks in the attached file, just below "Failures over the last 4 weeks, but not every week. Ordered most-recent first:”



> On Aug 12, 2020, at 2:06 AM, David Smiley <[hidden email]> wrote:
>
> Do we have any long term (aka "longitudinal") pass/fail rates for tests?
>
> SharedFSAutoReplicaFailoverTest in particular is kinda-sorta tied to HDFS, and that's going away to a plug-in for 9.0.  The shared file system notion isn't well supported in SolrCloud, I think.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Aug 3, 2020 at 7:26 AM Erick Erickson <[hidden email]> wrote:
> There are several tests that are causing a lot of noise:
>
> SharedFSAutoReplicaFailoverTest is failing 90%+ of the time.
> TestBulkSchemaConcurrent 31%
> StressHdfsTest  16%
> SchemaApiFailureTest 13.88%
>
> I encourage people to look at: http://fucit.org/solr-jenkins-reports/failure-report.html and see if anything looks like it is affected by recent work. TestBulkSchemaConcurrent has been failing off and on for a long time, but the failure rate picked up dramatically in the last couple of weeks. Ditto SchemaApiFailureTest.
>
> Do we even care about Hdfs? Are we deprecating it or not?
>
> Holding relatively steady otherwise:
>
> Raw fail count by week totals, most recent week first (corresponds to bits):
> Week: 0  had  82 failures
> Week: 1  had  94 failures
> Week: 2  had  502 failures
> Week: 3  had  19 failures
>
>
> ********Failures in Hoss' reports for the last 4 rollups.
>
> There were 562 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
> These tests were NOT BadApple'd or AwaitsFix'd
>
> Failures in the last 4 reports..
>    Report   Pct     runs    fails           test
>      0123   0.3     1271      8      RollingRestartTest.test
>      0123  93.3       41     36      SharedFSAutoReplicaFailoverTest.test
>      0123   3.5      627     16      TestCircuitBreaker.testBuildingMemoryPressure
>      0123   1.0      627      8      TestCircuitBreaker.testResponseWithCBTiming
>      0123   5.8     1483     79      TestContainerPlugin.testApiFromPackage
>      0123   2.3     1335     23      TestDistributedGrouping.test
> ********************************************
>
>
> Full report:
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: BadApple report, but please read the first bit

Erick Erickson
In reply to this post by Erick Erickson
Didn’t think at first (only one cup of coffee). Here’s the Emails that test appears in, the formatting is poor…

After that is the raw data from Hoss’ rollups that might be easier to ingest.

I have 1.3G of this kind of historical data, I’ve had vague thoughts about putting it someplace accessible to others but haven’t done anything with it.

I suppose, wrapped around this, is the entire question of how much value it’ll have depending on what happens with Mark’s reference impl...

“Suite” fails are things like object tracker failures.

e-mail-2018-03-26.txt:SharedFSAutoReplicaFailoverTest.java
e-mail-2018-04-02.txt:SharedFSAutoReplicaFailoverTest.java
e-mail-2018-04-09.txt:SharedFSAutoReplicaFailoverTest.java
e-mail-2018-04-16.txt:SharedFSAutoReplicaFailoverTest.java
e-mail-2018-04-30.txt:SharedFSAutoReplicaFailoverTest.java
e-mail-2018-05-21.txt:SharedFSAutoReplicaFailoverTest.java
e-mail-2018-06-11.txt:   SharedFSAutoReplicaFailoverTest.test
e-mail-2018-06-11.txt:        3 100.0        2      2      SharedFSAutoReplicaFailoverTest(suite)
e-mail-2018-06-11.txt:SharedFSAutoReplicaFailoverTest.test
e-mail-2018-06-18.txt:     0    100.0        2      2      SharedFSAutoReplicaFailoverTest(suite)
e-mail-2018-06-25.txt:     01    74.1       29     22      SharedFSAutoReplicaFailoverTest(suite)
e-mail-2018-06-25.txt:     0      5.9       34      2      SharedFSAutoReplicaFailoverTest.test
e-mail-2018-07-02.txt:     012   74.1       56     42      SharedFSAutoReplicaFailoverTest(suite)
e-mail-2018-07-02.txt:     01     5.1       73      4      SharedFSAutoReplicaFailoverTest.test
e-mail-2018-07-09.txt:     0123  74.1       83     62      SharedFSAutoReplicaFailoverTest(suite)
e-mail-2018-07-09.txt:     012    2.3      117      5      SharedFSAutoReplicaFailoverTest.test
e-mail-2018-07-16.txt:     0123  74.1      108     80      SharedFSAutoReplicaFailoverTest(suite)
e-mail-2018-07-16.txt:     0123  17.6      151     11      SharedFSAutoReplicaFailoverTest.test
e-mail-2018-07-23.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2018-07-23.txt:SharedFSAutoReplicaFailoverTest.test
e-mail-2018-07-30.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2018-07-30.txt:SharedFSAutoReplicaFailoverTest.test
e-mail-2018-08-06.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2018-08-06.txt:SharedFSAutoReplicaFailoverTest.test
e-mail-2018-08-14.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2018-08-14.txt:SharedFSAutoReplicaFailoverTest.test
e-mail-2018-08-20.txt:   SharedFSAutoReplicaFailoverTest.test
e-mail-2018-08-20.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2018-08-20.txt:SharedFSAutoReplicaFailoverTest.test
e-mail-2018-08-27.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2018-09-03.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2018-09-10.txt:     0     20.0        5      1      SharedFSAutoReplicaFailoverTest.test
e-mail-2018-09-10.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2018-09-18.txt:     01    33.3        8      2      SharedFSAutoReplicaFailoverTest.test
e-mail-2018-09-18.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2018-10-08.txt:        3  33.3        3      1      SharedFSAutoReplicaFailoverTest.test
e-mail-2018-10-08.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2018-12-24.txt:     0  3  33.3        6      2      SharedFSAutoReplicaFailoverTest.test
e-mail-2018-12-24.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-01-08.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-01-15.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-02-12.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-02-18.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-03-04.txt:      1    33.3        3      1      SharedFSAutoReplicaFailoverTest.test
e-mail-2019-03-04.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-03-11.txt:       2   33.3        3      1      SharedFSAutoReplicaFailoverTest.test
e-mail-2019-03-11.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-03-18.txt:        3  33.3        3      1      SharedFSAutoReplicaFailoverTest.test
e-mail-2019-03-18.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-03-25.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-04-01.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-04-08.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-04-15.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-04-22.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-05-06.txt:     0    100.0        1      1      SharedFSAutoReplicaFailoverTest.test
e-mail-2019-05-06.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-05-20.txt:       2  100.0        1      1      SharedFSAutoReplicaFailoverTest.test
e-mail-2019-05-20.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-06-03.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-06-10.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-06-17.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-06-24.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-07-01.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-07-29.txt:      1    18.2       11      2      SharedFSAutoReplicaFailoverTest.test
e-mail-2019-07-29.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-08-05.txt:       2   18.2       11      2      SharedFSAutoReplicaFailoverTest.test
e-mail-2019-08-05.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-08-12.txt:        3  18.2       11      2      SharedFSAutoReplicaFailoverTest.test
e-mail-2019-08-12.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-08-19.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-09-16.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-10-28.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-11-04.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-11-11.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-11-18.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-11-25.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-12-02.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-12-09.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-12-23.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2019-12-30.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-01-06.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-01-13.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-01-20.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-01-27.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-02-03.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-02-10.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-03-16.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-03-24.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-03-30.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-04-06.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-04-13.txt:     0     80.0       10      8      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-04-13.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-04-20.txt:     01    50.0       16     11      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-04-20.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-04-27.txt:     012   80.0       21     15      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-04-27.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-05-04.txt:      123  80.0       21     15      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-05-04.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-05-11.txt:       23  80.0       11      7      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-05-11.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-05-18.txt:        3  80.0        5      4      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-05-18.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-05-25.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-06-01.txt:     0     57.1        7      4      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-06-01.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-06-08.txt:      1    57.1        7      4      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-06-08.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-06-15.txt:     0 2   50.0        9      5      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-06-15.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-06-22.txt:      1 3  50.0        9      5      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-06-22.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-06-29.txt:     0 2   90.0       12     10      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-06-29.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-07-06.txt:     01 3 100.0       16     14      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-07-06.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-07-13.txt:     012  100.0       18     17      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-07-13.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-07-20.txt:     0123  57.1       25     21      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-07-20.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-07-27.txt:     0123  93.3       30     26      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-07-27.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-08-03.txt:     0123  93.3       41     36      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-08-03.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite
e-mail-2020-08-10.txt:     0123 100.0       38     33      SharedFSAutoReplicaFailoverTest.test
e-mail-2020-08-10.txt:SharedFSAutoReplicaFailoverTest.SharedFSAutoReplicaFailoverTest suite


Hoss’ weekly rollups:
HOSS-2018-03-26.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","14.2857142857143","7","1"
HOSS-2018-04-16.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","100","2","2"
HOSS-2018-04-25.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","80","5","4"
HOSS-2018-05-21.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","100","2","2"
HOSS-2018-06-18.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","100","2","2"
HOSS-2018-06-25.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","74.0740740740741","27","20"
HOSS-2018-06-25.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","5.88235294117647","34","2"
HOSS-2018-07-02.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","74.0740740740741","27","20"
HOSS-2018-07-02.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","5.12820512820513","39","2"
HOSS-2018-07-09.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","74.0740740740741","27","20"
HOSS-2018-07-09.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","2.27272727272727","44","1"
HOSS-2018-07-16.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","74.0740740740741","27","20"
HOSS-2018-07-16.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","17.6470588235294","34","6"
HOSS-2018-07-23.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","62.5","16","10"
HOSS-2018-07-23.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","4.34782608695652","23","1"
HOSS-2018-08-06.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","66.6666666666667","3","2"
HOSS-2018-08-13.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","100","2","2"
HOSS-2018-09-10.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","20","5","1"
HOSS-2018-09-10.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","50","4","2"
HOSS-2018-09-18.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","66.6666666666667","3","2"
HOSS-2018-09-18.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","33.3333333333333","3","1"
HOSS-2018-10-02.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","100","4","4"
HOSS-2018-12-03.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","100","3","3"
HOSS-2018-12-03.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","33.3333333333333","3","1"
HOSS-2018-12-10.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","150","2","3"
HOSS-2018-12-24.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","33.3333333333333","3","1"
HOSS-2019-01-28.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","60","5","3"
HOSS-2019-02-25.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","33.3333333333333","3","1"
HOSS-2019-05-06.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","100","1","1"
HOSS-2019-06-17.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","100","3","3"
HOSS-2019-07-08.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","18.1818181818182","11","2"
HOSS-2019-12-23.csv:"true","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","","100","3","3"
HOSS-2020-04-13.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","80","10","8"
HOSS-2020-04-20.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","50","6","3"
HOSS-2020-04-27.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","80","5","4"
HOSS-2020-06-01.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","57.1428571428571","7","4"
HOSS-2020-06-15.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","50","2","1"
HOSS-2020-06-29.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","90","10","9"
HOSS-2020-07-06.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","100","4","4"
HOSS-2020-07-13.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","100","4","4"
HOSS-2020-07-20.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","57.1428571428571","7","4"
HOSS-2020-07-27.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","93.3333333333333","15","14"
HOSS-2020-08-03.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","93.3333333333333","15","14"
HOSS-2020-08-10.csv:"false","org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest","test","100","1","1"


> On Aug 12, 2020, at 9:03 AM, Erick Erickson <[hidden email]> wrote:
>
>> SharedFSAutoReplicaFailoverTest


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: BadApple report, but please read the first bit

David Smiley
In reply to this post by Kevin Risden-3
Thanks Kevin; clearly I missed the link to that which I can now see at fucit.

I was worried I may have worked on something that could have perturbed this recent issue but no -- I don't think so.

~ David Smiley
Apache Lucene/Solr Search Developer


On Wed, Aug 12, 2020 at 9:08 AM Kevin Risden <[hidden email]> wrote:
http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

David for that specific test you asked the failures are recent with as far as I know no change to HDFS stuff. Starting June/July failing regularly.

Kevin Risden



On Wed, Aug 12, 2020 at 9:03 AM Erick Erickson <[hidden email]> wrote:
I have the weekly rollups (with a few gaps) going back to about April 2018, but nothing’s been done to try to make them generally available. Each BadApple report has rates for the last 4 weeks in the attached file, just below "Failures over the last 4 weeks, but not every week. Ordered most-recent first:”



> On Aug 12, 2020, at 2:06 AM, David Smiley <[hidden email]> wrote:
>
> Do we have any long term (aka "longitudinal") pass/fail rates for tests?
>
> SharedFSAutoReplicaFailoverTest in particular is kinda-sorta tied to HDFS, and that's going away to a plug-in for 9.0.  The shared file system notion isn't well supported in SolrCloud, I think.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Aug 3, 2020 at 7:26 AM Erick Erickson <[hidden email]> wrote:
> There are several tests that are causing a lot of noise:
>
> SharedFSAutoReplicaFailoverTest is failing 90%+ of the time.
> TestBulkSchemaConcurrent 31%
> StressHdfsTest  16%
> SchemaApiFailureTest 13.88%
>
> I encourage people to look at: http://fucit.org/solr-jenkins-reports/failure-report.html and see if anything looks like it is affected by recent work. TestBulkSchemaConcurrent has been failing off and on for a long time, but the failure rate picked up dramatically in the last couple of weeks. Ditto SchemaApiFailureTest.
>
> Do we even care about Hdfs? Are we deprecating it or not?
>
> Holding relatively steady otherwise:
>
> Raw fail count by week totals, most recent week first (corresponds to bits):
> Week: 0  had  82 failures
> Week: 1  had  94 failures
> Week: 2  had  502 failures
> Week: 3  had  19 failures
>
>
> ********Failures in Hoss' reports for the last 4 rollups.
>
> There were 562 unannotated tests that failed in Hoss' rollups. Ordered by the date I downloaded the rollup file, newest->oldest. See above for the dates the files were collected
> These tests were NOT BadApple'd or AwaitsFix'd
>
> Failures in the last 4 reports..
>    Report   Pct     runs    fails           test
>      0123   0.3     1271      8      RollingRestartTest.test
>      0123  93.3       41     36      SharedFSAutoReplicaFailoverTest.test
>      0123   3.5      627     16      TestCircuitBreaker.testBuildingMemoryPressure
>      0123   1.0      627      8      TestCircuitBreaker.testResponseWithCBTiming
>      0123   5.8     1483     79      TestContainerPlugin.testApiFromPackage
>      0123   2.3     1335     23      TestDistributedGrouping.test
> ********************************************
>
>
> Full report:
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]