Re: [JENKINS] Lucene-Solr-repro-Java11 - Build # 1162 - Still Failing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [JENKINS] Lucene-Solr-repro-Java11 - Build # 1162 - Still Failing

Jan Høydahl / Cominvent
The repro python script is failing on Jenkins due to SSL cert trust issues when fetching the console output

   File "dev-tools/scripts/reproduceJenkinsFailures.py", line 110, in fetchAndParseJenkinsLog

The SSL cert of Jenkins seems legit, so it could be that Python somehow does not have access to the CA certs of the host (anymore)?
We could ‘pip install certifi’ or add code to not verify certs at all?

Jan

19. jun. 2020 kl. 00:08 skrev Apache Jenkins Server <[hidden email]>:

Build: https://builds.apache.org/job/Lucene-Solr-repro-Java11/1162/

[...truncated 32 lines...]
[repro] Jenkins log URL: https://builds.apache.org/job/Lucene-Solr-Tests-master/4813/consoleText

[...truncated 48 lines...]
   raise RuntimeError('ERROR: fetching %s : %s' % (url, e))
RuntimeError: ERROR: fetching https://builds.apache.org/job/Lucene-Solr-Tests-master/4813/consoleText : <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

---------------------------------------------------------------------
To unsubscribe, [hidden email]
For additional commands, [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [JENKINS] Lucene-Solr-repro-Java11 - Build # 1162 - Still Failing

Jan Høydahl / Cominvent
The same happens with other python scripts like the smoke tester:

  [smoker] Retrying download of url https://archive.apache.org/dist/lucene/KEYS after exception: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>

I also have a customer where this suddenly started happening, so I suspect a Linux upgrade or perhaps a Python upgrade messed with the default CA cert location or something.
Uwe, have you ever seen similar on Policeman Jenkins?

Jan


19. jun. 2020 kl. 00:24 skrev Jan Høydahl <[hidden email]>:

The repro python script is failing on Jenkins due to SSL cert trust issues when fetching the console output

   File "dev-tools/scripts/reproduceJenkinsFailures.py", line 110, in fetchAndParseJenkinsLog

The SSL cert of Jenkins seems legit, so it could be that Python somehow does not have access to the CA certs of the host (anymore)?
We could ‘pip install certifi’ or add code to not verify certs at all?

Jan

19. jun. 2020 kl. 00:08 skrev Apache Jenkins Server <[hidden email]>:

Build: https://builds.apache.org/job/Lucene-Solr-repro-Java11/1162/

[...truncated 32 lines...]
[repro] Jenkins log URL: https://builds.apache.org/job/Lucene-Solr-Tests-master/4813/consoleText

[...truncated 48 lines...]
   raise RuntimeError('ERROR: fetching %s : %s' % (url, e))
RuntimeError: ERROR: fetching https://builds.apache.org/job/Lucene-Solr-Tests-master/4813/consoleText : <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

---------------------------------------------------------------------
To unsubscribe, [hidden email]
For additional commands, [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: [JENKINS] Lucene-Solr-repro-Java11 - Build # 1162 - Still Failing

Jan Høydahl / Cominvent
@Bruno

Note that the smoke tester is not happy, so someone needs to fix it before the release can be made.
I sent this email to dev list June 19th but no progress.

Looking into it again, I see that some Jenkins smoketest runs succeeds and some fail.
I think I have found a pattern that all the failures are on build slave «lucene2» and all builds on «lucene» succeed.
and this that succeeds on «lucene».

So my theory is still that the CA certificates Python3 uses by default on «lucene2» are messed up, either by a Linux of Python upgrade on that box. Can someone verify?

Jan

19. jun. 2020 kl. 20:39 skrev Jan Høydahl <[hidden email]>:

The same happens with other python scripts like the smoke tester:

  [smoker] Retrying download of url https://archive.apache.org/dist/lucene/KEYS after exception: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>

I also have a customer where this suddenly started happening, so I suspect a Linux upgrade or perhaps a Python upgrade messed with the default CA cert location or something.
Uwe, have you ever seen similar on Policeman Jenkins?

Jan


19. jun. 2020 kl. 00:24 skrev Jan Høydahl <[hidden email]>:

The repro python script is failing on Jenkins due to SSL cert trust issues when fetching the console output

   File "dev-tools/scripts/reproduceJenkinsFailures.py", line 110, in fetchAndParseJenkinsLog

The SSL cert of Jenkins seems legit, so it could be that Python somehow does not have access to the CA certs of the host (anymore)?
We could ‘pip install certifi’ or add code to not verify certs at all?

Jan

19. jun. 2020 kl. 00:08 skrev Apache Jenkins Server <[hidden email]>:

Build: https://builds.apache.org/job/Lucene-Solr-repro-Java11/1162/

[...truncated 32 lines...]
[repro] Jenkins log URL: https://builds.apache.org/job/Lucene-Solr-Tests-master/4813/consoleText

[...truncated 48 lines...]
   raise RuntimeError('ERROR: fetching %s : %s' % (url, e))
RuntimeError: ERROR: fetching https://builds.apache.org/job/Lucene-Solr-Tests-master/4813/consoleText : <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

---------------------------------------------------------------------
To unsubscribe, [hidden email]
For additional commands, [hidden email]



Reply | Threaded
Open this post in threaded view
|

Re: [JENKINS] Lucene-Solr-repro-Java11 - Build # 1162 - Still Failing

Mike Drob-3
I had previously opened https://issues.apache.org/jira/browse/INFRA-20452 and https://issues.apache.org/jira/browse/LUCENE-9412 for this. Any more details posted there would be appreciated!

On Tue, Jun 30, 2020 at 8:23 AM Jan Høydahl <[hidden email]> wrote:
@Bruno

Note that the smoke tester is not happy, so someone needs to fix it before the release can be made.
I sent this email to dev list June 19th but no progress.

Looking into it again, I see that some Jenkins smoketest runs succeeds and some fail.
I think I have found a pattern that all the failures are on build slave «lucene2» and all builds on «lucene» succeed.
and this that succeeds on «lucene».

So my theory is still that the CA certificates Python3 uses by default on «lucene2» are messed up, either by a Linux of Python upgrade on that box. Can someone verify?

Jan

19. jun. 2020 kl. 20:39 skrev Jan Høydahl <[hidden email]>:

The same happens with other python scripts like the smoke tester:

  [smoker] Retrying download of url https://archive.apache.org/dist/lucene/KEYS after exception: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>

I also have a customer where this suddenly started happening, so I suspect a Linux upgrade or perhaps a Python upgrade messed with the default CA cert location or something.
Uwe, have you ever seen similar on Policeman Jenkins?

Jan


19. jun. 2020 kl. 00:24 skrev Jan Høydahl <[hidden email]>:

The repro python script is failing on Jenkins due to SSL cert trust issues when fetching the console output

   File "dev-tools/scripts/reproduceJenkinsFailures.py", line 110, in fetchAndParseJenkinsLog

The SSL cert of Jenkins seems legit, so it could be that Python somehow does not have access to the CA certs of the host (anymore)?
We could ‘pip install certifi’ or add code to not verify certs at all?

Jan

19. jun. 2020 kl. 00:08 skrev Apache Jenkins Server <[hidden email]>:

Build: https://builds.apache.org/job/Lucene-Solr-repro-Java11/1162/

[...truncated 32 lines...]
[repro] Jenkins log URL: https://builds.apache.org/job/Lucene-Solr-Tests-master/4813/consoleText

[...truncated 48 lines...]
   raise RuntimeError('ERROR: fetching %s : %s' % (url, e))
RuntimeError: ERROR: fetching https://builds.apache.org/job/Lucene-Solr-Tests-master/4813/consoleText : <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

---------------------------------------------------------------------
To unsubscribe, [hidden email]
For additional commands, [hidden email]