Not adding badapples this week.

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Not adding badapples this week.

Erick Erickson
With the long weekend and the fact that the number of non-BadApple
tests is fairly small this week, I'll skip adding more BadApple tests
until next week.

We're scarily close to the non-BaApple'd tests coming under  control.
If we're lucky, we can draw a line in the sand soon then start working
on the backlog. I'll be encouraged if we can start shrinking the
BadApple'd tests.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Not adding badapples this week.

Adrien Grand
Thanks for helping on this front Erick. I noticed a significant decrease in noise since you started badapple-ing bad tests, but I'm observing that our smoke-release builds still keep failing because of Solr tests (10 out of the last 10 builds) in spite of the fact that they disable bad apples:
 - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/
 - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/

Do you know whether there is something that makes these jobs any different?

Le mar. 29 mai 2018 à 18:12, Erick Erickson <[hidden email]> a écrit :
With the long weekend and the fact that the number of non-BadApple
tests is fairly small this week, I'll skip adding more BadApple tests
until next week.

We're scarily close to the non-BaApple'd tests coming under  control.
If we're lucky, we can draw a line in the sand soon then start working
on the backlog. I'll be encouraged if we can start shrinking the
BadApple'd tests.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Not adding badapples this week.

Erick Erickson
Adrien:

"Do you know whether there is something that makes these jobs any different?"

Unfortunately no. I'm not really very well versed in the various test
environments, maybe Uwe or Steve Rowe or Hoss might have some insight?

On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand <[hidden email]> wrote:

> Thanks for helping on this front Erick. I noticed a significant decrease in
> noise since you started badapple-ing bad tests, but I'm observing that our
> smoke-release builds still keep failing because of Solr tests (10 out of the
> last 10 builds) in spite of the fact that they disable bad apples:
>  - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/
>  - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/
>
> Do you know whether there is something that makes these jobs any different?
>
> Le mar. 29 mai 2018 à 18:12, Erick Erickson <[hidden email]> a
> écrit :
>>
>> With the long weekend and the fact that the number of non-BadApple
>> tests is fairly small this week, I'll skip adding more BadApple tests
>> until next week.
>>
>> We're scarily close to the non-BaApple'd tests coming under  control.
>> If we're lucky, we can draw a line in the sand soon then start working
>> on the backlog. I'll be encouraged if we can start shrinking the
>> BadApple'd tests.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Not adding badapples this week.

sarowe
I looked at the way that tests are run, an tthe only difference I see in the smoke tester jobs is that tests are run twice, once each for Java8 and Java9.  Compared to non-smoke-tester jobs, this will double the likelihood of overall failure.

I looked at the suites that failed in the last ten runs on those two smoke tester jobs.  Except for SearchHandlerTest, which I have since (hopefully) fixed, there are seven suites with failed tests - here they are along with their packages:

  TestExecutePlanAction     o.a.s.cloud.autoscaling.sim
  TestComputePlanAction     o.a.s.cloud.autoscaling.sim
  TestTriggerIntegration    o.a.s.cloud.autoscaling.sim
  IndexSizeTriggerTest      o.a.s.cloud.autoscaling
  CreateRoutedAliasTest     o.a.s.cloud
  ReplaceNodeTest           o.a.s.cloud
  MetricsHistoryHandlerTest o.a.s.handler.admin

Some of those are pretty regular offenders AFAICT from http://fucit.org/solr-jenkins-reports/failure-report.html .

Andrzej Białecki did some work on IndexSizeTriggerTest (SOLR-12392) and un-bad-apple’d its tests, but at least one of them is still failing since then - I’ll go add a comment on the issue.

--
Steve
www.lucidworks.com

> On Jun 4, 2018, at 11:16 AM, Erick Erickson <[hidden email]> wrote:
>
> Adrien:
>
> "Do you know whether there is something that makes these jobs any different?"
>
> Unfortunately no. I'm not really very well versed in the various test
> environments, maybe Uwe or Steve Rowe or Hoss might have some insight?
>
> On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand <[hidden email]> wrote:
>> Thanks for helping on this front Erick. I noticed a significant decrease in
>> noise since you started badapple-ing bad tests, but I'm observing that our
>> smoke-release builds still keep failing because of Solr tests (10 out of the
>> last 10 builds) in spite of the fact that they disable bad apples:
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/
>>
>> Do you know whether there is something that makes these jobs any different?
>>
>> Le mar. 29 mai 2018 à 18:12, Erick Erickson <[hidden email]> a
>> écrit :
>>>
>>> With the long weekend and the fact that the number of non-BadApple
>>> tests is fairly small this week, I'll skip adding more BadApple tests
>>> until next week.
>>>
>>> We're scarily close to the non-BaApple'd tests coming under  control.
>>> If we're lucky, we can draw a line in the sand soon then start working
>>> on the backlog. I'll be encouraged if we can start shrinking the
>>> BadApple'd tests.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>



Begin forwarded message:

> From: Erick Erickson <[hidden email]>
> Subject: Re: Not adding badapples this week.
> Date: June 4, 2018 at 11:16:56 AM EDT
> To: [hidden email]
> Reply-To: [hidden email]
>
> Adrien:
>
> "Do you know whether there is something that makes these jobs any different?"
>
> Unfortunately no. I'm not really very well versed in the various test
> environments, maybe Uwe or Steve Rowe or Hoss might have some insight?
>
> On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand <[hidden email]> wrote:
>> Thanks for helping on this front Erick. I noticed a significant decrease in
>> noise since you started badapple-ing bad tests, but I'm observing that our
>> smoke-release builds still keep failing because of Solr tests (10 out of the
>> last 10 builds) in spite of the fact that they disable bad apples:
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/
>>
>> Do you know whether there is something that makes these jobs any different?
>>
>> Le mar. 29 mai 2018 à 18:12, Erick Erickson <[hidden email]> a
>> écrit :
>>>
>>> With the long weekend and the fact that the number of non-BadApple
>>> tests is fairly small this week, I'll skip adding more BadApple tests
>>> until next week.
>>>
>>> We're scarily close to the non-BaApple'd tests coming under  control.
>>> If we're lucky, we can draw a line in the sand soon then start working
>>> on the backlog. I'll be encouraged if we can start shrinking the
>>> BadApple'd tests.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Not adding badapples this week.

Adrien Grand
Thanks Steve. IndexSizeTriggerTest seems to be the test that fails most often on this build indeed.

Le lun. 4 juin 2018 à 19:30, Steve Rowe <[hidden email]> a écrit :
I looked at the way that tests are run, an tthe only difference I see in the smoke tester jobs is that tests are run twice, once each for Java8 and Java9.  Compared to non-smoke-tester jobs, this will double the likelihood of overall failure.

I looked at the suites that failed in the last ten runs on those two smoke tester jobs.  Except for SearchHandlerTest, which I have since (hopefully) fixed, there are seven suites with failed tests - here they are along with their packages:

  TestExecutePlanAction     o.a.s.cloud.autoscaling.sim
  TestComputePlanAction     o.a.s.cloud.autoscaling.sim
  TestTriggerIntegration    o.a.s.cloud.autoscaling.sim
  IndexSizeTriggerTest      o.a.s.cloud.autoscaling
  CreateRoutedAliasTest     o.a.s.cloud
  ReplaceNodeTest           o.a.s.cloud
  MetricsHistoryHandlerTest o.a.s.handler.admin

Some of those are pretty regular offenders AFAICT from http://fucit.org/solr-jenkins-reports/failure-report.html .

Andrzej Białecki did some work on IndexSizeTriggerTest (SOLR-12392) and un-bad-apple’d its tests, but at least one of them is still failing since then - I’ll go add a comment on the issue.

--
Steve
www.lucidworks.com

> On Jun 4, 2018, at 11:16 AM, Erick Erickson <[hidden email]> wrote:
>
> Adrien:
>
> "Do you know whether there is something that makes these jobs any different?"
>
> Unfortunately no. I'm not really very well versed in the various test
> environments, maybe Uwe or Steve Rowe or Hoss might have some insight?
>
> On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand <[hidden email]> wrote:
>> Thanks for helping on this front Erick. I noticed a significant decrease in
>> noise since you started badapple-ing bad tests, but I'm observing that our
>> smoke-release builds still keep failing because of Solr tests (10 out of the
>> last 10 builds) in spite of the fact that they disable bad apples:
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/
>>
>> Do you know whether there is something that makes these jobs any different?
>>
>> Le mar. 29 mai 2018 à 18:12, Erick Erickson <[hidden email]> a
>> écrit :
>>>
>>> With the long weekend and the fact that the number of non-BadApple
>>> tests is fairly small this week, I'll skip adding more BadApple tests
>>> until next week.
>>>
>>> We're scarily close to the non-BaApple'd tests coming under  control.
>>> If we're lucky, we can draw a line in the sand soon then start working
>>> on the backlog. I'll be encouraged if we can start shrinking the
>>> BadApple'd tests.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>



Begin forwarded message:

> From: Erick Erickson <[hidden email]>
> Subject: Re: Not adding badapples this week.
> Date: June 4, 2018 at 11:16:56 AM EDT
> To: [hidden email]
> Reply-To: [hidden email]
>
> Adrien:
>
> "Do you know whether there is something that makes these jobs any different?"
>
> Unfortunately no. I'm not really very well versed in the various test
> environments, maybe Uwe or Steve Rowe or Hoss might have some insight?
>
> On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand <[hidden email]> wrote:
>> Thanks for helping on this front Erick. I noticed a significant decrease in
>> noise since you started badapple-ing bad tests, but I'm observing that our
>> smoke-release builds still keep failing because of Solr tests (10 out of the
>> last 10 builds) in spite of the fact that they disable bad apples:
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/
>>
>> Do you know whether there is something that makes these jobs any different?
>>
>> Le mar. 29 mai 2018 à 18:12, Erick Erickson <[hidden email]> a
>> écrit :
>>>
>>> With the long weekend and the fact that the number of non-BadApple
>>> tests is fairly small this week, I'll skip adding more BadApple tests
>>> until next week.
>>>
>>> We're scarily close to the non-BaApple'd tests coming under  control.
>>> If we're lucky, we can draw a line in the sand soon then start working
>>> on the backlog. I'll be encouraged if we can start shrinking the
>>> BadApple'd tests.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Not adding badapples this week.

Andrzej Białecki
Yesterday I committed several fixes to the simulation package, which should improve reliability of these tests. If these failures still persist we should BadApple the ones that keep failing.

On 5 Jun 2018, at 10:14, Adrien Grand <[hidden email]> wrote:

Thanks Steve. IndexSizeTriggerTest seems to be the test that fails most often on this build indeed.

Le lun. 4 juin 2018 à 19:30, Steve Rowe <[hidden email]> a écrit :
I looked at the way that tests are run, an tthe only difference I see in the smoke tester jobs is that tests are run twice, once each for Java8 and Java9.  Compared to non-smoke-tester jobs, this will double the likelihood of overall failure.

I looked at the suites that failed in the last ten runs on those two smoke tester jobs.  Except for SearchHandlerTest, which I have since (hopefully) fixed, there are seven suites with failed tests - here they are along with their packages:

  TestExecutePlanAction     o.a.s.cloud.autoscaling.sim
  TestComputePlanAction     o.a.s.cloud.autoscaling.sim
  TestTriggerIntegration    o.a.s.cloud.autoscaling.sim
  IndexSizeTriggerTest      o.a.s.cloud.autoscaling
  CreateRoutedAliasTest     o.a.s.cloud
  ReplaceNodeTest           o.a.s.cloud
  MetricsHistoryHandlerTest o.a.s.handler.admin

Some of those are pretty regular offenders AFAICT from http://fucit.org/solr-jenkins-reports/failure-report.html .

Andrzej Białecki did some work on IndexSizeTriggerTest (SOLR-12392) and un-bad-apple’d its tests, but at least one of them is still failing since then - I’ll go add a comment on the issue.

--
Steve
www.lucidworks.com

> On Jun 4, 2018, at 11:16 AM, Erick Erickson <[hidden email]> wrote:
>
> Adrien:
>
> "Do you know whether there is something that makes these jobs any different?"
>
> Unfortunately no. I'm not really very well versed in the various test
> environments, maybe Uwe or Steve Rowe or Hoss might have some insight?
>
> On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand <[hidden email]> wrote:
>> Thanks for helping on this front Erick. I noticed a significant decrease in
>> noise since you started badapple-ing bad tests, but I'm observing that our
>> smoke-release builds still keep failing because of Solr tests (10 out of the
>> last 10 builds) in spite of the fact that they disable bad apples:
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/
>>
>> Do you know whether there is something that makes these jobs any different?
>>
>> Le mar. 29 mai 2018 à 18:12, Erick Erickson <[hidden email]> a
>> écrit :
>>>
>>> With the long weekend and the fact that the number of non-BadApple
>>> tests is fairly small this week, I'll skip adding more BadApple tests
>>> until next week.
>>>
>>> We're scarily close to the non-BaApple'd tests coming under  control.
>>> If we're lucky, we can draw a line in the sand soon then start working
>>> on the backlog. I'll be encouraged if we can start shrinking the
>>> BadApple'd tests.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>



Begin forwarded message:

> From: Erick Erickson <[hidden email]>
> Subject: Re: Not adding badapples this week.
> Date: June 4, 2018 at 11:16:56 AM EDT
> To: [hidden email]
> Reply-To: [hidden email]
>
> Adrien:
>
> "Do you know whether there is something that makes these jobs any different?"
>
> Unfortunately no. I'm not really very well versed in the various test
> environments, maybe Uwe or Steve Rowe or Hoss might have some insight?
>
> On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand <[hidden email]> wrote:
>> Thanks for helping on this front Erick. I noticed a significant decrease in
>> noise since you started badapple-ing bad tests, but I'm observing that our
>> smoke-release builds still keep failing because of Solr tests (10 out of the
>> last 10 builds) in spite of the fact that they disable bad apples:
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/
>>
>> Do you know whether there is something that makes these jobs any different?
>>
>> Le mar. 29 mai 2018 à 18:12, Erick Erickson <[hidden email]> a
>> écrit :
>>>
>>> With the long weekend and the fact that the number of non-BadApple
>>> tests is fairly small this week, I'll skip adding more BadApple tests
>>> until next week.
>>>
>>> We're scarily close to the non-BaApple'd tests coming under  control.
>>> If we're lucky, we can draw a line in the sand soon then start working
>>> on the backlog. I'll be encouraged if we can start shrinking the
>>> BadApple'd tests.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Not adding badapples this week.

Anshum Gupta-3
In reply to this post by Adrien Grand
I’ll take the blame for that one as I said I’d take a look but never got to it. I should get some time during my flight for a conference tomorrow (after long) and I’ll make sure I prioritize this right after my talk prep. Should have enough time for both. Sorry about the noise!

 Anshum


On Jun 5, 2018, at 1:14 AM, Adrien Grand <[hidden email]> wrote:

Thanks Steve. IndexSizeTriggerTest seems to be the test that fails most often on this build indeed.

Le lun. 4 juin 2018 à 19:30, Steve Rowe <[hidden email]> a écrit :
I looked at the way that tests are run, an tthe only difference I see in the smoke tester jobs is that tests are run twice, once each for Java8 and Java9.  Compared to non-smoke-tester jobs, this will double the likelihood of overall failure.

I looked at the suites that failed in the last ten runs on those two smoke tester jobs.  Except for SearchHandlerTest, which I have since (hopefully) fixed, there are seven suites with failed tests - here they are along with their packages:

  TestExecutePlanAction     o.a.s.cloud.autoscaling.sim
  TestComputePlanAction     o.a.s.cloud.autoscaling.sim
  TestTriggerIntegration    o.a.s.cloud.autoscaling.sim
  IndexSizeTriggerTest      o.a.s.cloud.autoscaling
  CreateRoutedAliasTest     o.a.s.cloud
  ReplaceNodeTest           o.a.s.cloud
  MetricsHistoryHandlerTest o.a.s.handler.admin

Some of those are pretty regular offenders AFAICT from http://fucit.org/solr-jenkins-reports/failure-report.html .

Andrzej Białecki did some work on IndexSizeTriggerTest (SOLR-12392) and un-bad-apple’d its tests, but at least one of them is still failing since then - I’ll go add a comment on the issue.

--
Steve
www.lucidworks.com

> On Jun 4, 2018, at 11:16 AM, Erick Erickson <[hidden email]> wrote:
>
> Adrien:
>
> "Do you know whether there is something that makes these jobs any different?"
>
> Unfortunately no. I'm not really very well versed in the various test
> environments, maybe Uwe or Steve Rowe or Hoss might have some insight?
>
> On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand <[hidden email]> wrote:
>> Thanks for helping on this front Erick. I noticed a significant decrease in
>> noise since you started badapple-ing bad tests, but I'm observing that our
>> smoke-release builds still keep failing because of Solr tests (10 out of the
>> last 10 builds) in spite of the fact that they disable bad apples:
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/
>>
>> Do you know whether there is something that makes these jobs any different?
>>
>> Le mar. 29 mai 2018 à 18:12, Erick Erickson <[hidden email]> a
>> écrit :
>>>
>>> With the long weekend and the fact that the number of non-BadApple
>>> tests is fairly small this week, I'll skip adding more BadApple tests
>>> until next week.
>>>
>>> We're scarily close to the non-BaApple'd tests coming under  control.
>>> If we're lucky, we can draw a line in the sand soon then start working
>>> on the backlog. I'll be encouraged if we can start shrinking the
>>> BadApple'd tests.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>



Begin forwarded message:

> From: Erick Erickson <[hidden email]>
> Subject: Re: Not adding badapples this week.
> Date: June 4, 2018 at 11:16:56 AM EDT
> To: [hidden email]
> Reply-To: [hidden email]
>
> Adrien:
>
> "Do you know whether there is something that makes these jobs any different?"
>
> Unfortunately no. I'm not really very well versed in the various test
> environments, maybe Uwe or Steve Rowe or Hoss might have some insight?
>
> On Mon, Jun 4, 2018 at 3:34 AM, Adrien Grand <[hidden email]> wrote:
>> Thanks for helping on this front Erick. I noticed a significant decrease in
>> noise since you started badapple-ing bad tests, but I'm observing that our
>> smoke-release builds still keep failing because of Solr tests (10 out of the
>> last 10 builds) in spite of the fact that they disable bad apples:
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/
>> - https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/
>>
>> Do you know whether there is something that makes these jobs any different?
>>
>> Le mar. 29 mai 2018 à 18:12, Erick Erickson <[hidden email]> a
>> écrit :
>>>
>>> With the long weekend and the fact that the number of non-BadApple
>>> tests is fairly small this week, I'll skip adding more BadApple tests
>>> until next week.
>>>
>>> We're scarily close to the non-BaApple'd tests coming under  control.
>>> If we're lucky, we can draw a line in the sand soon then start working
>>> on the backlog. I'll be encouraged if we can start shrinking the
>>> BadApple'd tests.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



signature.asc (849 bytes) Download Attachment