Solr Tests

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr Tests

Mark Miller-3
My boss has indicated I'm going to get a little time to spend on addressing our test situation. Turns out the current downward trend situation is a little worrying to some that count on the project down the road ;)

We can finally dig out though, I promise.

It's going to be a little bit before I'm clear and gathering steam, but I've got this figured out. I'm finally playing checkers instead of rock / paper / scissors.

I'm going to eat a lot of work to get there and I promise I can get there, but I'm going to need some help.

If you have an interest in changing our test situation, and I know some of you do, please join me as I invite others to help out more in a short time.

Beyond some help with code effort though, please get involved in discussions and JIRA issues around changing our test culture and process. We need to turn this corner as a group, we need to come together on some new behaviors, we need to generate large enough buy in to do something lasting.

You all need to learn 'ant beast' (I'm fixing it to work *much* better) and my beasting test script gist (each has there place). I'm going to force it on you even if you don't ;)


For a long time I put most of my attention towards SolrCloud stability and scale. From 2012-2015 or so, that felt like the biggest threat to the project while also getting relatively little attention. I felt like it was 50/50 in 2014 that I'd even be talking to anyone about SolrCloud today. That hurdle is done and over with. The biggest threat to Solr and SolrCloud now is our tests.

I'll be working from this umbrella issue: 

https://issues.apache.org/jira/browse/SOLR-12801 : Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.

Join the discussion.

- Mark
--
Reply | Threaded
Open this post in threaded view
|

Re: Solr Tests

Anshum Gupta-3
+1 Mark! I’m in.

  Anshum


On Oct 24, 2018, at 11:05 PM, Mark Miller <[hidden email]> wrote:

My boss has indicated I'm going to get a little time to spend on addressing our test situation. Turns out the current downward trend situation is a little worrying to some that count on the project down the road ;)

We can finally dig out though, I promise.

It's going to be a little bit before I'm clear and gathering steam, but I've got this figured out. I'm finally playing checkers instead of rock / paper / scissors.

I'm going to eat a lot of work to get there and I promise I can get there, but I'm going to need some help.

If you have an interest in changing our test situation, and I know some of you do, please join me as I invite others to help out more in a short time.

Beyond some help with code effort though, please get involved in discussions and JIRA issues around changing our test culture and process. We need to turn this corner as a group, we need to come together on some new behaviors, we need to generate large enough buy in to do something lasting.

You all need to learn 'ant beast' (I'm fixing it to work *much* better) and my beasting test script gist (each has there place). I'm going to force it on you even if you don't ;)


For a long time I put most of my attention towards SolrCloud stability and scale. From 2012-2015 or so, that felt like the biggest threat to the project while also getting relatively little attention. I felt like it was 50/50 in 2014 that I'd even be talking to anyone about SolrCloud today. That hurdle is done and over with. The biggest threat to Solr and SolrCloud now is our tests.

I'll be working from this umbrella issue: 

https://issues.apache.org/jira/browse/SOLR-12801 : Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.

Join the discussion.

- Mark
--

Reply | Threaded
Open this post in threaded view
|

Re: Solr Tests

Gus Heck
This situation has irritated me to no end and "something to help tests" is on my short list of things to contribute, along with "streaming expression stuff", "what ever happened to 3191" and of course continuing to help with TRA's

Count me in.

On Thu, Oct 25, 2018, 2:35 AM Anshum Gupta <[hidden email] wrote:
+1 Mark! I’m in.

  Anshum


On Oct 24, 2018, at 11:05 PM, Mark Miller <[hidden email]> wrote:

My boss has indicated I'm going to get a little time to spend on addressing our test situation. Turns out the current downward trend situation is a little worrying to some that count on the project down the road ;)

We can finally dig out though, I promise.

It's going to be a little bit before I'm clear and gathering steam, but I've got this figured out. I'm finally playing checkers instead of rock / paper / scissors.

I'm going to eat a lot of work to get there and I promise I can get there, but I'm going to need some help.

If you have an interest in changing our test situation, and I know some of you do, please join me as I invite others to help out more in a short time.

Beyond some help with code effort though, please get involved in discussions and JIRA issues around changing our test culture and process. We need to turn this corner as a group, we need to come together on some new behaviors, we need to generate large enough buy in to do something lasting.

You all need to learn 'ant beast' (I'm fixing it to work *much* better) and my beasting test script gist (each has there place). I'm going to force it on you even if you don't ;)


For a long time I put most of my attention towards SolrCloud stability and scale. From 2012-2015 or so, that felt like the biggest threat to the project while also getting relatively little attention. I felt like it was 50/50 in 2014 that I'd even be talking to anyone about SolrCloud today. That hurdle is done and over with. The biggest threat to Solr and SolrCloud now is our tests.

I'll be working from this umbrella issue: 

https://issues.apache.org/jira/browse/SOLR-12801 : Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.

Join the discussion.

- Mark
--

Reply | Threaded
Open this post in threaded view
|

Re: Solr Tests

Erick Erickson
In reply to this post by Anshum Gupta-3
Well, winter is coming on with long cold nights. What better thing to
do than hunch over my computer working with tests?

+1, I'll certainly add what I can


On Wed, Oct 24, 2018 at 11:35 PM Anshum Gupta <[hidden email]> wrote:

>
> +1 Mark! I’m in.
>
>   Anshum
>
>
> On Oct 24, 2018, at 11:05 PM, Mark Miller <[hidden email]> wrote:
>
> My boss has indicated I'm going to get a little time to spend on addressing our test situation. Turns out the current downward trend situation is a little worrying to some that count on the project down the road ;)
>
> We can finally dig out though, I promise.
>
> It's going to be a little bit before I'm clear and gathering steam, but I've got this figured out. I'm finally playing checkers instead of rock / paper / scissors.
>
> I'm going to eat a lot of work to get there and I promise I can get there, but I'm going to need some help.
>
> If you have an interest in changing our test situation, and I know some of you do, please join me as I invite others to help out more in a short time.
>
> Beyond some help with code effort though, please get involved in discussions and JIRA issues around changing our test culture and process. We need to turn this corner as a group, we need to come together on some new behaviors, we need to generate large enough buy in to do something lasting.
>
> You all need to learn 'ant beast' (I'm fixing it to work *much* better) and my beasting test script gist (each has there place). I'm going to force it on you even if you don't ;)
>
> https://gist.githubusercontent.com/markrmiller/dbdb792216dc98b018ad/raw/cd084d01f405fa271af4b72a2848dc87374a1f71/gistfile1.sh
>
> For a long time I put most of my attention towards SolrCloud stability and scale. From 2012-2015 or so, that felt like the biggest threat to the project while also getting relatively little attention. I felt like it was 50/50 in 2014 that I'd even be talking to anyone about SolrCloud today. That hurdle is done and over with. The biggest threat to Solr and SolrCloud now is our tests.
>
> I'll be working from this umbrella issue:
>
> https://issues.apache.org/jira/browse/SOLR-12801 : Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.
>
> Join the discussion.
>
> - Mark
> --
> - Mark
> about.me/markrmiller
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Solr Tests

Cassandra Targett
In reply to this post by Anshum Gupta-3
Hopefully they don't mind too much that I'm speaking for them, but on behalf of myself and the 7 other committers that make up the Solr Team at Lucidworks, we are also in and eager to help.

We've been trying to chip away at the problem for the past year+, but we need a systemic change to really make it better. I will do what I can to delay or defer some of our Lucidworks planned tasks to free up as much time for people as possible.

On Thu, Oct 25, 2018 at 1:35 AM Anshum Gupta <[hidden email]> wrote:
+1 Mark! I’m in.

  Anshum


On Oct 24, 2018, at 11:05 PM, Mark Miller <[hidden email]> wrote:

My boss has indicated I'm going to get a little time to spend on addressing our test situation. Turns out the current downward trend situation is a little worrying to some that count on the project down the road ;)

We can finally dig out though, I promise.

It's going to be a little bit before I'm clear and gathering steam, but I've got this figured out. I'm finally playing checkers instead of rock / paper / scissors.

I'm going to eat a lot of work to get there and I promise I can get there, but I'm going to need some help.

If you have an interest in changing our test situation, and I know some of you do, please join me as I invite others to help out more in a short time.

Beyond some help with code effort though, please get involved in discussions and JIRA issues around changing our test culture and process. We need to turn this corner as a group, we need to come together on some new behaviors, we need to generate large enough buy in to do something lasting.

You all need to learn 'ant beast' (I'm fixing it to work *much* better) and my beasting test script gist (each has there place). I'm going to force it on you even if you don't ;)


For a long time I put most of my attention towards SolrCloud stability and scale. From 2012-2015 or so, that felt like the biggest threat to the project while also getting relatively little attention. I felt like it was 50/50 in 2014 that I'd even be talking to anyone about SolrCloud today. That hurdle is done and over with. The biggest threat to Solr and SolrCloud now is our tests.

I'll be working from this umbrella issue: 

https://issues.apache.org/jira/browse/SOLR-12801 : Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.

Join the discussion.

- Mark
--

Reply | Threaded
Open this post in threaded view
|

Re: Solr Tests

Mark Miller-3
Yeah, there has been a lot of chipping for a very long time :)

Many valiant attempts have been made to get ahold of this problem. I've been part of some of them. It's like zombie whack-a-mole.

That is why you could almost read my email as, oh, again the tests, sure,  that sounds exciting, no has ever tried to fix the tests before, see you next year.

But this time is different. If I get even a quarter the amount of promised help I've already heard about, we will get to solid tests this time, and we will add support to enforce we stay there.

I'm already further along the path than it looks, but that will become more clear soon.

- Mark

On Thu, Oct 25, 2018 at 11:55 AM Cassandra Targett <[hidden email]> wrote:
Hopefully they don't mind too much that I'm speaking for them, but on behalf of myself and the 7 other committers that make up the Solr Team at Lucidworks, we are also in and eager to help.

We've been trying to chip away at the problem for the past year+, but we need a systemic change to really make it better. I will do what I can to delay or defer some of our Lucidworks planned tasks to free up as much time for people as possible.

On Thu, Oct 25, 2018 at 1:35 AM Anshum Gupta <[hidden email]> wrote:
+1 Mark! I’m in.

  Anshum


On Oct 24, 2018, at 11:05 PM, Mark Miller <[hidden email]> wrote:

My boss has indicated I'm going to get a little time to spend on addressing our test situation. Turns out the current downward trend situation is a little worrying to some that count on the project down the road ;)

We can finally dig out though, I promise.

It's going to be a little bit before I'm clear and gathering steam, but I've got this figured out. I'm finally playing checkers instead of rock / paper / scissors.

I'm going to eat a lot of work to get there and I promise I can get there, but I'm going to need some help.

If you have an interest in changing our test situation, and I know some of you do, please join me as I invite others to help out more in a short time.

Beyond some help with code effort though, please get involved in discussions and JIRA issues around changing our test culture and process. We need to turn this corner as a group, we need to come together on some new behaviors, we need to generate large enough buy in to do something lasting.

You all need to learn 'ant beast' (I'm fixing it to work *much* better) and my beasting test script gist (each has there place). I'm going to force it on you even if you don't ;)


For a long time I put most of my attention towards SolrCloud stability and scale. From 2012-2015 or so, that felt like the biggest threat to the project while also getting relatively little attention. I felt like it was 50/50 in 2014 that I'd even be talking to anyone about SolrCloud today. That hurdle is done and over with. The biggest threat to Solr and SolrCloud now is our tests.

I'll be working from this umbrella issue: 

https://issues.apache.org/jira/browse/SOLR-12801 : Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.

Join the discussion.

- Mark
--

--
Reply | Threaded
Open this post in threaded view
|

Re: Solr Tests

Mark Miller-3
In response to Erick's question in another thread of: "So, what I can I do to help?"

I actually have a lot of thoughts on that, but I think I need a little time to get some things I have in the pipeline fully fleshed out and ready to share.

At a very high level though...

My thoughts on Solr tests are this:

You can point me to pretty much any failing test and I can fix it. Probably I'll find 10 things to improve when I do it.

That doesn't seem to be a widespread skill and where it does exist , it is often under utilized.

So my dilemma is this: I can fix the tests. One by one, over weeks and months, I can fix the tests. But I cannot stop you from breaking them. Sometimes I can't stop myself.

It follows that I need a way to stop you and me from breaking the tests once they are fixed.

And I also need to spread the skill of actually fixing a test in combination with making it easier so that that skill is learned and not under utilized. Same as the skill of writing a good test - which is currently too hard, no doubt.

I have some major ideas on these problems, some of them are in the works.

So at a broad level, you can help by discussing and helping in any of the JIRA issues I've opened around SOLR-12801. Or jump on a test problem you know of.

Beyond that, once I've started actually fixing and defending tests, you can help by fixing tests and driving a culture of passing tests. Commits that change this should be reverted. 

I plan to fix and defend tests by package.

In a little time, I'll have much more specific requests.

- Mark


--
Reply | Threaded
Open this post in threaded view
|

Re: Solr Tests

Shawn Heisey-2
In reply to this post by Mark Miller-3
On 10/25/2018 12:05 AM, Mark Miller wrote:
> If you have an interest in changing our test situation, and I know
> some of you do, please join me as I invite others to help out more in
> a short time.

I'm interested in helping.

There's a lot of code that I'm not at all familiar with, and I've found
that I get *very* lost when I try exploring the rabbit hole to try and
*become* familiar with the code.  I learn a little more every day, which
mostly serves to inform me just how much I *don't* know.

If I see something I think I can do, I will work on it.  If anybody sees
a suitable candidate that I miss, feel free to prod me towards it.

> Beyond some help with code effort though, please get involved in
> discussions and JIRA issues around changing our test culture and
> process. We need to turn this corner as a group, we need to come
> together on some new behaviors, we need to generate large enough buy
> in to do something lasting.

I will add my two cents, and hopefully I won't aggravate everyone TOO
much. :)

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]