# Random testing performance (theoretical)

4 messages
Open this post in threaded view
|

## Random testing performance (theoretical)

 An interesting link was given to me today: "The Central Limit Theorem Makes Random Testing Hard" http://blog.regehr.org/archives/660Just recently I had a talk at a local university (http://idss.cs.put.poznan.pl/site/idss-en.html) and the low theoretical probability of hitting _any_ sensible bug was also raised as a concern. This is in stark contrast to what we see in practice, isn't it? It is an interesting phenomenon because it means that (among other possible explanations) the parameter/ search space of random tests is just ridden with failures and exceptions and bugs and even with close-to-zero probability we still hit a lot of them :) Dawid --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]
Open this post in threaded view
|

## RE: Random testing performance (theoretical)

 Hi Dawid, I think we do not do completely random testing at all! Random testing (with the limits mentioned in the article) would mean that we do very random testing with also randomly generated tests. In contrast our tests all have a non-random background, means our tests are written with a specific use-case in mind (there are some exemptions, I agree, see below). So with any random parameter they basically test what a non-random junit test would also test. In most cases the randomness is only added on top (to extend the space of parameters we test) - so the chances to find bugs get greater. If we would test Lucene only with also randomly generated tests, I agree we would find no bugs at all. On the other hand, classical junit tests have no randomness at all, they just test that a specific "requirement" is implemented correctly and returns the correct result. The problem with those tests is the fact, that by fixing a bug to do the correct thing in test can break something else (this is especially important for IR tests, they rely on statistics!!!). You could write a test that expects a specific output of a lucene query and you can of course fix all the scoring logic to exactly create that result. Of course any other similar query can return bogus results. So by executing random queries, we can also test statistical things like ranking function works also for a larger set of queries or larger number of documents. There are some exceptions: The famous TestRandomChains falls exactly in the category you mention: They just build a random chain of analyzer components and the number of combinations you could create is huge! With a few Tokenizers and a list of 150 TokenFilters you can create an insane amount of TokenStream combinations (just start to multiply..., it is the formula for the binomial coefficient). In that case, the chance to find a new bug is really small. The number of failures at the beginning really shows that the TokenStreams were in a very bad state! Now we have only few failures (one more today!), but I am sure there are still billions (ore more) of combinations that break the TokenStream contracts! Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.deeMail: [hidden email] > -----Original Message----- > From: [hidden email] [mailto:[hidden email]] On Behalf Of > Dawid Weiss > Sent: Sunday, June 24, 2012 3:42 PM > To: [hidden email] > Cc: [hidden email] > Subject: Random testing performance (theoretical) > > An interesting link was given to me today: > > "The Central Limit Theorem Makes Random Testing Hard" > http://blog.regehr.org/archives/660> > Just recently I had a talk at a local university > (http://idss.cs.put.poznan.pl/site/idss-en.html) and the low theoretical > probability of hitting _any_ sensible bug was also raised as a concern. This is in > stark contrast to what we see in practice, isn't it? It is an interesting > phenomenon because it means that (among other possible explanations) the > parameter/ search space of random tests is just ridden with failures and > exceptions and bugs and even with close-to-zero probability we still hit a lot of > them :) > > Dawid > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [hidden email] For additional > commands, e-mail: [hidden email] --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]