The Fuzzer That Does Not Fuzz

From: http://www.codenomicon.com/news/newsletter/archive/2009-08.shtml

"The fuzzer that does not fuzz", was how Codenomicon test tools were described at Black Hat USA 2009. Without necessarily knowing it, the speaker made the biggest compliment to our tools anyone has given for years, if ever.

Before 1998, all fuzzers that at least I know about were entirely stateless, and purely random. You should not really even consider this approach any more.

After 1998, in the PROTOS project, we described an approach where no randomness was involved at all. We called it Robustness Testing, based on definitions we heard e.g. ETSI use for such a testing approach. Other names for similar approach are grammar testing (used e.g. by Wurldtech) and syntax testing (used by testing specialists everywhere).

In PROTOS we noticed that if a protocol was modelled using dynamic and thorough state machines and message descriptions, there was no need for randomness anymore. Actually, the incremental benefits of adding random tests to the systematically built tests was so insignificant that eventually we just left them out entirely. Everything was carefully optimized. Test execution times were extremely fast (from minutes to few hours), and test coverage was much better than with other techniques, even those in use today.

After almost ten years, block-based fuzzers were invented. They are a kind of cross-breed between the purely random, non-protocol-aware fuzzers of the early 90s, and robustness testing tools that are entirely based on protocol models and systematic test generation. A block-based fuzzer adds enough protocol awareness to its minimalistic model and state diagrams, to be able to somewhat limit the amount of random or semi-random changes it does. Why did the inventors include any randomness at all? Because a fuzzer is supposed to do random testing - or perhaps that is just what people thought.

So when someone calls us a "fuzzer that does not fuzz", they are finally understanding the difference between a fuzzer, and a Robustness Testing tool. Even though finally in 2008 we decided to call our tools fuzzers also, they really isn't anything fuzzy about our tools. And we are proud of that!