Software quality strongly depends on software testing. Due to the huge complexity of software systems they will always contain errors. Even for a single function with a few parameters you would need thousands or millions of tests to check the entire functionality. Software system contain a huge number of functions and additional these functions may have dependencies and will have different behavior depending on the system status and the sequence of called functions. Therefore, it is impossible to create a complete test for a software system and find all errors.
As consequence, software tests are focused on use case specific situations. They will primary test the software according to their intended use. But as soon as the software is released, there will be users which do not use the software in the intended way. This may result in situations not covered by the software tests. Such unexpected inputs or processes may discover errors.
As described before, the main part of software tests should focus on the intended use of the software and check concrete use cases, inputs and results. Focusing on the intended use will result in a manageable number of use cases which could be covered by according test cases.
But how can we simulate and test unexpected user inputs which are not according to the intended use? Here we come back to the issue that this results in a nearly unlimited number of possible test cases.
One idea to solve this issue is testing with random input parameters and random execution sequences. Such test procedures are called fuzz testing or robustness testing. They will call software interfaces or functions with random data and check the result in some general way. For example, in case a function returns an execution status, this status can be checked. Another common way is to check whether the function is executed within a defined timespan and without a software crash.
Therefore, fuzz tests are not used to find specific errors, like wrong results. They are used to find general issues like software crash or freeze. This will allow to create a huge number of automated tests with random input parameters and random execution sequences.
Another advantage of automatically generated tests is the possibility to create long test sequences. Standard test cases which are used to check the intended behavior are normally very short. They will often test one functionality only and then reset the software system to a start state. Of course, there should also be test cases which will check longer execution sequence, but these tests are relatively rare as it is much more time consuming to develop such complex test cases.
Fuzz tests offer a good way to compensate this issue. With fuzz tests it will become easy to randomly test short and long execution sequences. But of course, fuzz tests for execution sequences are no alternative to the normal use case specific test cases as they don’t find specific errors. But again, they are a good complement to the standard tests as they can find general errors like software crashes or freezes.
Evaluation and documentation of fuzz tests
Fuzz testing allows to create and executed a huge number of test cases within a short time span. As these tests are executed with random values and execution sequences, each test execution results in new test cases. Therefore, fuzz testing tools can normally record and repeat the test cases. Of course, in case of an error, it is mandatory to record and store the test case. But depending on the specific project needs it may not be necessary to store the details of successful test cases is you don’t want to repeat them in the same way.
The documentation of fuzz tests may follow the same principle. It should be possible to create a short report with detail information for the erroneous test cases and summarized information for the successful test cases. And it should be possible to create a full test report with detail information about all executed test cases. Of course, such a full report may become unmanageable very fast as it contains a lot of data.
Simple fuzz tests are based on random test parameters. This will lead to redundant test cases. Furthermore, there is no distinction between practical data and non-practical data. These disadvantages may result in inefficient fuzz tests. Of course, fuzz tests should use random data because they should find error situations that nobody thought of. But fuzz testing will become more efficient if it is possible to create test date with a higher focus on covering the areas around the main use cases. Such an optimization is called smart fuzzing.
Smart fuzzing is a kind of gray box testing. So, the test system must have knowledge about the software system to test. For example, it may know the internal state machine and the meaning of input parameters. This will allow to create practical test data and reduce the number of redundant test cases.
With smart fuzzing it is possible to increase the test quality. But, of course, you don’t get this for free. The test system must get the knowledge about the software system to test. So smart fuzzing comes with a high effort for set up and teaching of the test system.
Fuzz testing improves software quality. It allows automated tests of software functionality outside of the main use cases. It is mainly focused on tests with uncommon or unexpected data and unusual sequence of events. Simple fuzz test can be created with little effort. According testing tools will allow to create, execute, record and document such test with random input data. If you want to create efficient and high-quality fuzz tests you should spend more time and use tools which allow smart fuzz tests. This kind of tests need knowledge about the test system and therefore you should schedule a higher effort for configuration, teaching or even implementation of such smart fuzz test tools.