Within software development projects you may often heard questions or statements like these:

“We are in trouble with the timelines so we have to skip some of the software tests. How will this affect the software quality?”

“Our software quality is low. Can we add some software tests to increase the quality?”

These are typical questions. But I think there is a general misunderstanding between the relationship of software tests and software quality. Maybe you can ask yourself: Is there a relationship between these two attributes and if yes what kind of relationship? As you will see, this question is very difficult to answer.

Quality does not mean testing! There are some managers and even developers who think in this way but to set these two attributes into a strict relationship will not work. Please, don’t understand me wrong. There may be a relationship but to say quality is equal to or only depends on testing is wrong. But what if we weaken the connection between these two attributes a little bit? If we do so, it will become difficult to find a concreate statement whether testing affects quality or not. I know there are many different opinions regarding this question. And of course there is no final answer. So this article will show my opinion and should give you some ideas and thoughts regarding this topic.

What is a Feature?

At first I want to have a look on another software attribute beside the quality: the software feature. A feature realization is always a cyclic execution of implementation and testing. To implement a feature, its requirements and therefore the expected use cases will be defined. The implementation itself always contains coding and testing. These two steps will be done in few cycles until the feature is finished. Therefore testing is an integral part of feature implementation. It ensures that the software feature will fit with the specified behavior and can be used by the customer. Without testing the feature will never ever work in all expected use cases.

Of course, software will never be without bugs, but if a feature is realized you should be sure it will work in normal cases. In special cases it may result in errors as not all special cases are tested. The difference between “normal” and “special” cases is: “normal” use cases cover the specified functionality of the feature and “special” use cases will use the feature in a way which is outside of the specification. If I talk about tests I mean the tests to check the expected and specified functionality of the feature. If you skip such tests, the feature will not work in some or many of the expected cases. Therefore if you skip tests you reduce the feature set of your release and not the quality.

What is Quality?

There are many definitions for quality. I like the common definitions which are independent of the kind of product and not specific for software development. A simple definition is: “Quality is the grade how requirements are meet”. This means the requirement must be fulfilled and quality adds some additional benefit. So another good definition is: “Quality is the sum of customer benefit beside the functionality of the product.”

What does this mean in terms of software? In software projects we normally have two main types of customers: the end user and the product owner. Sometimes these may be the same persons. In most cases the product owner is the company you are working for and the end users are the persons which buy and use the software.

Quality for the end user is focused on using the software. There may be attributes like: reliability, usability and efficiency. For example the wish “The software must be easy to use” is a typical quality criteria, except the user guidance in the software is specified in a detailed manner and therefore is implemented as feature.

Quality for the product owner is focused on further developing of the software. In this case there are attributes like: changeability, maintenance, flexibility and documentation. So the quality from product owner point of view is mainly focused on the code quality.

Does testing affect software quality?

Now, as we found a definition for the term “Quality” we will come back to our initial question: “Does testing affect software quality?” As mentioned before, “standard testing” means check the feature specific use cases. This kind of testing will not increase the quality of the product as it is not focused on quality criteria’s. It is focused on the feature.

But what if we do additional tests? That means, test which are independent of the feature specific standard tests. Can we increase the product quality with such tests?

As said on the beginning of the article: Skip testing does not decrease the software quality it removes features or parts of features. And on the other hand we now think about the possibility to increase the quality with additional test. At first this may sound like a contradiction. But you have to separate the different kinds of testing and look at their goals. The feature related tests are needed to implement the features. The additional tests are done to check things outside of the main functionality. So they are done to check the possible customer benefit beside the functionality of the product. And that exactly matches with our definition of quality.

Such additional testing outside of the normal use cases will therefore increase the quality of the product. Really? The tests itself will help to evaluate the software quality. At next the product owner has to decide whether the software should be changed or not. Therefore the additional tests focused on quality are an entry point for quality changes. So these tests are similar to end user feedbacks and will help to increase the quality.

Summary

There are different points of view whether testing is a quality attribute or not. This depends on the kind of test you speak about. If someone speaks about software tests, in most cases the tests to check the intended behavior of a feature are meant. In my opinion such tests are part of the feature development itself. So these tests do not influence the software quality.

As conclusion, in case a product owner wants to reduce testing effort, you have to look carefully on the consequences and explain them in terms of feature loss (most often) and/or quality loss (rarely).