In this article I would like to give few basic thoughts on the topic of cognitive testing and clarify what is meant by this kind of testing, what cognitive software systems are, how to test such systems and what is meant by cognitive testing tools.

Cognition

It isn’t that easy to find an exact definition for the term “cognition” because it depends on the context. For example, in contrast to cognitive software systems the human cognition involves thinks. So, I want to use a simplified clarification, which is applicable to man and machine.

Cognition is based on knowledge. This knowledge can be built up by analyzing historical data (learning) and by analyzing actual situations (self-observation). A cognitive action will start with an analysis of the actual situation and therefore the actual information. This information can contain a lot of unstructured data. This data will be analyzed based on the knowledge of the cognitive system. The result of this process is an appropriate action or reaction.

A cognitive system accesses many sources and data, combines that data, transforms information, filters and evaluates that information out of its context, interacts with other systems (such as a human being), learns from information, context, and interaction. The system makes something new out of this data, even if the underlying and available information and its relationships are complex.

Cognitive Software Systems

The term “artificial intelligence” has been a popular catchphrase for a long time. But in the recent years this outworn catchphrase was replaced by the term “cognitive system” or “cognitive software”.

Cognitive software systems will process unstructured data and use artificial intelligence to draw conclusions. Currently the artificial intelligence is often based on neural networks and the knowledge is build up by machine learning or the special form of deep learning.

We will use such intelligent software systems nearly every day. For example, every speech recognition in our cars or smart homes uses technology based on neural networks.

But I think there is a major issue concerning the use of the term “cognitive software”: it is used for a wide range of software system types, starting with simple systems which simulate intelligence and ending with systems which think and act in a human way for example systems for autonomous driving. Only the second type is a real cognitive software. So, in my opinion, the term “cognitive software” is used to often, even for systems which have some simple kind of artificial intelligence, like speech recognition, but which don’t have real cognitive capability’s.

Cognitive Testing

After the short introduction of the term cognitive and of cognitive software systems, we can start with the main topic of this article: cognitive testing. So, at first let’s think about the main work steps of a software tester.

The software tester will specify a set of basic tests. These basic tests are very specific and strict. They contain a detailed description of the pre conditions, the input date, the execution steps and the expected result. These basic tests will be executed for each test candidate. This type of testing is widespread and used in many companies. Of course, the creation of the tests isn’t easy. The tester will use his knowledge, creativity and experience to develop the test cases. So, this working step is strongly based on his cognitive skills. In contrast, the execution of these basic tests is very easy and does not depend on cognitive skills. The work of a tester is often supported by software tools. You will find a lot of tools for the execution of basic tests. But interestingly it will be very difficult to find tools which strongly support the development of tests. Maybe because this part is strongly based on cognitive skills?

There exists a second type of testing which is called “free testing” or “explorative testing”. This type of testing is rarely used because it is time consuming and therefore expensive. Explorative or free testing means, the tester will not work according a fixed test specification. Instead he will use his knowledge about the testee and execute some new test sequences without a previous planning. Of course, these new sequences must be documented, and the tester must evaluate the result carefully because the expectation about the result depends on the knowledge of the tester. Therefore, such free testing can be very difficult even for an experienced software tester. This work, again, strongly depends on cognitive skill and so it is not a big surprise to find nearly no software tools for explorative testing.

Cognitive Testing Tools

The complexity of software systems is constantly rising. Therefore, there is a need for intelligent testing tools which strongly support the tester during development and execution of software tests. As a result of the rising complexity of the software to test, the testing tools must become smarter too. So, wee need testing tools with artificial intelligence and cognitive capabilities.

Such testing tools must offer some kind of intelligence, for example based on neuronal networks. Therefore, a new and important part of the daily routine of a tester will be to train these systems. This training or teaching must be done with use case specific data to create a wide and deep knowledge about the software to test. A good trained testing tool will be able to support the tester on development and execution of test cases or the testing tool will even be able to do this work alone without the need of a tester.

Testing of cognitive systems

As mentioned before, the complexity of software systems is constantly rising. So, it can be assumed to see the same growth for intelligent software systems. Over time the number of these software systems will rise, and they will evolve from system with simple artificial intelligence to systems with real cognitive capabilities.

Nowadays, most software systems are complex but have a well-defined behavior. Furthermore, it is rather rare to execute explorative test. Therefore, the development and execution of test cases isn’t that complex, but it is a time-consuming task. As a result, there exist a lot of powerful testing tools which support the work of the tester.

In the future, the number of intelligent or even cognitive software systems will increase. This will lead to a big change of the testing process. On one hand the development of the basis test cases will get more complex and on the other hand the importance of explorative test will increase sharply. The actual testing tools are not suitable for this new testing process. Cognitive software systems are a new kind of software which will lead to a new testing process and which will need a new type of tools: cognitive testing tools.

Current Situation

Software systems with artificial intelligence keep moving into more and more areas. But the current systems are limited to specific and simple use cases and are far away from real cognitive software systems. It will take a few more years before we will see and use cognitive software system in our daily life. Therefore, there is nearly no need for cognitive testing tools now. But over the next years the systems with artificial intelligence will increase in number and complexity. This will lead to an evolution within the test tools too. They will also become smarter and as soon as we start to use real cognitive software systems, we will need cognitive testing tools too.