Colophon

©2018 Sogeti Nederland B.V., Vianen

Editing, typesetting & ePub production LINE UP boek en media bv, Groningen

Cover design Axioma Communicatie, Baarn

ISBN
978 90 75414 87 5 (book)
978 90 75414 88 2 (ePub)

No part of this publication may be reproduced and/or made public (for any purposes whatsoever) by means of printing, photocopying, microfilm, sound tape, type of electronic system, or any other data retrieval system without prior written permission from Sogeti Nederland B.V.

Trademarks
TMap, TMap NEXT, PointZERO and TPI are registered trademarks of Sogeti Nederland B.V.

Foreword

In the current digital age, the quality assurance conundrum of what to test, and how much to test is magnified multifold. The convergence of physical with the cyber has added another layer of complexity to the testing activity. Product strategy is shifting from building discrete products to building connected eco-systems. The interface with wider eco-system entities has opened opportunities as well as vulnerabilities.

The book gives insight into all aspects of testing for digital solutions. It ranges from the use of artificial intelligence or machine learning in testing to cooperation between robots and humans. It also addresses the special considerations in testing 3D printed products and autonomous machines.

New age software development techniques embracing continuous delivery and integration (CD&I) can skew the focus towards “delivery” instead of the “deliverable”. The book stresses the use of feedback loop in learning from past cycles and building machine learning algorithms to manage the “what” and “how” of the testing scope. It introduces the concept of a test intensity table to determine the focus for business value and quality risk, striking a good balance between the deliverable and the delivery.

In the digital age we need to test for experience rather than functions or features. Building on this thought, the test engineer elaborates three additional quality characteristics. The characteristics intelligent behavior, morality and personality are needed in addition to the product quality characteristics already defined in well-known international standards.

As we enter the realm of early AI testing it is critical to build knowledge based on artifacts we already collect like defect log data, life cycle information, fields defects, and production events to improve effectiveness. The authors provide us with key insights on the concept of cognitive QA and how we can leverage analytics and artificial intelligence to improve decision making and the way we do testing.

The book concludes with an interesting perspective on the digital quality engineering skills that an AI quality engineer needs. This can be used as a ready reckoner when setting up cross-functional testing teams armed with the right digital test engineering skills.

As we forge ahead in deploying advanced and highly connected AI systems, this book provides practical guidelines for the corresponding testing activities. The comprehensive ideas cover all aspects that need to be considered to deliver a satisfying customer experience. In the last chapter interviews with influencers of a variety of leading companies confirm the value of the content of the book, and the vision of its authors, for testing in the digital age.

Sanjay Salunkhe
CEO,
Sogeti Product & Engineering Services,
India, Europe, Asia-Pacific, United States

Contents

Cover

Colophon

Foreword

1 Introduction

1.1 Reading guide

1.2 TMap Suite

1.3 AI makes the difference

2 Characteristics of the digital age

2.1 The digital age is the new industrial revolution

2.2 Digital explained

2.3 Digital help

2.4 Artificial intelligence era

2.5 AI terminology

2.6 The digital twin

2.7 Smart testing in the digital age

3 Set up your digital testing roadmap

3.1 A digital testing journey

3.2 Ultimately put “people first”

3.3 Digital test engineering: moving from functional to expertise mindset

3.4 Quality characteristics for conventional software testing

3.5 Why we require additional quality characteristics

3.6 New quality characteristics for intelligent machines

3.7 Existing quality characteristics applied to intelligent machines

3.8 Quality characteristics mapped to quality angles

3.9 Test intensity table

3.10 Test varieties

3.11 Five hops to digital testing

Hop 1 Automation and robots

4 Testing AI solutions

4.1 Six angles of quality for machine intelligence

4.2 What needs to be tested?

4.3 Test objects in AI Solutions

4.4 Structured learning

4.5 Examination

4.6 A/B testing

4.7 Metamorphic testing

5 Cobotics

5.1 Human-robot interface

5.2 Different reasons for robot solutions

5.3 Robotic process automation (RPA)

5.4 Robotic arms

5.5 Robots testing physical equipment

5.6 Cobotics in airplane testing

6 Robotesting

6.1 Testing of physical robots

6.2 Six angles of quality in robotesting

6.3 Testing of smart algorithms

6.4 Testing chatbots

Hop 2 Use the data

7 AI test data

7.1 Fitting data

7.2 Testing the input values

7.3 Feeding specific input data to see how the AI learns

7.4 Predefined test data sets

7.5 Monitoring the input of the AI

7.6 Underfitting data

7.7 Overfitting data

Hop 3 Go model-based

8 Virtual engineering

8.1 Test execution in a virtual engineering environment

8.2 Production and automation engineering

8.3 User centered product design

8.4 Mechanics of materials

8.5 Physics and mathematics

8.6 Checklist for virtual engineering testing

9 Quality in the product lifecycle

9.1 Digital archeology and testing

9.2 Product Lifecycle Management

10 Blockchain and testing

10.1 Blockchain explained

10.2 Blockchain QA and testing

10.3 Blockchain specific testing

10.4 Blockchain test strategy roadmap

Hop 4 Use Artificial Intelligence

11 Testing with AI

11.1 Testing with artificial intelligence using cognitive QA

11.2 The cognitive QA model

11.3 Self-learning and self-exploring testing tools take over thinking

11.4 Speed up with AI

11.5 Security issues regarding AI

11.6 Privacy, Big Data and AI

12 IoT testing

12.1 Three rules for IoT testing

12.2 Evolutionary algorithms help IoT testing

12.3 Example: Evolutionary algorithm to speed up selection of test environments

Hop 5 Test forecasting

13 Models help test forecasting

13.1 Systems of Systems

13.2 Models of Models

13.3 Multi-model predictions

13.4 Singularity meteorologists

13.5 Forecasting by an artificially intelligent elevator

14 Business and social impact

14.1 Business impact

14.2 Continuous everything

14.3 Change management

14.4 The impact of intelligent technology

14.5 Business impact example: additive manufacturing test strategy

14.6 Social impact

14.7 Testing in a cross-functional team

14.8 Digital test engineering skills

15 Interviews with leading companies on their digital vision

15.1 Interview with Werner Soeteman, Sr. IT Manager – Service Center Test at Air France - KLM Royal Dutch Airlines

15.2 Interview with Reindrich Geerman, Digital Architect – KPN Technium

15.3 Interview with Caroline Arkestein, QA and Support Manager – Wolters Kluwer

15.4 Interview with Peter Claassen, Delivery Manager at Rabobank

15.5 Interview with Willem-Jan van Tongeren, Director IT business support at PostNL

15.6 Interview with Paul Hesen, VP Custom Systems at TomTom

15.7 Interview with Martijn Tideman and Gwen van Vugt, Product Director and Director Mobility Center, at TASS International (a Siemens business)

15.8 Interview with Ton van Hamersveld, Director Engineering – Thales

Acknowledgements

References

Glossary

1
Introduction

1.1 Reading guide

You have just started reading the book Testing in the digital age where AI makes the difference. In five hops we explain how digital testing takes shape. The chapters in this book are grouped under these five hops. Organization maturity, personal interest or your project stage has influence on the part you want to read. The following table guides you to the right chapters (should you have no time to read it all).

Chapter or hop
Name
Summary
Must-read for:

1
Introduction
Here you read the reason this book was written. As part of the TMap suite this book describes the changes the digital age brings to the test discipline.
Everyone

2
Background
Background information on the term digital and a first glance into how smart testing can make a difference.
Everyone

3
Set up your digital testing roadmap
Testing in the digital age is about a new vision on test engineering, using new quality attributes that tackle intelligent machines and a roadmap split up in five hops.
Head of R&D
Test engineering
Project or team lead

Hop 1
Automation and robots
Automate everything possible in order to speed up the complete product development cycle using all possible means, even a robot taking over human test activities is a possibility.
Product development
Test engineering

Hop 2
Use the data
Digital test engineering needs to cope with data for the purpose of testing and monitoring. We need to be smart about it in order to tame the data beast growing out of control.
Data analysis
Test engineering

Hop 3
Go model-based
Modelling, using models and using the digital twin are crucial to keep up with continuous testing. It helps to build confidence in products operating in the field that we think have elusive characteristics.
Scientific engineering
Test engineering

Hop 4
Use artificial intelligence
The goal of using artificial intelligence in testing is not to take people out of the loop. The goal is to make testing easier and faster. This hop also gives insight into how to use AI solutions.
Everyone

Hop 5
Test forecasting
The test forecasting hop is aimed at being ahead of the test results. To get to that situation all previous hops need to be put into place.
Head of R&D
Test engineering

15
Interviews with leading companies on their digital vision
We have interviewed companies on their view of the digital age. The new risks and challenges, opportunities and the impact on their test activities now and in the future.
Everyone

Glossary
Clear terminology is key when communicating. This glossary is an extensive list of terms that supports you in all your test communications
Everyone

1.2 TMap Suite

TMap is an established name in testing since 1995. Since its introduction the TMap method has been continuously extended and improved. This book brings testing to the digital domain. It expands the TMap test methodology to situations where a strategy for testing digital solutions is needed.

All information about TMap together forms the TMap Suite. In addition to this book you are reading, the TMap Suite consists of:

• | The TMap.net website. This website contains the building blocks of TMap. Together with the building blocks in this book you can build your own testing method. On TMap.net templates and checklists are available for download.
• | The book Neil’s Quest for Quality: A TMap HD Story describes the human driven and quality driven test approach for modern, agile organizations.
• | The book TMap NEXT with the core test method, describes techniques and processes for testing in every part of the product development lifecycle. Its adaptiveness ensures that TMap can be applied in traditional, iterative and hybrid organizations.
• | The book IoTMap: Testing in an IoT Environment expands the TMap Suite to the IoT and high-tech domain. Together with the other elements from the TMap Suite a complete test strategy for your project, fitting your product development methodology can be set up.
• | Several other books about testing, quality assurance and test process improvement.

1.3 AI makes the difference

Digital companies rise quickly in traditional analog markets. Tesla is claimed the biggest car manufacturer of the “Big 3”! Traditional (call them analog) car manufacturers have been selling cars for over a century. The digital car manufacturer only started a decade ago. The digital age is testing the limits of companies. A demand for extremely quick market response, new technology and keeping up with emerging more agile competitors, is what they face right now. Keeping up with all this, is putting you on your toes. Running an experiment in the field is nothing new. It is more and more accepted to crowd test and immediately move to large scale market solutions.

The technology is here, and it will work. Digital unlocks a new world of possibilities and there is no time to sit back, relax and watch it unfold. With digital, huge amounts of data are unlocked. Everywhere a sensor is put, data is collected, transmitted and stored. Data lakes are flowing into oceans of 1’s and 0’s.

Digital is put in front of a lot of terms. This makes existing processes, products or solutions different from how we know them. Digital twin, digital manufacturing, digital assurance, digital transformation, digital age and digital enterprise are just a few examples of digital terminology. We even have digital football players!

Digital creates an environment of like-minded products. A digital aspect of a product makes it possible to communicate with another digital product in one way or another. Integration is the keyword here. A flurry of recent acquisitions of major API management vendors shows the world that APIs are part of a bigger market called integration. API management is much more an extension of integration, because that’s where they are developing their APIs. A clear indication that IT recognizes the value of APIs for application integration. The Internet of Things (IoT) is maturing fast, making integration of “Thing” data a reality. IoT brings IT and OT together, making it the biggest integration project of this day and age. Finally, blockchain technology is contributing to all these integration initiatives. Companies ranging from Fintech to healthcare are getting in line to adopt this new way for fast and secure payments.

Integration will be the key that ties these new initiatives into your business, adding new channels for secure and fast business. IT departments need to find a new way forward to support the enterprise infrastructure and maximize the potential benefits of having improved access to this data. With the IoT, APIs, blockchain and chatbots thrown into the mix, integration is no longer the problem of your IT department, it is everyone’s problem.

Digital is extending human possibilities, new ways of working, new thoughts and takes on existing products. Where new things are created, things are tried and tested. This is where testing of digital solutions comes into play. The common denominators with all digital terminology are:

• | Speed: Extremely fast market response
• | Data: Huge amounts of data are collected
• | Integration: Everyone needs to integrate with everything.

Figure 1 The digital age acts as an accelerator for all our product development activities.

Extremely high speed, huge amounts of data and an infinite amount of possibilities for integration are the elements we are facing for testing. They extend beyond our human capabilities.

One way to help us out is test automation. Test automation in the context of digital testing, is automating everything possible in order to speed up the complete product development cycle using all means possible means, even a robot taking over human test activities is a possibility. Further help can be found in combining it with another new technology not yet mentioned: artificial intelligence (AI). AI works with huge amount of data, finds smart ways through infinite possibilities and has the potential to hand us solutions quickly. Let us not forget that AI needs to be tested as well, but AI can make the difference here.

We can create robots that can do all the testing automatically and monitor data coming from the field. The robots also have to keep up with growing amounts of test cases, test data and test environments. Tests are executed on a system under test. In that sense it is a reactive (and time consuming in the release train) activity. Testing must keep up and shift from an executing activity, to a monitoring role, towards a quality-forecasting medium that is ahead of the game.

Figure 2 Testing is moving from a reactive activity (for example test execution) towards test monitoring (for example monitoring data from operating products). Eventually testing becomes the forecasting of faults.

2
Characteristics of the digital age

2.1 The digital age is the new industrial revolution

Information density is growing rapidly with everything being digital available. The information age brings exponential growth of (digital) data. Storage capacity is still keeping up with generation of data. There is enough communication bandwidth to transmit it all. Moore’s Law is giving us the computational power needed to process the increasing flow of data. The evolution of technology in daily life and social media leads to the fact that the modernization of information and communication processes becomes the driving force of social evolution.

The industrial revolution marked the transition to new manufacturing processes and with that, huge social growth took place. Working and financial conditions improved with big steps (for example better housing, improved mobility and a bigger wallet). There was some resistance against this new way of working. Traditional manual labor was favored for some time, but resistance broke quickly when the positive impact on living standards became clear. The digital age is showing similar resistance.

Take Fintech as an example. Fintech is the new financial industry that applies technology to improve financial activities. Traditional financial services have their trust and regulation. Fintech steers away from traditional financial services and not everyone is happy with this change. At first, this put Fintech in the shady area of doing business. In the digital age innovations are adopted at a high pace. Rules and regulations for Fintech are emerging right now, which eventually makes Fintech look better and adopted on a wide scale.

Figure 3 There is always a healthy resistance against new technology, FinTech is not different.

The Internet of Things (IoT) fares a similar fate in the digital age. The speed at which IoT is entering our lives and the industry, is quicker than factories powered by steam engines in the industrial age. There is opposition against IoT solutions in terms of breaching privacy. A good discussion is going on whether or not we still make our own decisions with smart IoT solutions around us. You could also argue that IoT solutions make our daily life easier in taking over repetitive tasks or boring activities. This is similar to the situation of the industrial revolution where machines took over the repetitive manual labor tasks.

Figure 4 IoT makes our lives easier, but can it fetch a cup of coffee for us?

The digital age is also called the information age. Starting in the 1980s with the introduction of the personal computer with subsequent technology introduced, providing the ability to transfer information freely and quickly. The digital age revolutionizes our environment and in it the way we think, work and act. Robots can see and hear. They process analog input and convert it to (for them) comprehensible digital information. Digital solutions are more and more invisible to the human eye. They must work their magic without us noticing it. The digital age brings exponential growth of (unknown) combinations of functionality and an increase in complex systems. In that sense, the digital age is very much the new industrial revolution. With digital systems working ubiquitous, the demand for quality is higher than ever before.

2.2 Digital explained

The digital watch with an LED display is a good marker of the start of the digital age. This was in the early 1970s. We have been living in a digital age for some time now. The rate at which digital is entering our lives has only sped up since then. Digital enables you to get the most out of your business, your people, your relationships, and your technology. The technology involved marks periods in time. Funnily enough, this starts with “digital first”. After that, we see “mobile first” and “cloud first” which is now moving to “AI first”.

All tech periods build on its predecessor. The digital basis for all tech periods defines the digital age we now live in.

Digital twin, digital sports, digital manufacturing, digital health, digital assurance, digital music and digital transformation are examples of terms that start with digital. Digital comes from digit. A digit converts an analogue value in the real world to a discreet value in the digital world. The step size of the discreet values determines things like accuracy, update time and size of a digital dataset that has any significance.

Take the watch again. Already in the 1920s the analogue clock dial is converted into discreet numbers that change every minute, every second or even faster. The first watch without dials had digits. A set of numbers 0 to 9 clicking in place to show the time. Actually, the digital watch already existed in the 1920s!

Figure 5 Digits are often associated with binary numbers. A clock with digits in stead of an analogue clock dial is a representation of the digital age as well.

Binary digits are the best-known example of digital. A 1 and 0 are the only discreet values a binary digit can have. A set of 1’s and 0’s can be the digital representation of a piece of text (a document) and evolved quickly to digital photographs and pieces of music, etc. All digital information we have around us is stored as binary digits.

We are bombarded with 1’s and 0’s that are transformed into some form of information. Every scrap of information is available in a digital form. All this information is serving us a purpose. While traveling, all information on connections, flights, trains and end location is available at our fingertips. When operating machines in a factory, all information is put in place to operate on time and with a high efficiency. In other situations, the digital information bombardment disturbs everyday activities like having diner or watching a movie. From the digital watch to digital twins of windmills, digital is all around us.

2.3 Digital help

A factory with robotic arms assembling products must work around the clock and cannot make one error. Robots must act like that. The interaction with a chatbot on an insurance website must also be available 24/7. This robot must give the user the experience of talking to a real person together with having all information at hand you can possibly think of.

These examples of digital solutions have in common their demand for a high level of quality. The customer operating the robots (whether it be factory or website) demands from his supplier an almost immediate response to problems found. A robotic arm stopping or doing the wrong operations, stops the assembly line. The chatbot going offline might mean losing potential new customers.

Digital solutions can help us in a great way. They are characterized by:

1. | a high level of quality
2. | the ability to receive updates quickly
3. | non-stop operation

There is a change happening towards the situation where robots are smart enough to detect their own defects and communicate about this. Eventually going so far as to fix the defect and continue operation.

There is a clear demand for increasingly smarter products and experiences. Digital can also help make that possible. Digital means an acceleration in product development. Speeding up product lifecycles brings solutions and fixes much quicker to end users. With digital the feedback loop can be put in place to learn from defects.

2.4 Artificial intelligence era

The digital age brings forth smart products and solutions. Moore’s Law gives us energy-efficient, faster computational power to execute increasingly complex AI solutions. AI in itself is just another algorithm to run on a processor. There is one thing needed for AI to work and that is data. It needs data to learn, data to improve and data to continuously run. Data is enriched into information and in turn is used for new intelligent solutions. Enriched data and intelligent solutions in their turn can be used to start creating forecasts.

The digital age has been around for some time. In this time a lot of data has been collected and stored safely. Now this data can be put to work for AI algorithms to learn. The AI decision-making process is based on a huge amount of information analysis through context-based expertise. Expertise taught by human experts of the said domain to AI. Computer science is facing its most challenging steps at the moment. It’s a beautiful adventure, but the journey is difficult even if we now have the technologies to do it.

AI becomes our supporting companion that gives us the information we need before we even knew we needed it.

Figure 6 Personal digital assistants are starting to help us. In no time they will start talking to each other.

Modern information technologies and the arrival of machines powered by artificial intelligence (AI) have already strongly influenced the world. Computers, algorithms and software simplify ordinary tasks, and it is impossible to imagine how, in the near future, most of our life could be managed without them.

2.5 AI terminology

To this point it must be clear that artificial intelligence is making the difference in testing in the digital age. We need AI algorithms to support us in finding our way and filtering through the growing amount of test products (data, test cases, etc.).

This book uses terms like artificial intelligence, machine learning, machine intelligence, cognitive IT and robotics. These new aspects of information technology are relevant in today’s world of digital assurance and testing. The following explains the terms in general and how we use them throughout this book.

2.5.1 Artificial intelligence

Artificial intelligence (AI) is a sub-field of computer science aimed at the development of computers capable of performing tasks that are normally done by people, in particular tasks associated with people acting intelligently.

AI is a system, built through coding, business rules, and increasingly self-learning capabilities, that is able to supplement human cognition and activities and interacts with humans naturally, but also understands the environment, solves human problems, and performs human tasks.

The term artificial intelligence was coined in 1956 at a conference at Dartmouth College. The mid-1950s ushered an era of optimism. Many of that era’s leading scientific minds attended the Dartmouth conference and contributed to the early advancement of the technology. Despite the early optimism, achieving artificially intelligent systems proved to be a challenge. Waves of enthusiasm were followed by troughs of disillusionment throughout the 1950s, 60s, 70s, and 80s.

The first test activity for AI is described by Alan Turing and has become known as the Turing test:

“Can a computer communicate well enough with a human to convince the human that it (the computer), too, is human.”

AI is not required to learn, it could be using pre-programmed rules to handle all possible outcomes. However, for systems with more than basic complexity, this has proved to be a task too large and too complex to handle (it has been tried and failed multiple times since the 1960s). In that sense, we distinguish between artificial general intelligence and artificial narrow intelligence.

2.5.2 Artificial general intelligence

Artificial general intelligence (or AGI) is an intelligence that can execute all the tasks that a human could execute. The most important aspect of AGI is that it can execute different complex tasks in a sequence. The coffee test by Wozniak should not be a problem for AGI:

“A machine is required to enter an average American home and figure out how to make coffee: find the coffee machine, find the coffee, add water, find a mug, and brew the coffee by pushing the proper buttons.”

When full AGI is here and how it will look like is still unclear to this moment.

2.5.3 Artificial narrow intelligence

All AI we use nowadays is categorized as artificial narrow intelligence (or ANI). This AI is focused on one task. It tries to execute this as good as possible. Examples are autonomous driving cars, natural language processing, or facial recognition done by a chatbot. The biggest breakthrough for ANI is neural networks. Neural networks mimic biological processes. The paths that are laid in the brains of animals serve as the basis for this technology. With neural networks, it is possible to build much more complex systems for our AI solutions.

The biggest advantage is the learning capability by feeding information (most used example here is learning to recognize a specific object by feeding it large amounts of pictures and telling it when the object is in the picture). With reinforcement learning, it is possible to add a reward function. ANI is then evolving to a much smarter system and growing towards an AGI solution.

2.5.4 Machine learning

Machine learning is one of the ways to achieve artificial intelligence. It contains different algorithms – each with its own strengths and weaknesses. The last major breakthroughs in the field of AI are based on machine learning or more specifically on “deep learning”, which uses an artificial neural network. Other popular algorithms are: Bayesian networks, Decision Tree, K-Means Clustering and Support vector machines.

Each has its own strengths and weaknesses. These algorithms are often grouped into three categories:

• | Supervised learning
• | Unsupervised learning
• | Reinforcement learning

Although recognizing these differences, we have not differentiated the algorithms in this book.

2.5.5 Machine intelligence

Machine intelligence (MI) is a unifying term for what others call machine learning (ML) and artificial intelligence (AI). We found that when we called it AI, too many people were distracted by whether certain companies were “true AI”, and when we called it ML, many thought we weren’t doing justice to the more “AI-esque”-like aspects, such as the various flavors of deep learning. So, machine intelligence is a term that combines “artificial intelligence”, “machine learning” and other related terms.

2.5.6 Cognitive IT

The word cognitive means “knowing and perceiving”. Cognitive information technology is not just rule-based but is able to react based on perception and knowledge.

In this book, “cognitive QA” is used to refer to the use of cognitive IT to assist quality assurance and testing.

2.5.7 Robotics

What is a robot? It’s a machine that gathers information about its environment by input from sensors and, based on this input, changes its behavior. Combined with machine learning and machine intelligence the robot’s reactions over time become more adequate. The use of Internet of Things (IoT), Big Data Analytics and cloud technology make a robot versatile.

Robots come in many different shapes and forms. It’s not just the metallic man. Robots may equally be a smart algorithm on social media (for example a chatbot or a digital agent), an autonomous vacuum cleaner, or a self-driving car.

2.6 The digital twin

The digital age shows similarities with the industrial revolution. One might say we are now in the digital revolution (or the 4th industrial revolution). New digital services and solutions are entering our lives in an increasing pace. Whether we want it or not these services are also becoming smarter and smarter.

The industrial age moved from horse and carriage to steam-powered machines. Only shortly after came cars and now we are shifting towards a road network where all electric cars drive themselves. In order to make this work we have to trust machines that will carry us around. To gain confidence and more importantly keep that confidence in the digital age, testing needs to evolve on two tracks:

1. | Fast testing must be in place to keep update time to a minimum. This calls for automation on all test levels.
2. | Continuous testing must be in place to support continuous delivery of new features and fixes (for example quick response to a security breach).

There is valuable information to be gained from the usage of products. The digital age brings us the means for communication, data gathering, storage but also the computational power to sift through all the information quickly. Operational data can be combined with historical data and new correlations can be found. Testing can be put in place throughout the entire product life cycle in this way. This is the second form of continuous testing called continuous monitoring.

Complex digital solutions evolve quickly. New features are added, bug fixes are done, and unexpected combinations of products lead to (sometimes intended) misuse. Classical static documentation in this rapidly changing environment cannot keep up. Dynamic documentation based on models of digital products enables the capacity of defining, developing and testing at high speed.

Using a model helps. A model can be adapted more quickly, and the implication of a change is easily displayed. Models can grow from rough conceptual ideas to refined representations of the real thing. They can be used as simulators to speed up testing. Together with operational data models they may serve as forecasters for situations yet to come. The digital twin is the term used to describe this mechanism.

2.7 Smart testing in the digital age

Software testing is an investigation process that verifies and validates the alignment of a software system’s attributes and functionality with its intended goals. It is a labor-intensive and costly process. Therefore, unsurprisingly, automated testing approaches are desired to reduce cost and time. Software testing is optimized with the help of machines powered by artificial intelligence (AI).

Developers are under relentless pressure to deliver innovation and software quickly to the market, whilst maintaining quality. Testing is a critical component of the software development lifecycle:

• | Testing delivers proof that the rules and regulations are met.
• | Testing is crucial for gaining confidence in the correct implementation of a defined set of needs and features.

The test expertise is expanding beyond its traditional definition. DevOps and Agile strategies are used to deliver with speed and quality.

This need for increased speed and innovation is seeing the relationship between testing and development changing from a service to a partnership. The lines between testing and development have blurred. Developers are doing more testing, while testers are being involved much earlier in the lifecycle. They also participate in development activities.

Now let us shift the testing activities carried out by humans to machines powered by AI. It will reduce the long-term testing costs and increases the speed of testing.

We need a lot of smart ideas to let testing keep up with the revolution that is the digital age. Let us not forget the testing of digital solutions that are smart themselves. Testing with AI is one thing, but the testing of AI needs to be supported as well. We need to make testing a lot smarter than the AI it is testing. Other mechanisms need to be put in place that build the confidence we need. Only in that way can testers work together with bots and move towards the situation where we have enough confidence in bots doing the work for us.

Current testing evolves into smart testing and AI is going to make the difference. The first step in the evolution of testing is to do smarter testing. The second step is to make all test activities contribute to the build-up in confidence (specifically when using AI in testing or when testing AI). Table 1 shows an overview of the test activities that makes testing ready for the phase after the digital revolution.

Table 1 Test activities and their contribution to building confidence in (smart) solutions in the digital age. The last column shows a reference to the specific “hop” sections within the digital test approach for further reading. See also section 4.11 for more details on the hops.

Test activity
Implementation
Contributes to
Hop reference

Automatically select the right test cases
Let AI algorithms learn to select the best test cases for each change made.
Support short release cycles
Hop 4: Use artificial intelligence

Automatically set and/or select the right test environment
Use AI to set test environment parameters to cover most of the field settings.
Infinite possibilities of test environment parameters are reduced to a manageable set
Hop 4: Use artificial intelligence

Convert the huge amount of test data (test cases, results, etc.) with smart algorithms
Use AI or machine learning algorithms for selecting relevant data or find complex and unseen relations.
Gain insight in quality
Hop 2: Use the data
Hop 4: Use artificial intelligence

Continuous testing
Test automation tooling on all test levels (unattended code review and testing, model usage, simulation, services virtualization).
Continuous delivery
Hop 1: Automation and robots

Continuous monitoring
Use operational data in test scenarios. Keep on testing during the entire product life cycle.
Growth of quality level
Hop 1: Automation and robots

Combine operational data with a model
Implement a digital twin that grows in detail and changes role throughout the product life cycle.
Keep up with quick changes
Hop 3: Go model-based

Create a test & forecast system
Make the best use of computational power to run large amounts of simulations with the use of operational data. Use models to run multiple scenarios and compare results.
Confidence in future workings
Hop 5: Test forecasting

3
Set up your digital testing roadmap

A digital age brings all kinds of new products. Existing products are dragged into the digital age and need to adapt to remain relevant. Accompanying the adaption of product development, testing needs to adapt to the digital age. All changes in the digital age and its effect on testing are given in Figure 7. This chapter discusses all changes in the digital age and its effect on testing. At the end of the chapter this converges into the roadmap for testing in the digital age. We divided it into five hops. A hop can be skipped if not relevant. Even one single hop can be used as a goal for testing in your organization.

3.1 A digital testing journey

The digital age did not appear in one day. We gradually became familiar with digital technology. Nowadays, we cannot imagine our daily life without all the digital solutions around us. Testing in the digital age is also not something that is put in place in one day. It is a journey following the rise of new technologies and changing use of existing technologies.

Figure 7 shows the journey from the growth of data around us up to test forecasting and the testing of AI solutions. Let’s follow all the steps and see what the journey brings us.

Figure 7 Infographic on the roadmap for testing in the digital age.

1 Digital data growing

Transformation to more digital solutions is accelerated by four drivers:

i. | Unlimited computer and storage capacities
ii. | Growth of available data around us
iii. | Exponential rise of the Internet of Things (IoT)
iv. | Birth of digital platforms

Figure 8 Transformation to more digital solutions is accelerated by four drivers.