Colophon

©2020 Sogeti Nederland B.V., Vianen, the Netherlands

Editing, typesetting & ebook production LINE UP boek en media bv, Groningen

Illustrations & cover design Axioma Communicatie, Baarn

ISBN
978 90 75414 89 9 (book)
978 90 75414 90 5 (ebook)

No part of this publication may be reproduced and/or made public (for any purposes whatsoever) by means of printing, photocopying, microfilm, sound tape, type of electronic system, or any other data retrieval system without prior written permission from Sogeti Nederland B.V.

Trademarks
TMAP and TPI are registered trademarks of Sogeti Nederland B.V.
SAFe is a registered trademark of Scaled Agile Inc.
ISTQB is a registered trademark of ISTQB aisbl.
IEEE is a registered trademark of the Institute of Electrical and Electronics Engineers.

Contents

Cover

Colophon

Foreword by Patrick Debois

Foreword by the authors

Acknowledgements by the product owner

Part 1 Introduction

1 What value will you find in this book?

1.1 The DevOps IT delivery model

1.2 Continuous quality engineering

1.3 README.TXT (reading guide)

1.4 TMAP evolution

2 Successful high-performance IT delivery depends on people

2.1 The Tayloristic view and the post-industrial mindset

2.2 High-performance IT delivery with cross-functional teams

2.3 How to move towards people-oriented quality engineering

2.4 Automate everything, as long as it is useful

2.5 From DevOps to AIOps to NoOps?

3 The VOICE model

3.1 Explanation of the VOICE model

3.2 The VOICE model applied to DevOps

3.3 Implementation remarks on the VOICE model

4 Examples of indicators in the VOICE model

4.1 Examples of indicators

4.2 How to select your indicators?

4.3 Use the Goal-Question-Metric approach to find indicators

5 Introduction to quality and testing

5.1 Do not implement a “fixing phase”

5.2 Measuring quality provides information for establishing confidence

5.3 Testing consists of verification, validation and exploration

5.4 Testing is about providing different levels of information

5.5 Static and dynamic testing

5.6 Testing is about assessing quality based on criteria

6 CI/CD pipelines and tooling

6.1 Example CI/CD pipeline

6.2 Continuous everything

6.3 Needed capabilities in a CI/CD pipeline

6.4 CI/CD tooling

Part 2 IT delivery models

7 Overview of IT delivery models

7.1 IT delivery models

7.2 IT delivery models and development activities

7.3 IT delivery models and “working agile”

8 Sequential IT delivery models

8.1 Waterfall

8.2 V model

9 High-performance IT delivery models

9.1 Scrum

9.2 DevOps

9.3 Other high-performance IT delivery “models”

10 Hybrid IT delivery models

10.1 Demand/supply model

10.2 SAFe model

Part 3 QA & testing topics

11 Introduction to QA & testing topics

11.1 Two groups of QA & testing topics

12 Introduction to organizing QA & testing topics

12.1 A brief description of the organizing topics

13 Introduction to performing QA & testing topics

13.1 A brief description of the performing topics

14 Topics plotted on the IT delivery models

14.1 Sequential IT delivery models

14.2 High-performance IT delivery models (especially DevOps)

14.3 Hybrid IT delivery models (especially SAFe)

Part 4 Organizing topics explained for DevOps

15 Quality & test policy

15.1 Quality and test policy subjects

15.2 Reasons to create a policy

15.3 Translate policy into tactical and operational levels

15.4 Generic Test Agreements (GTA)

16 Responsibilities & roles

16.1 Common roles

16.2 Alternative competence model

16.3 The work of the test professional changes

17 Monitoring & control

17.1 Indicators

18 Anomaly management in DevOps

18.1 Anomaly handling in a light-weight process

18.2 Tools to support anomaly management

18.3 Terminology related to anomalies

19 Reporting & alerting

19.1 What information do the stakeholders need?

19.2 Information based on indicators

19.3 Detailed reporting

19.4 Overview reporting

19.5 High-level reporting

19.6 How is the information communicated and how do alerts work?

20 Estimating the effort

20.1 DevOps (Agile) estimating techniques

20.2 Adapted traditional estimating techniques

21 Planning the delivery

21.1 Agile planning approaches

21.2 Prioritization approaches

22 Infrastructure

22.1 Infrastructure-as-code

22.2 Infrastructure verification

22.3 Easily set up environments

22.4 Workstations and other infrastructure

23 Tooling

23.1 Test tooling

23.2 DevOps tooling

24 Metrics

24.1 Fundamentals of good metrics

24.2 Metrics and continuous improvement

24.3 How to define a set of metrics

24.4 Effectiveness and efficiency metrics

24.5 DORA DevOps performance metrics

24.6 Top 20 QA metrics collected by Forrester

24.7 Long non-exhaustive list of raw metrics

25 Continuous improvement

25.1 How to establish a continuous improvement culture

25.2 What do we need to improve continuously?

Part 5 Performing topics explained for DevOps

26 Quality risk analysis & test strategy

26.1 Gather the DevOps team members

26.2 List all backlog items (e.g. user stories, features) of the current sprint

26.3 Identify the relevant quality characteristics per item

26.4 Analyze the possible impact and chance of failure for each combination of item and quality characteristic (is item risk)

26.5 Determining the test intensity per combination of item and quality characteristic

26.6 Allocation of quality measures per item to each combination of test intensity and quality characteristic

27 Acceptance criteria

28 Quality measures

28.1 All quality measures may relate to all DevOps activities

28.2 Three groups of quality measures

28.3 Overview of quality measures

29 Reviewing (static testing)

29.1 Static and dynamic testing

29.2 Overview static testing

29.3 Registering anomalies

30 Test design

31 Test data management (TDM)

31.1 What is test data?

31.2 What is test data management?

31.3 Which data items?

31.4 Test data management practices

32 Test automation

32.1 How to determine which tests should be automated and which test variety should be selected

32.2 Continuous testing

32.3 Test automation solutions

32.4 Test orchestration

32.5 The future: smart automated frameworks

32.6 “Everything as code” automation

33 Test execution

33.1 Explicit and implicit tests

33.2 Different test varieties have a different focus

33.3 What if the test cannot be executed?

34 Investigate and assess the outcome of testing

34.1 Investigating a failed test case

34.2 Steps for analyzing the failed test and creating an anomaly

34.3 Be aware of fault clustering

Part 6 Quality measures and skills

35 Description of quality measures

35.1 Root cause analysis

35.2 Specification and Example (SaE)

35.3 Test-driven development

35.4 Pair programming

35.5 Pairing

35.6 Reviewing

35.7 Test design techniques

35.8 Feature toggles

35.9 Monitoring of product quality

35.10 Parallel testing

36 Personal, interpersonal and team skills

36.1 Collaboration techniques

36.2 Expressing yourself

36.3 Team values

36.4 Unfavorable team behavior

36.5 Learn fast

36.6 Exploring

36.7 Skills matrix

36.8 High-performance teams get support from the staff organization

36.9 T-shaped, Pi-shaped, Comb-shaped and beyond

36.10 Test professionals: a change in mindset

Part 7 Test varieties

37 Make sure there is variety in your testing

37.1 Spheres of testing

37.2 Testing pyramid

37.3 Testing quadrants

37.4 Regression testing and progression testing

37.5 How to define your test varieties?

38 Performance testing

38.1 What is performance?

38.2 What is performance testing

38.3 Performance testing varieties

38.4 Performance testing as part of the IT delivery lifecycle

38.5 Addendum: Load model

38.6 Addendum: Iteration model

38.7 Addendum: Performance metrics plan

39 Usability testing

39.1 Roles in usability testing

39.2 Quality subcharacteristics

39.3 Usability test plan and usability testing techniques

39.4 Success factors

40 Security testing

40.1 What is security?

40.2 What is security testing?

40.3 Security testing approaches

40.4 Focus of security testing

41 Maintainability testing

41.1 What is maintainability?

41.2 Measuring maintainability both statically and dynamically

41.3 Testability

42 Mutation testing tests the test

42.1 What is mutation testing?

42.2 How does mutation testing work?

42.3 Example

42.4 How does this relate to test coverage?

Part 8 Test design

43 The many aspects of creating tests and exploring a test object

43.1 Two distinct ways of preparing and performing tests

43.2 Experience-based testing

43.3 Coverage-based testing

43.4 Always combine experience- and coverage-based testing

44 Test design entities

44.1 Test design entities relationship diagram

45 Test approaches applied

45.1 Coverage-based testing

45.2 Process-oriented test design

45.3 Condition-oriented test design

45.4 Data-oriented test design

45.5 Appearance-oriented test design

45.6 Selecting and combining approaches and techniques

46 There are many techniques, which one to use?

46.1 No risk – no test – no development

46.2 Example case: “The magic boat ride in TestLand”

46.3 Process-oriented testing: path coverage

46.4 Condition-oriented test design

46.5 Data-oriented testing with EP and BVA

46.6 Data combination test

46.7 Syntactic testing

46.8 Code coverage in many different intensities

47 Experience-based testing

47.1 Experience-based testing approaches

47.2 Checklist

47.3 Error guessing

47.4 Exploratory testing

47.5 Crowd testing

47.6 What are heuristics and when to use them?

48 Is there any value in unstructured testing?

48.1 What is unstructured testing?

48.2 Is unstructured testing useful?

48.3 Should you be happy with finding many faults?

48.4 What if the quality is good?

48.5 Then why do so many organizations use unstructured testing?

48.6 What is an example of an unstructured testing approach?

48.7 How can we avoid unstructured testing?

Appendix: Quality characteristics and non-functional testing

A.1 Functional testing

A.2 Non-functional testing

A.3 ISO25010 quality characteristics for product quality

A.4 ISO25010 quality characteristics for quality in use

A.5 TMAP extension quality characteristics for intelligent machines

References

Index

Foreword by Patrick Debois

Even though testers didn’t make it to the word DevOps, they have always been an integral part of it. In the old days they would act as the wall between dev and ops, the final gatekeeper to production. Borrowing from systems theory, we want to optimize the whole system and not just the separate parts to achieve better results. This means quality goes beyond the testers group and is a responsibility for everyone. Everyone has to be taken in a very broad sense, not merely on the technical side; it also includes the process on how things are done and what is most important to the business. Sales, marketing and more recently DevRel (developer relations), can be seen as way to get valuable feedback too. If you optimize for quality, you will eliminate bottleneck after bottleneck. This is a noble goal, but one has to be focused on the business value. That balance is tough though: how much quality is enough? Speaking with the different groups inside the company plus, most importantly, the customer who buys and uses it, is crucial: it’s all about gathering feedback to improve things.

In this book we see that modern organizations have changed: instead of relying on the strength of individuals, they aim for high-performing teams. By following “you build it, you run it”, these teams put quality dead center because they would feel the pain of issues themselves. On their trip across the whole enterprise the authors show us how DevOps made an impact on the concept of quality in all different aspects of the daily work of an IT worker. It is easy to confuse DevOps with faster delivery where in fact it is as much about fast as it is about quality releases.

All the automation happening inside the delivery pipeline has not resulted in having less people involved or even having people fired. It has given humans time to work on things that matter. This continuous process is a learning experience and can only be achieved when we work together to make our jobs better.

Patrick Debois
Author of The DevOps Handbook and organizer of the first ever DevOps conference.

Foreword by the authors

Deliver value with the right quality at speed. That’s what organizations ask of their IT teams. To achieve this, IT people need to work closely together. Cross-functional teams of business analysts, developers, testers, operations people, and other involved areas of expertise, join forces to work towards business objectives. Collaboration is key. Collaboration between people that have a shared responsibility and pick up the tasks that need to be done, based on competences and roles, no longer worrying about official functions.

The DevOps culture is the enabler for this way of high-performance IT delivery, using an automated CI/CD pipeline to deliver at speed. But how do you deliver value with the right quality using DevOps? This book, and our website www.tmap.net, enable teams to implement quality engineering practices using our body of knowledge that contains various approaches, practical examples and concise explanations. We trust that teams will benefit from this and enhance their collaboration to deliver value to their stakeholders.

Aligned with the DevOps culture of IT delivery – where roles are important and functions are not – keep in mind that if in this book you see a term such as developer or tester, we refer to the role of a team member at a specific moment in time, not to a function.

Exploration is an important activity today because in high-performance IT delivery there’s not enough time to first work out every detail; during the journey things need to be explored. As Apollo 8 astronaut Frank Borman put it: “Exploration is really the essence of the human spirit, and I hope we will never forget that.” [Kluger 2017]

We encourage you to use this book in the same spirit. Explore the book, see what you can use. And adjust it to your own needs and those of the team.

When we created this book, we were inspired by many people. We also used a lot of knowledge from books, articles, conference-presentations, websites and other sources. Where we directly quoted from a source, we have put the reference to that source in the text. But, as you can imagine, we have acquired many ideas and knowledge over the years and can’t exactly attribute these to a specific source. Therefore, we apologize to anyone who is of the opinion that we reused their idea without a direct reference, we trust you will see this as a recognition and appreciation of your good work. We have done our best to make the list of references as complete as possible.

Although the name of Leo van der Aalst is not mentioned on the front cover of this book, Leo has written large parts of this book and was of great inspiration to the other authors. Before this book was finished, Leo had the opportunity to switch his career to a new direction beyond Sogeti, therefore he won’t be able to support the other authors to convey the message in the future. The other authors are very grateful for his great contributions and collaboration.

We are grateful to all people that contributed during this endeavor (see our product owners’ words for details). We thank our managers for enabling us to make this book a reality, thank you Rob Vijverberg, Christiaan Hoos and Tinus Vellekoop!

We trust you will find value in this book and wish you success and joy in applying the contents in your daily work.

Rik Marselis, principal quality consultant
Leo van der Aalst, DevOps lead
Berend van Veenendaal, expert backend developer
Dennis Geurts, lead software architect
Wouter Ruigrok, Agile quality coach

Acknowledgements by the product owner

Quality at speed is the challenge for IT delivery teams today. To support all people involved in facing this challenge, we embarked on a journey about a year ago to create an updated version of our body of knowledge, published on the website www.tmap.net, and a book that specifically deals with the challenges around quality and value a DevOps culture brings to organizations. Previously, TMAP was mainly for QA and testing professionals. In DevOps, the delivery of a product, including the quality of that product, is the responsibility of the team as a whole, therefore our body of knowledge supports all people involved: the DevOps team members and also the other people in the organization, with knowledge, skills, tools and more.

In our project, which started in the Netherlands and quickly involved people across many countries, we adhered to the principles of DevOps; we shared the responsibility of creating this book, for example.

Open source is widely adopted today, we have always supported this idea of sharing knowledge so that all can benefit. The TMAP body-of-knowledge is open and available to anyone to use. And I also encourage you to contribute your examples, good practices, building blocks or whatever can be useful, so that we will build a strong quality engineering community together.

Our journey really started with a kickoff meeting in April 2019 where an international group of IT experts came together and spent two days to lay the foundations of this new view on high-performance IT delivery.

The participants were Robbert van Alen, Eveline Moolenaars-Koetsier, Bert Linker, Wouter de Kort, Gitte Ottosen (DK), Nicklas West (SE), Vincent Wijnen, Filip Joele, Joey van Hoek, Jurian de Cocq van Delwijnen and Jan Sleutjes. As a result of this meeting (which was organized and moderated by Rik Marselis and Leo van der Aalst), the team decided on the relevant QA and testing topics, created various definitions and defined the important starting points to further work out our view on high-performance IT delivery.

We had a very inspiring meeting with the following representatives of our clients: Paul Custers (NS), Werner Soeteman (KLM), Willem-Jan van Tongeren (PostNL), Reindrich Geerman (KPN), Edward Elgeti (UWV), Peter Claassen (Rabobank) and Caroline Arkesteijn (NIBC Bank). We are grateful for them clarifying their business challenges. They confirmed the direction to which we were headed, pinpointed specific subjects of interest, shared practical insights and made a number of valuable nuances.

Many people contributed to the contents of this book and the body of knowledge by inspiring the authors and by presenting useful examples, snippets of text and references to valuable sources.

For this we thank (in no particular order) Eva Holmquist (SE), Peter Betting (FR), Tom van de Ven, Andrew Fullen (UK), Barry Weston (UK), Fredrik Scheja (SE), Marianne Duijst, Paul van de Geer, Matthias Hamburg (DE), Sven Fanslau (DE), René Boswinkel, Arno Balemans, Albert Tort Pigubet (ES), Geert Vanhove (BE), Geert Jan Carpay, Gijs Op de Beek, Maurice Siteur, Marc Roekens, Martijn van der Salm, Erwin Riemersma, Mark van der Walle, Emil Wesselink, Charlotte Janus, Rianne Oorebeek – de Neef, Jan Sleutjes, Dirkjan Kaper, Jan-Willem van den Brink, Tinus Vellekoop, Marco van den Brink and Casper Schipper.

I would like to extend special recognition to Richard Ammerlaan, Randy Semeleer, Bas de Heer and Daan Kroese who largely contributed to the chapters about non-functional quality.

We also received contributions from the quality & testing community; thank you Derk-Jan de Grood, Ard Kramer and Bart Broekman. I would also like to recognize all participants to the 24th Testing Retreat for sharing their visions on this topic. The Testing Retreat is an annual independent informal meeting of international QA & testing experts.

When the book neared its completion, a great number of people invested their valuable time in reviewing the semi-finished text blocks and the complete manuscript. We especially thank the clients and partners that helped us: René Tuinhout (RDW), John Bertens (Achmea), Bart Enkelaar (Bol.com), Pascal Nicolakis (Micro Focus), Arno van der Velde (Micro Focus) and Patrice Chorot (Micro Focus).

Our colleagues contributed a great number of suggestions, comments and improvements, thank you Albert Tort Pigubet (ES), Andrew Fullen (UK), Antoine Aymer (FR), Barry Weston (UK), Blue Ityalam (IR), Eva Holmquist (SE), Fethi Mebrouk (FR), Gitte Ottosen (DK), Jürgen Beniermann (DE), Maheshwar Kanitkar (IN), Markus Niehammer (DE), Matthias Hamburg (DE), Mona Iversen (NO), Monish Pawar (IR), Nicklas West (SE), Peter Betting (FR), Sven Fanslau (DE), Torunn-Cathrine Ludvigsen (NO), Vishal Rai (IR), Arno Balemans, Bert Linker, Clemens Reijnen, Dirkjan Kaper, Erik Kuipers, Eveline Moolenaars-Koetsier, Filip Joele, Geert Jan Carpay, Gert Stad, Gijs Op de Beek, Hester van der Helm, Marco Jansen van Doorn, Marianne Duijst, Mark van der Walle, Maurice Siteur, Paul van de Geer, Pepijn Paap, Ralph Klomp, René Boswinkel, Richard Ammerlaan, Robbert van Alen, Stefan Gerstner and Mark Buenen.

Organizing this review process and merging all feedback was a job very conscientiously done by Sander van Logchem.

In the process of creating a book there are some activities that need to be done by specialists. We are very happy with the support we got from our publisher, LINE UP boek en media, in particular Peter ten Hoor, Minke Sikkema, Mirjam Kroondijk and Annelies Gallagher. The professional look and feel of this book can all be attributed to the Axioma design team, Richard Wouters, Lex Stuip and Pepijn van der Meer, and to our colleagues of marketing & communications Linda van Tilborgh, Jolien Dusseldorp-Schipper and Nicolette van der Heide.

Last but most certainly not least I would like to thank the authors of this book. Creating a book like this is a long journey. A journey that isn’t always a smooth ride; the road can be bumpy or even rocky sometimes, a journey of taking one step forward and two steps back by occasion. They persisted in achieving their goal to create a book accessible to all members in a high-performance team. Hats off for Rik Marselis, Berend van Veenendaal, Dennis Geurts and Wouter Ruigrok! Of course, special recognition for Leo van der Aalst and Richard Ammerlaan who contributed by supplying substantial content for this book.

I trust you will enjoy reading this book and hope you will be inspired to apply the knowledge and use the insights and tools to the benefit of your IT delivery process.

Rob Vijverberg, Head of Digital Assurance and Testing, Sogeti NL
Vianen, February 2020

Part 1 Introduction

1
What value will you find in this book?

Organizations can only be successful when properly supported by IT systems. People involved in the creation and use of IT systems face many challenges. Updates and changes have to be delivered faster than ever before. The right quality is vital to maintain service in our 24/7 economy. User experience should be right first time to retain customers. Teams need to deal with these challenges, integrating quality measures throughout the IT delivery process and using tools to deliver at speed.

Information Technology (IT) has made giant technological leaps forward and still evolves at an incredible pace. The way IT systems are created and delivered has also evolved. Today, IT systems are preferably delivered incrementally with frequent adaptation to new business needs and user expectations, and with increasing speed, supported by state-of-the-art tools. We should keep a close eye on what it is all about: delivering software – which will generate business value – at the right time with the desired quality!

This book provides information on how to build in quality and establish a level of confidence so that the pursued business value can be achieved, as described in our VOICE model.

We call the modern iterative-incremental way of creating IT systems: high-performance IT delivery.

High-performance IT delivery is an approach that enables cross-functional teams to continuously improve the products, processes and people that are required to deliver value to the end users.

The title – Quality for DevOps teams – demonstrates that this book is primarily meant for those that want to deliver quality solutions in a DevOps setting. Although IT is often about tools and techniques, people still make the difference between dissatisfaction or success.

Do you work in a high-performance IT delivery approach, such as DevOps, and do you want to make sure you deliver value to users by assuring the right level of quality? And do you want to establish the level of confidence that the pursued business value can be achieved? Then you will find great benefit in the valuable ideas, approaches and experiences in this practical book.

1.1 The DevOps IT delivery model

Since DevOps is the main IT delivery model in this book, let us briefly describe DevOps.

DevOps is a cross-functional systems engineering culture that aims at unifying systems development (Dev) and systems operations (Ops) with the ability to create and deliver fast, cheap, flexible and with adequate quality, whereby the team as a whole is responsible for the quality. Other areas of expertise, such as business analysis and quality assurance (including testing) are usually integrated in the team. A DevOps culture has an Agile mindset that can be supported/implemented by, for example, the Scrum framework.

Figure 1.1 DevOps cross-functional team.

Working in a cross-functional team means that the team as a whole is responsible for delivering value. The team has all competencies and skills to perform the necessary tasks and no team member has the monopoly on performing any task. This way the team can always go forward, even when a team member is temporarily not available. And of course, a team can work together with specialists from other teams or support groups for specific tasks.

Overview of IT delivery models

In addition to DevOps, Scrum is also considered one of the high-performance IT delivery models. Besides the high-performance IT delivery models, we distinguish two other IT delivery models (Figure 1.2): Sequential models (e.g. waterfall and V model) and a blend of sequential and high-performance IT delivery models, resulting in a hybrid IT delivery model (e.g. demand/supply and SAFe). You can read more about IT delivery models in Chapter 7, “Overview of IT delivery models”.

Figure 1.2 IT delivery models.

1.2 Continuous quality engineering

In the DevOps IT delivery model, there is continuous focus on quality engineering. Actually, commonly DevOps teams try to implement “continuous everything”, which means that they strive to automate as many tasks and activities as possible. This leads to, among other things, Continuous Integration and Continuous Deployment (commonly abbreviated to CI/CD).

To implement continuous quality engineering, of which continuous testing is a part, DevOps teams must use state-of-the art tools powered by artificial intelligence and machine learning. This will enable them to deliver quality at speed, for example by forecasting quality problems and solving them before anyone experiences a failure.

1.2.1 Vision: “built-in quality”

We strongly believe that high-performance IT delivery teams have capabilities as a team, the capabilities of the people in the team contribute to the team as a whole, and tasks can be performed by various team members. One team member may be more skilled than another, but that is no reason to only provide work to the team member with the most experience and skills. Think of all tasks as a team effort in which each team member plays a role. Specifically, we think that quality assurance (QA) and testing activities should be integrated in both the DevOps activities as well as the people involved.

1.2.2 DevOps activities

DevOps is an elusive phenomenon. There are discussions about who is the founder of DevOps and what the DevOps IT delivery model looks like exactly. To avoid this discussion, we use a simplified model in this book, on which the common activities are plotted (Figure 1.3).

Figure 1.3 The six DevOps activities.

We identify six DevOps activities:

• | Monitor
• | Plan
• | Code
• | Integrate
• | Deploy
• | Operate

These activities provide excellent support to explain the integration of DevOps activities with the QA and testing topics. For more information on the DevOps activities see the description in Chapter 9, “High-performance IT delivery models”.

1.2.3 Quality assurance & testing topics

In this book, we take a list of quality assurance & testing activities, grouped in “Organizing” and “Performing” topics (Figure 1.4), as our starting point. We describe how these topics relate to the DevOps activities and what the people involved need to know and need to do related to these various topics.

Figure 1.4 Quality assurance & testing topics and DevOps activities.

The topics are not directly and one-on-one related to the DevOps activities. Depending on the activity, you will see that one or more topics can relate to a specific activity. See Chapter 14, “Topics plotted on the IT delivery models”.

These topics can – or even should – be applied by all DevOps people. Not just by a tester or a person with a quality assurance role. In general, quality assurance and testing topics are integrated with all DevOps activities and executed by all people involved.

This book will support you in organizing and performing quality engineering tasks within DevOps, irrespective of whether you see yourself as a developer, an operations person, a business analyst, a quality engineer, a tester, a product owner, a coach, a manager or any other role or function.

1.3 README.TXT (reading guide)

Quality for DevOps teams is not a book to read from cover to cover. Also it is not intended as a step by step implementation of quality engineering activities in DevOps. Instead, this book provides relevant parts of the TMAP body of knowledge. Depending on your needs, interests and experience with quality engineering, parts can be read carefully, scanned quickly, or even skipped altogether.

The book is intended for all people involved in DevOps. But every person will want to use another subset from this book. The parts, chapters and sections provide a handy division. The table of contents at the beginning of the book and the index at the end of the book will also assist you in locating relevant knowledge.

You will find an extensive glossary with over 300 terms on the website. The relevant definitions are included in this book and are clearly shown in a textbox. If you are looking for a specific definition, please use the index to find the page where the term is explained or find the glossary on www.tmap.net.

1.3.1 Overview of the parts of the book

This book is divided into eight parts. A short overview:

•
Part 1 Introduction
In this part, we explain the reason why you should read this book, that successful high-performance IT delivery depends on people; we explain the VOICE model (confidence in pursued value), and give an introduction to quality assurance & testing. At the end of this part we describe the CI/CD pipeline.

•
Part 2 IT delivery models
In this part, we provide an overview of three IT delivery models: sequential, high-performance and hybrid. And, since this is the focus of our book, we provide an in-depth explanation of the high-performance IT delivery model DevOps.

•
Part 3 QA & testing topics
In this part, you will find an introduction to QA & testing topics, which are grouped into “Organizing” and “Performing” topics. Examples of cross references between these topics and the three IT delivery models can be found in this part as well.

•
Part 4 Organizing topics explained for DevOps
In this part, the eleven organizing topics are described (Figure 1.4). These organizing topics are aimed at arranging, planning, preparing and controlling QA & testing activities.

•
Part 5 Performing topics explained for DevOps
In this part, the nine performing topics are described (Figure 1.4). These performing topics are aimed at the operational, specifying and executing QA & testing activities.

•
Part 6 Quality measures and skills
In this part, a wide variety of quality measures is described. These can be used independently as part of quality assurance as well as to cover risks. In this part, you will also find a description of personal, interpersonal and team skills.

•
Part 7 Test varieties
In this part, we explain the concept of test varieties and describe four test varieties in more detail. The testing pyramid and the testing quadrants are also discussed.

•
Part 8 Test design
In this part, you will find a description of experienced-based testing and its four approaches (checklist, error guessing, exploratory testing, crowd testing). We also provide an explanation of coverage-based testing, its four groups (process, condition, data, appearance) and at least one application of a test design technique for each group.

1.3.2 Target groups of this book

The main target group of this book is people working in DevOps teams. Others with an interest in quality engineering in today’s IT delivery will also find a lot of interesting knowledge in this book.

If you have never read anything about TMAP, this book certainly will help you get an overview of what it has to offer. On the other hand, if you are experienced in applying TMAP in your work, you will notice that we have made significant additions and adaptations to make it completely fit in today’s IT world.

As an aid to choosing the parts that are of interest in connection with certain roles, we also provide you with reading suggestions.

In general, our suggestion is first to determine the challenges in your own situation, and then select the parts, chapters and topics in this book as a source of inspiration and information to implement improvements.

Since every person has a different need for information, we cannot specify which parts of this book are of interest to you. However, the following hints and tips may help you find your way in this book.

We advise everyone to start with Part 1 to understand the basic starting points.

If you are interested in different ways of IT delivery read Part 2, if you are only interested in DevOps, you may skip this part.

QA & testing consists of many activities; we have structured these activities in twenty topics as described in Part 3.

If you are involved in organizing QA and testing in DevOps (for example as a product owner, agile coach, test manager etc.), we refer to Part 4.

If you are involved in performing QA and testing in DevOps (for example as a business analyst, a developer, a tester or an operations person), we refer to Part 5.

Part 6 is valuable as a reference to specific quality measures; please select from the table of contents the quality measures that are relevant in your situation.

If you set up quality engineering, you have to make sure there is sufficient variety in testing; Part 7 will give you the information needed.

If you are involved in specifying and executing tests, you will find very useful information about test design and execution (and the test design techniques that TMAP is famous for) in Part 8.

If you need to define the focus of quality engineering, you will find an overview of quality characteristics in the appendix.

1.4 TMAP evolution

TMAP is a body of knowledge for quality engineering in IT delivery. The strength of TMAP can be largely attributed to the considerable practical experience that is the basis for the body of knowledge. This experience comes from thousands of IT professionals in as many projects over the last twenty-five years.

TMAP was created in 1995 and back then mainly contained a process description for testing. Over the years, TMAP has evolved from process-driven, business-driven and human-driven to a body of knowledge with which quality assurance & testing activities can be integrated in all IT delivery models. The body of knowledge contains a wide range of approaches, good practices, techniques and tools for organizing and performing of activities related to quality assurance and testing of IT systems.

Today we use TMAP as a term on its own, rather than a meaningful abbreviation. For this reason TMAP is now written with four capital letters.

TMAP website

The TMAP body of knowledge is easily accessible on the website www.tmap.net. There are also numerous books available in the TMAP series. Books are static, IT delivery landscapes are changing faster and faster. We therefore aim to keep the website up to date. You can find most of the content of this book on this website, as well as many in-depth explanations, additions to the content of this book and up-to-date overviews of relevant tools. In addition, the quality assurance & testing topics are worked out for other IT delivery models.

In summary, this book is supportive of the website, which is constantly updated and adapted to changed IT delivery landscapes and new insights.

We wish you a lot of useful insights and good luck with applying TMAP knowledge in your situation.

Quality assurance & testing topics and DevOps activities

2
Successful high-performance IT delivery depends on people

The IT industry is often still struggling how to leave the Tayloristic era of industrialization mindset (where people were not supposed to make their own decisions) behind. Managers are shouting about “digital, Agile, disruptive”, but many large systems development organizations still assume that value is created by tools and processes operated by people. Otherwise people cannot be trusted, estimated, or replaced.

In today’s modern world – in the age of global markets – human beings are the dynamic part of the value creation. People need to understand that they are in the driving seat, in all aspects, and in all contexts, and in all layers. Not only consumers, but also engineers, developers, operations people, quality engineers, testers, product owners, et cetera, are human first.

The Agile mindset [Agile Manifesto 2001] shows that people with a clear mission, working together in teams and across teams, can create awesome results (Figure 2.1). That is the main promise of high-performance IT delivery models such as DevOps.

Figure 2.1 Individuals and interactions over processes and tools. One of the four values of the Agile manifesto.

What we strive for is to move quality engineering & testing away from the industrial age and help organizations move quality engineering & testing into the age of global markets where people come first – at scale. This means picking up where the human-driven approach of TMAP HD started off and extrapolate that message even further. To go from “There is actually people working in your processes”, towards “Here is how you enable people to become awesome at delivering value to users.”

Empower the team; they build it, they run it. That is the main idea of bringing people together from various backgrounds into one team. This approach is called DevOps, a combination of the words Development and Operations. Keep in mind that DevOps is supposed to be a truly cross-functional way of working. In DevOps all competences that are needed to go from the very first ideas to the actual implementation of (a part of) an IT system have to be present. And teams need to collaborate if they create and support IT systems that are beyond the scope of one team. This can only be done by using the right tooling. DevOps teams strive to automate as much as possible, and what cannot be automated (yet), is done as early as possible. The organization must enable teams to work this way. This can only be done by good use of tooling. An example of this in DevOps is “infrastructure as code”, which demonstrates that everything must be controllable from the keyboard of the team members.

DevOps is not just about being cross-functional; it is also about the ability to create and deliver fast, cheap, flexible and with adequate quality. The team as a whole is responsible for the quality, they should be intrinsically interested in quality assurance and testing. This is one of the major breakthroughs in high-performance IT delivery. Everyone in the team is in some way involved in quality engineering & testing activities. An example of this is “refactoring” (improving the code without making functional changes), initiated by the team.

2.1 The Tayloristic view and the post-industrial mindset

Sequential IT delivery models (that we now want to move away from) are based on the Tayloristic view of optimizing processes by taking all initiative away from individual people. Tayloristic-focused approaches trust tools or processes to make the end result awesome:

• | “If we only have control over the requirements process, it would sort out the situation.”
• | “If we could get a smarter set of regression tests running continuously, we would find all necessary bugs.”
• | “If we introduce static source-code analysis tools and an extra quality gate before business acceptance, that would do the trick.”

A post-industrial mindset is relying on the view that humans should be enabled to create magnificent value:

• | “How can we enable our engineers to react to data integrity breaches before they occur?”
• | “How do we enable our team members to rapidly include new changes when priorities shift or feedback from customers directs us to?”
• | “How can we help our developers to get instantly notified when a critical incident happens in production?”
• | “How can we help our operations people to track user behavior in real-time?”
• | “How do we coach all our people to utilize an optimal set of tools so they can balance out time spent versus value gained?”

In short, industrialization was based on people serving the magnificent machine (process or tool). In the age of global markets, the machine should instead serve the magnificent human.

2.2 High-performance IT delivery with cross-functional teams

In today’s world various definitions are used describing the way IT should deliver results in a modern way. We use the following definition:

High-performance IT delivery is an approach that enables cross-functional teams to continuously improve the process, people and products that are required to deliver value to the end users.

This definition is used for Scrum, DevOps and similar frameworks, cultures, methods and approaches.

Figure 2.2 Multi-disciplinary team.

Figure 2.3 Cross-functional team.

In this definition, we clearly distinguish cross-functional teams (Figure 2.3). This means that all knowledge and skills must be available in the team, although not in the sense of a multi-disciplinary team (Figure 2.2), which consists of specialists that are very good at one task, but work in silos. A true cross-functional team can still function when one of the team members is temporarily unavailable, because all team members have some knowledge and skills of each of the tasks of the team. Of course, some team members are better at a specific task than others, but no team member has a monopoly on one specific skill.

A cross-functional team consisting of skilled and motivated people is a very effective way to create and maintain IT systems. This is the foundation of the DevOps culture as described in the previous chapter.

2.2.1 Six DevOps principles

Many people and organizations try to define DevOps, which leads to various views that sometimes even contradict each other. We have taken a practical approach and tried to describe DevOps in such a way that it is usable without wasting too much effort on theoretical pernickety.

For us the “DevOps Agile Skills Association” (DASA) has a usable frame of reference with their six DevOps principles [DASA 2019]:

1. | Customer-centric action
2. | Create with the end in mind
3. | End-to-end responsibility
4. | Cross-functional autonomous teams
5. | Continuous improvement
6. | Automate everything you can

You will notice that this is one of our inspirations throughout this book.

2.2.2 The three ways of DevOps

DevOps is an engineering culture that includes development and operations responsibilities and activities. It is based on key principles as described by Gene Kim in his blog post “The three ways: the principles underpinning DevOps.” [Kim 2012]

These three ways are:

1.
Systems thinking
This emphasizes the performance of the entire system.
We take the “pursued value” of the system as the measure of success.

2.
Amplifying feedback loops
Process improvement is an integral part of engineering to ensure necessary corrections can be made continuously.

3.
Culture of continuous experimentation and learning
Take risks and learn from failure leading to the mastery of skills. This requires that time is allocated to experiment and learn, in the end this will lead to both better products (generating higher value) and an optimal process.

2.3 How to move towards people-oriented quality engineering

Tayloristic testing is mostly testing in a separate step in the process. Either because it has its own lane on any board (which is a dreadful idea) or because it is done by someone in a dedicated, separate function (i.e. a person with the function of tester).

Post-industrial testing is where we test something because we are curious or require very specific feedback rather than because we need to test something because the organization demands it. Examples are exploratory testing, or situations where developers test themselves to see if their changes are valid, maybe supported by a team member with a lot of testing experience who helps them see the bigger context.

Quality engineering as a human activity is about taking responsibility as a group (team or similar) rather than executing testing as an activity. The endless, and frankly tiresome, focus on regression-testing-only, that can be found in many organizations is an excellent example of a Tayloristic approach. In the post-industrial age, we gather feedback because we need to assess the quality level at this specific moment – with or without tools – to help us get there faster. A definitive move of testing away from everyone except the team is taking place: no longer can the business demand it, no longer can the manager dictate it, no longer does the organization meddle with testing, because it is confined to the team. A nice example can be found close by: try finding a developer who lets anyone else do their unit tests. They do not exist, because for a developer, unit testing is part of developing; code written without unit tests is unfinished code. It should be the same with other varieties of testing: any form of testing is done because “delivering software” is not complete without testing (for more information on test varieties, see Part 7). The team decides on what, when and how much sometimes using risk consideration, sometimes using time-to-market considerations, etc. A test professional is no longer the one that specifies and executes the tests, it is the team member providing the knowledge about testing and cementing it as a team responsibility. Because only by putting the responsibility for quality 100% in the team can we make them feel responsible (even though the team may ask others for support on specific testing tasks like security or performance testing, they still maintain their responsibility).

2.4 Automate everything, as long as it is useful

And yes, you did not find anything about specific tools or automation in the piece above. That is because the tools as such do not matter. In DevOps, teams strive to automate everything. If the team needs tooling to fulfil certain capabilities, they should definitely use tools. But keep in mind that just implementing a tool seldom solves a problem, and one tool can be exchanged for another. To deal with the amount of changes/deliveries/complexities, we need to “intelligently” automate the quality engineering activities – across the IT delivery lifecycle. And this requires much better orchestration, organization and enablement than teams usually have today. Critical will be the integration with development tools and CI/CD tools. Other critical factors are flexibility and every team member being able to use tools.

The team needs to implement a way of working with which they can say with confidence that they are satisfied with the quality of the software they deliver and the speed with which it was delivered. The fundamental issues teams are facing are never technical; they are always human: confidence, trust, communication and cooperation. You will often find those at the heart of any poor use of a tool or a failure in production. The primary goal of quality engineering is therefore also human centered: exploring, learning a system, gaining confidence.

2.5 From DevOps to AIOps to NoOps?

This book is about DevOps. Many other terms related to the combination of development and operations are used. Two terms that in our opinion are worth mentioning are AIOps and NoOps as they may describe where the evolution of IT delivery is heading.

2.5.1 AIOps

AIOps platforms utilize big data, modern machine learning algorithms and other advanced analytics technologies to directly and indirectly enhance IT operations functions (e.g. monitoring, automation and service desk) with proactive, personal and dynamic insight. AIOps platforms enable the concurrent use of multiple data sources, data collection methods, analytical (real-time and deep) technologies, and presentation technologies. [Lerner 2017]

2.5.2 NoOps

“NoOps,” which stands for “No IT Operations,” is an approach that can greatly benefit companies with high technological maturity or that play a significant role in software development and maintenance under certain circumstances.

Although the idea is to fully automate all IT operations tasks, this NoOps situation still requires initial setup activities, so people with operations expertise will remain needed. The main assumption behind NoOps is that DevOps practitioners are no longer required to take care of operations and may concentrate on systems development.

Under NoOps, all operations are automated, minimizing or even eliminating the risk of human error, and automated processes are faster and less error prone. With this approach, IT Ops teams no longer need to be engaged in technology-related tasks on a daily basis, because everything that could be automated is already automated. [STX Next 2019]

3
The VOICE model

Today’s organizations expect that their IT systems will enable them to generate business value. This business value may be financial but can just as well be a value in any other quantitative or qualitative way. To be able to specify the pursued value, people in the organization will detail their objectives. These objectives must be measurable to some extent. Therefore, indicators define a way to gain insight into specific objectives. Measuring these indicators is done by testing and other quality and value measuring activities. This testing results in information based on which the people involved can form an opinion and determine to what extent they are confident that the pursued value can be achieved. After the decision has been made to use the IT system, the organization can experience the real value. This experience will be used to further improve the value in the organization.

The VOICE model is about establishing the level of confidence that the pursued business value can be achieved. It consists of 5 terms: Value, Objectives, Indicators, Confidence and Experience.

Figure 3.1 The VOICE model of business delivery and IT delivery.

This is visualized in the VOICE model. This is a generic model, relevant for all kinds of IT delivery lifecycles that support business delivery (Figure 3.1). VOICE is an acronym of Value, Objectives, Indicators, Confidence and Experience, terms that together describe how to implement delivery of business value.

High-performance IT delivery teams (such as in DevOps) also use the VOICE model as a foundation to structure and organize their work. In these teams one of the many task-groups is quality engineering. This book will support you in organizing and performing the quality engineering tasks within your high-performance IT delivery team, irrespective of whether you see yourself as a developer, an operations person, a business analyst, a systems architect, a tester, a product owner, a coach, a manager or whatever other role or function. We strongly believe that high-performance IT delivery teams have capabilities as a team, the capabilities of the people in the team contribute to the team as a whole, and tasks can be performed by various team members. One team member may be more skilled than another, but that is no reason to only provide work to the team member with most experience and skills. Think of all tasks as a team effort in which each team member plays a role. And keep in mind that teams have team skills, and team members have personal and interpersonal skills (read more about skills in Part 6).

3.1 Explanation of the VOICE model

The VOICE model for quality in IT consists of five terms:

•
Value: Any IT system aims to bring value to someone. This value must be defined, this often reveals implicit expectations and makes them explicit. Keep in mind that high quality alone is not always high value. Quality means different things to different (groups of) people. Sometimes there is more value in a system with a lower quality level that is quickly available than in a high-quality system that is available too late.

•
Objectives: To understand the purpose of an IT system and how to create and maintain it, there must be quantifiable objectives with an adequate level of detail.

•
Indicators: Whether the objectives are met, and whether the pursued value can be achieved, needs to be measured. To this end, indicators are specified and tests will be performed to measure these indicators.

•
Confidence: The result of measuring the indicators will be good information for stakeholders to gain confidence that the IT system will be able to achieve the pursued value.

•
Experience: After the IT system has been incorporated in the operational business processes, the people in the organization will experience the actual business value. Based on this, they may define further improvements and a new cycle of the VOICE model is triggered.

The duration and speed of the cycles of the VOICE model significantly differ depending on the IT delivery model that is used and the release cycles in such a model. In sequential IT delivery (such as waterfall or V model) the cycles may be months, whereas in high-performance IT delivery (such as Scrum or DevOps) the cycles will be weeks or days or even minutes.

3.2 The VOICE model applied to DevOps

The VOICE model is about establishing the level of confidence that the pursued business value can be achieved. By taking the Value as a starting point, you ensure that the focus is on the goal of IT delivery, and the people involved keep being reminded that IT delivery itself is not the goal.

While the VOICE model concerns supplying information, DevOps is about creating systems. The VOICE model therefore supports gathering information about IT systems during the time these systems are created and maintained. Continuous quality assurance and testing are essential to supply vital feedback to all people involved and to promote achieving valuable systems.

Of course the VOICE model can be applied to other IT delivery models as well (such as sequential and hybrid IT delivery), but that is outside the scope of this book.

The VOICE model applied to DevOps consists of:

•
VALUE
In today’s world, the main reason for building and maintaining information systems is to create business value to stakeholders. While building and maintaining the information system, the stakeholders need to have information to decide whether they are confident that the business value can actually be attained. The first step is to identify the pursued business value. The value will relate to creating a specific service or supplying a certain product for the stakeholders within or outside the organization.
The business value elaboration itself is not described by the VOICE model or in this book, since it is part of IT development activities beyond the scope of this book. The defined business value however is the starting point for quality assurance and testing.
The value mainly relates to the DevOps activities “monitor” and “plan”.

•
OBJECTIVES
When the pursued value is clear, the next step is to detail the business value into quantifiable objectives. The objectives define several different but coherent aspects that together will make the confidence in the business value measurable. In DevOps, people will use the objectives as guidance about what the IT system must achieve and how it must support the business processes. These objectives will amongst others refer to service levels to be met, quality characteristics to be implemented and the software delivery model (Bespoke / COTS / SAAS) used.
The objectives mainly relate to the DevOps activities “monitor” and “plan”.

•
INDICATORS
To measure whether the objectives are achieved, one or more indicators per objective are defined. These indicators are measured by means of data collection and data analysis, which in the short DevOps cycles can only be done effectively using tools that automate most of the collection, analysis and visualization.
Measuring these indicators is what we generally call testing, but other quality measuring activities are also used. Indicators may be caught in very exact measurements, but some indicators will also be based on opinions of people, which just as well contribute to establishing the confidence level.
The indicators are defined in the DevOps activity “plan” and measured in all other DevOps activities.

•
CONFIDENCE in pursued value
With the results of testing and other quality measuring activities, the team interprets the indicators and reports using a confidence monitor (see Chapter 19, “Reporting & Alerting”). The stakeholders digest the information, which results in the confidence whether or not the pursued business value is attainable. This conclusion generally supports the decision whether or not (a changed version of) an IT system can go live.
Confidence is built by the people involved, such as the product owner, during all DevOps activities, but mainly during “deploy”, “operate” and “monitor”.

•
EXPERIENCE the real value
When an IT system is used to support a business process, the users experience the real value. They will experience this real value based on the outputs and results of the system, but also by monitoring (mostly using tools) the indicators in a live operation of the system. They will compare all the gathered information with their initial expectations as described in the pursued business value. Based on this experience and the comparison, new ideas will arise that are input for changes or new developments. A new cycle of the VOICE model is then triggered.
“Experience the real value” mainly relates to the DevOps activity “monitor”.

3.3 Implementation remarks on the VOICE model

The VOICE model does not describe all IT activities. These will be defined in the specific IT delivery model that the team(s) use(s), for example in the DevOps activities as referenced in the previous section. We use the VOICE model as an overarching model. The IT delivery models are grouped as sequential, high-performance and hybrid. For more information see Chapter 7, “Overview of IT delivery models”.

Value in this context refers to achieving the business case (or a lower-level result such as a change-request), but also to reach it in an efficient and effective way. Translating value to objectives may be a very simple task when the scope is a very limited user story; on the other hand, it may be a huge effort when the value refers to creating a complete IT system.

The stakeholders will consist of a broad group of people, varying from business managers, end users and customers, to the business analysts, developers, testers and operations people, amongst many others. All stakeholders are involved in some way in defining the objectives and indicators. The team uses those objectives and indicators to determine which activities they need to perform at what point in time. Depending on the IT delivery model (such as DevOps or sequential or anything else) the team will plan their work and supply information to the stakeholders so that the stakeholders can establish their feeling of confidence that the pursued business value will be attainable.

Different parts of the VOICE model will be relevant to different roles. Table 3.1 shows which role is mainly involved in what parts of the VOICE model.

Table 3.1 Main focus of team roles to parts of VOICE model.

DevOps role | Value | Objectives | Indicators | Confidence | Experience
Product owner | X | X |  |  | X
Business analyst | X | X | X |  |
Systems architect | X | X | X |  |
Developer |  | X | X |  |
Tester |  | X | X | X |
Operations |  |  | X | X | X
Agile coach |  | X | X | X |
User | X | X |  |  | X
Customers |  |  |  |  | X

4
Examples of indicators in the VOICE model