Software Engineering: Scaling Agile: The Value of Independent Testing

In Agile projects of all sizes including large-scale projects, software testing plays an important role. In our industry, there is a school of thought that proclaims that there are no developers or testers in an Agile team and everyone is a ‘team member’. Also, it challenges the need for independent testing. My intention is not to argue on that but to share my thoughts on the value of independent testing in this post.

First, in reality we do not develop standalone applications every time. A vast majority of software applications or products we create are complex with dependencies or integration needs on internal or external systems. They involve multiple technologies and compatibility requirements too. Second, nowadays, many projects involve two or more cross-functional teams. As the number of teams increases the situation becomes complex in spite of cross-functional teams focusing on the ‘Definition of Done (DOD)’. It is not enough if multiple cross-functional teams deliver working software that satisfies DOD. We need an integrated build that works! We need an integrated build that satisfies all functional requirements. We need to make sure that it satisfies all non-functional requirements as well – everything related to performance, security, integration, and so on. We need an independent testing team or independent verification team to take care of this.

That is obvious, known and make sense. However the questions are,

1. What should be the size of this independent testing or independent verification team?

2. When should we introduce this team - in the first iteration itself or closer to the first release?

3. Do we need this team in all iterations throughout the project?

4. How can we justify the costs?

5. What can go wrong even if we have an independent verification?

The size of the independent team typically varies between 10 to 15% of the total team size. For example if the size of your large-scale agile team is 200, that will include an independent testing team of 20 team members. That is just a guideline. And this team will include automation experts who focus on test automation for end-to-end functional tests, performance tests etc. The size of the team will increase based project complexity (multi-browser, multi-device, complex architecture, complex business rules), dependencies, non-functional requirements, target users (number of countries, regions, languages, etc.).

‘When should we introduce this team?’ is a great question. It is very important to explore test strategies and identify an optimal one. Your test strategy exploration needs to address several questions - What is our need? What are the possible approaches to meet our needs? What are the tools and techniques? What are the measures? What is going to be the governance? What are the pros and cons of strategy A, B, and C? When you do this you will get a clear idea on when to introduce this team.

‘Do we need an independent verification team throughout the project?’ The answer is ‘Yes’. You have multiple cross-functional teams that deliver features in short iterations. You need an independent verification team to test the integrated builds. You need them all the time. How do we justify costs? Well, don’t you have to test the integrated builds? Of course, yes. Independent verification directly relates to the cost of quality.

‘What can go wrong even if we have an independent verification team?’ Good question. Several things can go wrong. First, feature team can become lax on delivering tested code to meet DOD or product owners may relax DOD to speed up feature delivery assuming that independent verification team can find all defects. With that you enter a vicious cycle and independent verification team will slog in finding all kinds of defects. Feature teams will do firefighting to fix defects and that will potentially open more defects. This is a huge risk on release quality as well as timelines. Second, with no or weak test strategy, independent verification team may lose traction on test automation. Third, with lack of focus on fundamental testing principles, the team may dilute or miss traceability and coverage. There can be many other things that can go wrong. Please feel free to add those in your comments.

‘Are we going to have an Independent Verification Team? That sounds like waterfall!’ You must have heard this too. It is not waterfall. It is large-scale Agile. All team members are involved from the initial stages. The goals are clear. They follow Agile principles and practices. Teams do not operate in silos. Their timelines are not squeezed. And they don’t compromise in any way in terms of quality to make a release happen!