Who Cares About Software Quality?



This blog is about my observation and interpretation on the keynote address, ‘The Future of Quality’, at Belgium Testing Days 2012 by Goranka Bjedov. A moment of discontent or restlessness in her professional life related to the state of the industry and engineering practices during 2008 triggered several events that culminated in a realization, ‘Quality is Dead’, in 2009. This is because there were (and are) software products and applications out there in the market. Most of them are buggy. Customers are end-users accept such products. Buggy products become a source of revenue as they fetch maintenance and upgrade fees. According to her, practitioners went wrong with the thought that testing is all about checking by means of executing tests and finding defects. She emphasized that testing is more than this and there are several dimensions to testing.

Her question “What happens when value of quality is less than the price of quality” triggered additional questions. Is functional testing adequate? Do we do too much of functional testing (and hence spend a lot)? Is that cost justified? Do end-users pay more and get less? She emphasized that it is imperative for testing teams to ensure that nothing escapes, to provide information about the current state of quality, and stop the release of bad products. It is possible when testers understand product quality. Testers have to carry the right purpose and approach to make this happen.

We have seen software malfunction and failures in many industries. Businesses have paid huge sums of money to settle related charges and court verdicts. The keynote included several examples.

With these facts in front of us, she introduced ‘Productivity Testing’. According to her 'Productivity Testing' is all testing that is focused on faster development. Its main purpose is preventing developers from checking in bad code, and allowing for changes to the code with certain "peace of mind".

Unit tests, micro-benchmarks, "sanity" tests, etc. are examples of Productivity Testing. According to her these are small, fast, cheap to write and maintain, analysis-free (when they fail, it is clear where and why), perfect for automation, fantastic for gaming the system, managers love them (code coverage, metrics), and "technical" (are we using mocks, fakes, doubles, or something else?).

In order to stress on Agility, she shared Brian Marick’s quote

"The lore of testing is full of people who spent weeks improving test automation libraries without ever, you know, quite getting around to automating any tests. The trick is to make improvements in small steps while simultaneously continuing to frequently deliver the business value that makes the project worth funding. There's a real skill to moving gradually and continuously and simultaneously toward several larger goals."

This keynote revealed or reemphasized points such as

1) There is no point in claiming the number of test cases executed or talking about the coverage or number of test cases automated or the types of tools and frameworks used unless we demonstrate that we do the right things to add value to stakeholders at regular intervals.

2) Customers do not want everything right in a product to begin with (unless the product is going to impact their lives or critical money). They want products faster. They want the base quality right. Everything else does not matter. They want products free (and are ready to pay for services and upgrades).

3) When you have the base functionalities in place, it is ok if you deal with bugs as and when you find them. That is how many products are sailing through their life cycle in our industry.

4) We do get into a denial mode and hide ourselves claiming that because of high complexity in products, we are in this state.

I liked the way she concluded. She hinted that the current state of quality is the consequence our thought process, practices and attitude. In order to make the future of quality brighter we must have to think what we can do better. We can reduce development efforts (and costs) by writing productivity tests. We can bring in quality by adding smart system tests in the right places. We must think performance, scalability, usability, and optimization (virtualization, cloud). Also we must understand the economics of software and demonstrate the value of our work (in dollar amounts) to stakeholders.

Publishing volumes of test standards and guidelines do not ensure positive results unless practitioners understand the basics and apply context specific practices.

I agree. Our industry is evolving. We have seen a small number of top notch products and services in this evolution. Our industry will continue to evolve. We will continue to see top notch products and services. Either directly or indirectly customers define what is acceptable and what is not. Keynotes and discussions like these make us stop, think and revalidate our beliefs. Testers (and everyone in project teams for that matter) can enable the delivery of high quality products. Customers can demand quality. When both entities accept the unacceptable, we will continue to see bad quality products, and bugs that result in litigations and huge penalties.

When I shared this blog with Goranka, she added the following points.

1. All paid work should be about creating value - either in the short or in the long term.

2. Created value can be demonstrated in multiple different ways - but being able to attach monetary amounts to the value of work performed is the hardest one to argue with or dismiss.

3. Quality does not have a linear relationship with value. Once good enough is reached, the rest is obtained with effort that could have been better spent elsewhere.

4. ‘What is good enough’ is determined by the customer and not by internal test or QA department.

Additional Links:

     Goranka Bjedov‘s Keynote Abstract and Bio

     Brian Marick’s ‘Two Forgotten Agile Values: Ease and Joy’

     Goranka’s Keynote at STANZ 2011 (Video)