QA Research and Tackling The Causes of Defects - A CEO's Perspective

I often find it difficult to articulate what testing, bug detection and quality assurance means at the C-suite level. I know for a fact that I am not the only one and the inability to articulate our benefits to the bottom line is a big problem among testers. We know that it reduces cost of rework and increases customer satisfaction (if done properly), however how do you quantify or articulate that to the people at the top? Also, would the CEO of a large company, say a multinational, really understand what tackling bugs and why they happen does not just to the health of products and services but also to the overall health and strategy of the business? Would the CEO understand why it is a priority deserving of major R&D, and be proud to discuss it with investors?

Then I found an example of a CEO who, for want of a better word, "gets it". I watched a recent interview with Richard White, the CEO and founder of the Australian logistics software company WiseTech Global. The part where he talked about quality enhancements (12:00 onwards) as their major R&D factor is impressive.



What made this stand out to me was that when asked about R&D and having the edge on competitors, he spent time talking about the continual work done with his CTO to find out why defects occur and waste occurs in software development. He was proud of their continual work in "squeezing out" the failure points and cleaning up technical debt (something that I have never heard a CEO even mention in an investor interview, and something that is de-prioritized often in favour of working on new features).

To me, the most powerful thing he said was that he saw their investment on the above and the plummeting defect rates as providing a higher "yield" on each dollar spent, using the example of a 32% yearly increase in product updates done with just a marginal increase in staff.

This is a tech company thus software development is the core of their work, however there have been more than a few tech companies that have neglected quality assurance. What this shows is a CEO and technology team who see QA as a major priority to the bottom line and have taken steps to quantify it, allocate R&D resources to drive defect rates down and then be able to present and proudly talk about it on a show aimed at investors.

If a CEO can articulate the value that working on QA and defect resolution and prevention can talk about it so proudly, precisely and articulately, we testers should too.

On Minor Bugs, or Death by a Thousand Cuts

Defects like other products in IT have a lifecycle. There are variations but in most places I have seen and worked the lifecycle for defects that are fixed, retested and pass retest is largely as below -

Found -> Logged or Reported -> Triaged, Assigned Severity -> Allocated to Developer -> Reported Resolved -> Retested -> (if pass retest) Closed

This is not the only way. Some testing schools, particularly Rapid Software Development recommend talking with the developer to see if a quick fix can be added efficiently without the need to log or report the issue. One with access to source code and sufficient knowledge may have the development knowledge to implement one's own fix. The one problem with this approach is that these incidents would not appear in defect counts in test summary reports.

In a large number of cases the reporting and triage/severity assignment of a defect are done in the same step. That is a problematic approach and assumes that the tester has the background and knowledge to make the same judgement about defect severity as a user or stakeholder. There are numerous times I have misjudged the user impact of some defect and been subsequently corrected.

Typically defects are graded based on an enumerated severity score. One I use is -

Sev 1 : Blocker.

Catastrophic failure of the application or major functionality. No further testing can proceed. Defect must be resolved immediately before delivery.

Sev 2 : Critical.

A failure of a functionality or loss of data which cannot be worked around. Defect must be resolved before delivery.

Sev 3 : Medium

Failure of a functionality for which a workaround exists. Delivery may be done if PM or stakeholders accept the risk.

Sev 4 : Minor, Trivial

Cosmetic defects. Styling issues, spelling mistakes, layout and minor workflow errors etc.

Defects are then prioritised based on available resources, the impact of other work, timelines and fixes applied to Sev 1 and 2 defects first, then 3 and 4. Defects 3 and 4 may be resolved following a production release dependent on the acceptable risk...

..Except that the pressures of timelines, staffing and resourcing and the need to release other features make it very difficult for us to return to work on these lower severity issues. They often get forgotten - that minor useability issue that customers have to live with or that styling defect that makes the application look somewhat less than professional.

This is a dangerous state for the application and the project. While high severity defects mean that the customer cannot do some task or other and thus do typically get prioritised and fixed quickly, the less severe look-feel and useability defects left unresolved are what will annoy your customers and users in the long term, leave a bad impression of the company and at worst will leave your product as shelfware. These are the things that they grumble about in user groups or forums on the web, mention in app and product reviews and tell their friends and colleagues - each a lost avenue that may overwise have lead to a new customer. Once the complaints and dissatisfaction reach a crescendo these initially small issues become major ones and you will have to deal with them in a fast, probably stressful manner.

So how can we manage these? Some ideas I have seen are -

1) Endeavouring to resolve all of them before the first production release. This is likely to be impractical, negates the idea of priorities and severities in the first place and the sudden redevelopment will require more regression testing (automated or not). However it would get the job done.

2) An agile cycle ringfenced to resolve all outstanding non-severe defects. This is a good approach but not perfect. Agile development cycles centre around the release of features and you would lose a cycle of new features in the process. It will thus require buy-in from the stakeholders of the project, who may themselves not be so concerned about minor defects vs key features.

3) A blended approach of fixes drip-fed over time. This may be particularly good where there already exists a continuous integration and deployment process such that the said fixes can be deployed speedily and seamlessly. However this still impacts on resources otherwise likely to be working onthe development of features and there would still need to be a schedule devised such that all the lower severity defects are resolved as soon as possible.

The above was written as a point of discussion and I would love to know your views.