Sunday, 26 March 2017

On Minor Bugs, or Death by a Thousand Cuts

Defects like other products in IT have a lifecycle. There are variations but in most places I have seen and worked the lifecycle for defects that are fixed, retested and pass retest is largely as below -

Found -> Logged or Reported -> Triaged, Assigned Severity -> Allocated to Developer -> Reported Resolved -> Retested -> (if pass retest) Closed

This is not the only way. Some testing schools, particularly Rapid Software Development recommend talking with the developer to see if a quick fix can be added efficiently without the need to log or report the issue. One with access to source code and sufficient knowledge may have the development knowledge to implement one's own fix. The one problem with this approach is that these incidents would not appear in defect counts in test summary reports.

In a large number of cases the reporting and triage/severity assignment of a defect are done in the same step. That is a problematic approach and assumes that the tester has the background and knowledge to make the same judgement about defect severity as a user or stakeholder. There are numerous times I have misjudged the user impact of some defect and been subsequently corrected.

Typically defects are graded based on an enumerated severity score. One I use is -

Sev 1 : Blocker.

Catastrophic failure of the application or major functionality. No further testing can proceed. Defect must be resolved immediately before delivery.

Sev 2 : Critical.

A failure of a functionality or loss of data which cannot be worked around. Defect must be resolved before delivery.

Sev 3 : Medium

Failure of a functionality for which a workaround exists. Delivery may be done if PM or stakeholders accept the risk.

Sev 4 : Minor, Trivial

Cosmetic defects. Styling issues, spelling mistakes, layout and minor workflow errors etc.

Defects are then prioritised based on available resources, the impact of other work, timelines and fixes applied to Sev 1 and 2 defects first, then 3 and 4. Defects 3 and 4 may be resolved following a production release dependent on the acceptable risk...

..Except that the pressures of timelines, staffing and resourcing and the need to release other features make it very difficult for us to return to work on these lower severity issues. They often get forgotten - that minor useability issue that customers have to live with or that styling defect that makes the application look somewhat less than professional.

This is a dangerous state for the application and the project. While high severity defects mean that the customer cannot do some task or other and thus do typically get prioritised and fixed quickly, the less severe look-feel and useability defects left unresolved are what will annoy your customers and users in the long term, leave a bad impression of the company and at worst will leave your product as shelfware. These are the things that they grumble about in user groups or forums on the web, mention in app and product reviews and tell their friends and colleagues - each a lost avenue that may overwise have lead to a new customer. Once the complaints and dissatisfaction reach a crescendo these initially small issues become major ones and you will have to deal with them in a fast, probably stressful manner.

So how can we manage these? Some ideas I have seen are -

1) Endeavouring to resolve all of them before the first production release. This is likely to be impractical, negates the idea of priorities and severities in the first place and the sudden redevelopment will require more regression testing (automated or not). However it would get the job done.

2) An agile cycle ringfenced to resolve all outstanding non-severe defects. This is a good approach but not perfect. Agile development cycles centre around the release of features and you would lose a cycle of new features in the process. It will thus require buy-in from the stakeholders of the project, who may themselves not be so concerned about minor defects vs key features.

3) A blended approach of fixes drip-fed over time. This may be particularly good where there already exists a continuous integration and deployment process such that the said fixes can be deployed speedily and seamlessly. However this still impacts on resources otherwise likely to be working onthe development of features and there would still need to be a schedule devised such that all the lower severity defects are resolved as soon as possible.

The above was written as a point of discussion and I would love to know your views.

Some Security Podcasts I Like

As a tester and a current student of web security and pentesting I have recently tried to update and improve my knowledge on general security issues. I am a big fan of podcasts, being able to listen to them while I work and travel. Having a set of podcasts on the go is I find an easy win in terms of time and opportunity - and there are some free, good and enlightening ones...

OWASP 24-7

The official podcast of the OWASP security consortium. A podcast released around once a month. Conversations with speakers on big and important areas such as security, devops, continuous integration and project management along with details on individual OWASP projects and the APPSEC conferences. I find the style engaging and informative and almost never dull. Not to be missed.

TWiT Security Now

The Daddy of the genre. IT Guru and broadcaster Leo Laporte and anti-spyware pioneer Steve Gibson spend two hours every week discussing the big IT security issues of the day. Recent topics include the huge Struts 2 and Cloudflare vulnerabilities, the leaking of CIA's Vault 7 and cyber warfare against Poland and the Ukraine. Once again, interesting conversations and information to while away the hours.

Defensive Security Podcast

Security experts Jerry Bell (@maliciouslink) and Andrew Kalat (@lerg) talk about the big security issues of the day and how companies can protect themselves. Covers similar topics to TWiT but in a shorter time and at a somewhat more technical level. A decent and informative podcast, however I would not believe that there is much value in listening to this and TWiT Security Now every week.

Paul's Security Weekly / Hack Naked News

​A large mixture of podcasts and webcasts produced by Paul Asadoorian aimed at the jobbing security tester, with demos, news items, online interviews and advice entertainingly delivered. The Hack Naked News cast comprises short (no more than 10 min) news episodes about the world of hacking and security. Great for those who are busy or reticent about listening to the longer recordings above. Worth checking out..