On Minor Bugs, or Death by a Thousand Cuts

Defects like other products in IT have a lifecycle. There are variations but in most places I have seen and worked the lifecycle for defects that are fixed, retested and pass retest is largely as below -

Found -> Logged or Reported -> Triaged, Assigned Severity -> Allocated to Developer -> Reported Resolved -> Retested -> (if pass retest) Closed

This is not the only way. Some testing schools, particularly Rapid Software Development recommend talking with the developer to see if a quick fix can be added efficiently without the need to log or report the issue. One with access to source code and sufficient knowledge may have the development knowledge to implement one's own fix. The one problem with this approach is that these incidents would not appear in defect counts in test summary reports.

In a large number of cases the reporting and triage/severity assignment of a defect are done in the same step. That is a problematic approach and assumes that the tester has the background and knowledge to make the same judgement about defect severity as a user or stakeholder. There are numerous times I have misjudged the user impact of some defect and been subsequently corrected.

Typically defects are graded based on an enumerated severity score. One I use is -

Sev 1 : Blocker.

Catastrophic failure of the application or major functionality. No further testing can proceed. Defect must be resolved immediately before delivery.

Sev 2 : Critical.

A failure of a functionality or loss of data which cannot be worked around. Defect must be resolved before delivery.

Sev 3 : Medium

Failure of a functionality for which a workaround exists. Delivery may be done if PM or stakeholders accept the risk.

Sev 4 : Minor, Trivial

Cosmetic defects. Styling issues, spelling mistakes, layout and minor workflow errors etc.

Defects are then prioritised based on available resources, the impact of other work, timelines and fixes applied to Sev 1 and 2 defects first, then 3 and 4. Defects 3 and 4 may be resolved following a production release dependent on the acceptable risk...

..Except that the pressures of timelines, staffing and resourcing and the need to release other features make it very difficult for us to return to work on these lower severity issues. They often get forgotten - that minor useability issue that customers have to live with or that styling defect that makes the application look somewhat less than professional.

This is a dangerous state for the application and the project. While high severity defects mean that the customer cannot do some task or other and thus do typically get prioritised and fixed quickly, the less severe look-feel and useability defects left unresolved are what will annoy your customers and users in the long term, leave a bad impression of the company and at worst will leave your product as shelfware. These are the things that they grumble about in user groups or forums on the web, mention in app and product reviews and tell their friends and colleagues - each a lost avenue that may overwise have lead to a new customer. Once the complaints and dissatisfaction reach a crescendo these initially small issues become major ones and you will have to deal with them in a fast, probably stressful manner.

So how can we manage these? Some ideas I have seen are -

1) Endeavouring to resolve all of them before the first production release. This is likely to be impractical, negates the idea of priorities and severities in the first place and the sudden redevelopment will require more regression testing (automated or not). However it would get the job done.

2) An agile cycle ringfenced to resolve all outstanding non-severe defects. This is a good approach but not perfect. Agile development cycles centre around the release of features and you would lose a cycle of new features in the process. It will thus require buy-in from the stakeholders of the project, who may themselves not be so concerned about minor defects vs key features.

3) A blended approach of fixes drip-fed over time. This may be particularly good where there already exists a continuous integration and deployment process such that the said fixes can be deployed speedily and seamlessly. However this still impacts on resources otherwise likely to be working onthe development of features and there would still need to be a schedule devised such that all the lower severity defects are resolved as soon as possible.

The above was written as a point of discussion and I would love to know your views.

Some Security Podcasts I Like

As a tester and a current student of web security and pentesting I have recently tried to update and improve my knowledge on general security issues. I am a big fan of podcasts, being able to listen to them while I work and travel. Having a set of podcasts on the go is I find an easy win in terms of time and opportunity - and there are some free, good and enlightening ones...

OWASP 24-7

The official podcast of the OWASP security consortium. A podcast released around once a month. Conversations with speakers on big and important areas such as security, devops, continuous integration and project management along with details on individual OWASP projects and the APPSEC conferences. I find the style engaging and informative and almost never dull. Not to be missed.

TWiT Security Now

The Daddy of the genre. IT Guru and broadcaster Leo Laporte and anti-spyware pioneer Steve Gibson spend two hours every week discussing the big IT security issues of the day. Recent topics include the huge Struts 2 and Cloudflare vulnerabilities, the leaking of CIA's Vault 7 and cyber warfare against Poland and the Ukraine. Once again, interesting conversations and information to while away the hours.

Defensive Security Podcast

Security experts Jerry Bell (@maliciouslink) and Andrew Kalat (@lerg) talk about the big security issues of the day and how companies can protect themselves. Covers similar topics to TWiT but in a shorter time and at a somewhat more technical level. A decent and informative podcast, however I would not believe that there is much value in listening to this and TWiT Security Now every week.

Paul's Security Weekly / Hack Naked News

​A large mixture of podcasts and webcasts produced by Paul Asadoorian aimed at the jobbing security tester, with demos, news items, online interviews and advice entertainingly delivered. The Hack Naked News cast comprises short (no more than 10 min) news episodes about the world of hacking and security. Great for those who are busy or reticent about listening to the longer recordings above. Worth checking out..

On Vegan Cooking and What Testers can Learn from it

This admittedly odd idea for a blog came from an irreverent Twitter discussion between testers Jean Ann Harrison (@JA_Harrison), Mike Talks (@TestSheepNZ), Gem Hill (@Gem_Hill) and I that touched on vegan cookbooks and cooking, binding agents, testing conferences and random twitter followers! The idea came up that there could be some concept X that works in vegan cooking and yet could be readily applied to good software testing practice - and that someone should submit a testing conference abstract on this!

I haven't yet accepted the challenge to write a conference abstract, however as a tester, occasional blogger on eccentric testing topics and vegan I did offer to think hard about this and blog some ideas. This is my attempt at a testing article using vegan cooking as a metaphor. This blog post could crash and burn a horrible vegan death, taking my reputation with it, however fortune favours the brave and all that....

I have been a vegetarian since 2006 and a vegan since late 2012. I have never been comfortable with the idea that animals are there for our exploitation and consumption, and took the vegetarian and eventually vegan path for moral reasons. I am disappointed that it hasn't changed my waistline much - a combination of the large number of vegan confectionaries out there and my lack of  willpower - therefore I cannot vouch for it as a weight loss plan!

I spent some time thinking about this tonight while cooking dinner. Note that cooking and software testing are obviously very different activities and therefore the concepts I outline are somewhat nebulous, general and already well known, however that does not detract from their importance. Most cooking concepts listed below apply to some degree to other restricted diets and non-vegan cuisine as well.

My Dinner - Vegetable, Basil and Mock Chicken broth with rice.

1) Constraints Abound. Get Used to Going Off-Script and Exploring.

There is no national cuisine that fully lends itself to veganism (except maybe Vietnamese and some Middle Eastern cuisine). Vegan recipes and cookbooks are typically developed by either taking existing recipes and exploring suitable plant-based replacements for animal-based ingredients or developing from scratch.

This in itself is not perfect. Take baking for example. Baking usually uses egg as a binding agent and milk and butter to provide flavour, structure and texture. There are alternatives to these - binding agents such as ground flaxseed and mashed banana and nut and soy milks and margarines - however these are not a seamless replacement and can alter the taste, texture and preparation of food.

Another example is meat itself. A plethora of soy, seitan, mushroom, gluten and konjac-based meat replacements exist (SE Asian and Indian supermarkets are best for these although most supermarkets stock some brands), however they have their own properties, are not quite the same in flavour or texture and not to everybody's taste. For recent adherents to vegetarianism who are fond of the taste of meat (as I was), this can be troubling.

What this has resulted in is a great culture of exploration and experimentation. Lots of cookbooks were developed by taking an existing cuisine or set of recipes and add radical ingredients, not obvious at first glace, gained from iterative trial, error and creative adjustment. The recipe is just the starting point. We take vegetables, herbs, legumes and other ingredients and add them until we get the satisfaction we want. It can be long and laborious over many meals but it is a lot of fun and when it works the results are dramatic! As an example, the cookbook "Appetite for Reduction" by vegan chef Isa Chandra Moskowitz contains recipes from such combinations as red wine, kalamata olives and tempeh and garlic dressing made from tahini, nutritional yeast flakes and miso - and they taste amazing!

Our well-thumbed cookbooks.

In the world of testing it is well known that exploratory testing methods are more effective at finding defects than adherence to traditional scripted test documentation - I also find them more interesting and motivating - and yet so many testers still draw up test cases with detailed steps in advance and execute them narrowly and slavishly. This also requires the existence of such items as a complete, static requirements specification - which is most often unlikely to exist in all but highly structured waterfall projects. The key attributes of a tester (in my opinion) are curiosity about the software under test, a critical eye, passion for quality, persistence and creativity - so feel free to deviate from the path, come up with new test ideas and go for it!

2) Keep Your Environment Protected, Clean and Maintained (Preferably by You)

My wife and I live with my meat-eating father. We use the same fridge and kitchen. We take steps to protect our environment.

Having uncooked meats and unpackaged herbs, fruits and vegetables (especially to be used in raw salads) in the same fridge is a sanitary nightmare. Contaminated blood from the meats could make us all sick from food poisoning. We deal with this by ensuring that all meats are in sealed plastic containers on a different (lower) shelf to uncovered vegetables.

Also my wife and I are no longer fond of any lingering taste or smell of animal products in our cooking - and sometimes even the best cleaning doesn't get rid of these from cookware. We therefore bought my father his own pots and pans to use and allocated separate kitchen cupboards for him, something that has worked very well.

Our test environments, data and harnesses must similarly be protected and looked after. They should be easily reverted to a pre-test execution state. Although not perfect this helps to provide assurance that the defects we find are product defects and not caused by some external application, unwittingly corrupted data set or misconfiguration of settings. This also allows us to quickly remove test data and final states from an earlier test and re-execute a regression or defect retesting suite quickly and efficiently. I would also recommend that these be maintained by the testing team.

Modern Continuous Integration tools such as Docker, VM tools such as VirtualBox and Vagrant and automated database restores and migration serve to make the above simple and efficient. There are no excuses for having a sloppily maintained, uncontrolled test environment.

3) Have a Variety of Good Tools at Your Disposal. Know When, Where and How to Use Them.

Every decent cook, vegan or not, invests in a set of sharp, durable knives and good non-stick pans. I would also include a good wok. I'm a pretty mediocre cook at best and I still need them...

I do a lot of Asian stir-fry cooking. I like my carrots, capsicum and zucchini cut into strips or julienned and my herbs finely chopped up. My wife likes her eggplant finely cubed. To do this properly, especially with carrots and eggplant, we need sharp, durable knives and choppers of different sizes. These are expensive and need to be regularly sharpened. Similarly among meat eaters, every lover of steak swears by a good set of serrated steak knives.

Sliced bok choy, capsicum and basil with julienned carrot.

Similarly, we have a wide and disparate range of flavourings, herbs and condiments in our kitchen. These include apple cider and balsamic vinegars, kecap manis, soy sauce, miso, various powdered herbs such as basil, chinese five spice and oregano, dukkah and nanami togarashi. They provide a wonderful inclusion to the flavours in our cuisine, however used wrongly they can ruin a meal. I don't always instinctively know the best dishes to use them in (I have a predilection to sweet soy sauce, kecap manis, and use it in enough places for my wife to wonder if I am secretly pregnant...) but I do see it as a priority to learn.

A selection of our condiments

There are many tools available to the tester, from data creation through test planning execution, environment management and setup to automation. Everybody has an opinion of where and when to use them and the internet and books are full of guidance, however the decision of what, when and how to use a tool or tool suite depends entirely on the local context and must be decided by you. We do not have infinite time or budget or expertise available to us and the wrong tooling decisions uncorrected too late can ruin a project.

An example of wild claims and abuse of tools is test automation - far too often seen as a panacea or substitute for the hard graft of testing. Get HP ALM or Selenium Webdriver in, automate everything, we don't need manual testers anymore! Similarly write enough unit tests get 100% unit test coverage and we will never have a quality issue ever again! How often have we heard such complete rubbish?? It is me and the kecap manis again!

So many tools are available at low cost (or free, open source). There is also lots of good information on the internet on examples of how and where to use them. Take advantage of it and get the right tools for your project.

4) We Are (Mostly) Friendly, Come and Meet Us!

When I became a vegan in 2012 I struggled greatly despite my previous experience as a lacto-ovo-vegetarian. Veganism is a much broader and different set of values, not just in terms of how to follow a plant-based diet but also avoiding wool and silk, leather shoes, household cleaning and men's grooming products that contained animal products - even my choice of beer when I am in the pub on social events. It is very hard to do on one's own.

Luckily there are great online and in-person vegan support groups out there. Most large cities contain at least one vegetarian / vegan meetup (here in Sydney, Australia both Vegan Australia NSW chapter and Kym Staton's Sydney Vegan Club run great events) and I find that in almost all cases people are willing to offer advice and relate their experiences to newcomers. Online there are great initiatives (paid and free) such as Animals Australia Unleashed, Colleen Patrick-Goudreau's 30 Day Vegan Challenge and any number of Facebook groups and forums where people can find information and talk with others. Sadly not every group is friendly to newcomers however there is enough support and friendship out there to every person to find a niche.

Similarly, new testers and those without the ability to connect and chat in their workplaces have opportunities. So many cities nowadays have a Testers Meetup via meetup.com, usually running free or low cost talks and events. As an example the Sydney Testers Meetup, for which I am the social media officer has over 2000 members and almost all of its events are free and after work hours. Online Slack channels for testing such as http://testersio.slack.com and the Ministry of Testing Slack are busy with people discussing testing practice and offering tips. Most of the testing thought leaders are also active on Twitter and Linkedin and easy to find. Join up - we were all new once and happy to connect!

On the Joys of Sleep (Tester Edition)

Tonight in the part of Sydney, Australia I live in it is 11pm at night and 26°C with 65% humidity. Last night I couldn't sleep well and tonight is likely to be worse. The aircon isn't making much difference and in any case is bloody noisy. Some random alarm just went off outside. Instead of visiting the land of Nod I am up and feel a bit rubbish. I am no stranger to insomnia at the best of times but this takes the biscuit. What a better time for an impromptu testing rant blog?

Of course performance in every job short of professional sleep deprivation test subject is diminished by sleep deprivation, however I find that the particular skills required to do testing properly are shattered by a sleep deprived brain. Doing manual and exploratory testing, I struggle to concentrate for any length of time at all without good sleep and my productivity nosedives. I find myself nodding off at meetings (same if I have a big lunch), or at the very least listless. My ability to communicate to other testers or management or choose the right words to say or write is diminished to the point where people start wondering if I have mysteriously forgotten how to speak English - not good if you are a consultant and expected to be sharp and knowledgeable all the time. Meetings and conferences are interrupted by my incessant yawning. I act like a moody brat. And so on...

Think of the things that we do on a daily basis that require us to be mentally on top of our game. We need to be focused and observant for long periods, creating in coming up with new test scenarios, precise in our plans, defect logging and status reports, analytical in our study of requirements, able to communicate precisely and tactfully with management, technical staff and  stakeholders in providing the data they need to make go no-go and project decisions.

And yet the demands of IT, the pressure of overloaded agile development cycles and crammed final testing periods in waterfall, the need to test production deployments out of hours, the delays we get due to development overruns and tough, large defect retest backlogs, non-negotiable release deadlines - in some stints I have had resulting in working many lates, one or two all-nighters in a row and giving up weekends - are such that we are often made to push ourselves to the limit without suitable rest. The effect and disruption this causes on our family lives and health and the stress this causes mean that we are even less likely to have a good night's sleep. The cycle perpetuates. While it is true that jobs need to be done and deadlines need to be met, the culture of expecting productivity from people who are heading for spectacular burnout and mental exhaustion does nothing for the success of our projects. Of course my insomnia tonight has nothing to do with the above, however I see it as motivation to write about a much wider problem in the industry.

Why is this so unusual, it happens in other fields? Project managers, developers, business analysts, CIOs also have it. That is true and I have every sympathy, however being the author of a testing blog I have the licence to write uniquely from a tester perspective. I do believe that the particular nature of testing as a role imposes demands that even a small amount of mental exhaustion make difficult to achieve.

It is hard to speak up about it. How many of us, often contract testers and consultants working on client sites are able to challenge the processes, workload and deadlines of management? We don't want to "let the side down" or appear weak to our bosses and colleagues, we want to be seen to be team players and professionals. Maybe we fear the next round of offshoring and redundancies - even the cancellation of our visas and deportation if tied to our employer. The stakes are high and to speak out takes guts and persuasion. Luckily I work for excellent managers on an easier project and am able to voice my opinion about timelines and resourcing, but my experience in IT has told me that these things are a privilege not widely shared and projects have a momentum of their own that can slide into stress and trouble if not managed well.

So what is required? More considerate industry practices? Better timelines and test estimation by test leads? Testers to be more assertive? More caring and proactive management? Better scoping? A testers union maybe? I don't have the answers but would love to hear your thoughts.

In the meantime, I will try to hit the sack. It's hot and I don't fancy my chances but we shall see. Goodnight ;-)

Dubious but Still Fairly Common Reasons to Implement Test Automation

1) The budget is low and the boss says we have to lower tester headcount by any means possible...

2) Our agile cycles are so loaded with work and release deadlines so short that we can't actually fit manual testing in the schedule...

3) I'd secretly love to be a developer and doing lots of test automation is the best way I can learn programming in office hours...

4) (somewhat linked to 3) I read somewhere that manual testing is dead and if I don't become a test automation guru overnight I'll be unemployable, sad and poor forever...

5) The CIO/Test Manager was recently seduced into spending $20k QTP/Test Complete by a salesperson. The company QA standard is now to automate everything in existence and get our money's worth out of the licence fee.

6) I never actually liked the whole manual and exploratory testing pahlaver and can now get super high test coverage without ever manually testing anything again!

7) Test Automation is the way of the future and will solve all problems, regardless of what they are!

I have probably missed some others...

Back to Blogging

Dear All,

After a gap of about 2 years and lots of Twitter posts I have decided to return to blogging. I will try to blog about my thoughts and experiences in testing and IT at least once a week. I hope you enjoy the future content! :-)

On Being an "Impostor"...

I have always taken an interest in testing as an activity ever since I was given the testing role in my last job some years ago. I was the first dedicated tester the company had, and had to learn it from scratch without help for training. I managed to achieve good things in the role and set up some effective if crude processes although I did make some horrendous mistakes. l have spent time meeting other testers in industry meetups, read the blogs, followed the online discussions that interested me etc. However the more I read, the more I have realised how much I know about this area of IT- still very little. There is nothing like speaking to the highly skilled testers and studying more in the field to install a sense of humility and the idea that I have so much more to learn.

That isn't in itself a problem except that it does get me down that I haven't done or learnt more as a full-time tester, or that I still find it hard to think of myself as a "good" test analyst. I find every day something new to add to the pile of things to improve and learn. I like the challenge and stimulus but I seem to never catch up !. Is there a point in this industry when you can finally call yourself good or even adequate? Is it after X years of working or crossing off experience in enough strategies such as exploratory testing, static analysis, ATDD etc? After a good employment review, a good rep by a testing expert or an invitation to speak at a conference? After the nth ISTQB advanced/expert cert, maybe?

l recently came across the idea of the "Imposter" Syndrome, a common condition affecting high achievers where they cannot come to terms with and internalise their achievements and skills - thus believing themselves frauds and undeserving of their success. The term was coined for the first time back in 1978 and it is thought that up to 70% of all people will suffer from it at some point.

Left unchecked, this can have a damaging effect on one's career - Carol Pinchevsky in her SmartBear.com article Impostor Syndrome in the Workplace—and a Few Ways to Overcome It describes them as follows -

"Charlie, who manages a team of developers, explained why he requested anonymity. “An implication that someone thinks of themself as incompetent or an impostor runs the risk causing other people to believe that as well. Needless to say, this can cause increased attention/lack of trust issues from bosses, increased friction with colleagues at the same level, and potentially morale issues from subordinates."

Noted Microsoft software engineer and blogger Scott Hanselman, in his blog post on the subject, describes his experiences affected by the Imposter Syndrome and states his mindset in overcoming it -

"The important thing is to recognize this: If you are reading this or any blog, writing a blog of your own, or working in IT, you are probably in the top 1% of the wealth in the world. It may not feel like it, but you are very fortunate and likely very skilled. There are a thousand reasons why you are where you are and your self-confidence and ability are just one factor. It's OK to feel like a phony sometimes. It's healthy if it's moves you forward."

While I don't imagine that a third party assessment of my skills and achievements would reveal me as a "high achieving" tester (whatever that means) I do find that I don't take as much pleasure in a job "well done", or a good review, or praise from a manager as I would like. Some months ago I went through a spell for a few months when I made sloppy errors and bad decisions in the workplace; where I felt that I could do nothing right. Instead of shrugging it off and thinking of ways of recovering quickly, I allowed myself to end up in a slump - for a short while under the impression that I had been "found out" as some awful tester and lead. For some weeks I went through the motions, fearful of making the next
 embarrassing error. Luckily, I was able to get out of the slump eventually with the help of one of my colleagues.

It is difficult, I find, to talk about issues of struggle or disappointment or failure (of confidence or otherwise) in the general company of other testers at conferences or testing events. Conversations at these events tend to be more about one's successes and advice than one's disappointments and concerns. It is equally hard discussing this with colleagues and managers, since being a test lead you are expected to be poised, confident and in control. There aren't many testing blog posts I come across where any sort of vulnerability is mentioned.

British journalist Oliver Burkeman mentions in his Guardian article on "imposterism" as he calls it that -

    "The only solution, many experts say, is for higher-ups to talk about their own insecurities much more. ("When people see those they respect struggling, or     admitting they didn't know everything when they started, it makes it easier to have realistic opinions of their own work," says the Ada Initiative, which     supports women in technology.)"

The inspiration for this blog came about a few years ago from a piece of advice from a notable testing expert who gave a talk in Sydney, and who out of respect shall remain nameless. After the talk I had a chat with him about his blog and expressed concern that if I wanted to write my own blog, I couldn't add much new to the body of testing knowledge. His advice, just writing about your thoughts and experience - the good and the bad - should be enough. Well here they are! Hopefully they may touch and relate with others who have felt the same in the past or feel the same now.

 Please let me know your thoughts...

 

 -------------------------------------------------------------------------------------------------------

 

I will be writing a follow up blog with my opinions and experiences about various approaches to combatting the Imposter Syndrome, bearing in mind that I do still have it to a degree. In the meantime, for further info on the Imposter Syndrome, the additional resources below could be useful -

Caltech Counselling Center - Imposter Syndrome

 

APA - Feel like a Fraud?

 

Harvard Business Review - Overcoming Imposter Syndrome

 

Dr. Valerie Young's Blog