Sunday, 30 December 2018

On Being Called a Testing "Expert" or "Guru"

About a year ago someone contacted me on LinkedIn to thank me for a talk I did last year on test data  frameworks for automated UI tests. His words were "Thanks.. You're a Test Automation Guru!"

I was a bit stunned, although of course flattered and honoured, by this. The person who contacted me had about ten years of experience in UI test automation with what appeared to be a wide range of tools. I had between three and four years on and off, using a smaller motley of tools and frameworks from the rather redundant White Framework for .NET to Selenium (to good and bad results), along with being reasonably competent at knocking out code in C#, Java and Python. The difference between he and I is that I once had to solve some problem by using a very easy specimen test generation framework written by someone else off the internet and then got invited to speak at a few conferences on it. Much of my career (including my last few projects until changing jobs) has been in manual (or very limited automation) testing gigs. To me, he was the "guru"!

Not long after that I got a notification on Twitter that somebody had added me to a list of "Testing Thought Leaders". My first thought was "Have they confused me with someone else?"

In the last project before my current job I mentioned in the interview that I spoke at a few conferences and have a blog on testing. After I was appointed, in the first month or so I had at least three or four of my fellow testers telling me (in a quite reverential tone) that they heard I am a famous testing "guru" or "expert" and that they were desperate to learn from me. I downplayed it, saying that I had a blog and had done a few talks at conferences and meetups on testing - that was all. While I was lauded as particularly good at spotting and raising defects I don't believe that my performance in that place was vastly better than those of my colleagues who had never been to much less spoken at a testing conference, or blogged or tweeted obsessively about testing as I do.

The idea of being considered an "expert" or "guru" in tech or even testing is quite unsettling. I'm a long way from being considered a well-known and respected name in the testing industry - much less the next James Bach, Michael Bolton, Angie Jones or Rex Black. I'm not sure that even the really big names in our industry are comfortable to wear the title of "guru", such a loaded term as it is. Nobody has some irreproachable insight into achieving quality in all situations - there are no gods in this industry - and there are as many exceptions as there are rules.

Whilst I would imagine that being called an "expert" or "guru" provides benefits in having testers respect your opinions, getting gigs as a consultant or contractor, writing high-selling books or being invited as a keynote speaker to any number of conferences and seminars, it also comes with enormous expectation. In my last gig, while being labelled a "guru" by my colleagues I was petrified of making mistakes or stating an opinion that could be wrong or challenged. I also felt as if I needed to continually promote new ideas in meetings and be seen to be "cutting edge", regardless of the state of the testing processes, the resources available or whether people were capable of or interested in acting on those ideas.

The idea that some people may consider me as a "guru" or testing "thought leader" (as opposed to how I see myself as someone who just has ideas and likes to talk about testing topics, caveat emptor) has sometimes resulted in my censoring what I write about in this blog or post on Twitter - the idea being that if my thoughts and writings are not completely polished then I will be judged accordingly and should refrain from posting them.

I consider myself a good tester with an interest and active involvement in the field, however the field of testing and the fields it touches upon (development, devops, process management, reporting etc.) are so vast and change so rapidly that even in my ten years of working in the field I only know a small amount. I am about as much an expert as we are all - experts of the specific areas and projects that touched upon our own experience and interests at the times we did them - and our knowledge and our relevance becomes more redundant by the passing day  - a quite intimidating and humbling fact that I related to in my first "Tester Mid-Line Crisis" article. Self-study does help to counteract this and keep one relevant, however without the opportunity to put what you study into practice in a work context (a struggle I have had in some projects) it only gets one so far.

I also think if we put those who have had the privilege to speak at conferences and who write blogs one the pedestal of "testing experts" or "gurus" then we intimidate newcomers who want to do the same - many of whom are more than capable to adding great advice and perspective. James Bach, in his 2002 StickyMinds article "Becoming a Testing Expert" describes it so -

"What we're left with, a lot of times, are candidates who are overconfident in their ability to find good bugs. The same holds true for conference speakers. I've met dozens of interesting people at conferences who have helpful advice and interesting experiences. But when I encourage them to get up and speak or to write an article, most of them say, "Well, I'm not an expert. I don't know the right answers. I haven't read all the books." Valuable insight thus remains bottled up inside the self-skeptical minds of many excellent testers, while too many people who do speak and write could stand to put their ideas through a little more testing."

This is a shame as nobody is ever the polished article at the beginning (I am nowhere near it now) and testing as a profession and practice gains much from its diversity of background, experience and opinion. Saying that, what I have seen and quite liked in testing conferences is less of speakers relaying grand, abstract ideas in the style of university lecturers and more speakers talking about and critiquing the application of various testing practices in their environments, projects and teams - along with what they learned. In this vein, we are all experts in our part and have something others can gain from.

Monday, 24 December 2018

The Fear of Age and Irrelevancy - On the Tester's Midlife Crisis (1)

I have managed some notable achievements this year which I am proud of. I have spoken at three conferences (to mixed responses admittedly), been invited as a guest onto a couple of podcasts and got a new job with great colleagues that I am very happy with - only my third employer in testing in a ten year career. Thay was after seven years working for a really good testing consultancy on various projects.

However despite the achievements above I am just as uncertain about my career and progress as before. My father, who I was the main carer for was battling cancer for the past year, sadly passing away on the 3rd December. In order to look after and be there for him I stood down from various activities and roles I was proud of - notably my co-chair role in the Sydney Testers Meetup and my blog and test automation live stream. I also withdrew from my postgrad studies in CS for a semester, my second withdrawal in 18 months - delaying my graduation by a year.

My Worries Set In

I don't begrudge any of the above - one's family is invariably more important than one's job, extracurricular test-related activities and university studies - however I do feel angst about it. Angst that I am getting older and my technical skills in testing (learning Postman aside) haven't developed in the past year. Angst that everything is changing and my experience is no longer relevant - a redundant dinosaur. Angst that there are so many tools and techniques I haven't learned or had much commercial experience in - Devops, CI/CD, NoSQL, Protractor, mobile automation etc.

Angst that in a tech industry that values - even fetishises - youth and dynamism and where change can wipe out the benefits of experience in a heartbeat, I feel close to being seen as "past it" and "over the hill" as I imminently approach my 40s - to be replaced by younger hotshots on the cutting edge of testing.

Is this just Paul's paranoia getting to him? Maybe. However anecdotally, in chatting with Sydney Testers and conference attendees I have come across similar angst in others. Testers in their middle age concerned about being left on the scrapheap or not finding a job at all, or alternatively desperate to learn automation even if they don't like programming particularly. I have spoken to a few who have noticed their colleagues being younger, more technically skilled, clued up on the latest buzzwords and willing to put in long hours. Sydney, Australia where I live is about as brutal and competitive a tester job market as you can find with 300-400 job applications per advertised role a routine occurrence.

I have also come across people around my age and older who are greatly dissatisfied or apathetic with testing. They feel that they aren't getting anywhere in their careers or are tired of the constant learning to stay relevant. They feel that they are being poorly treated or paid much less than their developer colleagues even though they all work in the same teams. They hate the low status of testing compared to other areas of software development. They regret not choosing other employers or doing something else earlier.

A Tester's Mid-Life Crisis?

Within the past month, and especially with the inevitable introspection arising from my father's death,  I have begun to think of the above as a sort of tester's mid-life crisis - a career or professional identity crisis caused by one's approach to middle age. What are the features of a mid-life crisis?

According to Wikipedia, typical career related mid-life crisis symptoms may be -

  • a deep sense of remorse for goals not accomplished
  • a fear of humiliation among more successful colleagues
  • longing to achieve a feeling of youthfulness
  • ennui, confusion, resentment or anger due to their discontent with their marital, work, health, economic, or social status
  • ambitious to right the missteps they feel they have taken early in life

The term "Mid-Life Crisis" itself is contentious. Evidence that it exists at a grand scale at all or the ages when it starts is mixed. According to this article it happens anywhere between between the ages of about 37 and 50.

While it is anecdotal and not scientific there is some evidence that the particular nature of the tech industry makes things worse and harder. This article in KrAsia uses the stories of various high fliers in the Chinese tech industry where long hours and high pressure, ageism, rapid change arising from company acquisition and restructuring and resulting periods of unemployment forced them into a mid-life crisis while in their mid to late 30s. This article in CNBC, following on a from a post on Quora, reports that Silicon Valley tech staff even no more than 35-40 years of age can feel over the hill in the tech job market. This was deemed partly down to ageism, an interview process that favours young recent CS grads and being surrounded by younger talent. Encouragingly it did also list success stories of those who were in their 40s and above who were progressing in their careers in the big Silicon Valley tech firms.

A Twitter Chat

I posted on Twitter the following update -

I was surprised at the number of responses I had from testers far more established and famed in the industry than I who offered consolation or stated that I was not alone. Posts of being unhappy with being a tester or that the testing community was great but stifling.


This is the first of a sequence of articles detailing what I consider my "tester's mid-life crisis". Based on the feedback from this and resources existing to alleviate the kinds of angst and worry listed above I will write further articles in the coming weeks. I would like to know your opinions and your experience - if you can share them then I would be most grateful.

In the meantime I wish you a Merry Christmas and a Happy New Year!

Friday, 29 June 2018

On What the Famous John Barnes Rap Can Teach us about Software Engineering Practice

A Short and Tongue-In-Cheek Treatise on what the famous John Barnes Rap (from England's World Cup 1990 official song) can teach us about development and Software Engineering practice.

1) "You've got to hold and give but do it at the right time"

Knowing what and when to delegate is a critical skill of any dev or test lead. Be always aware of the demands placed on your staff and ensure that you delegate work they have the skills and time to finish and no more.

2) "You can be slow or fast but you must get to the line"

There are times when it is understandable to prioritise speed over some quality (i.e. for a new startup with limited resources pushing to be the first to market), however generally it is good to prioritise project quality and delivery over fast completion for the sake of it. Be wary of demands by management for unrealistic or deleterious deadlines. Getting to project completion with good quality is all important.

3) "They'll always hit you and hurt you, defend and attack."

It is true that implementing process change in software engineering may well cause friction in teams. You may face resistance, be it by fearful defensive colleagues or naysayer managers. Influencing skills are crucial.

4) "There's only one way to beat them, get round the back..."

To implement quality principles you may have to alternatively solicit support and resources from PMs, stakeholders and CIOs.

You could also implement a pilot or smaller solution yourself and demonstrate it.

5) "So catch me if you can cause I'm your England man"...

The English software engineering job market is competitive and favours those with quick thinking and the ability to hit the ground running.

6) "And what you're looking at is the master plan"

Overarching master development and test plans are discouraged in agile development but can be sometimes useful. If you do use them, share and explain them with your team and appropriate stakeholders.

7) "We ain't no hooligans. This ain't a football song"

Hooliganism and other antisocial behaviour is deeply unprofessional and should play no part in well functioning software teams. Also remember that loud football chants in quiet workplaces may not be appreciated by colleagues.

Monday, 25 June 2018

10 Minute Rant 2: Ten Skills that are Probably More Important for Testers than Tools and Test Automation

My second ten minute rant blog, designed to break out of a spate of blogger procrastination.

(With credit to Sprint 4: The Bloggers Club on the Club, Ministry of Test)

1) Analysis and Systems Thinking

2) The Ability to Communicate with Tech Colleagues, Support, Management, Business People and Stakeholders

3) Observation, Modelling and Note Taking

4) Curiosity and a Desire for Exploration

5) Tenacity

6) Empathy for the User

7) Business Domain Knowledge

8) Formulating Rational and Consistent Arguments

9) Persuasiveness and Influencing Skills

10) Common Sense

Each of these so-called "soft skills" are critical for testers to truly perform well (IMHO) and yet most generic testing or QA job ads I see these days gloss over the above (if they are even mentioned at all) and involve a checklist of test case management and defect reporting tools, programming and test automation technologies. People can learn tools and how to program - it is much harder to rectify deficiencies in most of the above.

I suspect that the reason for this is that employers and recruiters tend to think that programming and test automation can either overcome deficiencies in the above or that testers are fungible and all have some default level of skill in the above simply by virtue of the fact that they call themselves testers.

Sad really…

Saturday, 14 April 2018

Reflections on the Scripted Test Case

I have my reservations about the pre-scripted test case. I am more happy to use exploratory testing charters, automating the checking part, in my work. Being quite confident in coming up with test ideas on the fly, I find it far more interesting and personally find it results in more defects.
In my career I have only ever worked in one company that used exploratory in a major way (and it was I who implemented it as test lead!). In my experience, where manual approaches are taken the majority of the work is done using pre-scripted tests - based on the examination of requirements documents or user story in some tracking system like JIRA, approved in advance by a test manager or stakeholder and then executed sequentially. The test lead or project manager would monitor the execution of each test, produce lovely graphs and may even set rough targets of “X” tests completed per day.

In fact, according to this 2017 survey on behalf of Practitest, 58% of respondents still used scripted testing (the second highest testing methodology behind exploratory) and 56% created detailed scripted test cases - the latter having actually gone up slightly from 2016.

As a tester I have an uneasy and visceral relationship with this approach, although I have and still do work in projects as a tester with scripted tests. The idea of testing as crossing off a long list of manual checks is a very poor definition – especially in an agile context. When test execution metrics are applied and a daily test execution target set the incentive is not to enquire and investigate but to get through as many tests as possible. We are judged on our efficiency, if this is such a measure of it, and not on our ability to evaluate the product, think outside the book, come up with new real-time strategies – the tasks that should be the “bread and butter” of the testing role. It also relegates our conception and reporting of the product to a sum of minute behaviours taken separately from other. A product is fit for purpose when “enough” of its test pass. Test cases may be regarded as a type of document but as far as documentation goes they are usually pretty poor. If they weren’t then we would be including them in manuals and posting them in our site support section, however this is never done without substantial rewrite.

They also make great presumptions about the product before its release, especially when detailed steps are included. It takes an incredible amount of detailed documentation or a very precise oracle to come up with test case steps that are still relevant at the end of the development process. Done afterwards, the current behaviour of the product can bias their steps and expected outcomes, not to mention the perverse fact that to write the test case steps you have to “test” the product anyway.
Nevertheless, from the perspective of a busy test manager these can be seen as a dream, especially in reporting, communicating and negotiation with the PM, development team and key stakeholders. They provide easy (if somewhat naïve) metrics, almost instant evidence that the test team is “doing something”, and fast answers.

Test cases are a deliverable in themselves (and I have often seen them required as a deliverable to stakeholders before allowing the start of testing).

What are testers doing right now? Writing Test Cases! Are they testing what we want? Take a look for yourself, read the test cases! Is testing efficient? We completed an average of X tests per day! How serious is that defect? It breaks Y test cases, it must be serious! Why do you need more testing staff? Our "velocity" shows that we will only have 75% test case completion by the end of the test cycle, without the extra hands we can’t complete testing!

Test cases, in much the same way as widget creation in factories, act to de-skill the testing process. The real mental effort is involved in coming up with testing scenarios beforehand, however once the cases are done, depending on their detail they can be given to anyone, tester or not, to complete. In this point the tester becomes a pair of hands, an admin assistant with a checklist. Once we decide we want to, we can replace the tester with an automation script – think of all the money saved!

Also, despite its inherent inefficiencies, deskilling, ennui and lack of real depth and flexibility a test strategy of rote execution of a suite of manual scripted checks can be effective and I have worked in successful projects where this has been done. Much of this depends on how much the initial requirements do not change between their elucidation into test cases and the completion of test execution, how “high level” and open to exploration the test scenarios are and the amount of time and human resources available.

The above reasons are why scripted manual tests are so common in IT teams outside of tech companies and those relying on heavy automation, CI and fast regular releases. In fact, where I see a test approach consisting of mostly scripted manual checks, it is nearly always requested from top-down or mandated as part of an overall quality strategy. Otherwise they are enforced by the use of tools such as QTP which have very prescriptive views of testing.

Scripted testing methods are very easy to manage and report on. I do not believe that they are ever going away unless we can apply reporting of exploratory, bug bash and other testing methods that address all of the management communication points above. James Bach and Michael Bolton, in Session Based Test Management, have provided some answers to the reporting challenges above – thus showing that addressing these is not intractable.

Friday, 23 February 2018

On “Testing vs Checking” and other Oppositions - Dialectics and Deconstruction

Software testing appears to be riddled with contradictory narratives argued from opposing sides. We argue about what is "testing" vs "checking", "traditional" vs "modern" testing approaches, "scripted" vs "exploratory" and make arbitrary distinctions to argue about "automated" vs "manual" testing. In these cases, the opposition is not just based on semantics and meaning but also on value (i.e. that one is wholly “better” than the other). Despite efforts of testing practitioners to debate these topics online or at conferences it is arguable that the arguments on either side are not significantly more sophisticated than before and no more likely to be resolved by consensus. In the second of my blogs touching on postmodernism and testing, I look at the nature of oppositional debate and touch on why that could be.

I also, through concepts developed by the philosophers Hegel and Derrida, theorize that we can question the oppositions above when we see them in a text by a process of dialectic and  deconstruction to develop more sophisticated, contextual and progressive propositions.

On Dialectics, the study of Discourse

The reasoned discourse between two opposing arguments being debated is known as a dialectic, a philosophical term dating from the classical Greek period of Socrates and Plato. Among the earliest noted examples are the Socratic dialogues, where statements are clarified via questioning and their logical consequences studied until a contradiction arises that refutes these statements. Socrates' aim was to find truth, therefore he had no qualms about changing his arguments to further that aim. It is closely related to rhetoric (the public defending of beliefs and attacking opponents) but seen as a counterpart to it (notably by Aristotle). In the middle ages it was included within the teaching of Logic.

In modern times the dialectic was revived by the great German philosophers JG Fichte and later, GWF Hegel. Based on ideas by Immanuel Kant, Fichte described the dialectic as the triad Thesis-Antithesis-Synthesis (later misattributed to Hegel), with the definitions below -

• Thesis - Some Statement or Proposition

• Antithesis - A statement that negates the Thesis

• Synthesis - A process in which the Thesis and Antithesis are reconciled to form a new proposition.

In Fichte’s definition, the Synthesis could be used as a Thesis in a future dialectic. The continual synthesis of thesis and antithesis in a chain would constitute more sophisticated understanding and progress.

Credit to

Example 1: Hegel's Being and Nothing

Hegel's book "Logic" provides the following example regarding existence.

•Thesis - Existence must be posited as pure Being

•Antithesis - Pure Being, upon examination, is found to be indistinguishable from Nothing

•Synthesis - When it is realized that what is coming into being is, at the same time, also returning to nothing (in life, for example, one's living is also a dying), both Being and Nothing are united as Becoming.

Example 2: Marx’s Theory of History

Marx and Engels took the Hegelian dialectic from a context purely of ideas to a materialist context (that of the real world of production and economics) to denote how they saw change and progress would develop in societies. In Marx’s theory of history, the thesis and antithesis were different classes within society and the synthesis would be societal transformation resulting from inevitable conflict between these classes.

Image result for marxist dialectic
Credit to

Each of the above involves some overcoming of the negation (antithesis), resulting in keeping the useful parts of an idea to develop a more sophisticated proposition (which Hegel called Aufhebung or Sublation). Hegel stated that this "negation of the negation" would result in incorporating the opposing idea into itself. In doing so this was the foundation of the Hegelian dialectic, providing a counter-approach to the skeptical Platonic dialectical method of “reducto ad absurdum”, where following a contradiction the thesis would be rejected, leaving us with nothing. Hegel believed that contradictions were a necessary outcome of reasoned argument, thus only a synthetic approach would bring us to a complete truth.

Semiotics - A Dualist Notion of Words and Signs

Later in the 1800s, Ferdinand de Saussure, the so-called Father of Linguistics, proposed a structured interpretation of the use of words, signs and utterances. He called the word or phrase used the "Signifier", and the concept or object referred to by the Signifier the "Signified". His great contribution was to note that the Signifier did not necessarily have to have a connection to the Signified - this is completely arbitrary and the brain acts to combine the signifier word to the signified concept based on some customary or agreed relationship. In doing so, the brain synthesizes and provides meaning to the sign. This is a sort of application of the Hegelian Dialectic. The study of the relationships between and use of signs and words as described above is known as Semiotics. For the purposes of this essay, “word” and “sign” will be treated as synonymous.

How can a word and its meaning have no actual connection? Saussure made the case that words only have meaning relative to other words. Therefore the word "tree" only has meaning when distinguished from the words "bush" or "shrub" - only having use as part of an overarching structure of synonyms and antonyms. Also the relationship between Signifier and Signified changes over time, such that some words become dated as circumstances change. In this respect, Saussure was considered a pioneer in the field of "Structuralism".

Derrida - Spot the Différance

Jacques Derrida

In 1963 the great French Postmodern philosopher Jacques Derrida took de Saussure's ideas to the next level. He stated that signs (including words) can never in themselves denote what they mean. They can only do so by deferring to a potentially endless chain of other words (signifiers) that they differ from. Also our understanding of word meaning changes with each reading and over time with the changing definition of existing and new introduction of new words.

This results in what he called "espacement" or spacing, stating that this differentiation causes binary oppositions and hierarchies.

Derrida coined the term "différance" (sic), a deliberate misspelling to denote not just the difference between words but the concept of hierarchy and deferral (to defer in French is "differer"). It includes the fact that the word can only have meaning within the context of the text ("there is no outside text").
According to Derrida, this chain of signifiers will never get to a point of a complete meaning that transcends context and is understood at all times and by all parties (the so called "transcendental signified", used in philosophy as an ontological argument for the existence of God).

However, he pointed out that the opposition of words and signs with others express not just meaning but values.

"On the one hand, we must traverse a phase of overturning. To do justice to this necessity is to recognize that in a classical philosophical opposition we are not dealing with the peaceful coexistence of a vis-a-vis but rather with a violent hierarchy. One of the two terms governs the other (axiologically, logically etc.) or has the upper hand.

To deconstruct the opposition, first of all, is to overturn the hierarchy at a given moment, To overlook this phase of overturning is to forget the conflictual and subordinating structure of opposition."

Binary Opposition

Many word pairs in binary opposition have not just logical but also value oppositions depending on their use. Examples include "Christian" and "Pagan", "Left" and "Right" (philosophical/political sense), "Speech" and "Writing", "reality" and "fiction" and "reason" and "passion". Different authors and readers at different times will value one over the other. Depending on their understanding by the reader, their definition at the current time and their use in the text, the differences in meaning and value between the two are variable and changing. Derrida called these discrepancies "traces", thus stating that a word not only denotes its own meaning but, by the its partner in binary opposition, what it does not mean (which he referred to as the "absence of the present").

This "deconstruction" was the ongoing practice denoted by Derrida for all texts to be studied for conceptual and temporal oppositions and these broken down to develop new meaning. It formed his life's work and is heavily influential in postmodern philosophy, law, linguistics, LGBT studies, psychoanalysis and literary theory. Note that the structures of word opposition remain and are required for any sort of communication, however the act of deconstruction allows for them to be broken apart and the gaps between the implied sense of each word in opposition to be analysed and understood within the context of the text.

It must be stated that according to Derrida, unlike Hegel's Dialectic, these oppositions (Thesis and Antithesis) are irreducibly complex and unstable. They could be understood and their différance noted and determined, however they could not be synthesised into a whole as Hegel proposed.

"The oppositions simply cannot be suspended once and for all. The hierarchy of dual oppositions always re-establishes itself. Deconstruction only points to the necessity of an unending analysis that can make explicit the decisions and arbitrary violence intrinsic to all texts."

The above ideas are controversial and have been disputed (particularly by John Serle and Alan Sokal) however they can be used as a model to approach the contradictions and arguments stated in my introduction.

Applications to Testing Discourse

By applying the concepts above we can make assertions about the oppositions so heavily debated in testing. The pairs "testing" vs "checking", "scripted" vs "exploratory" and "automated" vs "manual" testing, in the context of how they are usually debated, are treated as binary opposites. As stated in the introduction, arguments around them also imply not just logical or semantic opposition but value opposition (i.e. that one is "better" or more worthy than the other).

An Example - Testing vs Checking

Let us take a look at the first example, “testing” vs “checking”.

Personally I would define “testing” as a systematic exploration and investigation of the application, less algorithmic and more a constant thinking process. I would equally define checking as a more systematic procedure of experimentation and comparison of a scenario outcome with some expected result or test oracle, then moving on. However my definitions are meaningless without appeal to the implied meanings of the other words in my propositions above - and the words they are related to. I also admit that the simple words “testing” and “checking” are probably inadequate in explaining what I had in mind. In taking sides or commenting on a debate of “testing” vs “checking” it may be that I state or imply a hierarchy of value - i.e. that “testing” is better (or worse) than “checking”.

"Testing" defers meaning in its relation to other words such as "exploration", "analysis", "scrutiny", "thinking", "investigation" - which themselves can only be defined by relation to others. Equally the word "checking" also only has meaning in its relationship to other words such as "control", "find out", "learn", “comparison” - each deriving meaning by relation to other words. At some point the "web of language" (another term defined by Derrida) may overlap and both words may derive meaning from at least some other common words, however this does not necessarily mean that they are the same.

They are (contentiously) defined by what they are not. "Testing" is "not" "checking" (same as "Automated" is not "Manual" etc). Their meanings cannot be determined outside of the context of the text they were written in, the time they were written and the fickle views and understandings of the writer and reader. No transcendental meaning exists outside of this.

These represent a dialectic, and from a Hegelian perspective an effort should be made to synthesize them into a whole. However a deconstructionist perspective would dictate that these differences are necessary (at least until some new words are developed with more sophisticated meanings) and irreconcilable and contain an inherent violence of meaning and value. This may explain why after many years we are still having angry debates online and at conferences about these oppositions.

So what can be done? The way to address this is to deconstruct each use of the words in their context and time.

• For example, when we see the word "testing", does this mean a specific incident of testing or the general practice?

• Instead of testing, what other words can be used to describe what the practitioner is doing? Is the word "testing" adequate?

• Who is apparently doing the "testing" - one person, a machine, a group of people, a context driven tester?

• When was the text written and how was testing defined at that time?

• Can a single word apply to all investigative approaches in every company? What does the author think of testing and what other words are used to describe it? What about the reader?

• Are there instances where "testing" and "checking" could mean the same thing? What may these be?

• Regarding the "value" of testing vs checking, who says that one is better than the other? How is it, and under what circumstances can that hierarchy be overturned?

• What "testing" are they referring to, based on the questions above, and in what context? Has one always been better than the other?

What can those who engage in the debates do to clarify the points above to the reader? One way is to publicly state clear and sophisticated definitions of “testing” and “checking” as one sees them. James Bach and Michael Bolton, in their co-written 2013 article “Testing and Checking Refined”, go to great lengths to do just that - along with their definitions of related words that would fall within each one’s “web of language” (such as evaluation, exploration, learning etc.). Also included are narratives of the definitions along with various implications.

“Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc.

(A test is an instance of testing.)

Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

(A check is an instance of checking.)“

However, while this is better, from Derrida’s perspective the extensive web of language behind these definitions and their related words, in the specific text and time they were written in, will always prevent these definitions from being perfect and unchanging, the basis of a founding philosophy of software testing.

Michael Bolton, in a follow-up piece on his blog, also attempts to take the hierarchy of value out of the debate. He makes it clear that his (and James’) side of the debate is to reconcile checking in a more complex definition of testing that embodies but distinguishes between the human, reflective and thought driven tasks and algorithmic/mechanical and process driven tasks.

“I would like to emphasize our goals here. Our purpose is not to denigrate checking, nor to disparage the use of tools, nor to deplore those people who are asked to do human checking. On the contrary: we’re attempting to deepen our understanding of our craft; to show that checking is deeply embedded in testing; to emphasize that tools and the skilled use of them are essential to our work in many ways; to realize that humans will always inject human elements into the things they do; to realize the value of those human elements and the risks involved in asking humans to behave like machines. We must be clear on the differences between what humans do and how our processes and tools—media, as McLuhan would call them—do. Or more accurately, the differences between what we do and how our tools affect what we do.”

This removes the opposition entirely by making checking a subset of testing - arguably a successful Hegelian, synthetic approach.


The questions posed in the example of the binary opposition “testing” / “checking” can be applied to texts containing the other contentious testing oppositions mentioned.

The process above is necessary, if not to remove or synthesise oppositions out of the picture, to create propositions and assertions with more contextually accurate and sophisticated meanings. With the debates we are currently having, unless we deconstruct the concepts we are using in the texts they are written in we are arguing over shadows. Then can we make headway in some of the tense binary oppositions that have dogged the testing profession for many years.

Useful Links and References

James Bach, “We use tools”

Jermais Rossler, “Testing vs Checking, so what?”

James Bach (cowritten with Michael Bolton), “Testing and Checking Refined”

Michael Bolton “On Testing and Checking Refined”

Wednesday, 3 January 2018

Some of My 2018 Learning Goals, and Free Resources to Help in Yours

With us now in January 2018 and the Christmas pies and frivolities becoming a distant memory, it is traditional to see this time as that act of penance and discipline! We enrol for the local gym, take a month off the drinking and increasingly look to improve our skills.

My efforts to eliminate the Christmas gut being beyond the scope of a software testing blog, I wish to detail with you a few of my NY resolutions to develop my skills and achieve my various goals. In the hope that you may find them useful in your learning endeavours, I list various relevant free resources I have come across.

1) Practising Programming

I am studying a part time IT graduate diploma, with the option to articulate to a Masters degree if I choose to. This is already very programming-intensive (and I have used Python, SQL, C, Java, Bash, Prolog and Perl as part of my course), however my last module done in the first half of 2017 was programming-lite and I took the second semester of 2017 off entirely. Not working in an automated testing role at the moment and not having a lot of side projects on, my programming skills are (hopefully temporarily) rusty.

In order to hit the ground running when my course restarts in late Feb, and also to develop my coding generally, I am endeavouring to devote an hour per day to learning coding - for application development and test automation (more on that later).

Whilst there are more online resources for this then I could possibly list, in just about any language existent, I am particularly fond of the professional Massive Open Online Courses (MOOCs) at Pluralsight and Edx.

Pluralsight is my personal favourite of all the various online tutorial sites, and its videos and tutorials are generally of a very high quality. It is not free, however for those with an Office Online email account it is easy to obtain a free 3 month subscription to this via registering on Microsoft's Visual Studio Dev Essentials site.

Edx, a well known academic MOOC provider, contains many courses but I recommend a truly excellent introduction to programming - MIT 6.001x Introduction to Computer Science and Programming Using Python - which is self paced and can be audited (done without certification) for free. Completing it is how I learned and got a grounding in Python. It is not an easy course and the weekly programming challenges and quizzes are quite challenging and need to be completed on time. It is however extremely rewarding. The follow-on course for those who complete the above (and which I have not completed myself), MIT 6.002x Introduction to Computational Thinking and Data Science, uses Python to apply various concepts from Computer Science such as graph optimization and dynamic programming before moving onto statistical methods such as Random Walks and Monte Carlo. I plan to return to MIT 6.002x at some point this year.

2) Test Automation

With respect to test automation resources, I am keen to devote myself to learning API testing before returning to UI automation.

In view of this, since I already have some experience of using SoapUI I will try to learn Postman API. Danny Dainton provides some great tutorials and examples on his Github project, which I plan to work through.

3) Machine Learning

I took an AI module a few years ago and was fascinated by it. It was going to study a machine learning module as part of my degree however am not able to do it this year due to missing a certain prerequisite. I therefore wish to, and coming from a strong maths background should be able to study Machine Learning is my own time this year and see how proficient I can get.

I have been avidly watching and doing a few homework exercises from two courses by Nando de Freitas, a professor of Statistics and Machine Learning at the University of Oxford and the University of British Columbia. His lecture style is engaging and easy to follow and the courses are quite entertaining.

The UBC undergraduate machine learning course from 2012, for which all lectures are on youtube and exercises and projects are available, encompasses material from basic probability theory right up to Bayesian Classification and Deep Learning. With the exercises requiring a knowledge of Python, various tutorials are linked to and included.

As a follow on, de Freitas' Oxford course in Deep Learning goes much deeper into the theory, however whilst the videos are freely available the course notes and exercises are sadly hidden to non-students. It is still worth checking out however.


The above projects are what I plan to work on for the first half of this year, my academic studies permitting. If you have any ideas of other resources or projects that I could learn from, I would be most grateful.