A QA Interview We Never Used
About five years ago I was working on a QA interview process and got the following suggestion: “You’ve just arrived at the airport and have two hours before your flight to Hawaii. Make a list of everything that could go wrong between now and when the flight lands.”
No information about how to assess the candidate’s response was provided, and I had another concern. I remembered growing up in Maine and taking a vacation once a year: we’d rent a house on a lake for a week and drive to it in the family car. For my parents, this type of vacation was a huge luxury, something the generation before had never been able to do; I doubt flying to Hawaii, or anywhere else, ever really entered their minds. At the age of 23, I had flown a total of three times in my life, and always between New England and Arizona.
All this is to say that I was worried about cultural bias in the interview, a worry I admit I wouldn’t had thought of if it had simply been presented as, say, a road trip. But it seemed that frequent flyers, who could readily access detailed images of airport parking lots, terminals, and the planes themselves, would have a distinct advantage that was unrelated to the testing skills I wanted to measure, so I didn’t include it in my set of interviews.
But there was something else I liked about the interview. It got at what it’s like to be a tester. Looking for risk and points of failure. Questioning assumptions. Viewing every transition from moment to moment or state to state (house, car, pre-security, post-security, boarding, flying, deplaning…) with skepticism, wondering where things might come apart and replaying them over and over, looking for faults. And as one of those people who often reflexively lists things that might go wrong in regular life for no reason and when no one asked, perhaps I felt this interview honored a behavior that annoys others. So a few years later, on a flight to San Francisco, I made my own version, which I then promptly forgot about. A few years after that, the QA team was redesigning parts of our interview process, and I remembered it and shared it with my colleagues.
Testing mindset interview
Introduction
In this interview, you’ll be presented with a scenario and asked to offer reasons for how the information presented in the scenario could be possible. To avoid a list of very similar reasons, we ask that you stretch your imagination and try to consider as many different categories of reasons as possible. What does categories mean?
Dividing anything into categories can be arbitrary, of course. To give you an idea of how we’ve divided things into categories, consider the example scenario below (we don’t want to spoil answers by discussing the real scenario yet). The example scenario is similar to the one we’ll present as the real interview question.
Example Scenario
Suppose you were asked to make a list of every danger you might face while hiking through a forest. To start, you might say, I could get attacked by a wolf, or trampled by a deer, or struck from above by a hawk. While these experiences are undoubtedly vastly different, they all involve an animal attacking you, and we’d say they fit into a single category. You might also suggest, for example, that you could be attacked by an empty suit of armor; while somewhat more poetic, this still falls, in our opinion, under the category of “something attacks me”, and doesn’t require a new category to hold it. If you mentioned accidentally ingesting a poisonous mushroom, though, we wouldn’t try to argue that the mushroom belongs in the same class as the other attacking entities — this is a new category, perhaps something along the lines of “ingesting dangerous things”.
For this exercise, try to keep our rough ideas of categories in mind. We’ll ask you to provide examples for a specific situation, and we’ll be looking for how many different categories your examples cover. While it doesn’t hurt you to list many things from the same category, it doesn’t help, either. We encourage you to write down everything that comes to mind, but focus on trying to think of new ideas that aren’t closely related to what you’ve already written.
The Real Scenario
You enter an empty cabin. The entrance is to a kitchen, where a table holds three identical bowls of porridge. On the stovetop, a much larger pot of porridge sits over a low heat. As you brazenly devour the porridge, you notice that the contents of the bowls are not identical after all — the porridge in the first bowl is scalding hot, while the porridge in the second bowl is freezing cold, and the porridge in the third bowl is a pleasant temperature, somewhere between the two extremes of the other bowls.
What are some possible reasons that the porridges were not all the same temperature?
I liked my scenario because I felt it didn’t require too much knowledge from any particular domain. Deep knowledge of cooking, heating and cooling, or fairy tales wouldn’t, I guessed, confer much advantage. It also offered a certain level of whimsy that I felt would serve as an appropriate warning for anyone who might have to work with me.
But a concern remained. It was one of my two concerns from five years ago: how to assess the candidate’s response? Was there a fair and objective way to do it? I intended to go through the candidate’s list (“someone added ice to one bowl and hot water to another”, “something is wrong with my nerve endings”) and check off every distinct category they’d identified (“interference from outside actor”, “error in measurement”), but even if I created a near-perfect list of categories (I felt close), could I really communicate what size of categories I had created to the candidate without giving away answers? (Probably not). When I actually practiced the interview with a colleague, things deteriorated further, as many of her answers fell into a gray area as to whether or not they should count for a particular category.
Time ticks on, cooling our porridge and bringing deadlines rushing to meet us. I kept thinking about the problem, but an epiphany never came. With no solution to problem of a fair scoring system, I scrapped the interview and wrote this blog post instead.
Do you give (or have you taken) any kind of QA interview like this? Let me know on one of the social platforms below.
Alexander Roy QA Analyst | NoRedInk
Thanks to Kristine Horn for her help while I was still trying to salvage the interview, and to Michael Glass and Charlie Koster for reviewing this blog post.