Testing a software product?
Then qualify – not quantify – your findings.
(No matter who is demanding quantified data)
Here's why
🧵👇🏼
Say you were asked to summarize your tests of a feature, by producing an aggregate, like "32 tests passed, 1 failed", or "97% pass-rate".
But is 97% good or bad? 🤔
Doesn't the answer to that depend on WHAT works or fails?
Is a cosmetic issue failing, yet customers can still gain tremendous value from the product?
Example: You ordered food, the food is on the way, yet the map displays ugly colors 🎨 or has some street names cut off the screen 📺
That's very different than when the map renders perfectly, yet the restaurant sends your food… to the wrong address 🧭
Or your order shows pending, even though it was never submitted to the restaurant 🤷🏼♂️
Hungry end-users aren't happy users. They're not repeat customers.
I get it. It's easier to say "98% pass-rate is preferred to a 93% pass-rate".
But such a statement is pointless.
Easier or not, you get paid to solve hard problems. And so is your boss, and their boss, up to the CEO.
Produce information that informs business decisions.
Your title may even have the word QUAL-ity in it, not QUANT-ity.
Don't present quantity information as qualitative.
Assess quality.
Once your management experiences what that looks like, they'll never ask for meaningless aggregates again.
tl;dr
♦ quantification may be easier
♦ and often is handily automatable
♦ but qualification is more informative
♦ so qualify, not quantify, your test results
If you valued this thread:
• Share with your following so they benefit too
• Follow me for more. That's how I know to keep producing.
I examine models in software testing, management, and society, to discern truth from narrative, so as to build more effective human systems.