Have you heard about "flaky tests" in your team?

"Just rerun the tests and keep trying until it's green", do you hear this in your team?

What are flaky tests?

It's a test that sometimes fails, if you retry it few times it passes eventually.

Every team deals with these flaky tests differently. The problem with the flaky tests is that it becomes difficult to understand if it's failing due to a bug or if its just a flaky test.

These are the 3 small mistakes:

Mistake #1: Not trying to find the problem

When you see a red build don't just focus on turning it green by rerunning the build. Try to analyse the problem. Try to understand if the problem is in code or is it failing due to a bug. Have enough observability in your test automation that can help in finding the problem or root cause.

👉 I would recommend this course "Introduction to Observability for Test Automation" on TAU by Abby Bangser (@a_bangser)

👉 Talk by João Proença (@jrosaproenca) "Observability meets flaky test"

Mistake #2: Adding sleep or waits

Don't just add waits or sleeps in the code and keep your fingers crossed to see a green build. It would only add more flakiness to the existing flaky tests and doesnt help.

👉 Talk by João Proença (@jrosaproenca) "Should we just...delete it?"

Mistake #3: Commenting out the tests

Don't just try to comment out the tests to make the tests pass and to have a green build. Investigate further to find more information around the failure. Document the flaky test to understand how often it fails.

What common mistakes have you seen when dealing with flaky tests? Do you have any resources that you'd like to share?

Share them in the comment below 👇