For a long time, we tracked latency using average. We know that average is not a great metric to track because it is sensitive to outliers. Instead many use percentiles.

Percentiles are dangerous and may lead to unnoticed regressions. Here’s an example:

You start your journey by looking at your current state using p99 and you want to stop regression and incentivize improvements. "We won't release a feature that adds latency."

You build a new feature and look at the latency impact. It didn't move. Ship it!

You add a new feature and look at the latency again. Latency is the same. Ship it!

Customer review starts coming in... "It's slow!" ... "It used to be fast."

So what happened? Let's take a different look at it. Instead of using p99, let's remove the fastest 1% and the slowest 1% to remove the outliers and then calculate the average. That's called trimmed mean.

It is >3x slower... oh no! Revert it. By using trimmed mean, we can represent the expected latency for 99% of the users. We went from 39ms expected latency to 131ms. That's not good. Meanwhile, the p99 remained unchanged.

Percentiles are understats. Percentiles only tell you how many people are bellow a threshold. Percentiles are terrible at representing where people are relative to that threshold. If your p99 is 1s, are people experiencing 0.99 seconds or are they experiencing 0.1s (or 100ms)?

Stop using percentiles to track your goals and start using trimmed mean.

Track your latency goal using average with this one little twist.

For a long time, we tracked latency using average. We know that average is not a great met

The all-in-one writing platform.

Write, publish everywhere, see what works, and become a better writer - all in one place.