Benchmarking Properly

file 10 · 2:22 · subtitles burned in

Chapters

1. 0:00Introduction
2. 0:09Write this down first
3. 0:24Sweeping concurrency
4. 0:41Where p95 crosses the target
5. 0:55The sweep
6. 1:09Three ways to get it wrong
7. 1:23Worked example: the table you produce
8. 1:43Mean versus p95
9. 2:01Recap

In this video Why one speed number misleads, and the table a field engineer hands a customer instead.

What it explains: traffic profiles, concurrency sweeps, percentiles

3 key points

1. 1

   Write down the traffic before you test anything.

   Four lines: tokens in, tokens out, requests per second, and the speed promise with its percentile (for example p95: the wait 95 out of 100 requests stay under). The video’s example: 1,024 tokens in and 256 out per request, tested with 200 requests at each crowd size.

2. 2

   Test at 1, 2, 4, 8, 16 and 32 people at once, and report the curve, not one number.

   In the video’s table, total output climbs from 24 to 264 tokens per second (11 times). Meanwhile the p95 wait for the first token (95 of 100 wait less) grows from 210 ms to 1,850 ms.

3. 3

   The answer is the biggest crowd whose p95 wait stays under the promise. The average hides the problem.

   With a 1-second promise the video sizes at 16 people (720 ms). From 8 to 32 people the average wait only went from 290 to 410 ms, while the p95 went from 380 to 1,850 ms, almost 5 times. An average-only report would have called 32 people fine.

Used in the course

• Day 3 · Step 1 · Concurrency sweep

The title card in the video says “Video 10 of 18”: that is the file order. The course plays the videos in the order of its days.

Download mp4 (8.9 MB)