Prefix Caching

file 11 · 2:37 · subtitles burned in

Chapters

1. 0:00Introduction
2. 0:13Prompts as a tree
3. 0:30Reusing the shared prefix
4. 0:49What improves
5. 1:06Turn it on
6. 1:20Caveats
7. 1:34Worked example: 20-turn agent session
8. 1:54The same session, cached
9. 2:15Recap

In this video Why agent traffic is mostly repeats, how servers reuse the repeated part, and what that does to the wait and the bill.

What it explains: shared prefixes, RadixAttention, prompt caching

3 key points

1. 1

   Most of what an agent sends is the same opening again.

   In the video’s 20-call example, 66,500 of the 72,000 tokens sent repeat the opening: 92%. The video shows 93% because it rounds 66,500 up to 67,000.

2. 2

   Reusing the opening cuts both the wait and the bill.

   In the video’s session, 95 of 100 calls see their first word within 0.35 seconds instead of 1.9. The input costs about a fifth of a cent instead of 2 cents.

3. 3

   Put the parts that never change first, or there is nothing to reuse.

   One changing line at the top, such as the time, makes every call new: calls 2 to 12 took 712 ms instead of 36 ms in the lesson’s sample.

Used in the course

• Day 5 · Step 1 · Prefix caching

The title card in the video says “Video 11 of 18”: that is the file order. The course plays the videos in the order of its days.

Download mp4 (9.6 MB)