Short answer
Because the fast requests are being served from cache and the slow ones are not. The cache is hiding the real problem rather than fixing it. The number that matters is how long an uncached page takes, and on a healthy site that should be under about a second.
The symptom everyone misreads
You load the site and it appears instantly. A customer says it took forever. You reload and it is fast again, so the report gets filed under “their internet”.
It usually is not. Most sites sit behind at least one cache. Once a page has been requested, it is stored and served from storage, which is fast regardless of how slow the site underneath is. The visitor who happens to be first to request a page — after a cache clear, after a plugin update, after a product changes — pays the real cost.
So the useful question is never “how fast is my homepage”. It is “how long does an uncached page take”, and almost nobody measures that.
How to measure the number that matters
Add a random query string to any URL on your site — something like ?x=12345. Most caches treat that as a page they have never seen, so you get the uncached response.
Load it and watch how long the server takes to send the first byte. Under a second is healthy. Three seconds is a problem. Anything above ten and something is genuinely broken rather than merely slow.
Do it a few times with different random values, and do it when you are not also crawling the site yourself — measuring during your own crawl gives you the number for a server under load, which is a different question.
What is usually underneath
A broken object cache. This is the one that catches people out, because it fails while reporting success. We found a store whose object cache backend could never resolve at all — every lookup timed out before falling back. It measured 281 times slower than serving straight from memory, and the plugin's own connection test reported it healthy. A cache that is slower than no cache is worse than no cache.
A database configuration nobody revisited. One site was being killed roughly every forty minutes because its buffer pool was configured at five times the RAM the server actually had. The server looked idle between crashes, which is exactly why nobody suspected the database.
A full opcode cache. When PHP's compiled-code cache fills, the site keeps working and just recompiles constantly. The tell is autoloader frames dominating your slow query log — those are not really slow queries, they are a symptom.
Scheduled tasks that stopped. One site had them dead for 23 days. Nothing alerted; the only visible evidence was a 492 MB sessions table that nobody was clearing.
A CDN pointed at the wrong continent. A free-plan CDN was serving an India-hosted site out of Singapore. Turning the proxy off made it four and a half times faster.
The order to check things in
Measure uncached response time first, so you have a number to improve against.
Then the server and database — PHP version, memory, opcode cache, buffer pool, slow queries. This is where most of the seconds actually are.
Then each cache layer, one at a time. Three layers hide each other's faults, so turning them all on at once tells you nothing.
Front-end work — images, fonts, render-blocking scripts — comes last. It is the most visible and usually the least of the problem.
What not to do
Do not install a second caching plugin. Two caches produce behaviour neither one documents.
Do not buy a bigger server before you know what is slow. We have watched a resource upgrade change nothing at all, because the constraint was configuration rather than capacity.
Do not trust a plugin's own health check. Several report success on backends that can never work.