Every engineer who has operated a production system knows that moment when alerts start firing all at once. Response times climb. Database server CPU hits 100%. Requests pile up. Users report errors. The cause almost always follows the same pattern. A volume of traffic above what was expected hit a system that was not prepared for it, and the database became the point of collapse.
This pattern is common enough to have a name in distributed systems literature. It is called progressive degradation under load, and it happens when one component starts responding more slowly, the components that depend on it slow down as well, new requests arrive before previous ones complete, and the queue grows until the system starts rejecting work. Understanding why this happens, and why cache is the most frequent response to this kind of crisis, is the starting point of this series.
How a relational database behaves under pressure
A relational database like PostgreSQL or MySQL is a system designed to store data persistently and return it with consistency guarantees. To do so, it follows, for each query received, a sequence of steps ranging from SQL parsing and permission checking to data access in memory or on disk, filtering, and assembling the final result. Each of these steps carries a computational cost that, individually, is small. The problem appears when many queries arrive at the same time.
PostgreSQL maintains a buffer pool in RAM to store recently accessed data pages. When a query needs data already in that buffer, the response is fast. When data must be read from disk, the cost increases dramatically. Jeff Dean catalogued in presentations on large-scale systems at Google the orders of magnitude of latency between different types of access, numbers that have become a reference in distributed systems literature. A RAM read takes around 100 nanoseconds, while a sequential read from a magnetic disk takes around 20 milliseconds, a difference of 200,000 times. Even with modern SSDs, which operate in the tenths of a millisecond range for random access, the gap relative to RAM is still two to three orders of magnitude.
When request volume grows beyond what the buffer pool can absorb, the database starts going to disk more frequently. Each disk access increases query response time. Queries that take longer keep connections open longer. Connections that stay open longer exhaust the available connection pool. New requests start waiting for a free connection. Latency rises. And this cycle feeds itself.
graph TD
A[Request volume grows] --> B[Buffer pool cannot absorb]
B --> C[Database reads from disk more frequently]
C --> D[Queries take longer]
D --> E[Connections stay open longer]
E --> F[Connection pool exhausts]
F --> G[New requests wait for a connection]
G --> H[Latency rises]
H --> AThis phenomenon is called resource contention. It occurs when multiple processes or threads compete for the same limited resource at the same time. In the database context, the contested resource can be CPU time, space in the buffer pool, access to a specific data page, or connections available in the pool.
"As the number of concurrent transactions increases, the system spends a growing fraction of its time managing conflicts and a shrinking fraction doing useful work."
— Jim Gray and Andreas Reuter, Transaction Processing: Concepts and Techniques, Morgan Kaufmann, 1992
As the number of simultaneous connections grows, the coordination cost among them grows too, and the effective throughput of the system starts to fall even when the hardware still has capacity available.
Why the same queries run thousands of times
A frequent pattern in web systems is that most requests end up being for a small set of data. In e-commerce platforms, the most popular products receive the majority of accesses. In streaming services, trending content is requested by many users at the same time. In news applications, the most recent articles concentrate the reads.
This access distribution follows what systems literature calls a power law distribution. Vilfredo Pareto, a nineteenth-century Italian economist, observed that 80% of Italy's wealth belonged to 20% of the population. This regularity, called the Pareto distribution, proved consistent across such different domains that it came to function as a design heuristic in computing systems. Breslau et al. documented in 1999, in the paper Web Caching and Zipf-like Distributions: Evidence and Implications, that web access patterns follow Zipf-type distributions, where the probability of accessing the nth most popular item is inversely proportional to n (P(n) ∝ 1/n^s), with the exponent s ranging between 0.6 and 1.0 depending on the domain. In practice, this means the most popular item is accessed roughly twice as often as the second most popular, three times as often as the third, and so on.
The practical consequence of this distribution is that the database can receive the same query, with the same parameters, dozens or hundreds of times per second. Each execution follows the same path of SQL parsing, data retrieval, result assembly, and response delivery. The result is identical across all executions, but the cost is paid in full each time. That is the computational waste cache proposes to eliminate.
What cache is and why it works
Cache is the act of storing the result of a costly operation so it can be reused when the same request appears again. The idea is simple, but its effectiveness depends on an observable property of computing systems, locality of reference.
"A program during any time interval t uses only an active subset of its pages. We call this subset its working set."
— Peter Denning, The Working Set Model for Program Behavior, Communications of the ACM, 1968
Denning observed that programs tend to repeatedly access the same set of data within a given time interval, a phenomenon he called temporal locality. Additionally, accessing a piece of data tends to be followed by accessing adjacent data, a phenomenon called spatial locality. These two properties are what makes cache viable. If a piece of data was requested just now, there is a good probability it will be requested again shortly. Storing it in a faster-access location, therefore, carries high expected value.
In web systems practice, the most common cache for this type of problem is an in-memory storage server like Redis. Redis is a key-value store designed for very low latency access. While a relational database executes a query with the overhead of parsing, planning, disk access, and transaction management, Redis returns a value stored in memory in under 1 millisecond. The latency difference between the two is two to three orders of magnitude for cases where data is already in cache.
The most common usage pattern is cache-aside, also called lazy loading. In it, the application checks the cache first. If the data is there, the result is used directly, with no call to the database. This is called a cache hit. If the data is not in the cache, the application queries the database, stores the result in cache with a defined expiration time, and returns the result to the client. This is called a cache miss. From the second access to the same data onward, within the expiration period, the database is no longer involved.
sequenceDiagram
participant C as Client
participant A as Application
participant R as Redis
participant B as Database
C->>A: Request
A->>R: Check cache
alt Cache Hit
R-->>A: Return data
A-->>C: Response (< 1ms)
else Cache Miss
R-->>A: Data not found
A->>B: Query database
B-->>A: Return data
A->>R: Store in cache with TTL
A-->>C: Response
endThe component view shows where each piece lives and how Redis positions itself as an intermediary layer between the application and the database.
graph LR
subgraph clientes [Clients]
C1([" "]) & C2([" "]) & C3([" "])
end
A[Application]
subgraph camada_cache [Cache Layer]
R[(Redis)]
end
subgraph banco [Database]
BP[Buffer Pool]
PG[(PostgreSQL)]
BP <--> PG
end
C1 & C2 & C3 --> A
A -- "check cache" --> R
R -- "cache hit" --> A
A -- "cache miss" --> PGThe impact of this pattern in a degradation scenario is significant. If 80% of database requests are now served by the cache, the volume of queries reaching the database drops to 20% of the original. The database CPU recovers. The buffer pool stops suffering pressure. Connections are freed. Latency falls. The system returns to responding within acceptable parameters.
What cache solves and what it does not
This is the point where the conversation about cache needs to go beyond the incident, because the immediate recovery of the system can create a misleading impression that the problem was solved. Cache resolved the symptom, but the structural problem continues to exist underneath.
The structural problem, in this scenario, is that the system was designed without accounting for request volume growing beyond a certain range. When volume grew, the database became the bottleneck because the architecture placed all reads directly against it, with no absorption layer. The cache added in response to the crisis covers that bottleneck, but the architecture still has the same vulnerable point. If cache fails, if Redis becomes unavailable for any reason, all requests go straight back to the database, and the original problem reappears immediately.
By adding Redis as a cache layer, the architecture gained a new critical dependency. Redis became a component in the path of every request, and its availability became as important as the availability of the database. This is a point that tends to be underestimated. Redis needs to be monitored, maintained, updated, and have its capacity managed over time. As data volume grows, it needs more memory. As traffic grows, it may be necessary to add instances or configure replication. In cloud environments, this set of needs translates into a non-trivial line in the infrastructure budget, with a cost that grows in direct proportion to the memory allocated and the number of instances. Adding Redis quickly in response to an incident is understandable, but the cost of operating it properly needs to be planned and justified.
This set of consequences is what software architecture literature calls accidental complexity. Fred Brooks, in his essay No Silver Bullet: Essence and Accidents of Software Engineering, published in 1987 in IEEE Computer, distinguishes the essential complexity of a system, which arises from the problem it solves, from the accidental complexity, which arises from implementation decisions. Adding Redis solves a problem, but brings with it accidental complexity in the form of one more component to operate, one more dependency to manage, one more source of failure to monitor, and one more cost to justify.
"Accidental complexity is not inherent to the problem but arises from the solutions we build today."
— Fred Brooks, No Silver Bullet: Essence and Accidents of Software Engineering, IEEE Computer, 1987
What cache-driven recovery reveals about the underlying architecture
When cache resolves a degradation-under-load problem, it simultaneously reveals something about how the system was built. In general, systems that collapse under traffic spikes for lack of cache were designed with the implicit assumption that request volume would stay within a predictable range and that the database would be able to respond to all of them directly.
This assumption is not necessarily a design mistake. In new systems, with limited users and tight budgets, making direct database queries for everything is simple, correct, and sufficient. The problem appears when the system grows beyond the point where that simplicity remains viable, and no cache layer had been planned for that transition.
What the incident reveals is that the system crossed a scale threshold it was not prepared for. The architecture had a single pressure point, and any volume above that threshold saturated it. This finding opens a territory of decisions that goes beyond adding Redis.
There are distinct layers where the problem may reside. At the data layer, table modeling and the way relationships were structured determine the cost of each operation before any cache enters the picture. At the project architecture layer, the problem may lie in the code that accesses the database. N+1 queries, for example, arise when an operation that should result in a single database query becomes dozens or hundreds, consuming connections and CPU in a way that cache cannot address directly. At the solution layer, the system can be reorganized to separate read paths from write paths, or to distribute data across multiple instances instead of concentrating it in one.
The risk is in choosing the wrong response for the problem at hand. Treating a modeling problem with cache is postponing a bill that will grow. Reorganizing the entire architecture for a volume that does not yet exist is spending ahead of time. What guides this choice is understanding which layer the problem is at and how much change the current context can absorb, considering team capacity, budget, and growth rate.
Cache bought time. The work of building a system that scales with confidence still lies ahead.
What comes next
This chapter covered the problem of degradation under load and presented cache as the most common and most immediate intervention. But cache is a subject extensive enough to fill an entire book, because each aspect of it, when examined closely, reveals trade-offs and architectural decisions that matter.
In the next chapters, the path starts from the computational principles that explain why cache works, from the processor memory hierarchy to content delivery networks at global scale. Then, the architectural patterns that determine how cache integrates with the database, surrounding services, and end users. After that, the structural, operational, and cognitive costs that cache introduces, with special attention to the problem of invalidation, which is widely recognized as one of the hardest to solve well in distributed systems.
"There are only two hard things in computer science: cache invalidation and naming things."
— Phil Karlton, cited by Martin Fowler in TwoHardThings, martinfowler.com
