During development, everything works. The query returns in milliseconds against a database with 200 records. The endpoint responds in acceptable time with a single user. The import batch finishes quickly on the local environment, with an SSD and no concurrency. The code passes its tests, the PR is approved, the deploy happens.
Then production volume starts to grow. An N+1 query that produced no signal in tests starts generating hundreds of database calls per request. A connection pool that was never configured hits the default limit and starts queueing. A timeout that was never defined causes an HTTP call to an external service to hold the thread for 30 seconds before failing. A JSON serialization that cost microseconds starts costing milliseconds when the payload grows from 2KB to 2MB. The system has not changed. What changed is the scale at which it operates, and with it, the weight of every IO operation the code executes.
The diagnosis that surfaces in those moments tends to be vague. "The database is slow." "The network is unstable." "We need more machines." And in war rooms under pressure, adding machines often works. Latency drops, alerts stop, the incident is closed. But adding capacity without understanding the cause treats the symptom. The N+1 keeps generating hundreds of queries. The pool remains misconfigured. The timeout remains absent. The problem was diluted across more hardware without being solved. When volume grows again, the same pain returns, now more expensive. The investigation that should have gone deeper to find the root cause, which can be something as specific as IOPS saturation on a shared disk volume or a garbage collector pausing at the worst possible moment, stops the moment the graph returns to normal.
The cost of that cycle is that it teaches reaction without teaching understanding. Knowing that IO is expensive is useful. Knowing why IO is expensive, when it is expensive, and under what circumstances it stops being the primary bottleneck is what separates a developer who fears complexity from an engineer who commands it.
The first attempt to solve the problem. Stop waiting
That disparity between what code does and what the system can deliver is not new. Since the earliest computing systems, processor speed outpaced input and output devices by orders of magnitude. In 1969, IBM introduced into OS/360 the concept of spooling, a technique that allowed the operating system to queue print and card-reader jobs to disk while the CPU continued processing other tasks, rather than sitting idle waiting for the printer or card reader to finish. If the CPU sat idle during a disk read, why not give it something else to do while the operation completed? That question, which sounds trivial when phrased in a single sentence, marks the starting point for decades of evolution in programming models.
Callbacks were one of the first materializations of that idea in high-level languages. Instead of calling a function and waiting for the result, the programmer passed a reference to another function that would be invoked when the result was ready. Node.js popularized this model in the early 2010s, and anyone who lived through that era remembers the phenomenon known as callback hell. The idea itself was sound, but the ergonomics of the implementation did not keep pace with the complexity of real-world flows. Chaining ten asynchronous operations with nested callbacks produced code that no human could maintain with confidence.
Asynchronous programming evolved. Futures and Promises brought an abstraction that could represent a value that did not yet exist but would exist at some point in the future. The idea came from theoretical computer science. In 1977, Henry Baker and Carl Hewitt published "The Incremental Garbage Collection of Processes", where they described the concept of a future as a placeholder for a pending computational result. When languages like Java (with CompletableFuture), JavaScript (with Promises and async/await), and Python (with asyncio) adopted variations of that idea, the mental model shifted. The programmer no longer needed to think about "when will this callback be called", but instead "this value is not ready yet, and I can keep doing other things until it is".
Reactive programming took that philosophy further, treating entire sequences of events as observable streams. The Reactive Manifesto, published in 2014 by Jonas Bonér, Dave Farley, Roland Kuhn, and Martin Thompson, articulated the principles of responsive, resilient, elastic, and message-driven systems. The core motivation was handling IO in a non-blocking way at scale, particularly in distributed systems where every component depended on network calls to function.
Coroutines, popularized by languages like Kotlin, Go (with goroutines), and Python (with async/await), brought even better ergonomics. A coroutine lets the programmer write code that looks sequential but internally suspends its execution when it encounters an IO operation, freeing the thread to run another coroutine. Melvin Conway introduced the concept of coroutines in 1963, and Donald Knuth formalized it in 1968 in "The Art of Computer Programming", but it took nearly half a century for mainstream languages to adopt them in practical form.
The motivation behind that entire evolution was never to make I/O faster. A disk does not read faster because the code is asynchronous, nor does the network transmit packets with more speed because a Promise sits in the middle. What asynchronous programming does is free the CPU to execute other tasks while the hardware completes the data transfer. Ignoring that distinction leads to decisions that only appear technical, such as adopting a reactive framework, rewriting code with async/await, or switching languages. If the bottleneck was never the CPU waiting idle, none of those changes solves the problem.
The question almost nobody asks. What is IO?
After years of dealing with callbacks, Promises, async/await, and all the tools the industry built to handle I/O, it is worth revisiting a basic question: what exactly is I/O?
The immediate answer is almost reflexive. IO is reading and writing, Input and Output, but input of what? Output to where? When someone says an operation is "IO-bound", what exactly is binding the execution time? When a profiler marks a function as spending 80% of its time in IO, what is happening during those 80%? Is the processor computing something? Is it waiting? Waiting for what?
The difficulty in answering those questions precisely exposes a limitation in how software engineering is learned. Tools for handling I/O are learned before understanding its nature. It is like learning to use an umbrella without understanding what rain is. It works until the day the problem requires a roof, a drainage system, or a reservoir.
What is seen when measuring IO depends entirely on where one looks and which tools are used. Is an await in the code IO? Is the time between sending a packet and receiving the response IO? Is copying data from a kernel buffer into the process's memory space IO? For all of those questions, the answer is "it depends", and that ambiguity bothers any engineer who seeks precision. I/O appears as a chain of events that crosses layers of software and hardware, and each layer defines in its own way what counts as "input" and "output".
IO is not just networking. Breaking the simplified definition
The most common association engineers make with IO is networking. HTTP calls, queries to a remote database, communication with an external service. That association makes sense, as those are the operations with the most visible latency in code, but limiting IO to networking narrows the definition Tanenbaum presents in "Modern Operating Systems".
When a program reads a file from the local disk, that operation is IO. The processor issues an instruction to the disk controller, which must locate the correct blocks, transfer the data into the controller's buffer, then into main memory, and only then can the program access the data. Even with modern SSDs, which eliminated the mechanical latency of hard drives, that chain of operations involves transfers between distinct physical components, each with its own speed and queue.
Inter-process calls on the same operating system are also IO. When two processes communicate via pipes, Unix sockets, or shared memory, there is a data transfer that crosses kernel boundaries. The operating system must intermediate that communication, copy data between separate address spaces, and signal the receiving process. All of that is IO, even if no network packet was sent and no cable was involved.
Even operations that appear purely computational can involve IO in non-obvious ways. When a program allocates memory beyond what the available physical RAM supports, the operating system resorts to virtual memory, moving pages between RAM and disk (swap). A simple pointer dereference that should cost nanoseconds may cost milliseconds if the referenced page is on disk. The program does not know that happened. From the code's perspective, it was a memory access. From the hardware's perspective, it was disk IO.
Andrew Tanenbaum, in the classic "Modern Operating Systems", defines IO as any data transfer between the CPU and the world external to it. That definition is broad enough to capture all the cases above and precise enough to exclude purely computational operations such as arithmetic or register manipulation. The CPU is an island, and everything that enters and exits that island is IO. What happens inside it is computation.
When exactly does an IO call begin?
To see what truly happens in IO, it helps to follow an operation from the CPU's perspective. Imagine a program reading a block of data from a file. At the highest level, the code calls something like read() or fread(). That call looks atomic to the programmer, but between the moment the function is invoked and the moment the data is available, a sequence of events crosses multiple layers of the system.
The first step is the transition from user space to kernel space. That transition, called a system call (syscall), is the moment the program asks the operating system to do something it does not have permission to do on its own. Accessing hardware is one of those things. No user program talks directly to the disk controller or the network card; it asks the kernel, the kernel intermediates, and that transition has a cost. The CPU registers must be saved, the execution context switches, and the kernel code takes control. That cost is small in absolute terms, on the order of hundreds of nanoseconds, but it is the first sign that something other than pure computation is happening.
Once in the kernel, the operating system checks whether the requested data is already available in some cache. The Linux page cache, for example, keeps in memory pages from recently accessed files. If the data is in the cache, the operation can return almost immediately, without ever touching the disk. In that case, there was a syscall and a context switch, but the disk IO itself did not happen. That is an important subtlety. The call looked like IO, the code treated it as IO, but the storage hardware was never involved.
sequenceDiagram
participant App as Application
participant Kernel as Kernel (OS)
participant Cache as Page Cache
participant Disk as Disk / Network
App->>Kernel: syscall read()
Note over App,Kernel: Context switch ~100ns
Kernel->>Cache: Data in cache?
alt Cache hit
Cache-->>Kernel: Data available
Kernel-->>App: Immediate return
else Cache miss
Kernel->>Disk: Hardware request
Note over Kernel,Disk: IO starts here
Disk-->>Kernel: Interrupt + data via DMA
Kernel->>Kernel: Copy buffer kernel → user space
Kernel-->>App: Thread/coroutine resumed
endIf the data is not in the cache, the kernel issues a request to the block subsystem, which translates it into commands for the disk controller. From that moment, the CPU has nothing useful to do in that context. It issued the instruction and now must wait for the hardware to respond. That is the exact instant when IO begins. The CPU has stopped executing useful instructions for the program and is awaiting a response from a component external to it. The time between issuing the request and receiving the response is pure IO time, the time that no software optimization can eliminate, because it is determined by the physics of the hardware involved.
When does an IO call end?
The intuitive idea that IO ends "when the response arrives" is a simplification that hides significant steps. What happens when the disk finishes reading the requested data involves a chain of events the programmer generally does not see, but which consume real time.
The first event is a hardware interrupt. When the disk controller completes the data transfer into the DMA (Direct Memory Access) buffer, it signals the processor through an interrupt. The processor interrupts whatever it is doing, saves the current context, and executes the kernel's interrupt handler. That handler checks what the interrupt means, locates the associated pending IO request, and marks the operation as complete.
But the data is not yet where the program needs it. It sits in the kernel buffer, in a memory space the user program cannot access directly. The kernel must copy that data from the kernel buffer into the user-space buffer the program provided in the original call. That copy, depending on the size of the data, can be trivial or significant. In high-throughput scenarios, such as file servers or video streaming, copying data between buffers can itself become a bottleneck, which motivated the creation of mechanisms like sendfile() and splice() in Linux, which allow transferring data between file descriptors without copying to user space.
After the copy, the thread or coroutine that was blocked or suspended must be woken up. In a blocking model, the thread was sleeping and must be moved back into the ready-to-run queue (run queue). In a model based on an event loop, such as epoll on Linux or kqueue on BSD, the file descriptor is marked as ready and the event loop notifies the associated callback or coroutine on the next iteration. In either case, there is an interval between the moment when the data is ready and the moment when the program code resumes executing. That interval depends on the operating system's scheduler, the machine's load, and the concurrency model in use.
The interval between "data is in the kernel buffer" and "program code has resumed executing" is time that simple profiling tools frequently do not capture, but in highly concurrent systems it can be the difference between a responsive application and one that suffers from tail latency.
Is waiting for a response IO? Or did IO already happen before?
Now that the mechanics between request and response at the hardware level are clear (syscall, interrupt, DMA, buffer copy, scheduler), it becomes possible to question what is actually being measured when looking at code. Confusion arises when the visible wait time in the code starts being interpreted as IO. When a program executes await fetch(url) and stalls for 200 milliseconds, it is natural to think those 200 milliseconds are "the IO", but that perception mixes two distinct phenomena.
The IO itself started before the await. The moment the runtime called the syscall to open the socket, resolve the DNS, establish the TCP connection, and send the HTTP request bytes, IO was already underway. The await is simply the point in the code where the programmer decided the result is needed to continue. It is the synchronization point, the moment the logical flow of the program cannot progress without the data, but the hardware was already working before that.
It is like checking the mailbox, where the act of looking inside does not represent the letter's transit. The transit already happened while other tasks were being done. Similarly, the await is the moment the program observes the result of IO that was already in progress in layers below the code.
That distinction has practical implications. If an engineer measures the time of an await and concludes that "IO is slow", they may be right or completely wrong. The time measured at the await includes the actual IO time (data transfer by the hardware), but also includes the time the coroutine spent in the queue waiting to be resumed, the time of the context switch, the time of buffer copies, and any runtime overhead. Optimizing "IO" in that case can mean completely different things depending on where the time is being consumed.
Rob Pike, one of Go's creators, summarized the idea plainly: "Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once." The await deals with concurrency. IO happens in parallel, in the hardware, independent of what the code is doing. Conflating the two leads to optimizations that attack the wrong point.
Can an IO call become CPU-bound?
The distinction between IO-bound and CPU-bound operations looks, at first glance, like a binary classification. Either the operation is waiting for external hardware, or it is consuming processor cycles. In practice, the same chain of operations can transition between those two categories throughout its execution.
Consider a common scenario. An application makes an HTTP request to an external API and receives a 2-megabyte JSON response. The network phase is clearly IO-bound. The time is determined by network latency, the physical distance between servers, and the link capacity. But when the response arrives and the program must parse that 2-megabyte JSON into a usable data structure, that phase is CPU-bound. The processor is iterating over each character, building the syntax tree, allocating memory for each node. There is no wait for external hardware; what limits the speed is the processor's computational capacity.
graph LR
A[HTTP Request] -->|IO-bound| B[2MB JSON Response]
B -->|CPU-bound| C[JSON Parsing]
C -->|CPU-bound| D[Data Structure]
style A fill:#4a90d9,color:#fff
style B fill:#4a90d9,color:#fff
style C fill:#e74c3c,color:#fff
style D fill:#e74c3c,color:#fffThat pattern repeats across many scenarios. Reading a CSV file followed by parsing and transformation, downloading an image followed by resizing or compression, receiving encrypted data followed by decryption. In each case, the first phase is IO and the second is CPU. Optimizing only the IO phase, using asynchronous calls and larger buffers, can make data arrive faster only to pile up in a queue awaiting CPU processing that cannot keep up with the rate.
Niklaus Wirth, creator of Pascal and Turing Award winner, published in 1995 a paper titled "A Plea for Lean Software", where he argued that "software is getting slower more rapidly than hardware becomes faster." That observation, made three decades ago, remains relevant. The tendency to add layers of abstraction, serialization, and transformation causes operations that were once dominated by IO to acquire significant CPU components. A microservice that receives a Protobuf message, deserializes it, validates it, transforms it, re-serializes it, and sends it to another service may spend more time in CPU than in IO, even though the developer intuitively classifies the entire operation as "communication between services".
The danger is in optimizing only one side. If the bottleneck is JSON deserialization and the engineer decides to add more instances to "handle the IO better", the new instances will have exactly the same CPU problem. The bottleneck shifts but does not disappear. And the infrastructure cost increases without proportional benefit.
The classic mistake. Measure CPU and conclude everything is fine
The average latency of a system tends to look acceptable even when the system has problems, and the reason is that averages compress the distribution. They sum all requests, fast and slow, and divide by the total. What stays hidden are the requests that take much longer than the majority, the so-called tail of the distribution. The 99th percentile of latency, abbreviated p99, is the value below which 99% of requests complete. It represents the worst case that is still statistically frequent, and it is where performance problems in production typically appear first.
Brendan Gregg, in "Systems Performance: Enterprise and the Cloud" (2013), documents variations of that pattern across multiple production environments. The following scenario synthesizes the recurring elements of those investigations. An engineering team responsible for a checkout API faced a recurring problem. During traffic peaks, checkout API response times degraded significantly. The p99 latency jumped from 300 milliseconds to 4 seconds. Alerts fired, customers abandoned carts, and the operations team entered war mode.
The main dashboard showed a reassuring metric: CPU at 25%. The immediate conclusion was that the system had plenty of capacity. If CPU was at 25%, the problem could not be computational, so the team started investigating other hypotheses. Perhaps it was the internal network, perhaps the load balancer, perhaps the inventory service that slowed under load.
The investigation consumed weeks, and each hypothesis led to a new monitoring tool, a new chart, a new meeting. Until a senior engineer decided to look at a metric nobody had examined carefully: the IO wait time for disk and network on the application instances. What he found explained everything. The instances were configured with general-purpose EBS (Elastic Block Store) volumes, which have an IOPS (input/output operations per second) limit that varies with volume size. The application's logging system wrote structured logs to disk synchronously before responding to each request. Under low load, this was imperceptible, but under high load, the disk writes competed for the available IOPS, creating a queue that no CPU- or memory-focused monitoring captured.
pie title CPU Time Distribution (illustrative scenario)
"Waiting for disk IO" : 75
"Processing requests" : 25The CPU was at 25% precisely because it spent 75% of the time waiting for disk IO. It was not "idle" in the sense of having no work. It could not do work because it was blocked waiting for the disk to accept more writes. The solution was to move logs to an asynchronous buffer and use a disk volume with provisioned IOPS for the local database. CPU climbed to 60% and latency dropped to normal levels. For the first time, the system was using its computational resources effectively.
The pattern this investigation documents has a precise name in the performance literature: the difference between utilization and saturation. Gregg formalized that distinction in 2012 in the USE Method (Utilization, Saturation, Errors), a diagnostic protocol for hardware resources. For each resource (CPU, disk, network, memory), the method proposes checking in sequence: utilization (the fraction of time the resource is active), saturation (whether the resource is being forced to queue work it cannot process immediately), and errors (whether failures are being reported by the hardware or driver). The 25% CPU indicated the processor was executing instructions for a quarter of the time, meaning low utilization. The disk had hit its IOPS limit and was queueing writes, indicating high saturation. A resource can have low utilization and be saturated at the same time. Without considering both dimensions during an investigation, the analysis stays fixed on the number that looks healthy while the queue grows without any metric pointing to it.
Eliyahu Goldratt, in "The Goal", formulated the principle that defines the Theory of Constraints: "any improvement made to a non-bottleneck is an illusion." That statement perfectly summarizes what happens when surface metrics obscure or fail to clarify what is happening. Measuring CPU without measuring IO is like measuring a patient's temperature without checking blood pressure. Everything can look normal on one metric and critical on another.
IO in the cloud. Why almost everything becomes IO
In a traditional data center with physical servers, most IO operations involved local disk and local network. The disk was physically connected to the server. The network connected servers within the same rack or building, latencies were predictable and relatively low. In that context, the ratio between CPU-bound and IO-bound operations tended to be more balanced.
The cloud changed that dynamic completely, and brought a historical irony. In 1969, spooling in OS/360 existed because the CPU sat idle waiting for the printer and the magnetic disk. More than half a century later, an EC2 instance sits idle waiting for EBS, RDS, and ElastiCache. The problem is structurally the same: the CPU has nothing to do while the external world does not respond. What changed is the scale and the volume of "external world" involved.
In an environment like AWS, Google Cloud, or Azure, almost everything that appears local is, in practice, remote. Block storage (EBS, Persistent Disk) is a network service. When an EC2 instance reads a block from its "disk", that read crosses the data center's internal network to reach the storage service managing that volume. The managed database (RDS, Cloud SQL, Aurora) is another network service. The cache (ElastiCache, Memorystore) is another network service. The queue system (SQS, Pub/Sub) is a network service. The authentication service, the configuration service, the secrets service. All of them are network calls.
Werner Vogels, CTO of Amazon, said in a re:Invent presentation that "everything fails, all the time." That statement, frequently cited in the context of resilience, has a less-discussed implication for performance. If everything is a network service, everything is subject to the latency, variability, and IO limitations of a network. An application that on a physical server made three IO operations per request (one disk read, one database query, one response to the client) may make ten or fifteen IO operations for the same request in the cloud, because each component that was once local is now remote.
graph LR
subgraph "Physical server: 3 IO operations"
A1[Application] -->|1. Local disk| D1[Disk]
A1 -->|2. Query| DB1[Local DB]
A1 -->|3. Response| C1[Client]
end
subgraph "Cloud: 10+ IO operations"
A2[Application] -->|network| EBS[EBS]
A2 -->|network| RDS[RDS]
A2 -->|network| CACHE[ElastiCache]
A2 -->|network| SQS[SQS]
A2 -->|network| AUTH[Auth Service]
A2 -->|network| CFG[Config Service]
A2 -->|network| C2[Client]
endMicroservices architecture amplifies that effect. A request that in a monolith was a local function call becomes, in a distributed architecture, an HTTP or gRPC call that crosses the network, passes through a load balancer, reaches another container, is processed, and the response travels the reverse path. If that microservice in turn calls two other services, and each of them queries the database, a single user request can generate a cascade of dozens of network IO operations.
Peter Deutsch, a researcher at Sun Microsystems, formulated in the 1990s the original fallacies of distributed computing, later expanded by James Gosling into the Eight Fallacies known today. The first is "the network is reliable". The second is "latency is zero". Those fallacies, written before cloud computing as we know it existed, describe the mistake many teams make when designing systems for the cloud. Assuming that a call between microservices has the same cost as a function call means disregarding that each of those calls is IO, with all the variability and overhead that implies.
Serverless, edge, and the IO you do not control
If in the traditional cloud the engineer at least chooses the instance type and disk type, the next layers of abstraction remove even those decisions. Serverless computing (Lambda, Cloud Functions, Azure Functions) took infrastructure abstraction to a level where the engineer does not even see the machine where the code runs. That abstraction has an IO cost that frequently surprises.
Cold start is the most well-known case. When a serverless function is invoked for the first time, or after a period of inactivity, the provider must allocate a container, load the runtime, initialize dependencies, and only then execute the code. That entire process is IO: container image transfer, package downloads, initialization of connections to dependent services. In functions that connect to databases, the cold start includes the TCP handshake and authentication, which can add hundreds of milliseconds to an operation the developer expected to complete in tens.
In traditional architectures, the application pays the cost of those initializations once and amortizes it across thousands of requests. In serverless, that cost can repeat at every traffic spike, at every scale-up, at every period of idleness followed by demand. Initialization IO, which in a long-running application is negligible, becomes the dominant latency factor in functions invoked sporadically.
Edge computing (CloudFront Functions, Cloudflare Workers, Deno Deploy) introduces another variable. Code executes close to the user, reducing network latency between client and server. That proximity makes a difference: a request that leaves São Paulo and hits an edge node in São Paulo instead of crossing the Atlantic to us-east-1 saves 150 to 200ms in RTT alone. But that gain exists only for the IO operation between the user and the edge. What happens next depends on the data.
When the code at the edge must query a database or an API that lives in a central region, the network IO between edge and origin happens regardless, now with one additional hop. The user's request goes to São Paulo, the edge code makes a call to us-east-1, waits for the response, and returns it to the user. The total network IO between the two endpoints may be greater than if the request had gone directly to the central server, because the latency between the edge node and the origin is not zero and it still pays the overhead of two connections instead of one.
When moving computation closer to the user does not include moving the data, the result can be the opposite of what was expected. Edge works when the code is self-sufficient with the data available locally: header transformation, routing, generating responses from a local cache, rendering HTML with data already at the edge. In those cases, the IO between client and edge dominates total latency, and moving code to the edge reduces that IO directly. When the code needs data that is elsewhere, total latency is determined by the IO between edge and origin, which the end user never sees on dashboards but feels in the experience.
A practical note: move computation close to the data, not close to the user. If the data cannot move to the edge, code at the edge will generate more IO, not less.
There is also the financial aspect. In the cloud, IO has direct cost. Data transfer between availability zones has a cost. Transfer between regions has a higher cost. Requests to managed services have a cost per operation or per volume. When an architecture multiplies IO operations without considering those costs, the bill grows disproportionately to the traffic.
In practice, the decision between serverless, edge, and traditional infrastructure comes down to the application's IO profile. Serverless works well when the function is lightweight, stateless, and invoked frequently enough to keep containers warm, such as a webhook that validates a payload and enqueues a message. The initialization IO amortizes across many invocations. It works poorly when the function needs persistent database connections or when the invocation pattern is sporadic. In those cases, the cold start dominates perceived latency and the initialization IO cost repeats at every traffic spike.
A team that understands where IO happens in its system can make architecture decisions that optimize not only latency, but operational cost. The first step is to understand the IO profile of the application itself, identifying whether time is dominated by waiting for external hardware or by CPU processing.
Thinking about CPU-bound vs IO-bound applications in practice
In theory, the distinction between CPU-bound and IO-bound applications looks like a simple classification. In practice, it is more fluid and depends on the execution context, the load profile, and even the time of day.
A REST API server that receives JSON requests, queries a database, and returns formatted responses is typically IO-bound. The largest portion of each request's time is spent waiting for the database to respond, the network to transmit the data, and the client to receive the response. The CPU is involved in parsing the JSON, building the query, and serializing the response, but those operations are orders of magnitude faster than the IO involved. Observing that server in production, CPU will be low (often below 20%), memory stable, and latency directly correlated with the database response time.
A video processing system that receives uploads, transcodes to multiple formats, and generates thumbnails is typically CPU-bound. The upload phase is IO, but transcoding is pure computation. Video compression algorithms like H.264 and H.265 involve discrete cosine transforms, motion estimation, quantization, and entropy coding. Each frame may require millions of arithmetic operations. Observing that system in production, CPU will be near 100%, with throughput limited exclusively by available computational capacity.
Between those two extremes lies a vast spectrum of applications that do not fit clearly into either category. A search system that indexes documents and answers queries is IO-bound during indexing (reading documents) and CPU-bound during ranking (scoring and sorting). A machine learning pipeline is IO-bound during data loading and CPU-bound (or GPU-bound) during training. The same application can be IO-bound during business hours, when request volume to the database is high, and CPU-bound overnight, when a processing batch consumes all available computational capacity.
Observed behavior in production is the most reliable indicator. When latency increases but CPU remains low, the system is likely IO-bound. When CPU reaches saturation and latency increases proportionally, the system is CPU-bound. When both metrics are moderate but latency is still high, the bottleneck may lie in lock contention, garbage collection, or limitations of the concurrency model in use.
Machine selection in the cloud. CPU or IO?
One of the most impactful and least discussed decisions in cloud systems architecture is the choice of instance type. Providers like AWS offer dozens of instance families, each optimized for a different load profile. Instances in the C family are optimized for compute, the R family for memory, the I family for storage, the M family for "general purpose". The most common temptation is to choose larger M family instances and expect the problem to resolve itself.
For CPU-bound applications, instances with faster processors and more cores make a direct difference. Doubling the number of vCPUs in a video transcoding system can reduce processing time by nearly half, assuming the software can parallelize the work. In that scenario, scaling vertically (a larger machine) is efficient up to the limit of the largest available instance type.
For IO-bound applications, the logic is completely different. Doubling the CPU of an instance that spends most of its time waiting for network IO will not reduce latency. The additional CPU will sit idle just as the previous one did. What helps an IO-bound application is increasing parallelism. Instead of one large instance waiting sequentially for IO responses, multiple smaller instances can issue IO requests in parallel, increasing the system's total throughput.
That difference explains why some teams spend fortunes on infrastructure without seeing performance improvement. If the bottleneck is network IO to a database already at its connection limit, adding CPUs will not help. If the database is the bottleneck, the solution may be adding read replicas, implementing caching, or redesigning the schema to reduce the number of queries. None of those solutions involves buying larger machines for the application.
John Ousterhout, Stanford professor and creator of Tcl and the RAMCloud storage system, argues in "A Philosophy of Software Design" that the greatest threat to software design is the complexity accumulated by decisions that, taken individually, appear correct. Scaling vertically is one tool and scaling horizontally is another. Each solves a different type of problem, and using the wrong one not only fails to solve the problem but adds unnecessary complexity and cost.
Vertical vs horizontal. The decision as a consequence of IO
The decision between scaling vertically (larger machines) and horizontally (more machines) is frequently treated as a high-level architectural choice, almost philosophical. "Microservices scale horizontally." "Databases scale vertically." Those statements appear in presentations and documentation as established truths, but they hide the reasoning that should support each specific decision.
A financial reporting application that aggregates millions of transactions in memory, calculates statistical indicators, and generates PDFs is intensely CPU-bound. That application benefits from vertical scaling. A machine with more cores and more memory allows processing to happen faster, with less context switching and without the complexity of distributing state across multiple nodes. Trying to scale that application horizontally would require partitioning the data, coordinating between nodes, and handling consistency and distributed aggregation. The added complexity rarely justifies the expense when the problem is strictly computational.
By contrast, a REST API serving a product catalog, where each request queries a database and returns JSON, is almost purely IO-bound. Scaling that API vertically (a larger machine) will not improve latency, because the time of each request is dominated by waiting for the database. But scaling horizontally (more instances of the same small machine) allows serving more simultaneous requests, because while one instance waits for IO, another can be processing a different request. What matters is IO parallelism.
graph TB
subgraph "CPU-bound → Scale vertical"
V1[1 large machine] -->|more cores, more RAM| V2[Faster processing]
end
subgraph "IO-bound → Scale horizontal"
H1[Instance 1] -->|waiting IO| DB[(Database)]
H2[Instance 2] -->|waiting IO| DB
H3[Instance 3] -->|waiting IO| DB
endProblems arise when the decision is made without that understanding. A team that scales horizontally a CPU-bound application without parallelizing the work will end up with multiple instances, each equally slow, with additional complexity from load balancing and coordination. A team that scales vertically an IO-bound application will have an expensive, underutilized machine, with the CPU waiting for IO exactly as the smaller machine did.
Fred Brooks, in "The Mythical Man-Month", observed that "adding people to a late project makes it later", and that analogy applies to infrastructure as well. Adding the wrong resources to a slow system may not only fail to solve the problem but, surprisingly, can make it slower, because it introduces coordination overhead in communication, increases the failure surface, and consumes budget that could have been invested in what would solve the problem.
Choosing the right resource requires knowing, first, where time is being spent. And knowing where time is being spent requires observing the system in production with the right tools.
Where to look. Making IO visible in your systems
The USE Method, introduced in the section about the saturated EBS case, works here as a formal diagnostic protocol. For each relevant resource, the sequence should start with errors, then saturation, and finally utilization. Starting with errors eliminates failures that could invalidate the other metrics. Checking saturation before acting on utilization avoids treating the wrong symptom: a resource with low utilization and high saturation, exactly what the disk showed in the earlier case, requires a different solution than a resource with high utilization and zero saturation. Without that explicit ordering, the investigation tends to be guided by what appears on the dashboard, rather than focusing on what is causing the problem.
To interpret what tools show within that framework, a reference for how much each type of operation costs is needed. Without that reference, a number like "2ms" says nothing; it can be fast for a network call or absurdly slow for a memory access.
The table below shows typical orders of magnitude. Exact values vary with hardware and context, but the proportions between them are stable and are what matter for architecture decisions:
| Operation | Typical latency | Reference |
|---|---|---|
| CPU register access | ~0.3 ns | Baseline |
| L1 Cache | ~1 ns | 3x register |
| L2 Cache | ~4 ns | 13x register |
| L3 Cache | ~12 ns | 40x register |
| RAM (main memory) | ~100 ns | 300x register |
| SSD (random 4KB read) | ~100 μs | 100,000 ns |
| SSD (sequential 1MB read) | ~1 ms | 1,000,000 ns |
| Network same AZ (round-trip) | ~0.5 ms | 500,000 ns |
| Network between AZs | ~1-2 ms | 1,000,000-2,000,000 ns |
| HDD (random read) | ~5-10 ms | 5,000,000-10,000,000 ns |
| Network cross-region | ~50-150 ms | 50,000,000-150,000,000 ns |
The difference between an L1 cache access and a cross-region network call is eight orders of magnitude. One hundred million times. When Tanenbaum's definition says IO is "any data transfer between the CPU and the external world", that table shows the cost of each centimeter of distance between the CPU and the "external world". That is why the cloud, by turning local disk into a network service, alters the IO profile of any application in an irreversible way.
At the operating system level, tools like iostat show the throughput and utilization of block devices. The %iowait field in vmstat or top indicates the fraction of time the CPU sits idle waiting for disk IO. That is exactly the scenario from the e-commerce case described earlier, with the CPU appearing free but actually blocked. High iowait is the most direct signal that the disk is the bottleneck. In practice, a vmstat 1 showing wa above 20% sustained is clear evidence the system is disk IO-bound:
$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 4 0 245612 18204 512048 0 0 8024 4512 1205 892 8 3 14 75 0
2 3 0 244580 18204 512048 0 0 7896 5124 1198 845 9 4 12 75 0The wa column at 75% shows exactly what was happening in the e-commerce case: the CPU spends three quarters of its time waiting for disk IO. The id (idle) column at 14% confirms that the CPU is not idle for lack of work; it is idle because it cannot advance without the disk.
To understand what a specific application is doing in terms of IO, strace (or dtruss on macOS) intercepts all syscalls and shows each read(), write(), connect(), sendto() with the time each consumed. It is the right tool to answer the question "where exactly is the time being spent". The cost is high, since syscall tracing slows the application, so it should be used for targeted diagnosis.
At the application level, distributed tracing (OpenTelemetry, Jaeger, Zipkin) shows the time decomposition of each request into spans. A span showing 400ms on a database call followed by 3ms of processing tells the complete story, with 99% of the time being IO. Without tracing, that information is lost in aggregated metrics that show averages that do not say much while the high percentiles tell a different story.
In cloud environments, provider metrics are the first line of investigation. In AWS, VolumeReadOps and VolumeWriteOps for EBS show whether provisioned IOPS are being consumed. NetworkIn and NetworkOut show network IO volume. DatabaseConnections in RDS shows whether the pool is near its limit. Each of those metrics answers a specific question about where IO is constraining the system.
The pattern to observe is always the same. When latency rises and computational resources are not saturated, some form of IO is dominating execution time. The question is which form, and those tools help answer it.
From diagnosis to action. Patterns that work
The industry has converged on a repertoire of patterns that address IO in complementary ways. None of them is universal, and choosing the wrong one can worsen the problem. Understanding the mechanism of each is what makes it possible to move from diagnosis to action.
Caching
Caching is the most direct pattern. If IO is expensive because the same data is fetched repeatedly, keeping a copy closer to the CPU eliminates subsequent trips. That is exactly what the kernel's page cache does with disk blocks, as covered in the section on when IO begins. At the application level, the logic is the same: a Redis layer between the application and the database turns a 5ms query into a 0.5ms network read. The risk is invalidation. Cache with stale data generates inconsistencies the code does not detect automatically. Caching works when the read pattern is predictable and tolerance for slightly outdated data is explicit.
For a deeper look at this topic, see our series of articles on caching optimization and challenges in distributed systems.
Batching
Batching reduces IO by grouping operations. Instead of ten individual database queries (ten network round-trips), a single query returning ten results (one round-trip). Instead of writing each log entry individually to disk, accumulating in a buffer and writing in bulk. That was exactly the pattern applied in the earlier scenario: the synchronous log became an asynchronous buffer. The motivation is the same that led to spooling in OS/360. If each individual operation pays the fixed setup cost (TCP handshake, context switch, buffer positioning), amortizing that cost across multiple operations reduces total overhead.
Prefetching
Prefetching anticipates IO before the program needs the data. The kernel does this automatically with readahead on sequential disk reads. At the application level, the pattern appears as preloading data that will be needed in future steps while the current step is still executing. A pipeline that knows it will process the next 100 records can start reading the next block while processing the current one, overlapping IO and CPU. It is the practical application of the distinction between await and IO itself: IO can begin before the point where the code needs the data.
IO parallelism
IO parallelism allows multiple operations to be in flight simultaneously. If a request must query three independent services, issuing all three calls in parallel and waiting for all of them (Promise.all, CompletableFuture.allOf, goroutines with WaitGroup) reduces the total IO time from sum to maximum. That is the difference between 3×100ms (sequential) and 1×100ms (parallel). This pattern is particularly relevant in the cloud, where each component is a network service and the number of IO operations per request has multiplied.
Choosing among these patterns depends on the diagnosis. Before applying any of them, three questions from what the instruments show are worth answering.
The first: is CPU high or low? High CPU with high latency indicates a CPU-bound system. The solution lies in optimizing computation (more efficient algorithm, less serialization, fewer copies) or scaling vertically. Low CPU with high latency indicates an IO-bound system, and the solution lies in the patterns above.
The second: is the IO repeated or unique? Data accessed repeatedly responds well to caching; data accessed only once needs batching or prefetching. Applying cache to data that never repeats wastes memory with no latency gain.
The third: is the IO sequential or parallelizable? Operations that depend on each other, where the result of A feeds the query of B, cannot be parallelized. Batching or caching are the options. Independent operations (querying inventory, calculating shipping, verifying credit) should be issued in parallel.
Those three questions do not require sophisticated tools to answer. A vmstat, a basic trace, and careful reading of the code are sufficient. Most teams have access to the tools. The mental model for knowing what to ask is what makes them useful.
Conclusion. IO as a mental model
Early in a career, IO is something to be avoided: a source of slowness, hard-to-reproduce bugs, incidents the code does not explain. That respect for the cost of IO is the foundation of any deeper understanding.
IO stops being an enemy when it becomes clear that it is an inherent characteristic of any system that interacts with the outside world. All useful software does IO, and a program that reads no data, communicates with no service, and writes no result is by definition useless. Understanding where IO happens, how much time it consumes, and how that time relates to total execution time becomes more important than trying to avoid it.
IO becomes a lens for seeing architecture. Looking at a system diagram and identifying where IO is dominant allows predicting where bottlenecks will appear, where latency will grow under load, and which components will scale well or poorly. Resolving production incidents contributes to that capacity, but without understanding the cause, experience accumulates recipes without building judgment. What builds that capacity is investigating with depth, combined with a repertoire that allows understanding causes and using that knowledge to predict problems before they arise.
Richard Feynman said that "the first principle is that you must not fool yourself, and you are the easiest person to fool." For software engineers, the most common form of self-deception in performance work is trusting metrics that confirm what is wanted to be believed. Low CPU appears to mean everything is fine. Acceptable average response time appears to mean customers are satisfied. But bottlenecks concentrate in the high percentiles, in the IO wait times that no standard dashboard shows, in the queues that form inside the operating system when disk or network saturate.
Understanding IO is understanding that the code written is only part of the story. Between the instruction the programmer writes and the result the user sees, there is a chain of data transfers, context switches, hardware interrupts, and buffer copies that determine, often more than the code's logic itself, the performance perceived by the end user.
In 1969, IBM engineers looked at a CPU wasting cycles waiting for a printer and created spooling. Today, engineers look at EC2 instances wasting cycles waiting for EBS, RDS, and dozens of network services, and apply caching, batching, prefetching, and IO parallelism. The abstractions changed, the languages changed, the scale changed, but the underlying problem is the same. The CPU is fast, the external world is slow, and performance engineering consists of deciding what to do with that interval. Mastering that understanding is what enables architecture, infrastructure, and design decisions that not only work, but work well under the conditions in which software operates in production.
