New computer systems, frameworks, and chips appear every year. Nobody can learn them all. But almost every one of them is built on the same small set of ideas. If you understand these ideas deeply, you can reason about any system, even one you have never seen before.

This post tries to follow first principles. That means two things. First, the base ideas must be independent: none of them can be explained by the others. Second, they must be complete: everything else can be explained from them.

The One Big Idea

Resources are limited, so every design choice spends one thing to save another.

This is called a trade-off. You cannot have everything. If you make one thing better, something else gets worse. Good designers do not try to escape trade-offs. They choose the best position for their situation.

This sentence is true, but it is too general to help you design anything. The rest of this post makes it concrete.

Four Root Causes

These are the reasons trade-offs exist. They come from physics, mathematics, and human nature. Nobody can remove them, so every system must work around them.

Root Cause 1: Resources Are Limited

Memory, time, energy, money, and capacity are all finite. A machine can only do so much work per second. A disk can only hold so much data. A budget can only buy so much hardware. So using more of one resource usually means saving or spending another.

This root explains why you must choose between memory and time, why copies cost money, and why sharing resources is attractive. It also explains why busy systems become slow: when capacity is limited, extra work must wait.

Root Cause 2: Distance Costs Time

Electrical signals and light travel at a limited speed. A big memory is physically big, so its far parts are far away. Sending a message across the world takes time, even at the speed of light. And big things cannot be close to everything at once. So small things can be fast, but big things must be slow.

This root explains why computers have many layers of memory, why every operation has a setup cost, and why machines in different places cannot agree instantly. When you see a sudden slowdown, ask: what got farther away?

Root Cause 3: The Future Is Uncertain

Systems must make decisions before they know what will happen. Will this data be needed again? Will two users change the same thing? Is this silent machine dead or just slow? Will the work stay the same next year? Machines crash and messages get lost without warning. Work arrives at random times.

This root explains why every cache, guess, and timeout is a bet. Some bets win, and some lose. It also explains why specialized tools are risky: they assume the future will look like the past.

Root Cause 4: Human Understanding Is Limited

People must design, build, and fix systems. Complexity causes bugs. A design that is perfect in theory but too complex to build correctly is often worse in practice than a simple design.

You could say this is just a special case of Root Cause 1, because human attention is a limited resource. That is true in a strict sense. But it is kept separate here because it explains a different kind of decision: why simple designs so often win.

Why Only Four?

You may have seen longer lists of root causes. Most extra items can be explained by these four.

For example, “agreement between machines is slow” comes from distance plus uncertainty. “Busy systems have long waiting lines” comes from limited capacity plus uncertain work. “Every operation has a fixed setup cost” comes from distance (a disk or network round trip) and from other trade-offs (like taking a lock). These are important results, but they are not roots.

Eleven Essential Trade-offs

Each trade-off below has five parts. Comes from shows which root causes create it. Also known as lists other common names, so you can recognize the idea in other places. Why it exists explains the cause. How it works gives a concrete example. What real systems do shows how different systems choose.

Trade-off 1: Memory vs. Time

Comes from: limited resources, uncertain future. Also known as: space vs. time, read vs. write optimization, indexing, precomputation, compression.

Why it exists. When you calculate an answer, you can save it for later, which costs memory. Or you can calculate it again each time, which costs time. The answer must live somewhere: stored in memory, or hidden in the steps that recreate it. There is an important special case. If the data changes, the saved answer must also be updated. So saving answers makes reads fast but makes writes slower.

How it works. Imagine you want to check if a word is in a list of one million words. With no extra memory, you must look at every word, which is slow. If you build a lookup structure called a hash table, you use more memory, but you find the word almost instantly. Compression works in the other direction. It saves memory, but you spend time to unpack the data.

For the read and write case, compare two ways to store data. A B-tree keeps data sorted all the time. Reads are fast, but each write must put data in the right place. An LSM tree adds new data quickly and sorts it later in the background. Writes are fast, but a read may have to look in several places. Adding a database index is the same choice: searches get faster, but every insert gets slower.

What real systems do. Databases build indexes on columns that people search often. Standard databases like PostgreSQL and MySQL use B-trees for a good balance. Systems with very many writes, like Cassandra, RocksDB, and logging systems, use LSM trees. Data warehouses are built almost only for reading, so they calculate summaries in advance. Small devices with very little memory usually recalculate. Big web services with cheap memory usually save everything. The simple rule is this: look at how often data is read and written, and put the work where it costs least.

Trade-off 2: Speed vs. Size (Locality)

Comes from: distance, limited resources. Also known as: the memory hierarchy, caching, locality.

Why it exists. Small memory can be fast, and big memory must be slow. Fast memory is also more expensive. So computers use many layers of memory, from tiny and fast to huge and slow. This only works if programs mostly use data from the fast layers.

How it works. Here are rough speeds. A CPU register takes about 0.3 nanoseconds. The smallest CPU cache takes about 1 nanosecond. Main memory (RAM) takes about 100 nanoseconds. A fast SSD takes 10 to 100 microseconds. A hard disk takes about 10 milliseconds. Sending data across a continent takes 70 to 150 milliseconds. Each step is much slower but much bigger.

A cache keeps a small, fast copy of data you used recently. This works because of locality. Programs usually reuse the same data soon (time locality), or use data that sits nearby (space locality). With good locality, most data comes from the fast layer, and the whole system feels fast. When a program jumps randomly around a huge amount of memory, almost every access is slow.

What real systems do. Every level of computing uses layers. CPUs have several caches. Operating systems use RAM as a cache for the disk. Databases keep their own memory caches. Content delivery networks keep copies of websites in cities near users, because distance on Earth is the biggest layer of all. Algorithms are also designed for layers. Database trees have many branches at each step, because each disk read is expensive, so you want to get as much information as possible from each read.

Important link. Caches create copies, and copies can disagree. So this is where the agreement problem (Trade-off 6) first appears. Even inside one CPU, the caches of different cores must stay in agreement.

Trade-off 3: Batching (Fast Response vs. Total Work)

Comes from: distance (which creates setup costs), limited resources. Also known as: latency vs. throughput, granularity, interrupts vs. polling, push vs. pull.

Why it exists. Almost every operation has a fixed setup cost that does not depend on how much work it does. A disk write, a network round trip, or taking a lock all have this cost. If you group work, you pay the setup cost less often, so the system does more total work per second. But each task must wait for its group. So you choose between fast response for each task and high total output.

This same idea appears in three forms. Batching means grouping tasks in time. Granularity means choosing the size of units, like locks, memory pages, or network packets. Big units have less overhead, but they waste resources and make others wait. Push vs. pull is about how you learn about new events: being told each time (push), or checking by yourself in groups (pull).

How it works. A database that saves to disk after every transaction might do 1,000 transactions per second, because each save takes about 1 millisecond. If it collects transactions for 5 milliseconds and saves them together (called group commit), it might do 50,000 per second. But each transaction waits up to 5 milliseconds longer.

For granularity, think about locks. A lock stops others from changing data while you use it. Locking a whole table is one cheap lock, but then only one transaction can use the table. Locking single rows lets many transactions work together, but you need millions of locks.

For push vs. pull, think about a network card. It can interrupt the CPU for every packet. This is fast and efficient at low traffic. But at 10 million packets per second, the CPU spends all its time handling interrupts. The other choice is polling: the CPU checks the card in a loop and handles packets in big groups. This is very efficient at high traffic, but it wastes a full CPU core when the network is quiet.

What real systems do. GPUs are built for total work, with thousands of slow workers. CPUs are built for fast response, with big caches and clever tricks. Big data systems care about total work. Trading systems and websites care about response time. Linux networking uses interrupts at low load and switches to polling at high load. Kafka lets you set how long to wait while collecting messages into a batch. Interactive programs like games turn batching off, because response time matters most. Microservices are the same choice at the architecture level: small services are flexible, but every conversation between them costs network time.

One note. In networks, latency and bandwidth are different things. You can buy more bandwidth by adding more cables. But latency has a limit set by the speed of light.

Trade-off 4: Busy vs. Predictable

Comes from: limited capacity, uncertain work arrival. Also known as: utilization vs. latency, queueing, tail latency, predictability vs. average performance.

Why it exists. Work does not arrive at a steady speed, and some tasks take longer than others. Because of this, waiting lines form even when the system is fast enough on average. Simple queue mathematics shows that waiting time grows very fast as a system gets busy. At 50% busy, a task takes on average about 2 times longer than the work itself. At 90% busy, about 10 times longer. At 99% busy, about 100 times longer. Free capacity is what absorbs surprises.

There is a second cause of unpredictability. Many tricks that make systems fast on average depend on normal patterns: caches, guessing, automatic memory cleanup, and background work. When the normal pattern fails, one task can suddenly take hundreds of times longer than usual.

How it works. A server at 95% busy looks efficient on a cost report. But when a burst of requests arrives, there is no free capacity, and the slowest responses become very slow. Highways work the same way. A few extra cars on a nearly full highway can cause a huge traffic jam.

Large systems make this worse. Imagine one user request goes to 100 servers, and each server has a 1% chance of being slow. Then about 63% of user requests will hit at least one slow server. The slowest responses are called the tail, and in big systems the tail controls what users feel. A Java service might answer in 2 milliseconds on average but sometimes pause for 200 milliseconds to clean memory (called garbage collection).

What real systems do. Web services often keep servers at 40% to 70% busy. Background jobs run near 100%, because nobody is waiting for one specific job. Some companies run both on the same machines: background jobs fill the free space and are paused when user traffic grows. Systems reject extra work when they are too busy (called load shedding). Big services sometimes send the same request to two servers and use the faster answer. Systems where a late answer is a wrong answer, like airplane controls and pacemakers, avoid caches and garbage collection completely, because they need predictable timing more than average speed.

Trade-off 5: Pay Now or Maybe Pay Later

Comes from: uncertain future. Also known as: eager vs. lazy, prefetching, binding time, compiled vs. interpreted, optimistic vs. pessimistic, speculation.

Why it exists. Many decisions have the same shape. You can pay a cost now, for sure, to avoid a possible cost later. Or you can skip the cost now and pay later, but only if it turns out to be needed. You cannot know the future, so every choice here is a bet.

This one idea appears in three common forms. Early vs. late work: do work in advance, or wait until the result is really needed. Early vs. late decisions: fix choices at build time so the system can optimize around them, or decide at run time to stay flexible. Pessimistic vs. optimistic: prevent conflicts before they happen, or let everyone work freely and fix conflicts if they happen.

This trade-off contains one of the deepest ideas in systems: work is rarely removed. It is usually only moved to a different time or place.

How it works. When a program creates a copy of itself, the operating system could copy all its memory right away. Or it could share the memory and copy each page only when one side changes it (called copy-on-write). Usually the new program replaces its memory soon anyway, so the late method saves a lot of work. This bet usually wins.

For decisions, compare programing languages. A compiled language with fixed types decides the meaning of each function call at build time, so the call becomes a simple jump. A dynamic language decides while running. This is more flexible, but every call needs extra checks.

For conflicts, imagine two people editing the same document. With pessimistic locking, the second person waits until the first finishes. This costs a little every time. With optimistic control, both edit, and the system checks a version number when saving. If someone else saved first, the second save fails and must be redone or merged. If conflicts are rare, optimism is faster because nobody waits. If conflicts are common, optimism wastes a lot of work on retries.

What real systems do. Operating systems are very lazy on purpose. They load memory only when used and delay writes to disk. Prefetching is the opposite bet: CPUs, disks, and browsers load data before you ask, because they guess you will need it. JIT compilers, like those in Java and JavaScript, try to get both: they start flexible, watch what the program really does, and then compile the busy parts into fast code. Traditional databases often use locks for busy data, while many modern databases keep several versions of data so readers never block writers. Git is optimistic: everyone works separately and merges later. CPUs are optimistic too: they guess which code will run next, start it early, and throw the work away if the guess was wrong.

How to choose. Ask two questions. How likely is the later cost? And how big is it? If it is likely or very expensive, pay now. If it is rare and cheap to fix, pay later.

Trade-off 6: Agreement vs. Speed

Comes from: distance, uncertain future (failures). Also known as: Amdahl’s Law, scalability, contention, centralized vs. decentralized, consistency vs. availability, CAP theorem, PACELC, eventual consistency, failure detection, split brain.

Why it exists. Whenever separate parts must agree, they must send messages, and messages take time. This is true for two threads on one chip and for two servers on different continents. The only difference is the distance, and the chance of failure. This is one of the most important trade-offs in computing, and it appears at every scale.

At small scale: more workers need more coordination. Splitting work among many workers adds capacity. But workers must talk to share data and combine results, and this talking happens one step at a time. Amdahl’s Law says that if 5% of the work cannot be split, you can never be more than 20 times faster, no matter how many workers you add. Often the cost of talking grows with every pair of workers, so at some point, adding workers makes the system slower.

In design: central vs. spread-out control. A central controller sees everything and decides simply and correctly. But it can become a bottleneck, and if it fails, everything stops. Spreading decisions across many machines removes these problems, but each machine acts with partial, possibly old information.

At large scale: consistency vs. availability. When data is copied across machines, sometimes the network breaks and splits them into groups that cannot talk (called a partition). Then each group must choose. It can refuse requests, which loses availability. Or it can answer with possibly old data, which loses consistency (all copies showing the same up-to-date data). This is the CAP theorem. Even when the network works, strong agreement needs messages back and forth, and messages are limited by the speed of light. This everyday version is called PACELC.

For failures: fast vs. correct detection. You notice a failure by silence, but you cannot tell the difference between a dead machine and a slow one. A short waiting time finds real failures fast but causes false alarms. A long waiting time avoids false alarms but leaves the system stuck when a machine really dies.

How it works. Imagine adding up one billion numbers with 64 threads. If every thread updates one shared total after each number, they all fight over the same piece of memory. This can be slower than using one thread. The fix is for each thread to keep its own total and combine them once at the end. Less agreement, more speed.

Now imagine a bank account copied in New York and London. These cities are about 5,500 km apart, and light in fiber travels about 200 km per millisecond. So one round trip takes at least 55 milliseconds. For strong consistency, both copies must agree before approving each withdrawal. If the connection breaks, one side must stop accepting withdrawals. The other choice is to let each side work alone and sync later. That is fast and always available, but someone could spend the same money twice.

For failure detection, imagine a 1-second waiting time. A healthy leader pauses for 2 seconds to clean memory. The group thinks it is dead and chooses a new leader. When the old leader wakes up, two machines may both think they are the leader (called split brain).

What real systems do. Fast systems avoid shared data that changes. They split data into separate pieces called shards, so each worker owns its own part. Many systems centralize the control plane, which makes decisions and changes slowly, and spread out the data plane, which carries the heavy traffic. Kubernetes works this way.

Systems like Spanner, etcd, ZooKeeper, and CockroachDB choose consistency. They use agreement methods called Paxos and Raft, and they accept higher latency. Systems like Dynamo, Cassandra, and DNS choose availability and accept temporary disagreement (called eventual consistency). Many systems let you choose for each request. Agreement systems use numbered “terms,” so a false alarm only causes delay, not damaged data. Inside a CPU, fully strict memory ordering between cores would be too slow, so modern CPUs use looser rules, and programmers add special instructions where order matters.

The right choice depends on how bad a mistake would be. A shopping cart that briefly shows a removed item is fine. Spending the same money twice is not.

Trade-off 7: Safety Through Copies vs. Cost and Speed

Comes from: uncertain future (failures), limited resources, distance. Also known as: durability vs. performance, redundancy, replication, write-ahead logging.

Why it exists. Parts fail without warning. Fast memory like RAM loses data when the power goes off. Permanent storage like SSDs, disks, or other machines is slow. Data is safe only after it reaches permanent storage, and confirming that takes time. Keeping several copies means the system survives as long as one copy survives. But every copy costs money, and copies must be kept in agreement, which brings in Trade-off 6.

How it works. A database that forces data onto disk after each transaction may wait milliseconds. Skipping this step can make it 100 times faster, but a power failure can lose recent transactions. Copies have a similar cost. Keeping a mirror copy on a second disk lowers the risk of losing data a lot, but storage costs double. A smarter method called erasure coding gives similar safety with about 1.5 times the space instead of 3 times, but rebuilding lost data is slower.

What real systems do. Databases give you settings to choose how safe each write should be. A common technique is the write-ahead log. The database first writes a small record of the change to the end of a log file, which is fast and safe, and updates the main data later. Cloud storage uses erasure coding for data that is rarely used and full copies for data that is used often. Airplanes use three computers and take a vote. The most important detail is correlated failure: copies in the same building, on the same power supply, or running the same software version can all fail together. So real systems place copies in different places. Caches can accept losing data. Bank records cannot.

Trade-off 8: General vs. Specialized

Comes from: uncertain future (assumptions may break), limited resources. Also known as: abstraction vs. performance, hardware acceleration, energy vs. performance.

Why it exists. A general tool must work for many cases, so it cannot use special facts about any one case. A specialized tool uses knowledge about the data, the usage pattern, or the hardware to do less work. But this knowledge is really a bet that the work will not change. When the assumptions break, the specialized tool breaks too. Abstraction layers hide details so systems are easier to understand. But hidden details are often exactly what you need to go faster.

Energy is closely connected. A general CPU spends a lot of energy figuring out what each instruction means and guessing what comes next. Special hardware skips this. Also, chip power grows very quickly with clock speed, roughly with the cube of speed. Around 2005, chips could not get faster without becoming too hot, which is why chip makers moved to many cores and special chips.

How it works. A general file system works okay for every program but perfectly for none. A database can skip the operating system’s cache and manage memory itself, because it knows its own patterns, but then it must build its own cache. A Bitcoin mining chip does one calculation thousands of times more efficiently than a normal CPU, but it can do nothing else.

What real systems do. Computing history moves back and forth between general and specialized. GPUs, AI chips, and video encoders appeared when those jobs became big and stable enough to be worth it. Phone chips have small efficient cores for background work and big fast cores for short bursts. Designers often add “escape doors” so experts can give hints or skip layers. The money logic is simple: specialization pays when the job is valuable, stable, and large. Generality wins when jobs change or are small.

Trade-off 9: Simple vs. Best

Comes from: human limits. Also known as: mechanism vs. policy, the end-to-end argument, layering.

Why it exists. The best solution usually needs more information, more special cases, and more code, and every piece of code can have bugs. A simple system that people can understand is often more reliable, and often faster in real life, than a complex system that is best in theory.

A related question is where each feature should live. A famous 1984 paper by Saltzer, Reed, and Clark, called the end-to-end argument, says that some features can only be done fully and correctly at the two ends of a system, the sender and the receiver. Only the ends know what “correct” really means. Lower layers can help as a speed optimization, but they cannot guarantee correctness alone. Putting features in the right place keeps each layer simple.

How it works. A cache must choose which item to remove when it is full. A simple rule, “remove the item used least recently” (LRU), works well for many programs. The perfect rule needs to know the future, which is impossible, and smarter rules usually gain only a few percent. Raft, an agreement method, was designed to be easier to understand than the older Paxos. It spread widely because engineers could build it correctly.

For the end-to-end argument, think about sending a file. Each network link may check its packets for errors, but data can still be damaged inside a router or a disk driver. Only a check of the whole file, made by the sender and verified by the receiver, truly guarantees the file is correct.

What real systems do. The Unix idea of “do one thing well” and the simple design of the internet chose simplicity, and they succeeded. The internet’s base layer (IP) just tries its best to deliver packets, and reliability (TCP) and encryption (TLS) live at the ends. This is a big reason the internet could support programs its designers never imagined. Many systems separate mechanism (how something is done) from policy (what should be done). Linux keeps mechanisms in its core and offers settings for policies. Butler Lampson’s advice is “make it work, then make it fast.” Start simple, measure, and add complexity only where it clearly helps.

Trade-off 10: Sharing vs. Separation

Comes from: limited resources, uncertain behavior of others. Also known as: isolation vs. efficiency, multi-tenancy, fairness vs. efficiency, starvation.

Why it exists. Sharing resources like memory, CPUs, and machines raises utilization and lowers cost, because unused resources go to whoever needs them. But you cannot know how other users will behave. When things are shared, one user can affect another through bugs, attacks, or simply using too much (called “noisy neighbors”). Keeping users apart needs protection, and every boundary has a cost to cross.

Fairness belongs here too. To get the most total work done, you usually favor the users who use resources best, like short jobs. Fairness means serving others too, which lowers total output. Without fairness, some tasks may wait forever (called starvation). Fairness is only possible when users are separated enough to be measured and limited.

How it works. Threads in one program share memory, so talking between them is fast, but one buggy thread can break the others. Separate processes are protected from each other, but talking between them is slower. Containers share one operating system core: they are light and fast, but one security hole affects everyone. Virtual machines each have their own core: they are much safer but use more resources.

For fairness, “shortest job first” scheduling gives the lowest average waiting time, but if short jobs keep coming, a long job may never run.

What real systems do. Cloud companies built very light virtual machines, like Firecracker, to get container-like speed with virtual-machine-like safety. Browsers started using a separate process for each website after the Spectre security problem, using more memory to protect users. Databases offer isolation levels, which trade how many transactions can run together against protection from strange results. Linux tries to give each process a fair share of the CPU. Many systems use aging: a task that waits longer slowly gets higher priority, so every task runs eventually.

Trade-off 11: Exact vs. Cheap

Comes from: limited resources (accuracy is also a resource). Also known as: approximation, probabilistic data structures, sampling, reduced precision.

Why it exists. Exact answers often need memory and time that grow with the amount of data, because you must remember every item. If you accept a small, known error, much smaller structures become possible. Many applications do not need perfect answers.

How it works. Counting unique website visitors exactly across billions of events needs a list of every visitor ID, which can take gigabytes of memory. A method called HyperLogLog estimates the count with an error of about 1% to 2% using about 12 kilobytes. A Bloom filter answers “have I seen this item before?” It never wrongly says “no,” and it sometimes wrongly says “yes,” at a rate you can choose. It uses only a few bits per item.

What real systems do. Databases use Bloom filters to avoid unnecessary disk reads. Analytics systems use approximate counts and percentages. Tracing systems record only a sample of requests. AI systems use less precise numbers, like 8-bit or 16-bit instead of 32-bit, because model quality drops only a little while speed and energy use improve a lot. Bank balances and cryptography must always be exact.

Why it matters. This is a way to escape Trade-off 1. By giving up a third thing, accuracy, you can save both memory and time.

How Systems Respond

Across all 11 trade-offs, good systems use a few common strategies.

Move work in time and space, because it rarely disappears. Do it in advance, delay it, group it, or move it to another layer or machine. Most “optimizations” do not remove work. They move it to a place or time where it is cheaper. When a system seems to get something for free, look for where the cost went.

Make the common case fast, and keep the rare case correct. Caches, guessing, and optimistic methods all bet on normal behavior. The hard part is making sure the unusual case still works correctly, even if it is slow.

Use locality. Keep related data and work close together, in memory, on disk, and across the network.

Make copies for speed and safety, then pay for agreement. Copies make reads fast and survive failures, but every copy must be kept in agreement. Decide clearly how much disagreement you can accept, and for how long.

Avoid coordination when possible. Split data so workers rarely need to talk. Use operations that give the same result in any order. Centralize only the small part of the system that makes decisions.

Leave free capacity. Run below full capacity, limit queue length, and reject extra work before the system is full. The last few percent of busyness are paid for with long delays.