Overview

Imagine you open a small restaurant. At first you cook, take orders, and serve customers all by yourself - easy, since maybe 10 people show up a day. Now imagine that restaurant suddenly gets famous and 10,000 people want to eat there every day. One kitchen and one chef can’t handle that anymore. You’d need more kitchens, more staff, a way to take orders without a huge line, a way to remember regular customers’ favourite dishes instantly, and a backup plan for when the oven breaks.

System design is this exact problem, but for software. It’s the skill of planning how all the pieces of a large application - servers, databases, caches, queues - fit together so the app stays fast, stays online, and doesn’t collapse when millions of people use it at once. It combines everything else you learn in computer science (networking, databases, operating systems, algorithms) into one practical skill. Whether you’re designing a URL shortener, a social media feed, or a global payment system, the same handful of ideas keep showing up - and this guide explains every one of them in plain English.

How to Approach System Design

Whenever you’re asked to design something, follow the same repeatable process instead of jumping straight to drawing boxes:

1. Clarify requirements - what must the system actually do (functional), and how fast/reliable/big does it need to be (non-functional)?
2. Estimate the scale - roughly how many users, requests per second, and how much data piles up over time?
3. Sketch the high-level design - the major components and how data flows between them, without going deep yet.
4. Deep dive - zoom into the one or two trickiest or riskiest components.
5. Identify bottlenecks - what breaks first under heavy load, and how do you fix it?
6. Talk through trade-offs - every decision gives something up; say out loud what you’re optimising for.

This single diagram is basically the skeleton most large systems share - everything below explains one box (or one arrow) from it in detail.

graph LR Client["Client Web / Mobile"] LB["Load Balancer"] API["API Servers"] Cache["Cache Redis"] DB["Primary Database"] Replica["Read Replicas"] Queue["Message Queue"] Worker["Background Workers"] Storage["Object Storage"] Client --> LB LB --> API API --> Cache API --> DB DB --> Replica API --> Queue Queue --> Worker Worker --> Storage

Scalability

Scalability simply means: can your system keep working well as more and more people start using it? There are only two real ways to get there.

Vertical Scaling (Scale Up)

You buy a bigger, more powerful machine - more CPU, more RAM, more storage. It’s simple to start with, but there’s always a hardware ceiling, and it’s still just one machine: if it fails, everything fails (a single point of failure).

Horizontal Scaling (Scale Out)

You add more (regular-sized) machines and spread the work across all of them. There’s no real ceiling - just keep adding boxes - but it’s more complex to manage, and you need a load balancer to spread traffic across them.

graph TD LB["Load Balancer"] LB --> S1["Server 1"] LB --> S2["Server 2"] LB --> S3["Server 3"] LB --> S4["Server 4"]

• Stateless servers - servers that don’t remember anything about you locally between requests. Any server can handle any request, which makes them easy to scale horizontally and freely add or remove behind a load balancer.
• Stateful considerations - if a server does remember things locally (sessions, in-flight work), scaling it horizontally gets trickier; you either need to route a user back to the same server every time, or move that “state” to a shared place like a database or cache.
• Back-of-the-envelope estimation - quick maths to sanity-check a design before you build it:
  • 1M daily active users × 10 requests/day ≈ 115 requests/second on average; multiply by 5–10× for peak traffic ≈ ~1,000 requests/second.
  • 1 KB per request × 1,000 requests/second ≈ ~1 MB/s of throughput to plan for.

Availability and Reliability

Availability asks “is the system actually reachable right now?” Reliability asks “when it responds, is the answer correct?” Both matter - a system can respond instantly with the wrong answer.

Availability	Downtime per year
99%	~3.65 days
99.9%	~8.7 hours
99.99%	~52 minutes
99.999%	~5 minutes

SLA

Service Level Agreement - the official promise made to customers, often with penalties if broken.

SLO

Service Level Objective - the internal target engineers design for, usually stricter than the SLA.

SLI

Service Level Indicator - the actual number you measure in production (e.g. “99.95% this month”).

A Single Point of Failure (SPOF) is any one component whose failure takes the whole system down. The fix is redundancy - keeping backups of critical components ready to take over, via failover.

Active-Passive

One standby instance sits idle. If the primary dies, the standby wakes up and takes over - simple and cheaper, but there’s usually a brief gap while it switches.

Active-Active

Multiple instances are all serving traffic at the same time. If one dies, the others simply absorb its share instantly - no noticeable downtime, but more infrastructure to run.

graph LR subgraph "Active-Passive" P1["Primary (serving)"] S1["Standby (idle)"] end subgraph "Active-Active" A1["Server A (serving)"] A2["Server B (serving)"] end

• Fault isolation - designing so a failure in one component doesn’t cascade into others, using patterns like bulkheads (watertight compartments) and circuit breakers (stop calling a service that keeps failing instead of hammering it further).
• Graceful degradation - when something fails, the system keeps working in a reduced way instead of crashing entirely (e.g. Instagram still lets you scroll your feed even if “likes” are temporarily broken).

The CAP Theorem

If your database is copied across two locations for safety, and the network connection between them ever breaks (it eventually will), you’re forced to choose between:

• Consistency (C) - every copy always shows the exact same, up-to-date answer.
• Availability (A) - the system keeps responding, even if some copies haven’t caught up yet.

You can’t have both perfectly during a network split - that’s the whole theorem (“P”, Partition tolerance, just means the network can split, which in real distributed systems is unavoidable, so you’re really always choosing between C and A).

CP System

Favours correctness - would rather show an error than risk showing wrong data. Good fit for things like banking ledgers.

AP System

Favours uptime - would rather show slightly outdated data than nothing at all. Good fit for things like a social media “like” counter.

Load Balancing

A load balancer sits in front of your servers and decides which one handles each request, so no single server gets overwhelmed - like a host at a busy restaurant deciding which table gets the next group.

graph TD LB["Load Balancer"] LB --> A["Server A"] LB --> B["Server B"] LB --> C["Server C"]

• Algorithms: Round Robin (cycle through servers in order), Weighted Round Robin (stronger servers get more traffic), Least Connections (send to whichever server is least busy), IP Hash (same user always reaches the same server), Random.
• Layer 4 vs Layer 7 - Layer 4 routes based on basic network info (like reading an envelope’s address); Layer 7 looks inside the request itself (like reading the letter) and can route smarter, e.g. images vs video differently.
• Health checks - the load balancer regularly pings each server and automatically stops sending traffic to any that stop responding.
• Session affinity (sticky sessions) - routing the same user back to the same backend every time; useful for stateful apps.
• Global load balancing - routing users to the nearest data centre using DNS or Anycast, so someone in Mumbai isn’t routed all the way to Virginia.
• Examples: NGINX, HAProxy, AWS ALB, Cloudflare Load Balancing.

Caching

Caching stores the result of something expensive (like a slow database query) somewhere fast, so the next time it’s needed you don’t redo the work - like keeping your most-used spices on the counter instead of walking to the pantry every time.

graph TD Req["Request"] --> Browser["Browser Cache"] Browser -- miss --> CDN["CDN Cache"] CDN -- miss --> AppCache["App Memory Cache"] AppCache -- miss --> Redis["Redis / Distributed Cache"] Redis -- miss --> DB["Database (slowest, always correct)"]

A cache hit returns data straight from the cache; a cache miss means fetching from the real source and (usually) storing it for next time.

• Cache Eviction Policies: LRU (throw away whatever hasn’t been used in the longest time), LFU (throw away whatever is used least often), FIFO (throw away the oldest item), TTL (items auto-expire after a set time).

Write-through

Write to cache and database at the same time. Always consistent, but writes are a bit slower.

Write-behind

Write to cache first, flush to the database later. Writes feel instant, but there’s a small risk of losing data if something crashes first.

Cache-aside

App reads from cache; on a miss it loads from the database and stores the result in cache. The most common pattern.

Read-through

Same idea as cache-aside, but the cache itself is responsible for fetching from the database on a miss.

• Cache Pitfalls: cache stampede (many requests miss at the same moment and hammer the database), cache poisoning (bad data gets stored and served to everyone), stale data (cache not updated after the real data changes).
• Redis is the most widely used distributed cache - it also supports pub/sub messaging, sorted sets, streams, and can persist data to disk.

DNS - How a Web Address Finds Your Server

Before any request can reach your servers, your browser needs to know where to send it. DNS (Domain Name System) is the internet’s phone book - it translates a human name like www.example.com into the actual numeric IP address of the server that should handle it.

graph LR Browser["Browser"] --> DNS["DNS Server"] DNS --> IP["IP Address"] IP --> LB["Load Balancer"]

Proxies, Reverse Proxies, and CDNs

A proxy (forward proxy) sits in front of clients and makes requests on their behalf, often to hide identity or filter content. A reverse proxy sits in front of servers instead and hides them from the outside world - every request appears to come from the proxy, not your actual backend machines.

graph LR Users["Internet Users"] --> RP["Reverse Proxy / CDN"] RP --> S1["App Server 1"] RP --> S2["App Server 2"]

A CDN (Content Delivery Network) is a special kind of reverse proxy with copies of your static files (images, videos, CSS, JS) stored on servers physically close to users worldwide, so a user in Tokyo gets the image from a nearby Tokyo edge server instead of from your origin server overseas - a huge speed win for anything that doesn’t change often.

Databases at Scale

graph TD Primary["Primary Database (handles writes)"] Primary --> R1["Read Replica 1"] Primary --> R2["Read Replica 2"]

• Read replicas - copies of the primary database that serve reads, while the primary handles all writes. Most apps read far more than they write, so read/write splitting dramatically increases how much traffic the database layer can handle.

graph LR Router["Shard Router"] Router --> S1["Shard 1: Users A-H"] Router --> S2["Shard 2: Users I-P"] Router --> S3["Shard 3: Users Q-Z"]

• Sharding - splitting data across multiple database nodes by a shard key (range, hash, or directory-based). Watch out for hotspots (a poor shard key sends uneven load to one shard) and cross-shard queries (expensive - avoid them with good schema design).
• Indexing - like the index at the back of a textbook: instead of scanning every row, the database jumps almost straight to the answer. Indexes speed up reads but cost extra storage and slightly slow down writes, so only index columns that are actually searched often.

ACID

Used by traditional SQL databases. Atomic (all-or-nothing), Consistent, Isolated, Durable. The gold standard when correctness matters more than raw speed - e.g. banking.

BASE

Common in many NoSQL databases. Basically Available, Soft state, Eventually consistent - favours staying online and fast over being perfectly up to date every millisecond. Good for things like view counters.

• Connection pooling - keeping a small set of database connections already open and reused, instead of opening a brand-new one for every request (PgBouncer, HikariCP).

SQL

Strong consistency, complex queries, proven at scale (PostgreSQL, MySQL, Aurora).

NoSQL

Flexible schema, horizontal scale, eventual consistency (MongoDB, Cassandra, DynamoDB).

NewSQL

SQL-style guarantees with horizontal scalability (CockroachDB, Spanner).

Consistent Hashing

If you shard data with a simple rule like “user ID modulo 3 servers,” adding a 4th server forces almost all data to reshuffle - a disaster at scale. Consistent hashing arranges servers and data on an imaginary ring; each piece of data belongs to the next server clockwise. Adding or removing a server only moves the small slice of data near it, leaving everything else untouched. This is how systems like DynamoDB and many caching layers handle growth smoothly.

Message Queues and Event Streaming

Message queues decouple components so they can communicate asynchronously, without waiting on each other.

graph LR Producer["Producer (e.g. Order Placed)"] --> Queue["Message Queue"] Queue --> Consumer["Consumer (e.g. Send Email)"]

• Producers send messages; Consumers read and process them. If a consumer is slow or down, messages simply wait safely in the queue - the producer never has to wait.
• Dead Letter Queue (DLQ) - messages that fail processing repeatedly are moved here for inspection instead of being lost or retried forever.
• Delivery guarantees: at-most-once (might lose a message, never duplicates it), at-least-once (never loses a message, might duplicate it), exactly-once (never lost, never duplicated - hard and expensive to guarantee).
• Message Brokers: RabbitMQ (flexible routing, delivery acknowledgements), Apache Kafka (high-throughput event streaming, durable, replayable by offset), AWS SQS / SNS (managed queues and pub/sub).

graph TD Pub["Publisher"] --> Topic["Topic"] Topic --> Sub1["Subscriber 1"] Topic --> Sub2["Subscriber 2"] Topic --> Sub3["Subscriber 3"]

• Pub/Sub pattern - publishers send to a topic; every subscriber gets a copy of each message (unlike a plain queue, where each message usually goes to only one consumer).
• Event Streaming vs Message Queue - streams retain messages for a window of time (hours/days) and multiple consumers can independently replay them; queues deliver each message to one consumer and then delete it.

Microservices vs Monoliths

graph TD subgraph Monolith["Monolith"] M["Users + Orders + Payments one app, one database"] end subgraph Microservices["Microservices"] U["Users Service"] --- UD["Users DB"] O["Orders Service"] --- OD["Orders DB"] P["Payments Service"] --- PD["Payments DB"] end

Monolith

All functionality in one deployable unit. Simple to develop, test, and deploy initially, but becomes unwieldy at scale - a bug can take down the whole system, and individual features can’t be deployed alone.

Microservices

Each business capability is its own service with its own database. Independent deployments, technology flexibility, and fault isolation, at the cost of distributed-system complexity, network calls, and more infrastructure.

• Service communication: synchronous (REST, gRPC - caller waits for a response) vs asynchronous (message queues, event streaming - caller fires and continues).
• Service Discovery - how services find each other’s current addresses (client-side: Eureka; server-side: NGINX, Consul).

graph LR Client["Client"] --> GW["API Gateway"] GW --> S1["Users Service"] GW --> S2["Orders Service"] GW --> S3["Payments Service"]

• API Gateway - a single entry point for all client requests; handles auth, rate limiting, routing, and logging.
• Strangler Fig Pattern - gradually migrating a monolith to microservices by replacing pieces one at a time, like a vine slowly growing around and eventually replacing a tree.

API Design

REST

Resources identified by URLs (GET /users/123, POST /orders). Stateless - every request carries all the info the server needs. Standard HTTP methods and status codes. Versioning via the URL (/v1/users) or headers.

GraphQL

The client specifies exactly what data it wants in a single query, avoiding over- and under-fetching. A schema defines types and relationships; resolver functions fetch the data. Great for complex, relationship-heavy data with varied client needs.

gRPC

Uses compact binary Protocol Buffers over HTTP/2 - much faster than JSON for internal services. Strongly typed, auto-generates client/server code from .proto schema files. Ideal for service-to-service communication.

• Pagination - cursor-based (pointing to “the item after this one”) scales much better than offset-based (“skip the first 10,000 rows”) for large, changing datasets.
• Rate Limiting - preventing clients from making too many requests (covered in detail below).
• Idempotency - safe to retry without causing duplicate side effects; POST requests should accept an idempotency key, e.g. so a double-click on “Pay $50” never charges twice.

Polling

Client repeatedly asks “anything new?” on a timer. Simple, but wasteful.

Long Polling

Client asks once; server holds the connection open until it actually has new data.

WebSockets

A persistent two-way connection - either side can send data anytime. Used for live chat.

Server-Sent Events

A persistent one-way connection where the server keeps pushing updates, e.g. live news or stock tickers.

Webhooks

Instead of you constantly asking another service “did anything happen?”, it calls your server automatically the instant something happens.

Event-Driven Architecture

Events are immutable records of something that happened. Event producers emit them; event consumers react and take action.

graph LR Order["Order Service"] -- "OrderPlaced" --> Payment["Payment Service"] Payment -- "PaymentConfirmed" --> Shipping["Shipping Service"]

• Benefits: loose coupling, easy to add new consumers without touching producers, a natural audit log.
• Challenges: eventual consistency, harder debugging, more complex error handling.
• Event Sourcing - storing every change as an event rather than just the current state; the current state is derived by replaying the events.
• CQRS - separating the write model (commands) from the read model (queries) so each can be optimised independently.
• Saga Pattern - managing a transaction that spans multiple microservices by chaining events, with compensating (“undo”) steps if something later in the chain fails.

Rate Limiting and Throttling

A rate limiter stops any client from making too many requests too fast - like a bouncer only letting a certain number of people in per minute, to protect everyone already inside.

• Token Bucket - tokens refill at a fixed rate; each request consumes one. Allows short bursts.
• Leaky Bucket - requests queue up and are processed at a fixed rate, smoothing out bursts.
• Fixed Window Counter - simple, but has edge cases right at window boundaries.
• Sliding Window Log - tracks individual request timestamps; accurate but memory-heavy.
• Sliding Window Counter - a practical compromise between the two above.
• Where to rate limit: API gateway, load balancer, or the application itself. For multiple servers, use a shared store like Redis for distributed rate limiting.

Storage Systems

Storage Type	What it’s for	Examples
Object Storage	Arbitrary blobs (images, videos, backups), accessed by key, horizontally scalable	AWS S3, Google Cloud Storage
Block Storage	Raw disk volumes attached to a VM, used for databases/OS disks	AWS EBS
File Storage	A shared file system accessed by multiple machines	AWS EFS, NFS

• Structured data with relationships -> relational database.
• Unstructured blobs (media, documents) -> object storage.
• Fast key-value access -> Redis or DynamoDB.
• Full-text search -> Elasticsearch.
• Time series data -> InfluxDB or TimescaleDB.

Security Basics Every System Needs

Authentication (AuthN)

Answers “who are you?” - proving identity, usually with a password or token.

Authorization (AuthZ)

Answers “what are you allowed to do?” - even once identity is known, the system still checks permission for the action.

• HTTPS/TLS - encrypts data while it travels between the user and your servers so nobody snooping on the network can read it.
• Encryption at rest vs in transit - “in transit” protects data moving across a network; “at rest” protects data sitting in storage, so a stolen disk can’t be read.
• OAuth - a standard that lets you log into one app using another account (e.g. “Sign in with Google”) without sharing your actual password with that app.

Monitoring, Logging, and Observability

You can’t fix what you can’t see.

• Logging - recording what happened, step by step, usually as text.
• Metrics - numerical measurements over time (requests/sec, error rate, latency), usually shown on dashboards.
• Tracing - following a single request through many services to see exactly where it slowed down or failed.
• Alerting - automatically notifying engineers when something crosses a dangerous threshold.

Together, logs, metrics, and traces are often called the “three pillars of observability.”

Latency Numbers Every Engineer Should Know

Operation	Roughly how long it takes
Reading from RAM	Nanoseconds - extremely fast
Reading from an SSD	Tens of microseconds
Reading from a spinning disk	Milliseconds
Round trip within the same data centre	~0.5 millisecond
Round trip across a country	Tens of milliseconds
Round trip across the world	100+ milliseconds

A network call to a faraway server is roughly a million times slower than reading from memory - exactly why caching and locating servers close to users (CDNs, global load balancing) matter so much.

Real-World Design Examples

Common systems to practise designing:

URL Shortener

Hashing, key generation, redirection, click analytics (e.g. bit.ly).

Paste / File Upload

Object storage, expiry, deduplication (e.g. Pastebin, S3).

Social Media Feed

Fanout-on-write vs fanout-on-read, timeline service, sharding (e.g. Twitter, Instagram).

Chat System

WebSockets, message ordering, group messaging, presence (e.g. WhatsApp, Slack).

Video Streaming

Adaptive bitrate, CDN, chunked encoding, recommendations (e.g. YouTube, Netflix).

Ride Sharing

Geospatial indexing, matching, pricing, real-time updates (e.g. Uber, Lyft).

Web Crawler

BFS traversal, politeness delay, distributed fetching, deduplication.

Notification System

Push vs pull, delivery guarantees, fan-out, retry logic.

Rate Limiter

Algorithms, distributed state, multi-tier limiting.

Search Autocomplete

Trie, top-K suggestions, caching, personalisation.

Key Trade-offs to Master

Every system design decision involves trade-offs. The best engineers are explicit about what they’re optimising for.

Dimension	Option A	Option B
Consistency vs Availability	Strong consistency (CP)	High availability (AP)
Read performance vs Write performance	Cache heavily (fast reads)	Write-through (fast, consistent writes)
Latency vs Throughput	Low latency (few, fast requests)	High throughput (many, batched requests)
SQL vs NoSQL	Rich queries, ACID	Horizontal scale, flexible schema
Sync vs Async	Simple, immediate response	Decoupled, higher throughput
Monolith vs Microservices	Simple to operate	Independent scale and deploy

How to Approach a System Design Interview

1. Don’t rush into boxes. Spend the first few minutes asking what the system needs to do and how many users are involved.
2. State assumptions out loud. “I’ll assume 1M daily users with reads far more common than writes” - this shows your thinking even if no one corrects you.
3. Start broad, then go deep. Sketch the high-level picture first, then pick the part that deserves the most detail.
4. Talk through trade-offs as you go, not just at the end.
5. Mention failure scenarios. What happens if the database goes down, or a service is overwhelmed?
6. It’s fine to say “I’m not 100% sure, but here’s my reasoning.” Interviewers care more about how you think than a memorised “correct” answer.

Quick Glossary

Term	Meaning
Latency	How long a single request takes
Throughput	How many requests the system can handle per unit of time
Node	A single machine/server in a larger system
Cluster	A group of nodes working together as one logical unit
Replica	A copy of data kept on a separate machine for safety or speed
Partition / Shard	A slice of the total data, stored on its own machine
Throttle	To deliberately slow down or limit something to protect the system
Idempotent	Safe to repeat without causing extra side effects
Eventual consistency	All copies of data will eventually match, just maybe not instantly
Heartbeat	A small, regular signal a service sends to prove it’s still alive

System design is not about finding the one right answer. It is about making informed decisions and communicating the trade-offs clearly. The more systems you study and practise designing, the faster your intuition for these patterns will develop.