Computing Architecture

In the earlier notes, we saw that managing physical servers has a lot of overhead - deciding CPU/RAM, IP addresses, registering with load balancers, and so on. This note explains how the industry solved that overhead, first with serverless computing, and then with containers and Kubernetes.

The Problem With Managing Servers Yourself

Managing your own servers means you constantly have to think about:

• How much CPU and RAM to give each machine
• Assigning IP addresses
• Registering new servers with the load balancer
• Installing an operating system and every required software package

This is a lot of overhead. At some point you have to ask: “Do I want to spend my time managing servers, or actually writing my application code?”

To solve this, cloud companies like AWS introduced a service that lets you skip managing servers entirely. This is called Serverless Computing.

What is Serverless Computing?

“Serverless” doesn’t mean there’s no server at all - it just means you don’t manage the server yourself. The cloud provider handles everything behind the scenes (CPU, RAM, auto-scaling, etc.) - you just write your code and hand it over.

Popular tool name: On AWS, this is called Lambda. Other providers have similar offerings (like Cloud Functions on GCP). The generic term is Serverless Functions.

Here’s how it typically works:

1. You write a small piece of code (a function) that handles a specific task.
2. You give this code to the serverless service.
3. It gives you back a public URL.
4. Whenever a request hits that URL, the serverless system automatically:
   • Loads your code
   • Runs it for that one request
   • Returns the result
   • Then shuts down (destroys) that instance

flowchart LR A[Request Comes In] --> B[Serverless Platform Spins Up Function] B --> C[Function Runs Your Code] C --> D[Returns Response] D --> E[Function Instance Destroyed]

If 10 requests come in at once, the platform automatically spins up 10 separate function instances, each handling one request, and destroys them once done. You never manually manage any of this.

Pros of Serverless

1. Very cheap for most use cases - many providers give a large free tier (e.g., 1 million free requests/month), and pricing beyond that is often just fractions of a cent per request.
2. No need to worry about scaling, configuration, or auto-scaling policies - the provider handles it.
3. You only pay when it’s actually used - unlike a traditional server where you pay per hour whether anyone uses it or not.

Cons of Serverless

1. Cold Start: If there’s been no traffic for a while (zero active function instances), the very first request has to “wake up” a fresh instance, which adds a small delay (maybe 1-2 seconds) for that first user. After that, it stays “warm” and responds quickly - unless traffic stops again for a long time.
2. Fixed Duration Limit: Since an API Gateway usually sits in front, there’s normally a maximum time limit (e.g., 15 seconds) for a single request to complete.
3. Can Struggle with Extreme Traffic Spikes (like a DDoS attack): since it will just keep spinning up more and more function instances, which can get very expensive very fast.
4. Vendor Lock-in: Once you structure your code specifically for one provider’s serverless system, it becomes very hard to move to a different cloud provider later - you’d have to rewrite a lot. Also, once you start using one serverless service, you often end up automatically adopting a bunch of other related services from the same provider (queues, gateways, domain management, storage, logging, etc.) just to make things work smoothly - which deepens the lock-in even more.
5. No deep configuration control: since you don’t manage the operating system level, you can’t tweak certain low-level settings.
6. Always Stateless: you can’t “remember” or store any ongoing data/state inside the function itself, because each instance gets destroyed after the request finishes.
7. Database Connection Issues: if a serverless function connects directly to a database (like MongoDB), a huge burst of requests can create a huge number of new database connections at once, sometimes overwhelming or dropping the database’s connections. To fix this, people often need a separate “connection manager” service, which brings back some of the complexity you were trying to avoid.

In short

Serverless is great for simple, spiky, cost-sensitive workloads, but it comes with real trade-offs (cold starts, vendor lock-in, limited control) that show up more once you’re running things at real production scale.

This whole style (managing physical servers yourself with fixed configuration) is called “Serverful” / traditional server architecture, as opposed to “serverless.”

The Next Problem: “It Works On My Machine”

Let’s go back to traditional (serverful) servers for a moment. Say you write code on your own laptop (Windows or Mac) that depends on some library - for example, a tool called FFmpeg (used for video processing).

When you try to deploy your code to an actual production server:

1. The production server usually runs a different operating system (commonly Linux/Ubuntu).
2. It might not have FFmpeg installed, or has a different version.
3. So your code that worked perfectly on your machine fails on the server.

This is the classic “it works on my machine” problem. And if you have 30-40 different dependencies/libraries in your project, keeping all of them in sync between your machine and every server becomes a nightmare - especially when you need to scale out fast (spinning up new servers under traffic pressure takes too long if you have to manually install everything each time).

Solution 1: Virtual Machines (VMs)

To solve this, the idea of Virtualization came in. A Virtual Machine (VM) lets you create a “mini-computer” inside your actual computer, complete with its own operating system.

You set up everything (OS, libraries, your code) inside this VM, test that it works, and then deploy the entire VM (as one packaged image) onto your production server.

flowchart TB subgraph Physical Machine subgraph VM["Virtual Machine"] OS[Full Operating System] L[Libraries/Dependencies] C[Your Code] end end

This solves the “works on my machine” problem, since the entire environment (not just the code) travels together.

But VMs have a downside: They are heavy. Since each VM includes a complete operating system on top of the host machine’s own operating system, they take up a lot of resources (RAM, storage) and take real time to start up (since booting a full OS can take a while). So while VMs solve the compatibility problem, scaling them quickly is still slow and resource-heavy.

Solution 2: Containers (Lightweight VMs)

To fix the “heaviness” of VMs, the industry came up with Containerization. Think of a container as a lightweight VM.

The key trick: remove the need for a full separate operating system. A container shares the host machine’s operating system kernel, but keeps all your code and dependencies isolated and packaged together.

flowchart TB subgraph Physical Machine - Shared OS Kernel subgraph C1[Container 1] L1[Libraries] Code1[Your Code] end subgraph C2[Container 2] L2[Libraries] Code2[Your Code] end end

Popular tool name: Docker is the most popular tool for building and running containers.

Since containers don’t carry a whole separate operating system, they are much smaller (for example, going from a few GB down to a few hundred MB) and much faster to start. This means:

• You can run many containers on the same physical machine.
• New containers can be created and destroyed very quickly, exactly matching the up-and-down nature of real traffic.
• If a container crashes, you can quickly spin up a new one to replace it.

Why containers were a big deal

Containers gave developers the best of both worlds: the compatibility guarantee of VMs (“works on my machine = works everywhere”), plus the speed and low resource usage needed for fast scaling.

The New Challenge: Managing Many Containers

Once you have many containers running across many machines, a new problem shows up: who manages all of this?

• Who decides when to create new containers (scale out) or remove them (scale in)?
• Who restarts a container if it crashes/errors?
• Who handles rolling out a new version of your code (replacing old containers with new ones without downtime)?

Doing all of this manually across dozens or hundreds of containers, spread across many machines (called a cluster), is extremely difficult. This whole problem area is called Container Orchestration - the process of automating the deployment, management, and scaling of containerized applications across a cluster of servers.

Deployment Strategies (a quick mention)

When you push new code, you don’t want to just delete all old containers at once (that would cause downtime). Instead, common strategies include:

• Rolling Updates: gradually replace old containers with new ones, one at a time, so the app never fully goes down.
• Blue-Green Deployment: run the new version alongside the old version, then switch traffic over once the new version is confirmed working.

How Kubernetes Was Born

Google faced this exact “container management” problem many years ago, at a massive scale, and built an internal system called Borg to manage it. Borg was never made public/open-source.

However, the team that built Borg was later asked to rewrite it from scratch as a new project, using everything they had learned. This new project was eventually open-sourced and donated to the Cloud Native Computing Foundation (CNCF). That project is called Kubernetes (sometimes shortened to “K8s”).

flowchart LR A[Google builds Borg internally] --> B[Team learns from Borg] B --> C[Rewrites from scratch as new project] C --> D[Open-sourced and donated to CNCF] D --> E[Kubernetes]

What is Kubernetes?

Kubernetes is an open-source container orchestration system that automates:

• Deploying containers
• Scaling them up/down automatically based on load
• Restarting failed containers
• Rolling out updates smoothly (zero downtime)
• Routing/load balancing traffic between containers

Kubernetes essentially acts as the “brain” that manages all your containers across a whole cluster of servers, so you don’t have to do it manually. It also comes with its own built-in load balancing and reverse-proxy capabilities, and strong support for handling incoming and outgoing traffic rules.

Note

This document only gives a high-level introduction to Kubernetes. A full breakdown of Kubernetes components (like the Control Plane, API Server, Scheduler, Worker Nodes, Kubelet, Kube-proxy, and Pods) deserves its own separate, dedicated note.

Putting the Whole Journey Together

Here’s how system design around deployment evolved over time, step by step:

flowchart LR A[Single Physical Server] --> B[Vertical Scaling - bigger server] B --> C[Horizontal Scaling - more servers + Load Balancer] C --> D[Virtual Machines - solves compatibility issue] D --> E[Containers - lightweight and fast] E --> F[Kubernetes - manages containers automatically]

Final Thoughts

Every time you watch a live cricket match with millions of viewers, or stream a video smoothly on YouTube, there is a massive amount of engineering behind the scenes - lots of trial and error, and even planned crash tests. Big companies (like Hotstar) often simulate fake heavy traffic a day before a big event (this is called load testing / stress testing) just to check how far their system can be pushed before it breaks.

This series covered the basic building blocks of a real, scalable system design:

• Servers, DNS, Load Balancers
• Vertical vs Horizontal Scaling
• Traffic patterns and why every company’s design is different
• Microservices, API Gateways, Message Queues, Pub/Sub, and Fan-Out architecture
• Rate Limiting, Database Read Replicas, Caching, and CDNs
• Serverless computing, Virtual Machines, Containers, and Kubernetes

With these basics clear, you now have a solid beginner-level foundation to understand how real-world, large-scale systems are designed and run.