2  Distributed Systems & Core Principles

2.1 What is a Distributed System?

A distributed system is any system that includes multiple components residing on multiple machines, coordinating through some mechanism (like message passing) to achieve a common goal.

Modern web applications are inherently distributed, with components like:

  • Web servers
  • Application servers
  • Databases
  • Cache servers
  • Load balancers
  • Message queues

All working together to serve user requests.

2.2 The 3 Core Principles of Distributed Systems

When designing distributed systems, three fundamental principles guide our architectural decisions:

2.2.1 1. Single Responsibility Principle

Each component in the system should have a single job to perform.

This principle ensures that components are:

  • Easier to understand
  • Simpler to test
  • More maintainable
  • Independently deployable

Single Responsibility Principle

Example:

Instead of having one service that handles user authentication, sends emails, and processes payments, separate these into:

  • Authentication Service
  • Email Service
  • Payment Service

Each service has a single, well-defined responsibility.

2.2.2 2. No Single Point of Failure Principle

The system should never have any component whose failure will result in the failure of the entire system.

Achieving this requires:

  • Redundancy at every level
  • Replication of critical components
  • Graceful degradation
  • Failover mechanisms

No Single Point of Failure

Strategies:

  • Multiple load balancers in active-active or active-passive configuration
  • Database replication with automatic failover
  • Redundant network paths
  • Multi-region deployment

2.2.3 3. No Bottleneck Principle

The system should not have performance bottlenecks that limit scalability.

To achieve this:

  • Design for horizontal scaling
  • Distribute load across multiple instances
  • Use asynchronous processing where appropriate
  • Monitor and identify bottlenecks early

No Bottleneck Principle

Common Bottlenecks:

  • Single database instance
  • Synchronous processing of long-running tasks
  • Insufficient network bandwidth
  • CPU or memory constraints on a single machine

Solutions:

  • Database sharding or read replicas
  • Message queues for async processing
  • Content delivery networks (CDN)
  • Horizontal scaling with load balancing

2.3 Guide for System Design

When approaching a system design problem, follow this systematic process:

2.3.1 1. Requirement Analysis

Functional Requirements: What should the system do?

  • Define core features
  • List user actions
  • Specify system behaviors

Non-Functional Requirements: How should the system behave?

  • Performance targets (latency, throughput)
  • Scalability requirements
  • Availability targets (99.9%, 99.99%)
  • Security requirements
  • Compliance needs

2.3.2 2. API Design

Define the interfaces exposed to the outside world:

  • API endpoint names
  • HTTP methods (GET, POST, PUT, DELETE)
  • Request parameters
  • Response formats
  • Error handling
  • Authentication/authorization

Example:

POST /api/v1/users
Request: { "username": "john", "email": "john@example.com" }
Response: { "id": "123", "username": "john", "created_at": "2024-01-01T00:00:00Z" }

2.3.3 3. Define Data Model

Create a representation of:

  • Object data structures
  • Associations between objects
  • Validation rules
  • Constraints

Example:

User:
  - id (UUID, primary key)
  - username (string, unique, indexed)
  - email (string, unique)
  - created_at (timestamp)
  - updated_at (timestamp)

2.3.4 4. High-Level Design

Create a diagram showing:

  • Component blocks (load balancers, servers, databases, caches)
  • Data flow between components
  • External integrations
  • Network boundaries

The high-level design visualizes how data flows from entry point to database and back.

Components typically include:

  • Load balancers
  • Web servers
  • Application servers
  • Databases
  • Caches (Redis, Memcached)
  • Message queues (Kafka, RabbitMQ)
  • CDN
  • Object storage

2.3.5 5. Scale the Design

After creating the initial design, optimize for scale:

  • Identify bottlenecks
  • Add caching layers
  • Implement database sharding
  • Use read replicas
  • Add CDN for static assets
  • Implement horizontal scaling
  • Consider geographic distribution

Questions to ask:

  • Can the system handle 10x current load?
  • What happens when a component fails?
  • Where are the performance bottlenecks?
  • How do we handle peak traffic?

2.4 Summary

Understanding these core principles and following a systematic design process provides a strong foundation for building distributed systems. The following chapters will explore specific components and patterns that implement these principles in practice.

For deeper understanding of distributed systems architecture and implementation patterns, readers are encouraged to consult comprehensive resources such as Kleppmann’s Designing Data-Intensive Applications (Kleppmann 2017) and practical guides like the System Design Interview series (Xu 2020, 2022).