Top 73 System Design Interview Questions & Answers (2026)

System design is the process of defining the architecture, components, modules, interfaces, and data flow of a system to satisfy specified requirements. It bridges the gap between high-level requirements and low-level implementation details. In software engineering, system design involves making decisions about: Architecture — monolith vs microservices, client-server vs peer-to-peer; Data storage — which databases, how to structure data, replication strategy; Scalability — how the system handles growth in users, data, and traffic; Reliability — how the system behaves when components fail; Performance — latency, throughput, response times; Security — authentication, authorization, encryption. System design interviews evaluate a candidate's ability to think at scale and make informed engineering trade-offs. The process typically involves: clarifying requirements, estimating scale (capacity planning), designing high-level architecture, deep-diving into components, identifying bottlenecks, and discussing trade-offs. There is no single "correct" answer — good system design is about demonstrating sound reasoning and awareness of constraints.

Scalability is the ability of a system to handle an increasing amount of work (users, data, requests) by adding resources. Two primary types: (1) Vertical scaling (Scale-up): adding more resources to a single machine — more CPU cores, more RAM, faster SSDs. Simpler (no code changes), no distributed systems complexity. Limits: hardware ceiling (can't add infinite RAM), single point of failure, expensive high-end hardware, downtime during upgrade; (2) Horizontal scaling (Scale-out): adding more machines to distribute the load. No theoretical limit — add as many servers as needed. More complex — requires load balancing, stateless application design, data partitioning, distributed coordination. Cloud-native, cost-effective (commodity hardware). Most large-scale systems (Google, Facebook, Amazon) use horizontal scaling. Elastic scalability: automatically scaling up/down based on demand (auto-scaling groups in AWS). Scalability dimensions: data volume (storage), transaction volume (throughput), geographic distribution. Trade-offs: scalable systems are more complex — distributed transactions, consistency challenges, network partitions. Good scalability design starts early — retrofitting scalability is expensive. Always ask: "What does the system need to scale to?" before over-engineering.

A load balancer distributes incoming network traffic across multiple servers to ensure no single server becomes overwhelmed, improving availability and responsiveness. It sits between the client and server pool, acting as a reverse proxy. Load balancing algorithms: (1) Round Robin: requests distributed in rotation — simple, assumes equal server capacity; (2) Weighted Round Robin: more requests sent to higher-capacity servers; (3) Least Connections: new requests go to the server with the fewest active connections — good for long-lived connections; (4) IP Hash: client IP determines which server — ensures the same client always reaches the same server (session affinity/sticky sessions); (5) Random: random server selection; (6) Resource-based: routes based on actual server CPU/memory metrics. Layer 4 LB: operates at TCP/UDP level — fast, no content inspection; Layer 7 LB: operates at HTTP level — can route based on URL, headers, cookies (content-aware). Health checks: continuously tests servers; removes unhealthy ones from rotation. Types: hardware (F5), software (Nginx, HAProxy), cloud-managed (AWS ALB/NLB). Benefits: high availability (no SPOF), horizontal scalability, SSL termination, DDoS protection. The load balancer itself can be a bottleneck — use redundant load balancers (active-passive or active-active) for high availability.

Caching stores copies of frequently accessed data in fast storage (memory) so future requests are served faster, reducing load on the origin (database, API). The fundamental trade-off: freshness (accuracy) vs speed. Cache levels: (1) Client-side: browser cache (CSS, images, JS — reduces network requests); (2) CDN: geographically distributed caches for static content; (3) Application cache: in-process cache (local variable, per-instance — fast but not shared); (4) Distributed cache: Redis, Memcached — shared across multiple app servers; (5) Database cache: query result cache, InnoDB buffer pool. Cache patterns: Cache-aside (Lazy Loading): app checks cache first, on miss reads from DB and populates cache — most common; Write-through: write to cache AND DB simultaneously — no stale data, but every write hits both; Write-back (Write-behind): write to cache only, flush to DB asynchronously — fastest writes, risk of data loss; Read-through: cache sits in front of DB, handles loading automatically. Cache eviction policies: LRU (Least Recently Used), LFU (Least Frequently Used), TTL-based expiry, FIFO. Cache invalidation is notoriously hard — when data changes, stale cached copies must be invalidated or expired. "There are only two hard things in CS: cache invalidation and naming things."

The CAP theorem (Brewer's theorem, 2000) states that a distributed data store can only guarantee two of the following three properties simultaneously: (1) Consistency (C): every read receives the most recent write or an error — all nodes see the same data at the same time. Like a single, up-to-date copy; (2) Availability (A): every request receives a response (not necessarily the most recent data) — the system is always up and responding; (3) Partition Tolerance (P): the system continues operating even if network messages are dropped or delayed between nodes (network partition). In a distributed system, partition tolerance is mandatory — network partitions happen and cannot be avoided. So the real choice is between C and A when a partition occurs: CP systems (Consistency + Partition Tolerance): during a partition, some nodes become unavailable to prevent inconsistency. Examples: HBase, MongoDB (strong consistency mode), ZooKeeper, etcd; AP systems (Availability + Partition Tolerance): during a partition, nodes stay available but may serve stale data. Examples: Cassandra, CouchDB, DynamoDB (default). The CA option is only possible in a single-node system (no partitions). Modern nuance: PACELC theorem extends CAP — even without partitions, there's a trade-off between latency and consistency.

A CDN is a globally distributed network of servers (edge nodes/PoPs — Points of Presence) that cache and deliver content from locations geographically closer to users, reducing latency and origin server load. How it works: when a user requests content, the CDN routes the request to the nearest edge server (based on DNS routing or Anycast). If the edge has the cached content, it serves it directly (cache hit). If not (cache miss), it fetches from the origin, caches it, and serves the user. Content served via CDN: static assets (images, CSS, JS, fonts, videos), large files, streaming media, and increasingly dynamic content (edge computing). Benefits: reduced latency (content served from nearby node), reduced origin server load (most requests served by CDN), improved availability (distribute traffic across many nodes), DDoS protection (distributed absorbs attack traffic), SSL termination at edge. Cache control: CDN respects Cache-Control and Expires HTTP headers set by the origin. Invalidation: purge specific files or paths via CDN API. Providers: Cloudflare, AWS CloudFront, Akamai, Fastly, Azure CDN. Cache-Control headers: Cache-Control: public, max-age=31536000 — cache for 1 year (immutable assets with hash in filename); Cache-Control: no-store — don't cache at all (sensitive data). CDN is usually one of the highest-impact, lowest-effort performance improvements for any web application.

SQL (Relational) databases store data in structured tables with predefined schemas, use SQL for queries, and enforce ACID transactions. Relationships are defined via foreign keys. Examples: MySQL, PostgreSQL, Oracle, SQL Server. Best for: complex queries with joins, strong consistency requirements, well-defined structured data, financial transactions. NoSQL databases store data in flexible, schema-less formats optimized for specific access patterns. Types: Document (MongoDB, Couchbase — JSON documents); Key-Value (Redis, DynamoDB — ultra-fast lookup by key); Column-family (Cassandra, HBase — wide rows, good for time-series); Graph (Neo4j, Amazon Neptune — relationship-heavy data). Best for: rapidly changing schemas, massive scale, specific access patterns, unstructured data, horizontal scaling. Key differences: Schema: SQL is rigid (ALTER TABLE); NoSQL is flexible (add fields anytime). Scaling: SQL scales vertically easily, horizontally is complex; NoSQL designed for horizontal. Consistency: SQL typically ACID; NoSQL often eventual consistency (BASE). Queries: SQL supports complex joins; NoSQL queries limited to access patterns the data model was designed for. Choosing: structured + complex queries + ACID → SQL; flexible schema + massive scale + simple access patterns + horizontal scaling → NoSQL. Many modern architectures use both — SQL for core business data, Redis for caching, Elasticsearch for search, Cassandra for time-series.

Database sharding is a horizontal partitioning strategy that distributes data across multiple database instances (shards), with each shard holding a subset of the total data. Each shard is an independent database that only stores its portion of the data. Sharding strategies: (1) Range-based: shard by value range (user IDs 1-1M → shard 1, 1M-2M → shard 2). Simple routing but risk of hot spots; (2) Hash-based: apply hash function to shard key (hash(user_id) % num_shards). Distributes evenly, but range queries require all shards; (3) Directory-based: lookup table maps records to shards — flexible, but lookup table is a bottleneck; (4) Geo-based: shard by geography — EU users on EU shard, US on US shard — data sovereignty compliance. Benefits: each shard handles less data and load, enabling horizontal scaling beyond one machine's limits. Challenges: cross-shard joins are complex or impossible; cross-shard transactions require distributed protocols; resharding when adding nodes is complex (consistent hashing helps); shard key selection is critical — wrong key causes hot spots. Choosing the shard key: high cardinality (many distinct values), uniform distribution, aligns with query patterns. Tools: Vitess (MySQL sharding), Citus (PostgreSQL sharding). MongoDB and Cassandra have built-in sharding.

Database replication maintains multiple copies of the same data on different servers to improve availability, fault tolerance, and read performance. Primary-Replica (Master-Slave): one primary accepts writes; replicas are read-only copies. Primary streams changes to replicas via write-ahead log or binary log. Benefits: read scaling (route reads to replicas), high availability (promote replica if primary fails), backup (take backups from replicas). Lag: replicas may be behind primary — reads from replicas may be stale. Primary-Primary (Multi-Master): multiple nodes accept writes and replicate to each other. Benefits: write scaling, geographic distribution (write to nearest). Challenges: conflict resolution when the same data is written to two primaries simultaneously. Synchronous replication: primary waits for replica to confirm before acknowledging the write — no data loss but higher write latency. Asynchronous replication: primary acknowledges immediately, replica updates later — lower latency but potential data loss on primary failure. Semi-synchronous: primary waits for at least one replica to confirm. Read-write splitting: application routes writes to primary, reads to replicas. Requires handling replication lag (read-your-own-writes: after write, read from primary briefly, then replicas). Examples: MySQL replication, PostgreSQL streaming replication, MongoDB replica sets, Cassandra multi-node replication with configurable consistency.

Microservices architecture structures an application as a collection of small, independently deployable services, each responsible for a single business capability with its own data store. Each service runs in its own process and communicates via lightweight APIs (HTTP/REST, gRPC) or message queues. Key characteristics: single responsibility (each service does one thing well); independent deployment (deploy one service without touching others); independent scaling (scale the payment service 10x without scaling user service); technology diversity (each service can use different language/database); decentralized data management (each service owns its data). Benefits: independent deployments, faster release cycles, isolated failures (one service crash doesn't bring down everything), team autonomy (each team owns their service), granular scaling. Challenges: distributed system complexity; network latency between services; distributed transactions (no ACID across services — use Saga pattern); service discovery; operational overhead (many services to monitor, deploy, version); testing complexity (integration tests span multiple services); data consistency. Communication patterns: synchronous (REST, gRPC — direct request/response); asynchronous (Kafka, RabbitMQ — event-driven, decoupled). When to use: large teams, complex domains, different scaling needs per component. When NOT to use: small teams, simple applications, early stage startups — start with a monolith and extract microservices as complexity grows.

An API gateway is a server that acts as the single entry point for all client requests to a microservices backend. It handles cross-cutting concerns so individual services don't have to. Functions of an API gateway: (1) Request routing: routes requests to the appropriate backend service based on URL path, headers, or other criteria; (2) Authentication/Authorization: validates JWT tokens or API keys once at the gateway, so services don't each need auth logic; (3) Rate limiting: throttle requests per client; (4) SSL termination: handle HTTPS at the gateway, forward plain HTTP internally; (5) Load balancing: distribute requests across service instances; (6) Request/response transformation: translate between client and service formats; (7) Caching: cache common responses; (8) Logging and monitoring: centralized observability; (9) Circuit breaking: stop forwarding to unhealthy services; (10) API composition: aggregate data from multiple services into one response (Backend for Frontend pattern). Examples: AWS API Gateway, Kong, NGINX, Traefik, Netflix Zuul, Apigee. BFF (Backend for Frontend) pattern: separate API gateways for different client types (mobile BFF, web BFF) — each gateway optimized for its client's needs. Trade-off: gateway adds a hop and becomes a potential bottleneck/single point of failure — deploy redundantly.

Consistent hashing is a technique for distributing data across a cluster of nodes such that adding or removing nodes requires remapping only a minimal fraction of keys. Problem it solves: with simple modulo hashing (key % N), adding or removing one server causes almost all keys to remap — triggering massive cache invalidation or data migration. How it works: imagine a circular ring (hash space) with values 0 to 2³²-1. Each server is hashed to a position on the ring. Each key is also hashed to a position. A key is assigned to the first server encountered going clockwise on the ring. Adding a server: only keys between the new server and its predecessor on the ring are affected — roughly 1/N of total keys. Removing a server: its keys move to the next server clockwise — only 1/N affected. Virtual nodes (vnodes): each physical server is mapped to multiple positions on the ring. Benefits: more uniform distribution (avoids hot spots); smoother addition/removal; different server capacities handled by assigning proportionally more vnodes. Real-world use: Memcached client-side sharding, Cassandra ring topology, DynamoDB (originally), Riak, Chord DHT, Akamai CDN. Applications: distributed caching, database sharding, load balancing. Consistent hashing is one of the most important concepts in distributed systems design.

REST (Representational State Transfer) exposes data as resources with fixed URLs and HTTP methods. Each endpoint returns a predetermined data structure. Simple to understand, widely supported, good HTTP caching. Problems: over-fetching (endpoint returns more data than needed), under-fetching (multiple requests needed for related data), versioning complexity (/v1/, /v2/), rigid endpoint structure. GraphQL exposes a single endpoint where clients specify exactly what data they need via a query language. The server returns exactly what was requested — no more, no less. Key differences: Fetching: REST returns fixed data; GraphQL returns only requested fields; Multiple resources: REST needs multiple requests; GraphQL fetches all related data in one request; Versioning: REST often needs /v1/, /v2/; GraphQL adds fields and deprecates old ones without versioning; Type system: GraphQL has a strongly typed schema; REST has no built-in type system; Caching: REST is naturally cacheable via HTTP; GraphQL queries (POST) require custom caching (persisted queries, CDN with cache headers); Tooling: GraphQL has GraphiQL, introspection; REST has Swagger/OpenAPI. When to use GraphQL: complex data graphs, multiple client types (mobile/web) needing different data shapes, rapid product iteration. When to use REST: simple CRUD, public APIs, heavy caching requirements, limited client diversity. Many companies use both.

Eventual consistency is a consistency model in distributed systems where, given enough time without new updates, all replicas of the data will eventually converge to the same value. It does NOT guarantee that all reads return the latest write immediately — there is a period of inconsistency after a write during which different replicas may return different values. Contrast with strong consistency: every read reflects the most recent write — reads may be slower (must check all replicas or wait for acknowledgment). Why eventual consistency? The CAP theorem shows that during network partitions, you must choose availability or consistency. AP systems (Cassandra, DynamoDB) choose availability — stay up and accept writes during partitions, with replicas syncing when the partition heals. This enables massive scale and global distribution. Examples of acceptable eventual consistency: social media likes/follower counts (stale by seconds is fine), DNS propagation (updates propagate over hours), shopping cart contents, collaborative document editing (eventually converges). Unacceptable for: bank balance (strong consistency needed), inventory count (must avoid overselling). Techniques for reading recent data in AP systems: read from multiple replicas and take the latest (quorum reads), or read from the primary. BASE model (Basically Available, Soft-state, Eventually consistent) contrasts with ACID as the philosophy behind eventual consistency systems.

A message queue is an asynchronous communication mechanism where producers publish messages to a queue, and consumers process them independently, decoupling the two in time. Core benefits: (1) Decoupling: producers don't know about consumers — add consumers without changing producers; (2) Async processing: accept user requests immediately, process heavy work in background — faster perceived response time; (3) Load leveling: absorb traffic spikes — if 10,000 orders arrive in one second, the queue buffers them and workers process at their natural rate; (4) Reliability: messages persist until consumed — if a worker crashes, the message is redelivered; (5) Fan-out: one message can be consumed by multiple services simultaneously (pub-sub). Use cases: email/notification sending, video transcoding, order processing, image resizing, payment processing, logging pipelines, real-time analytics. Message queue systems: RabbitMQ: AMQP protocol, flexible routing (exchanges and queues), good for task queues; Apache Kafka: distributed log, very high throughput, message retention (consumers can re-read), event streaming; AWS SQS: managed, simple, at-least-once delivery; Redis (BullMQ): lightweight queue on Redis; Google Cloud Pub/Sub. Delivery guarantees: at-most-once (may lose), at-least-once (may duplicate — consumers must be idempotent), exactly-once (harder, requires transactional producers/consumers).

A reverse proxy sits in front of backend servers and intercepts client requests before forwarding them to the appropriate backend server. The client interacts with the proxy, not directly with the backend — the backend server's identity is hidden. Benefits: (1) Load balancing: distribute requests across multiple backend servers; (2) SSL/TLS termination: handle HTTPS at the proxy, communicate with backends in plain HTTP — simpler certificate management, offloads crypto from app servers; (3) Caching: cache responses and serve without hitting the backend; (4) Compression: compress responses (gzip, brotli) before sending to clients; (5) Security: hides backend server IPs, protects against DDoS (absorbs traffic), Web Application Firewall (WAF); (6) Static file serving: serve static assets without hitting the app server; (7) URL rewriting: translate public URLs to internal paths; (8) Authentication: enforce auth at proxy level. vs Forward proxy: a forward proxy sits in front of clients — used for client anonymity, content filtering, corporate internet access control. A reverse proxy protects servers. Examples: Nginx (most common), Apache HTTP Server, HAProxy (high-performance LB), Traefik (microservices-aware), Caddy (automatic HTTPS). In most production setups: client → CDN → Load Balancer/Reverse Proxy → App servers → Databases.

Rate limiting controls how many requests a client can make within a time window, protecting the system from abuse, DDoS, and accidental overload. Algorithms: (1) Token bucket: a bucket holds N tokens, refilled at rate R. Each request consumes a token. If the bucket is empty, the request is rejected. Allows short bursts up to bucket size while maintaining average rate; (2) Leaky bucket: requests fill a fixed-size queue (bucket) processed at a constant rate — smooths out bursts but doesn't allow them; (3) Fixed window counter: count requests in fixed time windows (e.g., per minute). Simple but has edge case — a burst at the boundary of two windows allows 2N requests in 2× the interval; (4) Sliding window log: store timestamps of all requests in a log; count requests in the past [window]. Accurate but memory-intensive; (5) Sliding window counter: hybrid — use fixed window counts but weight by how much of the window has elapsed. Good balance of accuracy and memory. Implementation: typically using Redis atomic operations (INCR + EXPIRE for fixed window; ZADD + ZRANGEBYSCORE for sliding). Rate limit by: IP address, user ID, API key, endpoint. Response: return HTTP 429 Too Many Requests with Retry-After header. Distributed rate limiting: shared Redis ensures consistent limiting across multiple app servers. Use cases: API quotas, login attempt limiting (brute force prevention), scraping protection.

Synchronous communication: the caller sends a request and waits (blocks) for the response before continuing. The two services are temporally coupled — both must be available at the same time. Examples: HTTP/REST API calls, gRPC, database queries. Pros: simple to reason about, immediate response, easy error handling. Cons: tight coupling (if B is slow or down, A waits or fails), latency accumulates across service chains, cascading failures. Asynchronous communication: the caller sends a message and does not wait — it continues processing immediately. The response (if any) arrives later via a callback, event, or polling. Examples: message queues (Kafka, RabbitMQ), event-driven architecture, webhooks. Pros: temporal decoupling (B can be down; messages queue up), better resilience, enables reactive/event-driven patterns, better throughput. Cons: complex error handling (failures are delayed), harder to debug, eventual consistency, requires idempotency. When to use sync: real-time user interactions requiring immediate response (login, payment confirmation), simple read operations, when the caller needs the result to proceed. When to use async: long-running operations (email sending, video processing), fire-and-forget operations, when operations can be retried, when decoupling is important. Common patterns: request-reply async (correlate async response via correlation ID), event-driven (services emit events, others subscribe), CQRS (separate read and write paths).

A Single Point of Failure (SPOF) is any component in a system whose failure causes the entire system to fail. Eliminating SPOFs is fundamental to building highly available systems. Common SPOFs: single database server, single load balancer, single DNS server, single network switch, single power supply, single datacenter. Strategies to eliminate SPOFs: (1) Redundancy: add duplicate components — two load balancers (active-passive or active-active), database primary + replicas, multiple app server instances; (2) Failover: automatic switch to backup when primary fails — DNS failover, load balancer health checks remove failed servers; (3) Geographic redundancy: deploy in multiple datacenters/regions — regional failure doesn't bring down the whole system; (4) No shared state: stateless application servers — any server can handle any request (session data in Redis, not in-memory); (5) Chaos engineering: intentionally kill components to verify the system handles failures (Netflix Chaos Monkey). Availability calculation: system availability = product of all component availabilities for serial components. Two redundant components in parallel: 1 - (1 - A₁) × (1 - A₂). 99.9% + 99.9% in parallel = 99.9999%. Cost vs. availability: eliminating every SPOF is expensive — prioritize based on business impact. A system with 99.9% uptime has ~8.7 hours of downtime per year; 99.99% = 52 minutes; 99.999% = 5 minutes.

High Availability (HA) is the ability of a system to remain operational and accessible for a very high percentage of time, minimizing downtime. Measured as a percentage of uptime: 99.9% = "three nines" (~8.7h downtime/year); 99.99% = "four nines" (~52min/year); 99.999% = "five nines" (~5min/year). HA principles: (1) Eliminate SPOFs: redundant components at every layer (web, app, database, network); (2) Automated failover: system detects failures and reroutes automatically without human intervention — health checks, heartbeats, Kubernetes pod rescheduling; (3) Graceful degradation: when a component fails, serve a reduced-functionality response rather than a total failure — "circuit breaker" pattern; (4) Geographic distribution: multi-AZ (availability zone) deployments within a region; multi-region for disaster recovery; (5) Zero-downtime deployments: rolling updates, blue-green deployments, canary releases — deploy new code without taking the system down; (6) Health checks and self-healing: Kubernetes restarts crashed pods; load balancers remove unhealthy servers. Active-Passive HA: one active server + one standby — simple, wastes standby capacity. Active-Active HA: all servers active, load balanced — no waste, full capacity. HA vs DR (Disaster Recovery): HA prevents downtime from component failures; DR recovers from catastrophic failures (datacenter loss) with acceptable RTO (Recovery Time Objective) and RPO (Recovery Point Objective).

Latency is the time it takes to complete a single operation — from request to response. Measured in milliseconds (ms) or microseconds (μs). Types: network latency (time for data to travel), processing latency (time to compute), database latency (query execution time). Percentiles matter more than averages: p99 latency (the 99th percentile — 99% of requests are faster than this) reveals tail latency that averages hide. A p99 of 500ms means 1% of users wait 500ms — significant at scale. Throughput is the number of operations completed per unit time — requests per second (RPS), transactions per second (TPS), or data volume (GB/s). Measures overall system capacity. Relationship: they are related but different. A system can have: low latency + high throughput (good); low latency + low throughput (fast but limited capacity); high latency + high throughput (slow but handles many concurrent requests via batching). Little's Law: L = λW — average number of requests in the system = arrival rate × average time in system. Doubling throughput with same latency means more concurrent users. Trade-offs: optimizing for latency often reduces throughput (fewer concurrent requests); optimizing for throughput (batching) increases latency. Practical benchmarks: L1 cache reference ~0.5ns; SSD sequential read ~100μs; network round-trip same datacenter ~0.5ms; disk seek ~10ms; cross-continental round-trip ~150ms.

Fault tolerance is the ability of a system to continue operating correctly (possibly at a reduced level) even when some of its components fail. Fault tolerant systems detect, isolate, and recover from failures without manual intervention. Key concepts: (1) Redundancy: N+1 (one spare), N+2 (two spares), 2N (full duplication) redundancy strategies; (2) Replication: data replicated across multiple nodes — any node can serve the data if others fail; (3) Failover: automatic switch to backup component when primary fails — seconds-level RTO; (4) Circuit breaker: stops sending requests to a failing service, preventing cascade failures — returns to normal when service recovers; (5) Retry with backoff: automatically retry failed requests with exponential backoff + jitter; (6) Timeout: don't wait indefinitely for a response — fail fast; (7) Bulkhead: isolate failures so they don't spread — separate thread pools per downstream service (named after ship compartments that prevent entire sinking if one floods); (8) Graceful degradation: serve simplified response when a component is unavailable (show cached data, disable non-critical features). Difference from High Availability: HA focuses on uptime; fault tolerance focuses on correctness during failures. A fault-tolerant system may have brief unavailability but ensures data consistency. Byzantine fault tolerance: handles failures where nodes behave maliciously or send conflicting information — used in blockchain consensus algorithms.

Availability is the percentage of time a system is operational and accessible — usually expressed as uptime percentage (99.9%, 99.99%). It measures whether the system is up right now. A system that crashes every hour but restarts in 1 second has very high availability despite crashing. Formula: Availability = MTTF / (MTTF + MTTR), where MTTF = Mean Time to Failure (average time between failures) and MTTR = Mean Time to Repair (average time to restore service after failure). Improve availability by: reducing failure frequency (better hardware, testing), reducing repair time (automation, monitoring, on-call). Reliability is the probability that a system performs its intended function correctly over a specified period under specified conditions — no failures, no incorrect results, no data corruption. A system can be available (up) but unreliable (returning wrong results). A reliable system produces correct results consistently. Measured by: error rate, success rate, MTTF. Example: a calculator that's always on (100% available) but sometimes returns wrong answers (unreliable). A DNS server that's down for maintenance windows (lower availability) but always returns correct results when up (reliable). Designing for both: reliability (correctness) requires thorough testing, data validation, fault isolation, transactions. Availability requires redundancy, failover, and fast recovery. They often reinforce each other — a reliable system fails less often, improving availability.

A distributed system is a collection of independent computers that appear to users as a single coherent system, coordinating through message passing over a network. The components share state, divide work, and coordinate to achieve a common goal. Characteristics: (1) Components run on separate machines; (2) Communicate via network (unreliable, with latency); (3) No shared clock (cannot synchronize perfectly); (4) Partial failures (some nodes fail while others continue); (5) Concurrent execution. Key challenges: (1) Network unreliability: messages may be lost, delayed, duplicated, or reordered; (2) Partial failure: impossible to distinguish a slow node from a failed one; (3) No global clock: can't tell the exact order of events across machines (use logical clocks — Lamport timestamps, vector clocks); (4) Concurrency: race conditions, deadlocks across nodes; (5) Consistency vs availability trade-off (CAP theorem); (6) Byzantine failures: nodes may behave arbitrarily or maliciously. Fallacies of distributed computing (L. Peter Deutsch): 1. The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4. The network is secure. 5. Topology doesn't change. 6. There is one administrator. 7. Transport cost is zero. 8. The network is homogeneous. Understanding these fallacies guides robust distributed system design.

Idempotency means that performing the same operation multiple times produces the same result as performing it once. An idempotent operation can be safely retried without side effects. In distributed systems, where failures and retries are common, idempotency is critical for correctness. HTTP methods: GET, PUT, DELETE are idempotent (multiple identical requests have the same effect as one); POST is NOT idempotent by default (each POST may create a new resource). Why it matters: in distributed systems, you cannot always know if a request succeeded — the network may fail after the server processed the request but before the response arrived. Without idempotency, retrying a failed payment request could charge the user twice. Implementing idempotency: (1) Idempotency keys: client generates a unique ID per operation; server stores the result keyed by this ID; if the same key is received again, return the stored result without re-processing; (2) Natural idempotency: "SET balance = 100" is idempotent; "ADD 10 to balance" is not; rewrite as "SET balance = current + 10 where balance = current" (compare-and-swap); (3) PUT over POST: use PUT (replace) instead of POST (create) for updates; (4) Idempotent consumers: message consumers must handle duplicate messages (deduplicate using message ID + database unique constraint). Examples: Stripe's idempotency keys for payments, database upsert operations, S3 PUT operations.

The circuit breaker pattern prevents a cascade of failures in a distributed system by stopping requests to a failing service and allowing it time to recover. Named after electrical circuit breakers that trip to protect circuits from overload. Three states: (1) Closed (normal): requests flow through normally; failures are counted. When failures exceed a threshold within a time window, the circuit "trips" to Open; (2) Open: requests fail immediately without calling the downstream service — fast failure, no waiting for timeouts. After a configured timeout, transitions to Half-Open; (3) Half-Open: allows a limited number of test requests through. If they succeed, circuit Closes (service recovered); if they fail, circuit re-Opens. Why it matters: without circuit breaking, a failing service causes all callers to wait for timeouts (e.g., 30s each), exhausting threads/connections and causing cascade failures. The circuit breaker provides immediate "fail fast" response. Configuration parameters: failure threshold (e.g., 50% of requests fail), time window (measure failures over last 30s), timeout in Open state (try recovering after 60s), success threshold to close (3 consecutive successes in Half-Open). Implementations: Netflix Hystrix (deprecated), resilience4j (Java), opossum (Node.js), Polly (.NET). Combined with: retry (with exponential backoff), timeout, bulkhead, and fallback patterns for comprehensive resilience.

ACID properties ensure reliable database transaction processing: (1) Atomicity: a transaction is an all-or-nothing unit. Either ALL operations in the transaction succeed and are committed, or ALL are rolled back. No partial transactions. Example: bank transfer — debit and credit must both happen or neither; (2) Consistency: a transaction brings the database from one valid state to another valid state, respecting all defined rules (constraints, triggers, cascades). Data integrity is always maintained; (3) Isolation: concurrent transactions execute independently — intermediate states are invisible to other transactions. Four isolation levels: Read Uncommitted, Read Committed, Repeatable Read, Serializable. Higher isolation = more consistency, less concurrency; (4) Durability: once a transaction is committed, it persists even if the system crashes. Achieved via write-ahead logging (WAL) — changes are written to a log before being applied. ACID vs BASE: ACID is the philosophy of traditional relational databases (MySQL, PostgreSQL, Oracle). BASE (Basically Available, Soft-state, Eventually consistent) is the philosophy of many NoSQL databases that sacrifice strict consistency for availability and performance. Distributed ACID: ACID across multiple services/databases requires distributed transactions — 2-Phase Commit (2PC) provides this but is slow and has failure modes. Saga pattern provides eventual consistency across services without distributed locks.

Vertical scaling (Scale-up): upgrade a single server to a larger, more powerful machine — add more CPU cores, RAM, faster storage, better network. Pros: simple implementation (no application changes), no distributed systems complexity, lower latency (no network between parts of the system), suitable for databases that are hard to distribute (traditional SQL). Cons: hard ceiling — maximum machine size is limited by available hardware; expensive high-end hardware; single point of failure (if the server dies, the whole service is down); downtime required for hardware upgrade. Horizontal scaling (Scale-out): add more machines (nodes) to a system and distribute the load. Pros: theoretically unlimited — add as many machines as needed; commodity hardware (cheaper); high availability — if one node fails, others continue; cloud-native (easy with auto-scaling groups). Cons: more complex — requires load balancing, stateless application design, distributed data management; network overhead between nodes; data consistency challenges; some operations don't parallelize well. What to scale first: start vertical (simpler), scale horizontal when vertical limits are reached or cost is too high. Most large systems today are horizontal. Stateless design prerequisite: horizontal scaling requires stateless application servers — no user session in local memory. Use Redis for sessions, shared storage for files. The 12-factor app methodology promotes stateless services specifically to enable horizontal scaling.

A CDN reduces latency by serving content from servers geographically closer to the user, reducing the number of network hops and physical distance data must travel. The speed of light imposes a hard limit — data from New York to Tokyo takes ~70ms just for the round-trip light travel time, regardless of server speed. A Tokyo CDN edge serves content in <5ms. Latency reduction mechanisms: (1) Geographic proximity: edge servers in 50-200+ cities worldwide — serve from the nearest one; (2) TCP connection optimization: pre-established persistent connections from edge to origin; edge handles TCP handshake with user (3-way handshake overhead is local); (3) Protocol optimization: CDNs use HTTP/2 or HTTP/3 (QUIC) by default — multiplexing, header compression, faster connection establishment; (4) Reduced origin load: cache hits at edge don't reach origin — origin only handles cache misses; (5) DNS-based routing: CDN DNS returns the IP of the nearest edge node; (6) Anycast routing: same IP address announced from multiple locations — BGP routes to the nearest one. Cache hierarchy: browser cache → CDN edge → CDN regional cache (shield) → origin server. The shield layer reduces origin requests further. Dynamic CDN: CDNs like Fastly and Cloudflare support edge computing (Cloudflare Workers, Fastly Compute@Edge) — run custom code at the edge, enabling dynamic personalization with minimal latency.

Monolithic architecture: all application components (UI, business logic, data access) are in a single codebase, compiled and deployed together as one unit. Pros: simple development and debugging (single codebase), easy local development, simple deployment (one artifact), no network latency between components, simpler transactions (ACID). Cons: as it grows — hard to understand and modify, long build/deploy times, must deploy the whole app for a small change, one bug can crash everything, can't scale individual components (must scale the whole app), technology lock-in. Microservices architecture: application split into small, independent services each with its own codebase, data store, and deployment pipeline. Pros: independent deployment and scaling, technology flexibility per service, fault isolation (one service crash doesn't crash others), smaller codebases easier to understand, team autonomy. Cons: distributed system complexity, network latency between services, distributed transactions (no easy ACID), service discovery, operational overhead (many services to monitor), testing complexity. Migration path: start with a monolith (simpler, faster iteration early on); identify natural boundaries (domain-driven design); extract services as specific components need independent scaling, different technologies, or independent deployment. Strangler fig pattern: gradually migrate monolith to microservices by building the new service alongside the old, routing traffic incrementally.

Service discovery is the mechanism by which services in a microservices architecture find each other. In dynamic cloud environments, service instances start, stop, and change IP addresses constantly — hardcoded IPs don't work. Two patterns: (1) Client-side discovery: client queries a service registry (Consul, etcd, ZooKeeper, Eureka) to get available instances, then load balances and calls directly. Client must know the registry and implement load balancing. Examples: Netflix Eureka with Ribbon; (2) Server-side discovery: client calls a load balancer or API gateway that queries the registry and routes to an appropriate instance. Client doesn't need to know about discovery — simpler client. Examples: AWS ALB, Kubernetes Services. Service registry: a database of service names → available instances (IP:port). Services register on startup, deregister on shutdown, and send heartbeats to indicate health. The registry removes instances that stop sending heartbeats. DNS-based discovery: Kubernetes uses DNS — each service gets a stable DNS name (my-service.namespace.svc.cluster.local); DNS resolves to the service's cluster IP which is load-balanced to pods. Simple but lacks health check detail. Health checks: registry integrates with health checks — only healthy instances are returned. Types: HTTP health check endpoint, TCP connection check, command execution. Tools: Consul (popular, multi-datacenter), AWS Cloud Map, Kubernetes Services + CoreDNS.

Back-pressure is a flow control mechanism where a slower consumer signals an upstream producer to slow down, preventing the consumer from being overwhelmed and crashing. Without back-pressure, a fast producer overwhelming a slow consumer causes memory exhaustion, request queuing, and eventual failure. Analogies: a garden hose nozzle controlling water flow; car traffic congestion spreading back from a bottleneck. Strategies: (1) Blocking/synchronous back-pressure: the producer waits until the consumer is ready — simple but creates tighter coupling (TCP flow control window is an example); (2) Message queue with bounded queue: if the queue is full, the producer blocks or drops/rejects messages — the queue acts as a buffer with a capacity limit; (3) Rate limiting upstream: signal the producer via HTTP 429 (rate limit) or queue depth metrics to reduce throughput; (4) Load shedding: when overwhelmed, drop non-critical requests rather than crashing; (5) Reactive Streams: a standard for async stream processing with non-blocking back-pressure (Java: Project Reactor, RxJava; implemented in Akka Streams). In practice: Kafka consumers control their read rate independently; Node.js streams have pause()/resume() for back-pressure. Back-pressure is a fundamental concept in designing resilient data pipelines, streaming systems, and microservices under variable load.

Push architecture: the server/producer actively sends data to clients/consumers when new data is available. The server initiates the communication. Examples: WebSockets, server-sent events (SSE), webhooks, Kafka push to consumers. Pros: low latency (data delivered immediately when available), no polling overhead, real-time. Cons: server must maintain connections, can overwhelm slow consumers (need back-pressure), requires persistent connection (WebSockets) or pre-registered endpoint (webhooks). Pull architecture: the client/consumer periodically requests data from the server/producer. The consumer initiates the communication. Examples: REST API polling, Kafka consumer pulling from topic, database queries. Pros: consumer controls its processing rate (natural back-pressure), simpler server (stateless), works across firewalls/NAT. Cons: inherent latency (must wait for next poll), wasted requests when no new data (polling overhead). Long polling: hybrid — client sends a request, server holds it open until new data is available, then responds. Lower latency than short polling. When to use each: push when real-time delivery matters and connections are manageable (chat, notifications, live feeds); pull when consumers have varying processing speeds and connection management is complex (batch processing, ETL, Kafka consumers). Most systems use both patterns in different parts of the architecture.

Webhooks are user-defined HTTP callbacks — "reverse APIs" where a server pushes data to a client's URL when a specific event occurs, rather than the client polling for updates. How they work: (1) Client registers a URL with the server (the webhook endpoint); (2) Server stores the URL; (3) When an event occurs, the server makes an HTTP POST request to the registered URL with event data as the request body. Examples: GitHub sends a webhook to your CI server when code is pushed; Stripe sends webhooks when a payment succeeds or fails; Slack bots receive webhooks for slash commands; payment gateways notify merchants of transaction status. Advantages: real-time (no polling), efficient (no wasted requests), decoupled (server doesn't need to know client details). Challenges: (1) Delivery guarantees: if the webhook endpoint is down, events may be lost — need retry logic with exponential backoff; (2) Ordering: webhooks may arrive out of order — use timestamps to sort; (3) Idempotency: same event may be delivered multiple times — use event IDs to deduplicate; (4) Security: verify the webhook came from the expected sender — use HMAC signatures (sender signs payload with shared secret, receiver verifies); (5) Firewall/NAT: the client endpoint must be publicly accessible. Best practices: respond quickly (202 Accepted), process asynchronously, implement signature verification, retry failed deliveries.

Data partitioning (also called data partitioning or sharding) divides a large dataset into smaller, manageable pieces distributed across multiple storage nodes. This enables horizontal scaling of storage and query performance. Types: (1) Horizontal partitioning (sharding): different rows of the same table go to different partitions — user ID 1-1000 on shard 1, 1001-2000 on shard 2. Each shard has the same schema. Most common type; (2) Vertical partitioning: split a table by columns — user profile (ID, name, email) in one partition, user settings (ID, preferences, notifications) in another. Related data accessed together stays together; (3) Functional partitioning: data segregated by functional area — orders data in one cluster, product catalog in another. Similar to microservices data isolation. Partition strategies: range (value ranges), hash (hash function distributes evenly), list (specific values to specific partitions), composite (combination). Considerations: Hotspots: a partition receiving disproportionate load (hash-based helps avoid this); Cross-partition queries: joining data from multiple partitions is expensive — design data model to minimize this; Rebalancing: adding nodes requires moving data — consistent hashing minimizes this; Referential integrity: foreign key constraints across partitions are not enforceable by the database. Partitioning is often the last resort after exhausting vertical scaling, caching, and read replicas.

Data Warehouse: a centralized repository for structured, processed, and curated data optimized for analytical queries (OLAP — Online Analytical Processing). Data is cleaned, transformed, and loaded (ETL — Extract, Transform, Load) before being stored in a highly organized schema (star schema, snowflake schema). Query performance is excellent for predefined analytics. Examples: Snowflake, Amazon Redshift, Google BigQuery, Azure Synapse. Suited for: business intelligence, dashboards, financial reporting. Data Lake: a centralized repository that stores raw, unprocessed data in its native format (structured, semi-structured, unstructured — CSV, JSON, Parquet, images, logs, video). Uses a schema-on-read approach — apply schema when querying, not when storing. Much cheaper storage (S3, HDFS). Examples: AWS S3 + Glue/Athena, Azure Data Lake, Databricks. Suited for: machine learning, data exploration, storing everything for future unknown use cases. Data Lakehouse: hybrid architecture combining the storage flexibility of data lakes with the performance and ACID transactions of data warehouses. Examples: Delta Lake (Databricks), Apache Iceberg, Apache Hudi. Key differences: Data warehouse = structured + schema-on-write + fast analytics + expensive; Data lake = raw + schema-on-read + flexible + cheap + can be slow to query without proper formats. Most modern architectures use both, with data flowing from lake (raw ingestion) to warehouse (refined analytics).

Event-driven architecture (EDA) is a software design pattern where components communicate by producing and consuming events — notifications that something happened — rather than direct synchronous calls. Core components: Event producers: generate events when something happens (user registered, order placed, payment processed); Event broker: receives, stores, and routes events (Kafka, RabbitMQ, AWS EventBridge, SNS/SQS); Event consumers: subscribe to events and react (send welcome email, update inventory, generate invoice). Event types: Domain events: something that happened in the business domain (OrderPlaced, PaymentSucceeded); Integration events: crossing microservice boundaries; Commands vs Events: commands are directed requests ("send email to user 1"); events are facts ("user 1 registered" — anyone can react). Benefits: loose coupling (producers don't know consumers), scalability (consumers scale independently), resilience (consumer failures don't affect producer), extensibility (add new consumers without changing producers), event replay (reprocess past events for new features). Challenges: eventual consistency, complex debugging (trace event chains), event schema evolution, duplicate events (consumers must be idempotent). Patterns: Event Sourcing (store events as the source of truth, derive state), CQRS (separate read/write models), Choreography (services react to events) vs Orchestration (central orchestrator directs services).

Synchronous replication: the primary node waits for at least one replica to acknowledge that it has received and written the data before confirming the write to the client. The client doesn't receive a success response until the data is on multiple nodes. Pros: zero data loss — if the primary fails after acknowledging a write, at least one replica has the data; strong consistency. Cons: higher write latency — must wait for network round-trip to replica + disk write on replica; if the replica is slow or the network is congested, all writes slow down; if the replica crashes, writes may stall (must wait for timeout). Asynchronous replication: the primary acknowledges the write immediately after writing locally, then replicates to replicas in the background. The client gets a faster response. Pros: lower write latency (client waits only for local disk write); primary performance not affected by slow replicas. Cons: potential data loss — if the primary crashes after acknowledging but before replicating, the unconfirmed writes are lost. Replication lag — replicas may be behind the primary. Semi-synchronous replication: MySQL default — at least one replica must acknowledge before the primary commits. Compromise: less data loss risk than async, less latency than fully sync. Choosing: financial systems needing zero data loss → sync; high-throughput systems tolerating small data loss window → async; most production systems → semi-sync or sync with one fast replica.

A URL shortener converts a long URL to a short unique key (e.g., bit.ly/abc123) and redirects users. Requirements: functional: shorten URL, redirect short URL, custom aliases, expiry; non-functional: low latency reads, high availability, ~100M URLs/day. Scale estimation: 100M writes/day = ~1200 writes/sec; assume 10:1 read:write = 12,000 reads/sec; 100M URLs × 500 bytes = 50GB/day storage. URL shortening algorithm: Option 1: hash (MD5/SHA-256) → take first 7 chars → collision possible; Option 2: Base62 encoding of an auto-increment ID (62 chars: a-z, A-Z, 0-9) — 7 chars = 62^7 ≈ 3.5 trillion unique URLs. Better: distributed ID generator (Twitter Snowflake — time-based 64-bit IDs ensuring uniqueness and sortability without coordination). Database: schema: (short_url, original_url, created_at, expires_at, user_id); SQL or NoSQL (DynamoDB — simple key-value pattern). Redirection: 301 (permanent, browser caches — saves server load but can't track clicks) vs 302 (temporary, no browser cache — allows click tracking). Use 302 for analytics. Cache: cache popular short URLs in Redis (80/20 rule — 20% of URLs get 80% of traffic). Cache size: 20% of 100M = 20M URLs × 500 bytes ≈ 10GB. Architecture: client → load balancer → app servers (stateless) → Redis cache → SQL DB; ID generation service (centralized or per-region); CDN for assets.

A rate limiter restricts clients to N requests per time window. Requirements: limit per user/IP/API key; configurable limits per endpoint; accurate distributed limiting across multiple servers; low latency (adding <5ms per request). Algorithm choice: Token bucket: allows bursts up to bucket size, smooth average rate; Sliding window counter: accurate, memory proportional to request count; Fixed window: simple, slight boundary race condition. Token bucket is the most common for API rate limiting. Storage: must be shared across app servers → Redis. Redis provides atomic operations essential for thread-safe counting. Implementation (Fixed Window with Redis): key = "rate_limit:{user_id}:{current_minute}"; INCR key; EXPIRE key 60; if count > limit: reject; Must be atomic — use Lua script or Redis MULTI/EXEC to prevent race conditions. Sliding Window (Redis Sorted Set): ZADD with score = timestamp of each request; ZREMRANGEBYSCORE removes old entries; ZCARD counts current requests. Accurate but O(n) per request. Distributed considerations: race conditions with multiple app servers? Use Redis atomic operations (INCR is atomic). What if Redis is down? Either fail open (allow all requests) or fail closed (deny all) — typically fail open with logging. Response: HTTP 429 with headers: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, Retry-After. Advanced: tiered limits (free vs paid), per-endpoint limits, dynamic limits.

Designing a news feed is one of the most common system design questions. Core challenge: when user A opens their feed, show recent posts from people A follows, ranked by recency (or algorithm). Two approaches: Push (Fan-out on write): when user X posts, immediately push the post to all X's followers' feed caches. Feed reads are fast (just read from the user's pre-computed feed). Cons: celebrities with millions of followers write to millions of caches — too slow for hot users. Pull (Fan-out on read): when user A views their feed, query the latest posts from all A's followees, merge and rank in real-time. Simple writes. Cons: reads are slow (query N followees' posts, sort, deduplicate). Bad for users following thousands of people. Hybrid approach (Twitter/Facebook approach): push for regular users (up to 10K followers); pull for celebrities (defined threshold). Regular users get fast reads via pre-computed feed; celebrity posts are pulled and merged on read. Data model: users table, follows table (follower_id, followee_id), tweets table (tweet_id [Snowflake ID], user_id, content, created_at), feed_cache (user_id → sorted list of tweet_ids, stored in Redis sorted set with score = timestamp). Timeline storage: Redis sorted sets — ZADD timeline:{user_id} {timestamp} {tweet_id}; ZREVRANGE to fetch. Evict old entries (keep last 800 tweets per user). Media: images/videos in S3, served via CDN. Scale: read heavy (100:1 read:write), cache aggressively.

A distributed key-value store stores key-value pairs across a cluster of nodes. Design like Redis, DynamoDB, or Dynamo (Amazon's paper). Requirements: put(key, value), get(key), delete(key); horizontally scalable; fault tolerant (N failures tolerated); configurable consistency. Partitioning: consistent hashing — place nodes on a virtual ring; keys are assigned to the first node clockwise from the key's hash position. Virtual nodes (vnodes) per physical node for even distribution. Replication: replicate each key to the next N nodes clockwise. Replication factor N=3 is common. Any of the N nodes can serve reads/writes. Consistency (CAP trade-off): Quorum reads/writes — W write replicas must acknowledge; R replicas must respond for a read. W + R > N guarantees reading the latest write. Common: N=3, W=2, R=2. Lower W → faster writes, lower consistency. Conflict resolution: last-write-wins (LWW) using timestamps (risk: clocks skew); vector clocks (track causality, detect concurrent writes); CRDTs (conflict-free replicated data types — auto-merge without conflicts). Gossip protocol: nodes periodically exchange state with random neighbors — eventual propagation of membership info without central coordinator. Failure detection via heartbeats + phi accrual failure detector. Read repair: during a read, if replicas disagree, fix inconsistency. Merkle trees: efficiently detect differences between replicas for synchronization.

A notification system delivers messages across multiple channels (push notifications, email, SMS, in-app) to users based on events. Requirements: multiple channel types, high throughput (millions per day), at-least-once delivery, extensible for new channels, user preferences (do not disturb, channel preferences). Architecture: Event sources (payment service, social service, etc.) publish events to a message queue (Kafka); Notification service consumes events, applies user preferences, determines channels; Channel workers (email worker, push worker, SMS worker) each consume from their dedicated queue and deliver via third-party providers. Channels: Push: APNs (iOS), FCM (Android); Email: SendGrid, SES, Mailgun; SMS: Twilio, AWS SNS. Flow: event → Kafka → notification service (look up user preferences + device tokens) → fan out to channel-specific queues → workers call provider APIs → provider delivers to device. User preferences: store in DB: (user_id, channel, enabled, quiet_hours). Check before sending. Rate limiting: don't spam users — limit notifications per user per hour. Delivery tracking: store notification record (id, user_id, channel, status: sent/delivered/read, timestamp). Update status via provider delivery receipts/webhooks. Retry: providers may fail — retry with exponential backoff + dead letter queue for persistent failures. Template engine: store notification templates in DB, render with user-specific data. Unsubscribe/opt-out: honor immediately, store in DB, check before any send.

The CAP theorem states distributed systems can only guarantee two of: Consistency, Availability, Partition Tolerance. Since partition tolerance is non-negotiable in real distributed systems (networks fail), the practical choice is CP vs AP: CP databases (Consistency + Partition Tolerance): During a network partition, the system refuses requests rather than return potentially stale data. All nodes agree or the request fails. Examples and their CP properties: HBase/ZooKeeper: strongly consistent, may be unavailable during partition; MongoDB (majority concern): returns consistent data, unavailable on minority side of partition; Etcd/Consul: Raft consensus ensures strong consistency, minority partitions become unavailable; SQL databases with synchronous replication. Choose when: financial transactions, inventory counts, any case where stale data causes business problems. AP databases (Availability + Partition Tolerance): During partition, all nodes remain available but may return stale data. Examples: Cassandra: highly available, configurable consistency; CouchDB/Couchbase: always accepts reads/writes, merges later; DynamoDB (default): eventually consistent reads. Choose when: social media likes, shopping carts, product catalogs — stale by seconds is acceptable. Real-world nuance: most databases offer tunable consistency (DynamoDB strongly consistent reads, Cassandra quorum reads) — the database isn't fixed to one side of CAP; specific operations can be configured.

Event sourcing stores the state of a system as an immutable, ordered sequence of events (facts) rather than the current state. Instead of "user balance is $100," you store: "UserCreated," "Deposit $50," "Deposit $75," "Withdrawal $25" — and derive the current state by replaying all events. Key properties: (1) Immutable event log: never delete or update events — append only; (2) Event as source of truth: the current state is a projection/snapshot of past events; (3) Temporal queries: "what was the balance on Jan 15?" — replay events up to that date; (4) Full audit trail: every change is recorded with who made it, when, and why; (5) Replay: replay events to a new read model if you need a different view of the data; (6) Snapshots: periodically save the computed state to avoid replaying all events from the beginning. Benefits: complete audit log, temporal queries, easy debugging (replay to reproduce), multiple projections from same events, decouples state from storage, enables CQRS. Challenges: eventual consistency in projections, event schema evolution (old events must be readable by new code), performance of replaying large event streams (mitigated by snapshots), steep learning curve. Combined with CQRS: event sourcing naturally pairs with CQRS — write side stores events, read side projects events into query-optimized models. Used by: banking (every transaction is an event), e-commerce (order state machine), collaborative tools (Google Docs stores operations, not snapshots).

CQRS separates the model for reading data (queries) from the model for writing data (commands). Instead of one model that handles both, CQRS uses separate models optimized for each concern. Command side: handles write operations (Create, Update, Delete commands); validates business rules; updates the write database; publishes events about what changed. The write model is optimized for consistency and business logic. Query side: handles read operations; maintains one or more denormalized, read-optimized projections/views; updated asynchronously from command-side events. Multiple read models can serve different query patterns (a "recent orders" view and a "order analytics" view from the same write model). Benefits: (1) Each model can be independently scaled and optimized; (2) Read models can be highly denormalized for query performance without affecting write model; (3) Multiple read models for different access patterns; (4) Clear separation of concerns; (5) Simpler write model (focused on invariants, not queries). Challenges: eventual consistency — read models lag behind writes; more complexity (two models, synchronization infrastructure); not every system needs this complexity. When to use: high read:write ratio, complex query requirements, different scaling needs for reads/writes, event-driven architecture. Simpler CQRS: just use separate read and write methods/services without separate databases — already provides separation of concerns. Full CQRS with separate databases adds complexity but maximum performance separation.

The Saga pattern manages data consistency across multiple services in a microservices architecture without distributed transactions (2PC). A Saga is a sequence of local transactions, where each step publishes an event/message to trigger the next step, and each step has a compensating transaction to undo its effects if a later step fails. Types: (1) Choreography: each service publishes events and reacts to events from other services autonomously — no central coordinator. Pro: simple, loose coupling. Con: hard to follow the flow, risk of cyclic dependencies. Example: Order service publishes OrderCreated → Payment service processes payment, publishes PaymentProcessed → Inventory service reserves items, publishes InventoryReserved → Shipping service schedules delivery. On failure: each service listens for failure events and triggers compensation (PaymentFailed → Order service marks order cancelled); (2) Orchestration: a central Saga orchestrator explicitly tells each service what to do and reacts to replies. Pro: centralized flow, easy to understand. Con: risk of becoming a god object, more coupling. The orchestrator runs as a separate service/workflow. Compensating transactions: undo a successfully completed step — "cancel reservation," "refund payment." Must be idempotent. Key insight: Sagas provide eventual consistency, not ACID. Steps may briefly be in an inconsistent state. Design for "compensatable" transactions — not all operations can be undone (e.g., an email sent cannot be unsent). Tools: AWS Step Functions, Temporal, Conductor, Axon Framework.

A Write-Ahead Log (WAL) is a disk-based data structure used by databases and storage systems to ensure durability and consistency. The principle: before modifying data in the main storage, first write a description of the change to the WAL. On recovery after a crash, the system replays the WAL to restore data to a consistent state. How it ensures ACID properties: Durability: once a transaction commits, its WAL entry is flushed to durable storage — the data survives a crash; Atomicity: if a crash occurs mid-transaction, WAL entries for uncommitted transactions are discarded on recovery; Consistency: WAL entries contain enough information to redo committed transactions and undo uncommitted ones. WAL in PostgreSQL: all changes to data pages are first written to the WAL (redo log), then asynchronously applied to the actual data files. WAL entries are sequential writes (fast) vs random writes to data pages. This makes writes faster — sequential WAL write + background page update. Streaming replication: PostgreSQL sends WAL records to standby servers in real-time — they replay the WAL to stay in sync. This is the basis for all PostgreSQL replication. Logical decoding: tools like Debezium read PostgreSQL's logical WAL to capture all changes for CDC (Change Data Capture) pipelines to Kafka. WAL is also the foundation of: MySQL binlog, Kafka's log structure, HDFS edit log, Cassandra commit log.

Designing a full-text search system requires understanding inverted indexes and distributed search. Core data structure — Inverted Index: maps each word/term to the list of documents containing it. For "mongodb" → [doc1, doc4, doc7]. Elasticsearch architecture: Index: logical namespace (like a database); Shard: a Lucene instance — each index is divided into primary shards (for scale) and replica shards (for redundancy); Node: a single server in the cluster; Cluster: collection of nodes. Indexing flow: document → tokenization (split into terms) → normalization (lowercase, stemming, stop words) → build inverted index → store in Lucene segment. Search flow: query → parse → route to all relevant shards → each shard searches its local inverted index → merge and rank results (TF-IDF or BM25 scoring). Distributed search: query goes to a coordinator node → broadcasts to all primary shards → shards return top-N local results → coordinator merges and re-ranks → returns top-K global results. Near real-time: Lucene writes new documents to in-memory buffer → flushes to segments periodically (every ~1s) → new docs visible after flush. Relevance scoring: BM25 considers term frequency, inverse document frequency, field length normalization. Design decisions: number of primary shards (set at index creation, can't change later); shard size 20-40GB ideal; replica count for read scaling and HA.

The Two-Phase Commit (2PC) protocol ensures atomic commit across multiple distributed nodes — all commit or all rollback. Used when a transaction spans multiple databases or services. Participants: coordinator (transaction manager) and participants (resource managers — databases). Phase 1 — Prepare: coordinator sends "Prepare" to all participants; each participant executes the transaction locally but doesn't commit — writes to WAL, acquires locks; each participant responds "Yes, I can commit" or "No, I cannot commit." Phase 2 — Commit/Abort: if all participants voted Yes → coordinator sends "Commit" → all participants commit and release locks; if any participant voted No → coordinator sends "Abort" → all participants rollback. Problems with 2PC: (1) Blocking: if the coordinator crashes after Phase 1 but before Phase 2, participants are stuck waiting with locks held — the system is blocked until the coordinator recovers; (2) Latency: two network round trips per transaction — slow; (3) Single point of failure: coordinator failure blocks the system. Alternatives: Saga pattern (eventual consistency, no distributed locks), 3PC (three-phase commit — adds pre-commit phase to avoid blocking, but not widely used), consensus-based protocols (Paxos, Raft — for replicated state machines). When 2PC is used: relational databases (XA transactions), some distributed databases, financial systems where ACID across services is required. The CAP theorem explains why 2PC is unavailable during network partitions.

Raft is a consensus algorithm designed to be more understandable than Paxos, ensuring that a cluster of nodes agrees on a sequence of values even when some nodes fail. Used in: etcd (Kubernetes), CockroachDB, TiKV, Consul. Key concepts: (1) Leader election: one node is the leader — handles all client requests. If the leader fails, a new one is elected. Nodes start as followers; if they don't hear from a leader, they become candidates and request votes; candidate wins with majority votes and becomes the new leader; (2) Log replication: client writes go to the leader; leader appends to its log and sends AppendEntries RPCs to followers; when a majority of followers acknowledge, the entry is "committed" and the leader responds to the client; committed entries are applied to the state machine; (3) Safety: only nodes with up-to-date logs can win elections — ensures committed entries are never lost. Term: Raft time is divided into terms — each election starts a new term. Terms detect stale leaders (a node hearing from a node with a lower term knows it's stale). Guarantees: if N is the cluster size, Raft tolerates ⌊(N-1)/2⌋ failures — majority must be alive. Comparison with Paxos: Paxos is the theoretical foundation (more flexible but complex); Raft is a practical, implementable alternative. Multi-Raft: divide data into many Raft groups, each independently electing leaders — enables distributed databases to scale Raft.

A web crawler systematically browses the web to collect and index content. Requirements: crawl billions of pages, politeness (rate limit per domain), avoid duplicate URLs, prioritize important pages, handle redirects and errors, distributed scalability. Core components: (1) URL Frontier: queue of URLs to visit. Priority queue based on importance (PageRank estimate). Partition by domain to enforce per-domain rate limiting; (2) Fetcher: downloads the page content via HTTP. Respects robots.txt, rate limits, handles redirects. Multiple fetcher workers; (3) Parser: extracts links from HTML, cleans and normalizes URLs (canonicalization — remove tracking params, normalize encoding); (4) Duplicate detection: before adding a URL to the frontier, check if already visited. Use a distributed Bloom filter (space-efficient probabilistic structure) to check in memory; store canonical URL hash in database for definitive check; (5) Content deduplication: same page content from different URLs — use SimHash (locality-sensitive hash) to detect near-duplicates; store content fingerprints; (6) Storage: raw pages → S3 or HDFS; extracted content → Elasticsearch; URL state → distributed database (Cassandra). Politeness: robots.txt parsing; per-domain crawl delay; distributed URL frontier partitioned by domain (each worker owns a domain partition). Distributed coordination: multiple crawler workers, consistent hashing to assign domains to workers. DNS caching: DNS lookups are slow — cache DNS results aggressively.

MapReduce is a programming model for processing large datasets in parallel across a cluster of machines, introduced by Google in 2004. Inspired by functional programming's map and reduce. Two phases: (1) Map phase: input data is split into chunks; each chunk is processed independently on worker nodes by the map function; map emits key-value pairs; (2) Reduce phase: all values with the same key are grouped and sent to a reducer; the reduce function aggregates them into a final output. Example — word count: Map("document") → (word, 1) for each word; Shuffle → group by word; Reduce("word", [1,1,1]) → (word, 3). Execution: master node splits input, assigns map tasks to workers; workers write output to local disk; master assigns reduce tasks, which pull map outputs over the network; reducers write final output. Fault tolerance: if a worker fails, master re-runs its tasks on another worker — possible because map is deterministic. When to use: batch processing of large datasets (logs, user events), ETL pipelines, building inverted indexes, machine learning on large datasets. Modern alternatives: Apache Spark — in-memory processing (10-100x faster for iterative algorithms); much more expressive (SQL, ML, streaming). MapReduce has largely been replaced by Spark in practice. Apache Hadoop YARN still provides the resource management layer. Conceptual value: MapReduce introduced the "move computation to data" principle — process data where it's stored rather than moving data to computation.

A Bloom filter is a space-efficient probabilistic data structure that tests whether an element is a member of a set. It may produce false positives (says "element might be in the set" when it isn't) but never false negatives (if it says "not in set," it definitely isn't). How it works: a bit array of m bits (all initially 0) + k independent hash functions. Insertion: hash the element with each of the k functions → set those k bits to 1. Lookup: hash with k functions → check those k bits. If all k bits are 1 → probably in set (false positive possible). If any bit is 0 → definitely not in set. No deletion: setting bits back to 0 would affect other elements — use Counting Bloom Filter for deletions. Tuning: more bits (larger m) → fewer false positives; more hash functions → tradeoff between false positive rate and insertion speed. Optimal false positive rate depends on m, k, and n (elements inserted). 1% false positive: ~10 bits per element. Applications: (1) Web crawler: check if URL already visited — avoid recrawling; (2) Database query optimization: check if a value might exist before expensive disk lookup; (3) CDN: identify if content is cached at edge; (4) Distributed caching: check if key exists before fetching from DB; (5) Chrome safe browsing: check if URL is in malicious list; (6) Cassandra, HBase: avoid disk reads for non-existent keys. Space advantage: 1 million elements with 1% false positive rate ≈ 1.2MB vs HashSet ≈ 40MB.

Designing Uber-like ride sharing involves real-time location tracking, matching, and pricing. Core flows: driver shares location → rider requests ride → match driver to rider → ride in progress → payment. Location service: drivers send GPS updates every 4-5 seconds. 500K active drivers = 100K-125K location updates/second. Use Cassandra (time-series location: driver_id, timestamp, lat, lng) or Redis geospatial (GEOADD command stores driver positions, GEORADIUS queries nearby drivers in O(n+log n)). Matching service: when rider requests, find available drivers within radius, compute ETA for each, rank by ETA, offer ride to best candidate. Use geospatial indexes (S2 geometry library). Maps service: use third-party for routing/ETA (Google Maps Platform) or build own (HERE, OpenStreetMap + OSRM). ETA calculation crucial for matching and pricing. Surge pricing: real-time supply/demand ratio per geographic cell. Increase price multiplier when demand > supply × threshold. Trip service: state machine: REQUESTED → ACCEPTED → PICKUP → IN_PROGRESS → COMPLETED. Stored in PostgreSQL (ACID transactions for state transitions). Payment service: third-party (Stripe/Braintree). Payment authorized on match, captured on trip completion. Notification service: push notifications for driver-to-rider communication. WebSockets: maintain persistent connections with drivers for real-time location updates and ride assignments. Scale: separate services for location, matching, trips, payments, notifications.

A time-series database (TSDB) is optimized for storing and querying data indexed by time — sequences of data points with timestamps. Unlike general-purpose databases that treat time as just another column, TSDBs have specialized storage engines, compression, and query capabilities designed for temporal data. Characteristics of time-series data: data arrives in time order (monotonically increasing timestamps), high write throughput (thousands of metrics per second), queries often aggregate over time ranges (average CPU last hour), data retention policies (delete data older than N days/months), recent data accessed more than old data. Optimizations in TSDBs: chunk storage by time windows (recent chunks in memory, older chunks compressed on disk), time-based compression (delta encoding for timestamps, XOR compression for values — exploits small differences between consecutive values), efficient range queries with time-based indexing, automatic downsampling (keep raw data for 7 days, hourly aggregates for 1 year), TTL-based retention. Examples: InfluxDB, TimescaleDB (PostgreSQL extension), Prometheus, VictoriaMetrics, Druid, ClickHouse (OLAP, handles time-series well). Use cases: infrastructure metrics (CPU, memory, network), IoT sensor data, financial tick data, application performance monitoring (APM), user analytics (page views per minute), server logs. When NOT to use TSDB: data without a strong time dimension, data that needs complex joins with other data models.

Message queues (RabbitMQ, AWS SQS): designed for task distribution — a message is consumed by ONE consumer and then deleted from the queue. Pull or push delivery. Consumers compete for messages (competing consumer pattern). Messages are typically short-lived (consumed and gone). Use for: task queues (send one email per event), request-response patterns, work distribution among workers. Event streaming platforms (Apache Kafka, AWS Kinesis): designed as a distributed, persistent log — events are retained for a configured retention period (days to forever). Multiple independent consumer groups each read ALL events from their own position (offset) in the log — events are not deleted after consumption. Producers write to the end of the log; consumers read from any offset. Use for: event sourcing, audit logs, real-time analytics, multiple systems consuming the same event stream, event replay, stream processing (Kafka Streams, Apache Flink). Key differences: message queue — messages disappear after consumption, consumers are equivalent (one processes each message); event log — events persist, multiple consumers independently process all events. Kafka mechanics: topics split into partitions (parallel processing); each partition is an ordered, immutable sequence of records; consumers track their offset per partition; consumer groups — each partition is consumed by one consumer in the group (scalability); replication for durability. When to use each: need task distribution with one processor per task → message queue; need multiple systems to react to the same events, event replay, audit trail → Kafka.

HDFS (Hadoop Distributed File System) design principles: store very large files (TB-PB), optimized for throughput over latency, write-once read-many, commodity hardware. Architecture: NameNode (master): stores filesystem metadata — namespace (directory tree, file → block list, block → DataNode list). Holds all metadata in memory for fast access. Single master but with HA standby. DataNodes (workers): store actual data blocks. Report their block list to NameNode on startup (block report) and send periodic heartbeats. File storage: files split into large blocks (128MB default). Each block replicated to 3 DataNodes (configurable). Replica placement: one on local rack, two on a different rack — balances failure tolerance and bandwidth. Write flow: client contacts NameNode → NameNode selects DataNodes, returns pipeline (ordered list) → client writes to first DataNode → first forwards to second → second forwards to third (pipeline replication) → acknowledgments flow back → NameNode records successful replication. Read flow: client asks NameNode for block locations → NameNode returns closest replicas → client reads directly from DataNode. Fault tolerance: NameNode detects DataNode failure via heartbeat timeout → finds under-replicated blocks → schedules replication to restore factor. HA NameNode: active + standby NameNode sharing NFS/QJM (Quorum Journal Manager) for edit log. ZooKeeper for leader election/failover. Erasure coding: newer HDFS versions support erasure coding (like RAID) — better storage efficiency than 3× replication for cold data.

Designing Netflix-scale video streaming involves video processing, CDN, recommendations, and massive scale. Video ingestion pipeline: content team uploads source video → transcoding pipeline converts to multiple formats and resolutions (4K, 1080p, 720p, 480p) × multiple codecs (H.264, H.265/HEVC, AV1) × multiple bitrates (adaptive bitrate streaming) → stored in S3. HEVC/AV1 reduce bandwidth ~50% vs H.264. Adaptive Bitrate Streaming (ABR): video divided into small segments (2-10 seconds each). Player requests an HLS/DASH manifest → selects appropriate bitrate segment based on current bandwidth → if bandwidth drops, switches to lower bitrate seamlessly. This prevents buffering. CDN strategy: Netflix built its own CDN (Open Connect) — deploys servers inside ISP datacenters. The most popular content is pushed to edge servers (proactively cached). ~95% of traffic served from edge. Off-peak hours: pre-position popular content to edge nodes worldwide. Recommendation system: collaborative filtering (users who watched X also watched Y) + content-based filtering + A/B tested ML models. Offline training on Hadoop, online serving via low-latency model serving infrastructure. Database: user data, profiles → MySQL (sharded); viewing history → Cassandra; search → Elasticsearch; real-time events → Kafka → Spark Streaming → updates to all systems. Microservices: hundreds of microservices — each team deploys independently. Chaos Monkey intentionally kills services to test resilience. Multi-region: active-active in multiple AWS regions for global availability.

Consistent hashing with virtual nodes is the gold standard for distributing data across a dynamic cluster. Basic consistent hashing: imagine a ring of hash space [0, 2³²-1]. Each physical node N maps to one position: pos(N) = hash(N_id) mod 2³². Each key K maps to: pos(K) = hash(K) mod 2³². Key K is assigned to the first node clockwise from pos(K). Problem with basic consistent hashing: with few physical nodes, the hash positions are non-uniform — one node might own 40% of the ring while another owns 10% (uneven load). Also, node heterogeneity is unaddressed (powerful nodes should handle more load). Virtual nodes (vnodes): each physical node is mapped to V virtual nodes (V = 100-200 typically): pos(N, i) = hash(N_id + i) for i in [0..V]. The ring now has N×V positions. A key maps to the first vnode clockwise, which maps back to a physical node. Benefits: (1) Load uniformity: as V increases, the distribution of load per physical node converges to equal (law of large numbers — many positions per node averages out); (2) Heterogeneous nodes: powerful nodes get more vnodes; (3) Adding a node: new node steals roughly even shares from all existing nodes (not just neighbors); (4) Removing a node: its vnodes' keys are redistributed across all remaining nodes. Replication: key K is replicated to the next R distinct physical nodes clockwise. Real usage: Cassandra, DynamoDB, Riak, Voldemort all use vnode consistent hashing.

Google Spanner achieves the seemingly impossible: globally distributed, strongly consistent, ACID-compliant SQL database with high availability. Key innovations: TrueTime API: GPS receivers and atomic clocks in every datacenter provide a globally consistent time reference with a bounded uncertainty interval [earliest, latest]. The actual time is guaranteed to be within this interval. Spanner uses TrueTime to assign commit timestamps and guarantee external consistency: if transaction T1 commits before T2 starts, T2 sees T1's writes. Without TrueTime, you'd need Lamport clocks or coordination that adds latency. Architecture: data organized into tablets (contiguous key range shards); tablets stored in Colossus (distributed file system); replicated via Paxos groups across zones; multiple Paxos groups cover different key ranges; Spanner directory (smallest unit of replication — configurable per-database or per-table). Read-write transactions: two-phase locking + 2PC across Paxos groups; commit wait — after acquiring commit timestamp, Spanner waits until TrueTime.now().earliest > commit_timestamp, guaranteeing no future transaction can have an earlier timestamp. Snapshot reads: consistent reads at a specific timestamp — no locks needed, any replica can serve. Extremely scalable. F1 SQL: full SQL with JOINs, subqueries, interleaved table hierarchies (child table co-located with parent — avoids cross-partition joins). Impact: Spanner inspired CockroachDB (open-source, uses hybrid logical clocks instead of TrueTime) and YugabyteDB.

Pessimistic locking assumes conflicts will happen — acquires locks before reading or modifying data, preventing others from touching the data until done. Implemented via: row-level locks (SELECT FOR UPDATE in SQL), distributed locks (Redis SETNX, ZooKeeper ephemeral nodes, etcd). Pros: prevents conflicts entirely, simple application logic (just hold the lock). Cons: reduces concurrency, risk of deadlocks (multiple processes waiting for each other's locks), distributed lock failures (lock holder crashes, lease must expire before lock is released — adds latency). Good for: high contention scenarios, when conflicts are expected, when operations are long and re-doing them is expensive. Optimistic locking assumes conflicts are rare — reads data and remembers a version (version number, timestamp, ETag, or checksum). When writing back, checks if version matches — if someone else has changed it since the read, abort and retry. Implemented via: version column in SQL (UPDATE ... WHERE id=X AND version=Y), HTTP ETags (If-Match header), CAS (Compare-And-Swap) in Redis. Pros: high concurrency (no locks held during read-think-write cycle), no deadlocks, better for read-heavy workloads. Cons: optimistic concurrency conflicts require retry logic, can cause starvation under high contention (always retrying). Good for: low contention, read-heavy, short operations. Distributed lock implementations: Redis RedLock (controversial, uses multiple Redis masters), ZooKeeper ephemeral znodes (strong consistency), etcd leases (Raft-backed, highly reliable). Lease-based locks: lock expires automatically if holder crashes — prevents stale locks.

Google Bigtable (2006) is a distributed NoSQL database optimized for structured data at massive scale. It influenced HBase, Cassandra, DynamoDB. Data model: sparse, distributed, persistent, multidimensional sorted map. Key: (row_key, column_family:column_qualifier, timestamp) → value (bytes). Row keys are sorted lexicographically — enables efficient range scans. Column families group related columns, defined at schema creation. Multiple versions of each cell stored with timestamp. Architecture: Master: assigns tablets to tablet servers, detects tablet server failures, balances load. Not on the critical read/write path (clients cache tablet locations). Tablet servers: each serves multiple tablets (contiguous row ranges, 100-200MB each). Handle reads/writes directly from clients. GFS (Colossus): underlying distributed storage for persistent data. Tablet storage (LSM-Tree): writes go to memtable (in-memory sorted buffer) → periodically flushed to SSTable (sorted string table on GFS) → SSTables compacted periodically (merge sort, write new SSTable) to limit read amplification. Bloom filters per SSTable to quickly determine if a key is absent. Read path: check memtable → check recent SSTables → merge results from multiple SSTables using timestamps. Compaction makes reads faster (fewer SSTables to scan). Locality groups: store frequently-accessed column families together, rarely-accessed separately — avoids reading unnecessary data. Modern equivalent: Cloud Bigtable (managed), HBase (open-source Hadoop ecosystem), Cassandra (inspired by Bigtable + Dynamo).

The PACELC theorem extends the CAP theorem by acknowledging that even in the absence of network partitions, a distributed system must trade off between latency (L) and consistency (C). CAP only addresses the partition scenario; PACELC covers normal operations too. PACELC stands for: "In case of Partition (P), a system must choose between Availability (A) and Consistency (C); Else (E), even when operating normally without partitions, a system must choose between Latency (L) and Consistency (C)." Why latency-consistency trade-off? Achieving strong consistency requires coordination — waiting for all replicas to agree before responding (e.g., quorum write). This coordination adds latency. To minimize latency, respond as soon as the local write is confirmed (eventual consistency). System classifications: PA/EL: choose availability during partition, low latency normally. Examples: Dynamo, Cassandra (default). Very fast responses, eventual consistency always. PC/EC: choose consistency during partition, strong consistency normally. Examples: HBase, ZooKeeper, etcd. Slower responses (coordination required), strongest consistency. PA/EC: available during partition, consistent in normal operation. Examples: MongoDB (some configurations), Cosmos DB (strong consistency mode with partition tolerance trade-off). PC/EL: consistent during partition, low latency normally. Rare. Practical value: PACELC is more useful than CAP for designing systems because it acknowledges that latency/consistency trade-off is constant, not just during the rare partition event. Most systems optimize for the EL side since partitions are infrequent.

A global distributed cache serves low-latency data access across multiple geographic regions. Design like a multi-region Redis cluster. Topology options: (1) Regional caches: independent Redis clusters in each region (US, EU, APAC). Applications read from local region cache — zero cross-region latency for cache hits. Cache invalidation propagated across regions asynchronously. Risk: brief inconsistency between regions; (2) Hierarchical cache: L1 in-process cache (per app instance) → L2 regional Redis → L3 origin database. L1 has lowest latency (microseconds) but small size and no sharing between instances; L3 is slowest but authoritative. Cache invalidation across regions: when data changes, publish an invalidation event to Kafka → consumers in each region invalidate local cache entries. Eventual consistency — brief window where stale data serves. For strong consistency: use cache-aside with short TTLs and version validation. Partitioning: within a region, consistent hashing with vnodes across Redis cluster nodes — supports N nodes with minimal remapping on topology change. Redis Cluster uses hash slots (16384 total) assigned to nodes. Replication within region: each Redis primary has replicas — primary handles writes, replicas serve reads. Replica promotion on primary failure. Data eviction: LRU (Least Recently Used) for general caches; LFU (Least Frequently Used) for access-frequency-based eviction; TTL-based for time-sensitive data. Write strategies per region: write-through (write to both cache and DB) or write-behind (write to cache, async flush). Monitoring: cache hit rate, eviction rate, memory usage, replication lag per region.

A vector clock is a data structure for tracking causality and detecting concurrent events in distributed systems without a centralized clock. Each node maintains a vector of logical timestamps, one per node in the system. Structure: node A maintains VC_A = [A:2, B:1, C:0] — A is at time 2, last known B time is 1, last known C time is 0. Operations: (1) Internal event: increment local component: VC_A[A]++; (2) Send message: increment local component, attach VC to message; (3) Receive message: take element-wise maximum of local VC and received VC, then increment local component. VC_merge[i] = max(VC_local[i], VC_received[i]). Comparing events: event A happened before B (A → B) if VC_A[i] ≤ VC_B[i] for all i AND VC_A ≠ VC_B. Events are concurrent if neither precedes the other — some components larger, some smaller. Conflict detection: two writes are concurrent if their vector clocks are incomparable. The system can detect that a conflict exists and present both versions to the application for resolution. This is better than LWW (last-write-wins with timestamps) which silently discards data. Used in: Amazon DynamoDB (original), Voldemort, Riak. Dotted version vectors: improvement over vector clocks — avoids false conflicts by tracking which events caused each update. CRDTs (Conflict-free Replicated Data Types) go further — data structures designed so concurrent updates always merge without conflict (counters, sets, registers). Examples: G-Counter, OR-Set, LWW-Register. Used in collaborative editing (Google Docs), distributed databases, mobile offline sync.

An autocomplete system suggests completions as users type, with low latency (<50ms). Scale: 5B Google searches/day → 50% type at least 5 chars = 25B trie lookups/day. Core data structure — Trie: prefix tree storing all possible queries. Each node represents a prefix; leaf nodes or marked nodes are complete queries with frequency counts. Traversal from current prefix returns all completions, sorted by frequency. Top-K retrieval: store top K (usually 5-10) completions at each trie node — no need to traverse the entire subtree, just return the cached top-K. Build top-K by propagating from leaves upward during trie construction. Trie limitations at scale: trie too large to fit on one machine → partition by prefix (A-G shard 1, H-P shard 2, Q-Z shard 3). Data pipeline: log search queries → Hadoop/Spark aggregate frequencies weekly (or daily) → build trie from top-N queries → serialize and distribute to search servers. Storage: serialize trie to disk (binary format); load into memory on each search server. Redis also supports sorted sets for prefix matching: ZADD search:prefix:ty {score:frequency, member:query}. Caching: most prefix lookups are concentrated on popular prefixes (80% on 20% of prefixes) → cache top prefixes in L1 cache (browser, CDN, application). Personalization: layer personal search history over global completions — re-rank based on user's own patterns. Real-time updates: use a streaming pipeline (Kafka) to update frequencies for trending queries without full weekly rebuild. Latency: partition queries to appropriate shard → in-memory trie lookup → return top-K. Total: <10ms server-side.

Geo-sharding partitions data by geographic region, ensuring data for users/entities in a specific region stays in that region's data center. Critical for: data residency compliance (GDPR requires EU user data in EU), latency optimization (users read/write local data), disaster isolation. Implementation: (1) Region routing: DNS-based GeoDNS routes users to nearest regional cluster; API gateway in each region routes to regional database; (2) Data model: include region identifier in partition key: user_id = "EU_123456" or shard_key = (region, user_id). All data for a user stays in their region's shard; (3) Cross-region queries: avoided by design — each region is self-contained. If truly needed (global analytics), use async data export to central data warehouse; (4) User migration: when a user moves regions, migrate their data asynchronously while both regions serve the user temporarily. GDPR compliance considerations: right to erasure (delete all EU user data from all systems within 30 days), right to portability (export), data minimization (don't store EU data outside EU). Ensure backups, analytics pipelines, logs also respect geo-boundaries. Global entities: some data is inherently global (product catalog, shared configs). Options: replicate globally (eventual consistency), or designate a home region and cache elsewhere with TTL. Challenges: users traveling between regions, multi-region transactions (user in EU interacting with user in US), global uniqueness of IDs (use global UUID generation service, not auto-increment per region). Tools: Vitess (MySQL) with geo-partitioning, CockroachDB (built-in multi-region support), AWS Aurora Global Database.

A distributed lock service ensures mutual exclusion across processes on different machines — critical for leader election, distributed cron jobs, resource access control. Requirements: mutual exclusion (only one holder at a time), deadlock-free (locks auto-expire if holder crashes), fault tolerant (lock service must be highly available), low latency. Redis-based lock (Redlock): single Redis: SETNX key "lock_value" EX 30 (set if not exists, expire in 30s). Release: check value matches, then DEL (Lua script for atomicity). Problem: if Redis crashes, lock state is lost. Redlock (multi-node): acquire lock on N Redis instances (N=5), succeed if majority (3/5) acquired within timeout. Provides better fault tolerance. Controversy: Martin Kleppmann argued Redlock is unsafe for strong mutual exclusion (clock skew, GC pauses); use fencing tokens (monotonically increasing token, servers reject lower tokens). ZooKeeper-based lock: create ephemeral sequential znode under /locks/my-lock. The node with lowest sequence number holds the lock. Others watch the preceding node. When predecessor deleted (holder released or crashed — ephemeral), the next watches node is notified. Completely reliable for mutual exclusion. ZooKeeper Raft consensus ensures durability. etcd-based lock: similar to ZooKeeper via lease-based grants. Use etcd's Compare-And-Swap: PUT key value --prev-kv (only if current value matches). Leases expire if client crashes. Design principles: use TTL/lease expiration (prevents stale locks), use fencing tokens to prevent use of stale locks even if holder thinks it holds the lock, keep critical sections short.

Zero-downtime deployment updates a running system without users experiencing outage or errors. Critical for systems with strict SLAs. Strategies: (1) Rolling update: replace instances one at a time (or in batches). At any point, some instances run old version, some new. Kubernetes default. Simple but during deployment, old and new code run simultaneously — must be backward compatible. If new version is buggy, roll back by re-deploying old version; (2) Blue-green deployment: maintain two identical environments. Blue = current production, Green = new version. Deploy to Green, run tests, switch load balancer to point to Green. Instant rollback = flip LB back to Blue. Cost: need 2x infrastructure during deployment. Database migrations must be backward compatible (Blue must still work with migrated schema); (3) Canary deployment: route small percentage (1%, 5%) of traffic to new version. Monitor errors/latency. If healthy, gradually increase percentage. Roll back by routing all traffic back to old version. Advanced: route only certain user segments (internal users, beta group) to canary; (4) Feature flags: deploy code with feature disabled, enable flag progressively for user segments. New code is deployed but not activated until ready. Database migrations for zero-downtime: expand-contract (backward compatible) pattern: Phase 1 — add new column with nullable, deploy code that writes to both old and new column; Phase 2 — backfill old data; Phase 3 — deploy code that reads from new column; Phase 4 — drop old column. Never do a breaking schema change while old code is running.