Top 37 Kubernetes (K8s) Interview Questions & Answers (2026)

Kubernetes (K8s) is an open-source container orchestration platform originally developed by Google (based on their internal Borg system) and donated to the CNCF (Cloud Native Computing Foundation) in 2014. It automates deploying, scaling, and managing containerized applications across a cluster of machines. Core capabilities: (1) Self-healing: automatically restarts failed containers, replaces containers on failed nodes, kills containers that don't respond to user-defined health checks; (2) Horizontal scaling: scale applications up/down with commands, UI, or automatically based on CPU/memory usage; (3) Automated rollouts and rollbacks: deploy changes gradually, monitor, and automatically rollback if problems occur; (4) Service discovery and load balancing: expose containers using DNS or IP, load balance across replicas; (5) Storage orchestration: automatically mount storage systems (local, cloud, NFS); (6) Secret and configuration management: store and manage sensitive information; (7) Bin packing: efficiently schedule containers onto nodes based on resource requirements. Key terms: cluster (master + worker nodes), node (machine running containers), pod (one or more containers), workload (deployment, statefulset, daemonset), service (network endpoint for pods). Kubernetes is the de facto standard for container orchestration, supported natively by all major cloud providers (EKS, GKE, AKS).

Kubernetes follows a control plane + data plane architecture: Control Plane (Master): manages the cluster state. Components: (1) kube-apiserver: the front door — exposes the Kubernetes API. All communication goes through it. Validates and persists state to etcd. Horizontally scalable; (2) etcd: distributed key-value store — the single source of truth for cluster state (pod specs, deployments, secrets). Highly available consensus-based. Backup critical; (3) kube-scheduler: watches for unscheduled pods and assigns them to nodes based on resource requirements, affinity, taints/tolerations, policy; (4) kube-controller-manager: runs controller loops that reconcile desired state with actual state. Node Controller (handles node failures), Replication Controller, Endpoints Controller, Service Account Controller; (5) cloud-controller-manager: integrates with cloud provider APIs (provision load balancers, manage volumes, node registration). Worker Nodes (Data Plane): run application workloads. Components: (1) kubelet: agent on every node. Watches for pod specs assigned to the node; starts/stops/monitors containers; reports node and pod status to the control plane; (2) kube-proxy: network proxy on every node. Implements Service networking (iptables or IPVS rules for load balancing); (3) Container Runtime: software that runs containers (containerd, CRI-O, Docker). Cluster communication: all control plane components communicate via the kube-apiserver. Nodes communicate with the API server via TLS-authenticated connections.

A Pod is the smallest deployable unit in Kubernetes — a group of one or more containers that share: network namespace (same IP address, port space), storage volumes, and lifecycle (created, scheduled, and terminated together). Single-container pods: the most common case — one container per pod. Kubernetes manages pods, not containers directly; Multi-container pods: tightly coupled containers that must run together. Common patterns: Sidecar: helper container alongside the main app (log forwarder, proxy, auth sidecar — Istio envoy). Ambassador: proxy local connections to the outside world. Adapter: standardize output of the main container. Pod spec: apiVersion: v1 kind: Pod metadata: name: my-app labels: app: my-app spec: containers: - name: app image: my-app:1.0 ports: - containerPort: 3000 env: - name: ENV value: production resources: requests: memory: "128Mi" cpu: "250m" limits: memory: "256Mi" cpu: "500m" readinessProbe: httpGet: path: /health port: 3000 livenessProbe: httpGet: path: /health port: 3000 volumes: - name: config configMap: name: app-config. Pod lifecycle phases: Pending (scheduled but not running), Running (at least one container running), Succeeded (all containers exited with 0), Failed (at least one container exited non-zero), Unknown. Pods are ephemeral — don't use pods directly, use workload resources (Deployment, StatefulSet).

A Deployment is a higher-level abstraction that manages a set of identical Pods (called a ReplicaSet). It provides declarative updates, rollout strategies, and rollback. Deployment spec: apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: replicas: 3 selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: containers: - name: my-app image: my-app:1.0 ports: - containerPort: 3000 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0. Key features: ReplicaSet ensures N replicas are always running (self-healing); rolling updates with configurable strategy; rollback to previous revision. Kubectl commands: kubectl apply -f deployment.yaml kubectl scale deployment my-app --replicas=5 kubectl set image deployment/my-app my-app=my-app:2.0 kubectl rollout status deployment/my-app kubectl rollout history deployment/my-app kubectl rollout undo deployment/my-app kubectl rollout undo deployment/my-app --to-revision=2. Update strategies: RollingUpdate (default — gradually replace old pods with new ones. maxSurge: extra pods during update, maxUnavailable: pods that can be down); Recreate (kill all old pods before creating new — causes downtime, use for stateful). Deployment vs ReplicaSet vs Pod: always use Deployment (manages ReplicaSets for rollback history). Never create ReplicaSets or Pods directly in production. Pause/resume: kubectl rollout pause/resume deployment/my-app — pause during multi-step updates.

A Service provides a stable network endpoint (DNS name + IP) for accessing a set of Pods, regardless of which pods are running or their IPs (pods are ephemeral — their IPs change). Services use label selectors to identify target pods. Service types: (1) ClusterIP (default): virtual IP accessible only within the cluster. Used for internal service-to-service communication. DNS: my-service.my-namespace.svc.cluster.local; (2) NodePort: exposes service on each node's IP at a static port (30000-32767). Accessible externally: NodeIP:NodePort. Not for production — limited to specific nodes; (3) LoadBalancer: provisions a cloud load balancer (AWS ALB/NLB, GCP Load Balancer). Accessible externally via the LB IP. Standard for production external services. Cost: one LB per service; (4) ExternalName: maps service to an external DNS name (CNAME). No proxying — DNS alias to external service. Service spec: apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: my-app type: ClusterIP ports: - protocol: TCP port: 80 # Service port targetPort: 3000 # Container port. Headless service: clusterIP: None — no virtual IP, DNS returns individual pod IPs. Used with StatefulSets for stable pod DNS. Load balancing: kube-proxy uses iptables or IPVS to distribute traffic across pod endpoints. Session affinity: sessionAffinity: ClientIP.

Namespaces provide logical isolation within a Kubernetes cluster — dividing cluster resources between multiple users, teams, or applications. Objects in different namespaces can have the same name without conflict. Default namespaces: default — resources without a specified namespace; kube-system — Kubernetes system components (kube-dns, kube-proxy, dashboard); kube-public — publicly readable by all users; kube-node-lease — node heartbeat objects. Creating a namespace: kubectl create namespace development kubectl apply -f namespace.yaml. Working with namespaces: kubectl get pods -n development kubectl apply -f deployment.yaml -n development kubectl config set-context --current --namespace=development # set default namespace. Resource quotas per namespace: apiVersion: v1 kind: ResourceQuota metadata: name: dev-quota namespace: development spec: hard: pods: "20" requests.cpu: "4" requests.memory: 8Gi limits.cpu: "8" limits.memory: 16Gi services.loadbalancers: "2". LimitRange: set default resource requests/limits for pods in a namespace — prevents unbounded resource consumption. Network policies: by default, all pods can communicate across namespaces. Network policies restrict this: "pods in namespace A can only communicate with pods in namespace B on port 443." RBAC per namespace: Role (namespaced) and RoleBinding — grant permissions within a specific namespace. ClusterRole/ClusterRoleBinding for cluster-wide access. When to use: environment separation (dev/staging/prod in same cluster), team isolation, multi-tenant clusters.

ConfigMaps store non-confidential configuration data as key-value pairs. Secrets store sensitive data (passwords, tokens, keys) — base64 encoded (not encrypted at rest by default — must enable etcd encryption). ConfigMap creation: kubectl create configmap app-config --from-literal=DB_HOST=localhost --from-literal=LOG_LEVEL=debug --from-file=config.properties kubectl apply -f configmap.yaml. Secret creation: kubectl create secret generic db-secret --from-literal=DB_PASSWORD=mysecretpassword --from-literal=DB_USER=admin. Using ConfigMap as env vars: envFrom: - configMapRef: name: app-config # All keys as env vars env: - name: DB_HOST valueFrom: configMapKeyRef: name: app-config key: DB_HOST. Using Secret as env var: env: - name: DB_PASSWORD valueFrom: secretKeyRef: name: db-secret key: DB_PASSWORD. Mounting as volume files: volumes: - name: config configMap: name: app-config volumeMounts: - name: config mountPath: /etc/config # Creates files named after each key. Security best practices for Secrets: enable encryption at rest (EncryptionConfiguration in API server); use external secret managers (AWS Secrets Manager, HashiCorp Vault) with External Secrets Operator; RBAC to restrict secret access; avoid logging secret values; rotate secrets regularly. Sealed Secrets: encrypt secrets with public key — store in Git safely. Decrypt only inside the cluster by sealed-secrets controller.

A StatefulSet manages stateful applications — pods that require: stable, unique network identifiers; stable, persistent storage; ordered, graceful deployment and scaling. Unlike Deployments (pods are interchangeable), StatefulSet pods have a sticky identity (pod-0, pod-1, pod-2) that persists across restarts. Use cases: databases (MySQL, PostgreSQL, MongoDB, Cassandra), distributed systems (ZooKeeper, Kafka, Elasticsearch), any app requiring stable storage per instance. Key differences from Deployment: stable pod names (pod-0, pod-1, pod-2 — not random hashes); stable DNS per pod: pod-0.service-name.namespace.svc.cluster.local; ordered startup (pod-0 must be Running/Ready before pod-1 starts); ordered scale-down (reverse order); each pod gets its own PersistentVolumeClaim (via volumeClaimTemplates). StatefulSet spec: apiVersion: apps/v1 kind: StatefulSet metadata: name: mysql spec: serviceName: mysql selector: matchLabels: app: mysql replicas: 3 template: spec: containers: - name: mysql image: mysql:8.0 volumeMounts: - name: data mountPath: /var/lib/mysql volumeClaimTemplates: - metadata: name: data spec: accessModes: [ReadWriteOnce] resources: requests: storage: 10Gi storageClassName: gp3. Headless service: StatefulSets require a headless service (clusterIP: None) for DNS-based individual pod addressing. Update strategies: RollingUpdate (default — reverse order), OnDelete (manual trigger by deleting pods).

A DaemonSet ensures that all (or a subset of) nodes run a copy of a Pod. As nodes are added to the cluster, pods are added automatically. As nodes are removed, pods are garbage collected. Use cases: log collection (Fluentd, Filebeat — collect logs from every node's containers); metrics collection (node-exporter — scrape host-level metrics from every node); storage daemons (Ceph, GlusterFS); networking agents (Calico, Flannel, Cilium — run on every node); security agents (Falco, antivirus); monitoring agents (Datadog agent). DaemonSet spec: apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-elasticsearch spec: selector: matchLabels: name: fluentd-elasticsearch template: metadata: labels: name: fluentd-elasticsearch spec: tolerations: - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule containers: - name: fluentd image: fluentd:latest resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log volumes: - name: varlog hostPath: path: /var/log. Node subset: use nodeSelector or nodeAffinity to run only on specific nodes. Update strategy: RollingUpdate (default, with maxUnavailable) or OnDelete. Toleration for control plane: add toleration for control-plane taint to run on master nodes too (useful for system daemons). Kubernetes system components (kube-proxy, CNI plugins) themselves use DaemonSets.

Kubernetes resource management defines how CPU and memory are allocated to containers and how the scheduler places pods on nodes: Requests: the minimum resources the container needs — guaranteed by the scheduler. If a node has fewer resources than requested, the pod won't be scheduled there. Limits: the maximum resources the container can use. CPU is throttled if exceeded. Memory eviction (OOMKilled) if exceeded. Resource units: CPU: cores (1 CPU = 1000m millicores; 250m = 0.25 CPU); Memory: bytes (Mi = mebibytes, Gi = gibibytes). Resource spec: resources: requests: memory: "128Mi" cpu: "250m" limits: memory: "256Mi" cpu: "500m". Quality of Service (QoS) classes: Guaranteed: requests == limits for all containers — highest priority, last to be evicted; Burstable: requests < limits for at least one container — middle priority; BestEffort: no requests or limits — lowest priority, first evicted when node is under memory pressure. LimitRange: namespace-level default requests/limits: kubectl apply -f limitrange.yaml — prevents unbounded resource consumption in a namespace. ResourceQuota: total resource usage limit per namespace. Vertical Pod Autoscaler (VPA): automatically adjusts container resource requests based on actual usage. Best practices: always set requests (for scheduling); set limits (prevent noisy neighbor); monitor actual usage with kubectl top or Prometheus; use VPA recommendations to right-size. Avoid setting CPU limits in many scenarios — throttling can cause latency spikes even when CPU is available.

A Kubernetes Ingress manages external HTTP/HTTPS access to services in the cluster, providing: host-based routing, path-based routing, TLS termination, and load balancing — all with a single entry point (one load balancer for multiple services). Ingress spec: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: my-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / kubernetes.io/ingress.class: nginx spec: tls: - hosts: [api.example.com] secretName: tls-secret rules: - host: api.example.com http: paths: - path: /v1 pathType: Prefix backend: service: name: api-v1 port: number: 80 - path: /v2 pathType: Prefix backend: service: name: api-v2 port: number: 80. Ingress Controllers: Ingress resources require an Ingress controller to implement them. Popular controllers: NGINX Ingress Controller (most common, feature-rich); AWS Load Balancer Controller (ALB per Ingress); Traefik (auto-discovers services); HAProxy; Istio Ingress Gateway; Kong (API gateway features). Install NGINX: helm install nginx-ingress ingress-nginx/ingress-nginx. TLS: store certificate in a Secret → reference in Ingress tls section → automatic HTTPS. Use cert-manager for automatic Let's Encrypt certificates. Annotations: controller-specific behavior: rate limiting, auth, rewrites, timeouts, CORS. IngressClass: multiple Ingress controllers in one cluster — ingressClassName: nginx to select. Gateway API: newer, more expressive successor to Ingress — HTTPRoute, TCPRoute, TLSRoute, GRPCRoute resources. Better multi-team support, more features.

Labels are key-value pairs attached to Kubernetes objects used to organize, select, and filter resources. Selectors match objects by their labels. Labels are fundamental to how Services find pods, Deployments manage ReplicaSets, and scheduling works. Label conventions: metadata: labels: app: my-api environment: production version: "1.0.3" tier: backend owner: platform-team. Common label keys (recommended): app.kubernetes.io/name, app.kubernetes.io/version, app.kubernetes.io/component, app.kubernetes.io/part-of, app.kubernetes.io/managed-by. Label selectors in Services: selector: app: my-api tier: backend — routes traffic to pods with BOTH labels. Equality-based selectors: app=my-api, environment!=staging. Set-based selectors: environment in (production, staging), tier notin (frontend), !experimental. kubectl with selectors: kubectl get pods -l app=my-api,tier=backend kubectl get pods --selector="environment in (production)" kubectl delete pods -l app=old-app. Annotations: different from labels — key-value pairs for non-identifying metadata (deployment notes, build info, external tool config). Not used in selectors. Can contain larger amounts of data. Examples: deployment.kubernetes.io/revision: "3", kubernetes.io/description: "Main API server". Field selectors: filter by resource fields: kubectl get pods --field-selector status.phase=Running,spec.nodeName=node1.

The Horizontal Pod Autoscaler (HPA) automatically scales the number of pod replicas in a Deployment, ReplicaSet, or StatefulSet based on observed metrics (CPU, memory, custom metrics). HPA spec: apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: my-app-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: my-app minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 - type: Resource resource: name: memory target: type: AverageValue averageValue: 200Mi. How it works: every 15 seconds, HPA queries the Metrics Server (or custom metrics API) for current resource usage → computes desired replica count = ceil(currentReplicas × currentMetric / targetMetric) → scales up or down within min/max bounds. Scale-up cooldown: 3 minutes (default) before scaling up again — prevents flapping; Scale-down cooldown: 5 minutes (default) before scaling down. Custom and external metrics: scale on any metric via Prometheus Adapter (HTTP requests/second, queue depth, latency). Example: type: External external: metric: name: sqs_queue_depth selector: matchLabels: queue: orders target: type: AverageValue averageValue: "5". Metrics Server: must be installed in the cluster (not default): kubectl apply -f metrics-server.yaml. KEDA (Kubernetes Event-Driven Autoscaling): extends HPA with 60+ event sources (Kafka, SQS, Redis, Azure Service Bus, Prometheus). Scale to zero when no events. VPA: scale pod resource requests vertically (more CPU/memory) — can't run simultaneously with HPA on same metric.

kubectl is the Kubernetes command-line tool for interacting with Kubernetes clusters. It communicates with the kube-apiserver via REST APIs. Essential commands: kubectl get pods [-n namespace] [-A for all namespaces] [-o wide for more info] kubectl get pods -o yaml # full YAML kubectl get all # all resources in namespace kubectl describe pod my-pod # detailed info including events kubectl logs my-pod [-c container] [-f for follow] [--previous for crashed container] kubectl exec -it my-pod -- /bin/sh # interactive shell kubectl exec my-pod -- env # run command kubectl apply -f deployment.yaml # create/update from file kubectl delete -f deployment.yaml kubectl delete pod my-pod kubectl port-forward pod/my-pod 8080:3000 # local port forwarding kubectl port-forward service/my-service 8080:80 kubectl top nodes # resource usage kubectl top pods kubectl rollout status deployment/my-app kubectl rollout history deployment/my-app kubectl rollout undo deployment/my-app kubectl scale deployment my-app --replicas=5 kubectl set image deployment/my-app my-app=my-app:2.0 kubectl config get-contexts # list cluster contexts kubectl config use-context my-cluster # switch cluster kubectl create secret generic my-secret --from-literal=key=value kubectl create configmap my-config --from-env-file=.env kubectl run debug --image=busybox --rm -it -- /bin/sh # temporary pod kubectl debug pod/my-pod --image=ubuntu -it # ephemeral debug container. kubectl plugins: extend kubectl with plugins (kubectl-neat, kubectl-tree, kubectl-images). krew: kubectl plugin manager.

Kubernetes uses probes to monitor container health and determine if a container is ready to serve traffic: (1) Liveness Probe: determines if the container is running. If it fails, Kubernetes kills and restarts the container. Use for: deadlocked processes that aren't crashing but not making progress. Don't use liveness probes too aggressively — if they fail during startup (before app is ready), it causes restart loops; (2) Readiness Probe: determines if the container is ready to accept traffic. If it fails, the pod is removed from Service endpoints (stops receiving traffic) but NOT restarted. Use for: dependencies not ready (database not connected), warming up caches, temporary unavailability; (3) Startup Probe: delays liveness and readiness probes until the container has started. Prevents premature liveness probe failures during slow startup. Runs until success — then hands off to liveness probe. Probe types: livenessProbe: httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 30 periodSeconds: 10 failureThreshold: 3 timeoutSeconds: 5 readinessProbe: httpGet: path: /ready port: 8080 successThreshold: 1 startupProbe: httpGet: path: /startup port: 8080 failureThreshold: 30 periodSeconds: 10 # Allows 5 min for startup. Probe types: httpGet (HTTP/HTTPS), tcpSocket (TCP connection), exec (command in container, exit 0 = success), grpc. Best practices: readiness ≠ liveness — use different endpoints. Health endpoint should be fast (<100ms). Don't include dependencies in liveness (cascading restarts). Include database connectivity in readiness. Startup probe for slow-starting apps (avoid increasing initialDelaySeconds).

Kubernetes imposes fundamental networking requirements: every pod gets its own IP address; pods on the same node can communicate without NAT; pods across nodes can communicate without NAT; agents on a node can communicate with all pods on that node. This is achieved by the CNI (Container Network Interface) plugin. Pod networking: each pod has a virtual network interface (veth pair). One end in the pod's network namespace, other end in the host namespace. Connected to a bridge. CNI plugin manages routing across nodes. CNI plugins: Calico (network policy + routing, BGP or VXLAN, most popular); Cilium (eBPF-based, high performance, rich network policy, service mesh capabilities); Flannel (simple overlay network, VXLAN, limited network policy); Weave; Canal (Flannel + Calico network policy). Service networking (kube-proxy): ClusterIP services get a virtual IP (cluster-internal only). kube-proxy creates iptables (or IPVS) rules that DNAT traffic from ClusterIP → one of the pod endpoints. DNS: CoreDNS (deployed as a Deployment, every pod configured to use it). my-service.my-namespace.svc.cluster.local → ClusterIP. Network Policies: by default, all pods can communicate with all pods. Network Policies restrict traffic using pod selectors, namespace selectors, and port specifications. Both ingress and egress can be controlled. Requires a CNI plugin that supports Network Policy (Calico, Cilium, Weave). Service Mesh (Istio, Linkerd): adds mTLS between services, traffic management, observability — sidecar proxy (Envoy) injected into each pod. Cilium Mesh provides eBPF-based mesh without sidecar overhead.

Kubernetes RBAC (Role-Based Access Control) regulates access to Kubernetes API resources based on roles assigned to users, groups, or service accounts. Core RBAC objects: (1) Role: namespaced — defines allowed actions (verbs) on resources within a namespace: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: production name: pod-reader rules: - apiGroups: [""] # "" = core API group resources: ["pods", "pods/log"] verbs: ["get", "list", "watch"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list"]; (2) ClusterRole: cluster-wide — same as Role but non-namespaced. For: cluster-wide resources (nodes, PV), cross-namespace access; (3) RoleBinding: binds a Role (or ClusterRole) to subjects (users, groups, serviceaccounts) within a namespace: apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: read-pods namespace: production subjects: - kind: User name: alice apiGroup: rbac.authorization.k8s.io - kind: ServiceAccount name: my-app namespace: production roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io; (4) ClusterRoleBinding: binds ClusterRole to subjects cluster-wide. Service accounts: identity for pods — auto-mounted at /var/run/secrets/kubernetes.io/serviceaccount/. Use RBAC to grant service accounts specific permissions (Secrets access, ConfigMap read, Pod list). kubectl auth can-i: kubectl auth can-i create pods --namespace production --as alice. Principle of Least Privilege: grant minimum required permissions.

Helm is the package manager for Kubernetes — it defines, installs, and upgrades complex Kubernetes applications using packages called charts. Think of it as apt/yum/brew for Kubernetes. Core concepts: Chart: bundle of Kubernetes manifest templates + default values. Published to chart repositories (Artifact Hub, Docker Hub OCI). Release: installed instance of a chart. Multiple releases from the same chart (dev release, prod release). Values: configuration that overrides chart defaults. Common commands: helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update helm search repo nginx helm install my-nginx ingress-nginx/ingress-nginx \ --namespace ingress --create-namespace \ --values values.yaml \ --set controller.replicaCount=3 helm upgrade my-nginx ingress-nginx/ingress-nginx --values values.yaml helm rollback my-nginx 1 helm uninstall my-nginx helm list -A # all releases helm status my-nginx helm get values my-nginx helm get manifest my-nginx helm template my-nginx ingress-nginx/ingress-nginx --values values.yaml # dry run. Chart structure: mychart/ Chart.yaml # metadata values.yaml # default values templates/ deployment.yaml service.yaml _helpers.tpl # named template partials NOTES.txt # post-install notes charts/ # dependencies. Template functions: {{ .Values.image.tag }}, {{ .Release.Name }}, {{ .Chart.Version }}, {{ include "helpers.name" . }}, {{ toYaml .Values.env | nindent 12 }}, {{- if .Values.enabled }}, {{- range .Values.hosts }}. Helm 3 vs 2: Helm 3 removed Tiller (server-side component) — uses Kubernetes RBAC directly. Release state stored in Secrets. OCI registries: helm push mychart-1.0.0.tgz oci://registry/charts. Helmfile: declarative Helm releases management — deploy multiple charts in order with dependencies.

Kubernetes scheduling affinity controls where pods are placed: Node Affinity: prefer or require pods to be scheduled on nodes with specific labels. More expressive than nodeSelector: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: # Hard requirement nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/arch operator: In values: [amd64] preferredDuringSchedulingIgnoredDuringExecution: # Soft preference - weight: 100 preference: matchExpressions: - key: node-type operator: In values: [compute-optimized]. Taints and Tolerations: taints mark a node so pods won't be scheduled there unless they tolerate the taint. Used to dedicate nodes for specific workloads or mark nodes for special conditions. kubectl taint nodes gpu-node dedicated=gpu:NoSchedule # Taint kubectl taint nodes gpu-node dedicated=gpu:NoSchedule- # Remove. Pod toleration: tolerations: - key: dedicated value: gpu operator: Equal effect: NoSchedule. Taint effects: NoSchedule (don't schedule), PreferNoSchedule (avoid), NoExecute (evict existing pods without toleration). Pod Affinity/Anti-affinity: schedule pods based on where OTHER pods are running. Affinity: "schedule my app pods on nodes where Redis pods are running" (reduce latency). Anti-affinity: "spread replicas across nodes" (high availability). affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: my-app topologyKey: kubernetes.io/hostname # Don't schedule on same node. topologySpreadConstraints: modern replacement for pod anti-affinity — evenly spread pods across zones/nodes: topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: my-app.

Kubernetes Network Policies are Kubernetes objects that control traffic flow between pods, using pod/namespace selectors and port specifications. Without Network Policies, all pods can communicate with all other pods cluster-wide — the default is "allow all." Network Policy spec: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: api-network-policy namespace: production spec: podSelector: matchLabels: app: api # Apply to pods with this label policyTypes: [Ingress, Egress] ingress: - from: - namespaceSelector: matchLabels: environment: production # Same namespace - podSelector: matchLabels: role: frontend # From frontend pods - ipBlock: cidr: 203.0.113.0/24 except: [203.0.113.5/32] ports: - protocol: TCP port: 3000 egress: - to: - podSelector: matchLabels: app: postgres ports: - protocol: TCP port: 5432 - to: [] ports: - protocol: TCP port: 443 # Allow HTTPS egress to internet. Default deny-all pattern: create a NetworkPolicy selecting all pods in a namespace with empty ingress/egress rules → blocks all traffic. Then create specific allow policies. This is defense-in-depth. Requirement: Network Policies only work with CNI plugins that enforce them (Calico, Cilium, Weave — NOT Flannel alone). Namespaces and selectors: combine namespaceSelector + podSelector with AND logic (within one from element) vs OR logic (separate from elements). Cilium: extends Network Policy with L7 policy (allow only specific HTTP paths/methods, DNS-based policies).

Kubernetes observability requires metrics, logs, and traces: Metrics: Metrics Server — lightweight metrics pipeline. Enables kubectl top and HPA. Only current metrics (no storage). Prometheus — de facto metrics collection for Kubernetes. Pulls metrics from endpoints every 15s. Long-term storage. PromQL for querying. kube-state-metrics — exposes Kubernetes object metrics (replica counts, deployment status) to Prometheus. Node Exporter — host-level metrics (CPU, memory, disk, network). Prometheus Adapter — converts Prometheus metrics to Kubernetes custom metrics API for HPA. kube-prometheus-stack (Helm chart): installs Prometheus + Alertmanager + Grafana + node-exporter + kube-state-metrics. Standard monitoring stack. Grafana dashboards: pre-built dashboards for Kubernetes (Grafana.com IDs: 3119 for cluster overview, 6417 for pod resources). Alertmanager: handles alerts from Prometheus — routes to PagerDuty, Slack, email based on severity, time, labels. Logging: containers write to stdout/stderr → kubelet captures to node filesystem. Collect with: Fluentd or Fluent Bit (DaemonSet) → forwards to Elasticsearch/OpenSearch, CloudWatch, Datadog, Loki. Loki + Grafana: Loki indexes log labels (not full text), cheap storage — query with LogQL in Grafana. Tracing: Jaeger or Zipkin deployed in cluster. Applications instrumented with OpenTelemetry SDK → send spans to Jaeger. Kubernetes events: kubectl get events --sort-by=.lastTimestamp -n namespace — critical for debugging. Events expire after 1 hour — forward to persistent storage. Resource monitoring commands: kubectl top pods --containers kubectl top nodes kubectl describe node node-1 # Allocated resources, conditions, events.

GitOps is an operational model where Git is the single source of truth for declarative infrastructure and application configuration. Changes are made via Git commits; the cluster automatically reconciles to match the Git state. Benefits: full audit trail (Git history), easy rollback (revert commit), developer-friendly workflow, automated drift correction. Argo CD: declarative, GitOps continuous delivery for Kubernetes. Core concepts: Application (maps Git repo path to Kubernetes cluster namespace); sync (apply Git state to cluster); health (actual cluster state health); out-of-sync (cluster drifts from Git). Features: multi-cluster management, SSO, RBAC, app of apps pattern, ApplicationSet (generate many apps from templates), sync hooks (pre-sync, post-sync), resource health (per-resource custom checks), rollback. UI: visual application topology, sync status, resource health, diff view. CLI: argocd app create my-app --repo https://github.com/org/app --path k8s/ --dest-server https://kubernetes.default.svc --dest-namespace production argocd app sync my-app argocd app get my-app argocd app diff my-app. Flux v2 (GitOps Toolkit): Kubernetes-native GitOps toolkit. Multiple controllers: Source Controller (Git repos, Helm repos, OCI), Kustomize Controller (applies kustomizations), Helm Controller (Helm releases), Notification Controller (alerts). Bootstrap: flux bootstrap github --owner=myorg --repository=fleet-infra --branch=main --path=clusters/production. Argo CD vs Flux: Argo CD has a better UI and multi-cluster management; Flux is more flexible and Kubernetes-native. Both are CNCF projects. Choice depends on team preference.

Kubernetes security hardening across multiple layers: 1. API Server security: enable RBAC (default in modern clusters); disable anonymous auth; restrict API server access (private endpoint, IP allowlist); enable audit logging; use mTLS for component communication. 2. Pod security: Pod Security Admission (PSA) — replaces deprecated PodSecurityPolicy. Three standards: Privileged (no restrictions), Baseline (minimum restrictions), Restricted (hardened). Apply per namespace: kubectl label namespace production pod-security.kubernetes.io/enforce=restricted; Restrict capabilities: securityContext: runAsNonRoot: true runAsUser: 1000 readOnlyRootFilesystem: true allowPrivilegeEscalation: false capabilities: drop: [ALL]. 3. Network segmentation: NetworkPolicies for ingress/egress control; service mesh for mTLS; Cilium for L7 policies. 4. Image security: scan with Trivy, Snyk, or Clair in CI/CD; sign images with cosign; use digest pinning: image: my-app@sha256:abc123; use minimal base images (distroless); Admission Controllers: OPA Gatekeeper or Kyverno — enforce policies (no :latest tag, require resource limits, require labels, no privileged containers). 5. Secrets management: external secrets (AWS Secrets Manager, Vault) via External Secrets Operator or CSI Secrets Store Driver; encrypt etcd at rest; avoid env var secrets for very sensitive data (use volume mounts). 6. Runtime security: Falco — runtime threat detection (detect unexpected syscalls, container escapes, privilege escalation); Tetragon (eBPF-based, lower overhead). 7. Supply chain: SBOM generation; policy enforcement with Sigstore/cosign; dependency vulnerability scanning. CIS Kubernetes Benchmark: kube-bench tool checks cluster against CIS benchmarks.

Kubernetes cluster upgrades require careful planning and execution: Supported version skew: kube-apiserver must be at most 1 minor version ahead of kubelet. Upgrade control plane before worker nodes. Supported upgrade path: one minor version at a time (1.27 → 1.28 → 1.29, not 1.27 → 1.29 directly). Managed K8s (EKS/GKE/AKS): cloud providers handle control plane upgrade. You upgrade node groups separately. Blue/green node group upgrade is safest: create new node group with new K8s version → cordon and drain old nodes → delete old node group. Self-managed cluster upgrade (kubeadm): (1) Upgrade control plane: apt-get update && apt-get install -y kubeadm=1.29.0-00 kubeadm upgrade plan # Verify upgrade path kubeadm upgrade apply v1.29.0 apt-get install -y kubelet=1.29.0-00 kubectl=1.29.0-00 systemctl restart kubelet; (2) Upgrade each worker node: kubectl cordon node-1 # Mark unschedulable kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data # Evict pods # On node-1: apt-get install kubeadm=1.29.0-00; kubeadm upgrade node; apt-get install kubelet=1.29.0-00; systemctl restart kubelet kubectl uncordon node-1 # Mark schedulable. Pre-upgrade checklist: review CHANGELOG for breaking changes and deprecated API removals; test in non-prod first; backup etcd; check addon compatibility (Ingress controller, CSI drivers, CNI plugin); run pluto to detect deprecated API usage; scale up capacity to allow draining nodes. API deprecations: Kubernetes deprecates APIs 2 versions before removal. Use kubectl convert to migrate manifests. Rollback: kubeadm doesn't support downgrade — test upgrades thoroughly in staging first.

A service mesh provides a dedicated infrastructure layer for service-to-service communication, handling: mutual TLS, load balancing, circuit breaking, observability, and traffic management — without application code changes. Architecture: data plane — sidecar proxy (Envoy for Istio, linkerd2-proxy for Linkerd) injected into each pod, intercepts all traffic; control plane — manages sidecar configuration, certificate rotation, observability data. Istio features: mTLS between all services (automatic certificate management); Traffic management — traffic splitting (canary), retries, timeouts, circuit breaking, fault injection; Observability — distributed traces (Jaeger), metrics (Prometheus), access logs; Security — fine-grained authorization policies (allow ServiceA to call ServiceB on /api only); Ingress/egress gateways. Istio traffic management: apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata: name: my-service spec: hosts: [my-service] http: - match: [{headers: {x-canary: {exact: "true"}}}] route: [{destination: {host: my-service, subset: v2}}] - route: - destination: {host: my-service, subset: v1} weight: 90 - destination: {host: my-service, subset: v2} weight: 10. Linkerd: lighter weight, Rust-based proxy (lower latency), simpler to operate, strong security defaults. Less features than Istio. eBPF-based alternatives: Cilium Service Mesh — no sidecar overhead, network-level observability. Linkerd Ambient mode (sidecar-less, similar to Cilium). When to use service mesh: when you need mTLS between services, sophisticated traffic management, or need service-level observability at scale. Avoid for simple setups — adds operational complexity.

Kubernetes has multiple layers of autoscaling: HPA (Horizontal Pod Autoscaler): scales pod replicas within a deployment/statefulset based on metrics (CPU, memory, custom). VPA (Vertical Pod Autoscaler): adjusts container resource requests/limits automatically based on historical usage. Three modes: Off (recommendations only), Initial (apply only on pod creation), Auto (apply and evict pods for resizing). VPA + HPA on same metric causes conflicts — use VPA for CPU/memory, HPA for custom metrics. Cluster Autoscaler (CA): scales the number of nodes in the cluster. Adds nodes when pods are Pending due to insufficient resources; removes nodes when nodes are underutilized (default threshold: 50% for 10 minutes). Works with cloud provider node groups (AWS ASG, GCP MIG, Azure VMSS). Configure: helm install cluster-autoscaler autoscaler/cluster-autoscaler \ --set autoDiscovery.clusterName=my-cluster \ --set awsRegion=us-east-1 \ --set rbac.serviceAccount.annotations."eks.amazonaws.com/role-arn"=arn:aws:iam::123:role/CA. Karpenter (AWS): replacement for Cluster Autoscaler on EKS. Faster (seconds vs minutes), smarter (selects optimal instance type per workload requirements), cost-effective (automatically selects Spot or On-Demand, right instance size). Node provisioner CRD defines allowed instance types, zones, capacity type. Consolidation: actively replaces nodes with cheaper alternatives when underutilized. KEDA (Kubernetes Event Driven Autoscaling): event-driven autoscaling — scales pods to 0 when no work, scales up based on queue depth, Kafka lag, etc. Works as external metrics provider for HPA or as its own CRD (ScaledObject). 60+ scalers available.

The Kubernetes scheduler assigns pods to nodes through a two-phase process: Filtering (Predicate): eliminate nodes that don't satisfy pod requirements. Checks: nodeSelector, nodeAffinity (required), resource availability (requests fit node allocatable), port conflicts (hostPort), volume zone constraints, taints/tolerations, PodTopologySpread. Result: feasible nodes list. Scoring (Priority): rank feasible nodes and pick the best. Plugins: LeastAllocated (prefer nodes with fewer resources used — spreading), MostAllocated (bin-packing — consolidate on fewer nodes), NodeAffinity (preferred), PodTopologySpread (preferred), InterPodAffinity/AntiAffinity, NodeResourcesFit, ImageLocality (prefer nodes already having the container image). Scheduler plugins: Kubernetes scheduler uses a plugin framework (Scheduling Framework). Each scheduling phase has extension points: PreFilter, Filter, PostFilter, PreScore, Score, Reserve, Permit, PreBind, Bind, PostBind. Custom scheduler: write a custom scheduler for specialized placement logic. Specify in pod spec: spec: schedulerName: my-custom-scheduler. Multiple schedulers: run multiple schedulers in the same cluster for different workload types. Scheduler extender: add custom filter/score logic via webhook — call an external HTTP service during scheduling. Gang scheduling: all pods of a batch job must be scheduled simultaneously (AI/ML training, MPI jobs). Volcano or Scheduler framework Coscheduling plugin. Descheduler: rebalances pods after initial scheduling — evicts pods violating affinity constraints, on overutilized nodes, duplicates on same node. Runs periodically as a CronJob. Complement to scheduler (scheduler places pods, descheduler corrects drift).

etcd is the distributed key-value store that stores ALL Kubernetes cluster state — every pod spec, deployment, secret, ConfigMap, RBAC rule, service account. Loss of etcd data = loss of cluster state. etcd uses the Raft consensus algorithm — requires quorum (majority of members) to accept writes. A cluster of 3 members tolerates 1 failure; 5 members tolerate 2 failures. Always deploy an odd number. etcd data layout in Kubernetes: all data under /registry/ prefix. Pod specs: /registry/pods/namespace/podname. Secrets: /registry/secrets/namespace/secretname (should be encrypted at rest). Backup with etcdctl: ETCDCTL_API=3 etcdctl snapshot save snapshot.db \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key etcdctl snapshot status snapshot.db. Restore: etcdctl snapshot restore snapshot.db \ --data-dir=/var/lib/etcd-restored systemctl stop etcd mv /var/lib/etcd /var/lib/etcd-backup mv /var/lib/etcd-restored /var/lib/etcd systemctl start etcd. Backup strategy: backup before cluster upgrades; regular automated backups (hourly for production) to S3/GCS. Managed K8s (EKS, GKE, AKS) handles etcd backups automatically. etcd performance: CPU and disk I/O sensitive. Use SSDs. Dedicated nodes for etcd in large clusters. Monitor: etcd_disk_wal_fsync_duration_seconds (should be <10ms). Velero: application-level backup tool — backs up Kubernetes resources AND persistent volumes. Restore to same or different cluster. Supports S3, GCS, Azure Blob storage.

A Kubernetes Operator is a method of packaging, deploying, and managing a Kubernetes application — specifically stateful applications that require domain knowledge to manage correctly. Operators encode operational expertise into software that automates management of complex applications. Core concept: an Operator extends the Kubernetes API with custom resources (CRDs) and a controller that watches those resources and reconciles actual state to desired state. Example — Database Operator: define a PostgreSQLCluster CRD. Operator watches for PostgreSQLCluster objects → provisions primary + replicas, configures replication, handles failover, manages backups, handles scaling, performs rolling upgrades. The operator knows HOW to manage PostgreSQL; the user just declares WHAT they want: apiVersion: postgres.example.com/v1 kind: PostgreSQLCluster metadata: name: my-db spec: replicas: 3 version: "15" storage: 100Gi backupSchedule: "0 2 * * *". Building operators: Operator SDK: scaffold operator project, generate CRD manifests, Ansible/Helm/Go operators. Kubebuilder: framework for building operators in Go. KUDO: declarative operators with YAML. Level of operator maturity (Capability Model): Level 1: Basic Install; Level 2: Upgrades; Level 3: Full Lifecycle (backup/restore); Level 4: Deep Insights (metrics, alerts); Level 5: Auto Pilot (auto-scaling, self-healing, self-tuning). Production operators: Prometheus Operator (installs Prometheus, Alertmanager via CRDs), Cert-Manager (manages TLS certificates), External Secrets Operator, Argo CD, PostgreSQL operators (CloudNativePG, CrunchyData PGO), Kafka operators (Strimzi, Confluent).

Custom Resource Definitions (CRDs) extend the Kubernetes API with your own resource types. Once a CRD is registered, custom resources (CRs) of that type can be created, read, updated, and deleted like native Kubernetes resources — via kubectl, API, Argo CD, etc. CRD example: apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: websites.example.com spec: group: example.com names: kind: Website plural: websites singular: website shortNames: [ws] scope: Namespaced versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: url: type: string replicas: type: integer minimum: 1 maximum: 10 default: 1 domain: type: string status: type: object properties: availableReplicas: type: integer. Custom Resource: apiVersion: example.com/v1 kind: Website metadata: name: my-website spec: url: https://github.com/myorg/mysite replicas: 3 domain: example.com. Controller reconciliation loop: func (r *WebsiteReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { website := &examplev1.Website{} if err := r.Get(ctx, req.NamespacedName, website); err != nil { return ctrl.Result{}, client.IgnoreNotFound(err) } // Reconcile: create/update Deployment, Service for this Website if err := r.reconcileDeployment(ctx, website); err != nil { return ctrl.Result{}, err } // Update status website.Status.AvailableReplicas = getAvailableReplicas() r.Status().Update(ctx, website) return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil }. Admission Webhooks: validate or mutate resources before persisting. MutatingAdmissionWebhook (modify), ValidatingAdmissionWebhook (validate). Used by cert-manager (mutate to inject certs), Istio (mutate to inject sidecar), OPA Gatekeeper (validate policy).

eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that allows safe execution of sandboxed programs in kernel space without changing kernel source code. Cilium leverages eBPF to provide high-performance networking, security, and observability for Kubernetes. Why eBPF for networking: traditional networking (iptables/kube-proxy): iptables rules are evaluated sequentially — O(n) complexity — slow for thousands of services; no visibility into kernel-level traffic. eBPF: XDP (eXpress Data Path) processes packets at NIC driver level before even reaching the kernel network stack — extremely fast; programmable, can inspect and modify packets; kernel bypass for hot paths. Cilium capabilities: (1) Replacement for kube-proxy: eBPF maps for service load balancing — O(1) lookup vs O(n) iptables. Lower CPU overhead, better performance at scale; (2) L3-L7 Network Policy: not just IP/port-based — allow specific HTTP paths (/api/v1), HTTP methods, DNS names, gRPC methods; (3) Service Mesh without sidecars: Cilium Mesh uses eBPF to provide mTLS, L7 visibility, traffic management at kernel level — no Envoy sidecar per pod; (4) Hubble (observability): real-time network observability using eBPF. Hubble UI: visual flow map between services. CLI: hubble observe --namespace production; (5) Cluster Mesh: connect multiple Kubernetes clusters for cross-cluster service discovery and load balancing without VPN; (6) WireGuard encryption: transparent pod-to-pod encryption using WireGuard in the kernel; (7) Bandwidth Manager: QoS, egress rate limiting per pod; (8) Socket-level load balancing: bypass iptables entirely — process connections at socket level. Cilium is the recommended CNI for new Kubernetes deployments, especially at scale (Google GKE, AWS EKS-A use Cilium).

Advanced StatefulSet patterns for managing stateful applications: 1. Init containers for initialization: clone data, wait for leader, create configuration: initContainers: - name: clone-data image: gcr.io/google-samples/xtrabackup:1.0 command: ["/bin/bash", "-c", "nohup mysqldump --all-databases > /backup/all-databases.sql &"] volumeMounts: - name: data mountPath: /backup. 2. Sidecar containers for replication: xtrabackup sidecar alongside MySQL — handles backup/restore without modifying main container. 3. Pod Management Policy: podManagementPolicy: Parallel — create all pods simultaneously (default: OrderedReady — sequential). Use Parallel for bootstrapping replicas. 4. Update strategy OnDelete: updateStrategy: type: OnDelete — pods only updated when manually deleted. Fine-grained control over database rolling upgrades. 5. Headless service + DNS: each pod addressable at pod-{N}.service.namespace.svc.cluster.local. Configure application to use specific pod DNS for primary (pod-0) vs replicas. 6. VolumeClaimTemplates with StorageClass: each pod gets its own PVC automatically created by StatefulSet. PVCs are NOT deleted when pod is scaled down — data preserved for pod return. Manually delete PVCs if data should be removed. 7. Ordinal-based configuration: MY_POD_INDEX=$(echo $MY_POD_NAME | grep -o "[0-9]*$") — extract ordinal from pod name. Configure pod-0 as primary, others as replicas. 8. Stable network identity for cluster formation: ZooKeeper, Kafka, Cassandra use stable DNS names for initial cluster formation — critical that pod-0.my-service resolves consistently. 9. Graceful scale-down: StatefulSet stops highest-ordinal pod first. Application should drain connections before terminating (preStop hook, gracefulShutdown). 10. CloudNativePG operator: production-grade PostgreSQL on Kubernetes — manages HA primary/replica, automated failover, backup to S3, monitoring — better than bare StatefulSet for databases.

Kustomize is a Kubernetes configuration management tool built into kubectl (no external install needed) that enables template-free customization of Kubernetes YAML manifests. It works by layering "overlays" on top of a "base" configuration using strategic merge patches and JSON patches. Directory structure: ├── base/ │ ├── kustomization.yaml │ ├── deployment.yaml │ └── service.yaml └── overlays/ ├── development/ │ ├── kustomization.yaml │ └── patch-replicas.yaml └── production/ ├── kustomization.yaml └── patch-resources.yaml. Base kustomization.yaml: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml commonLabels: app: my-app. Production overlay: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namePrefix: prod- resources: - ../../base images: - name: my-app newTag: v2.0.0 replicas: - name: my-app count: 10 patchesStrategicMerge: - patch-resources.yaml configMapGenerator: - name: app-config literals: - LOG_LEVEL=error - ENV=production. Key features: namePrefix/nameSuffix: prepend/append to all resource names; images: override image tags without touching YAML; configMapGenerator/secretGenerator: generate ConfigMaps/Secrets from files/literals, auto-hash suffix for rollout; patchesStrategicMerge: merge patch on top of base resources; patchesJson6902: precise JSON patches; vars: substitute values across resources; components: reusable kustomization logic (like Helm library charts). Apply: kubectl apply -k overlays/production/ kubectl diff -k overlays/production/ kustomize build overlays/production/ | kubectl apply -f -. Kustomize + GitOps (Argo CD/Flux) is a standard pattern — each cluster/environment has its own overlay in Git.

Kubernetes disaster recovery encompasses multiple layers: etcd backup (cluster state): most critical — all resource definitions stored here. Backup strategy: scheduled CronJob running etcdctl snapshot save; ship snapshots to S3 with encryption; test restore regularly; for managed K8s (EKS/GKE/AKS) — provider manages etcd, but you still need application data backup. Velero (application backup): open-source tool for backing up and restoring Kubernetes cluster resources and persistent volumes. Features: scheduled backups, on-demand backups, selective backup (by namespace, label selector), cross-cluster restore, migrate to new cluster, disaster recovery. Installation: velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.7.0 \ --bucket my-velero-bucket --secret-file ./credentials-velero \ --backup-location-config region=us-east-1 \ --snapshot-location-config region=us-east-1. Usage: velero backup create daily-backup --include-namespaces production --ttl 720h velero backup get velero restore create --from-backup daily-backup velero schedule create daily --schedule "0 1 * * *" --include-namespaces production. PV backup: Velero uses volume snapshots (cloud provider snapshots via CSI VolumeSnapshot API). For cross-region recovery, ensure snapshots are copied to the target region. RTO/RPO considerations: etcd restore requires cluster downtime; Velero restore requires cluster to exist. Multi-region active-active (no downtime) vs backup-restore (minutes to hours). Infrastructure as Code: if infrastructure defined in Terraform/CDK + manifests in Git, entire cluster can be recreated from code. Combine: IaC for cluster creation + Velero for application state + etcd snapshots for cluster resources. Runbooks: document step-by-step recovery procedures; test regularly with chaos engineering (Chaos Mesh, LitmusChaos).