Kubernetes Deep Dive: Pod Startup and Failure Modes

"Walk me through what happens from kubectl apply to the first request hitting your container." It sounds like a warm-up question, but how you answer it can decide whether you land a senior or staff role. In this deep dive we trace a pod from creation to production and unpack the failure modes interviewers expect you to anticipate. You'll learn: - The full startup chain: API server, etcd, controllers, scheduler, kubelet and container runtime - Why init containers can't run in parallel, and why migrations belong in one for security - Startup, liveness and readiness probes, and the liveness trap that turns a 10-second database blip into an outage - Native sidecars in Kubernetes 1.29 (KEP-753) and the startup and shutdown races they fix - HashiCorp Vault: mutating webhook injection, TokenReview and system:auth-delegator, and dynamic database credentials - Zero trust with mTLS, SPIFFE identities and JWTs, plus the alg: none attack and audience checks - Service DNS, ndots:5, ClusterIP "phantom" IPs, iptables vs IPVS, and headless services for gRPC - Istio traps: DestinationRule ordering, empty AuthorizationPolicies and a stray YAML hyphen - Helm secrets, migration hooks, and a five-phase rollout for breaking API changes - A layered runbook for debugging intermittent 503s Chapters 00:00 Introduction 00:22 What interviewers are really listening for 01:31 From kubectl apply to "success" 03:06 The scheduler and the kubelet 04:18 Init containers 05:26 Why migrations belong in an init container 06:34 Startup, liveness and readiness probes 08:09 The liveness probe thundering herd trap 10:04 What a liveness probe should check 10:53 Startup probes for slow-booting apps 11:40 The sidecar startup and shutdown race 12:52 Native sidecars in Kubernetes 1.29 13:39 Vault injection with a mutating webhook 14:48 Vault's init and sidecar agents 15:34 Vault authentication and TokenReview 16:43 Dynamic database credentials 17:55 A Postgres quoting trap 18:44 Zero trust and mutual TLS 19:52 SPIFFE identities in Istio policies 20:17 mTLS vs JWT 20:39 Istio policy traps 21:46 The alg: none attack and audience checks 23:20 Service DNS and the phantom ClusterIP 24:31 iptables vs IPVS and headless services 25:41 Istio routing and deployment order 26:29 The YAML hyphen trap 26:53 Helm versions and secrets 27:41 Helm migration hooks 28:04 Five phases for a breaking API change 29:14 Debugging 503s step by step 30:00 Matching symptoms to causes 30:47 Final thought: abstractions leak #Kubernetes #DevOps #SRE #CloudNative #Istio #SystemDesign #InterviewPrep Interview Prep Podcast