System Design Interview: Payment Settlement Batch Processing

Design a batch processing system for end-of-day payment settlement at a payments company that processes 50 million transactions per day. The system must net merchant positions, calculate fees, and initiate fund transfers to merchant bank accounts within a strict bank cutoff window. Walk me through your design, covering reliability, scalability, and how you'd handle failures. Key Takeaways The outbox pattern is the canonical solution to the dual-write problem. Know it cold. Partition keys define ordering and parallelism — choose with intention, usually around the natural aggregation boundary (here, merchant ID). Throttling must be distributed when you have multiple workers — Redis-backed buckets or a sidecar. Idempotency is non-negotiable in payments. Design for at-least-once and dedupe. Retries are tiered: in-process for transient, delay-queue for slower-resolving, DLQ for terminal. Backpressure beats dropping — use Kafka lag as your buffer when downstream is slow. Reconciliation closes the loop — you don't know it worked until ground truth confirms. Corrections are new events — never rewrite history in a financial system.