Sampling Ad Exchange Events at Scale: What We Kept, What We Dropped, and Why

This story was originally published on HackerNoon at: https://hackernoon.com/sampling-ad-exchange-events-at-scale-what-we-kept-what-we-dropped-and-why. How we reduced Kafka stream volume by 80%, cut Spark workers from 240 to 30, and lowered reporting costs through sampling and executor tuning at Unity. Check more stories related to data-science at: https://hackernoon.com/c/data-science. You can also check exclusive content about #data-engineering, #apache-kafka, #apache-spark, #big-data, #data-pipelines, #adtech, #cloud-cost-optimization, #data-sampling-techniques, and more. This story was written by: @vikrantbachhav. Learn more about this writer by checking @vikrantbachhav's about page, and for more stories, please visit hackernoon.com. At Unity, our ad exchange was processing approximately 10 GB of Kafka event data per second, much of which wasn't needed for downstream reporting. By introducing auction-level sampling, retaining critical events at full fidelity, and optimizing Spark executor configurations, we reduced stream volume by 80%, cut Spark workers from 240 to 30, and lowered DSP reporting transform costs by approximately 41%. Along the way, we uncovered longstanding reporting inaccuracies and learned why understanding data usage is just as important as optimizing infrastructure.