Inference & Intelligence Lab

Two Ways to Measure Demand, and When the Market Lens Matters | EP4: Inference in the Wild

EP4: Two Ways to Measure Demand, and When the Market Lens Matters "Demand" can mean more than one thing. In day-to-day product analytics, it’s the chart right in front of us: sessions, searches, transactions, and conversion rates. This is Funnel Demand—the activity on our own surface. But there is a second, critical lens worth holding alongside it: Market Demand. In this episode of Inference in the Wild, we explore the friction between how data scientists count activity and how economists define markets. By the time a user reaches your website, they aren't a random sample; they are the heavily filtered "website survivors" let through by search engines, recommendation systems, competitors, and targeted ad delivery. If your team is relying solely on the funnel lens to drive high-stakes choices like pricing strategy, you are making decisions on a fundamentally biased population. In this episode, we discuss: Funnel Demand vs. Market Demand: Defining demand as a static count of internal metrics versus an economist's true price-to-quantity curve. The Subscription Pricing Blindspot: Why estimating price sensitivity from your own subscriber data tells you who will churn (funnel lens) but leaves you completely blind to the prospective customers you will never win (market lens). Choosing a Lens is Choosing an Estimand: The critical importance of naming your target population up front—from the inner ring of website visitors to the outer ring of the addressable market. GenAI and the Dark Funnel: How LLMs break traditional market measurement tools by stripping away impression logs, auctions, and visible ranking data. Define the Target Population Up Front: Before running the math, explicitly state your estimand. Are you analyzing the users on your surface, or the whole market that would consider you at a plausible price point? Look Where You Aren't Tracking: When assessing strategic risks, remember that the most price-sensitive people are often the ones who never enter your data because they were never your customers. Prepare for the Invisible Funnel: As discovery goes dark under LLM-mediated filtering, the future data challenge won't just be calculating conversion drops, but understanding the demand we never observed at all. 📖 Read the companion deep dive (with illustrations and takeaways): https://inferenceintel.substack.com/p/are-we-measuring-demand-or-only-the?r=7bs4uy About the Host Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and causal inference. 🤝 Connect with me on LinkedIn: ⁠⁠https://www.linkedin.com/in/linjia/ If you found this episode valuable, please consider: Following the Podcast: Tap the "+" or "Follow" button on Spotify to stay at the cutting edge of measurement strategy. Sharing the Episode: Know a Data Scientist or Product Manager trying to optimize strategy using only internal funnel metrics? Send this their way. Joining the Conversation: Share your thoughts on today’s topic on LinkedIn—let’s raise the standard of the DS craft together.

The Causality Gap: Measuring the True Impact of Voluntary Adoption in Digital Marketplaces

Across the tech industry, many of the most valuable features rely on voluntary adoption. A traveler chooses whether to join a loyalty program, or a marketplace seller decides whether to opt into a smart-pricing tool. Because you cannot force users to adopt a feature, standard A/B tests leave teams with a diluted, flat topline result. Genuinely great features get prematurely killed simply because the bottleneck was an adoption problem, not a product quality problem. In this special episode, we break down The Causality Gap. We expose the structural math flaws that cause standard observational methods (like PSM or regression adjustment) to fail in opt-in scenarios, and reveal how combining Randomized Encouragement Design (RED) with Double Machine Learning (DoubleML) provides a diagnostic map to save your highest-potential features. In this episode, we discuss: The "Opt-In" Trilemma: Why voluntary adoption, extreme user heterogeneity, and finite samples break traditional product feedback loops. The Collider Bias Trap: Why matching or conditioning on post-treatment adoption creates a spurious correlation that breaks your counterfactuals by design. Randomized Encouragement Design (RED): Leveraging randomized nudges as Instrumental Variables to build a clean causal chain reaction. Denoise First, Estimate Second: How DoubleML strips out immense marketplace noise while avoiding regularization bias through cross-fitting. ATT vs. ITT: How decomposing your rollout-level impact from your adopter-level impact tells you exactly whether to iterate on the feature or optimize the funnel. 📖 Read the deep dive on booking.ai medium blogpost (with illustrations and takeaways): https://medium.com/booking-com-data-science/the-causality-gap-measuring-the-true-impact-of-voluntary-adoption-in-digital-marketplaces-ea68b5a35120 About the Host Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience . Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints . A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and causal inference . 🤝 Connect with me on LinkedIn: ⁠⁠https://www.linkedin.com/in/linjia/ 🚀 Support the Craft If you found this episode valuable, please consider: Following the Podcast: Tap the "+" or "Follow" button on Spotify to stay at the cutting edge of measurement strategy. Sharing the Episode: Know a Data Scientist or Product Manager struggling to measure opt-in platform features? Send this their way. Joining the Conversation: Share your thoughts on today’s topic on LinkedIn—let’s raise the standard of the DS craft together.

Build the Camera — How Measurement Design Guides Statistical Testing | EP2: Inference in the Wild

EP2: Build the Camera — Why Measurement Design Trumps Statistical Testing Running a statistical test is simply pressing the shutter. But designing the measurement system? That is building the camera. In this episode, we challenge the industry’s obsession with "which test to run" and shift the focus to what actually matters: whether your metric captures meaningful change in user behavior. We explore why even a successful feature can "fail" a T-test (p=0.34) not because the feature failed, but because the raw metric amplified noise from outliers and suppressed the pattern that mattered. In this episode, we discuss: The Shutter vs. The Camera: Why statistical tests are secondary to how you define and reduce noise in your metrics. The Estimand Trade-off: How common transformations (like Log-Transforms) don't just change the distribution—they fundamentally alter the business question you are answering. Leverage through Design: Why the most successful Data Science teams focus on what to measure rather than just how to test. Case Study: Rank Transformation: Using ranks as a strategic design choice to neutralize outliers while preserving the directional "truth" of your data. Before choosing a statistical test, every practitioner should ask: The Business Question: What do stakeholders actually need to know (e.g., "by how many minutes" or simply "is it better")? Metric Topology: What does the distribution really look like, and where is the noise coming from? The Noise Reduction Strategy: Which approach preserves the "truth" while eliminating the interference of outliers? The Reliability Proof: Does simulation verify that this method achieves 80% power without inflating the False Positive Rate for this specific metric? 📖 Read the companion deep dive (with illustrations and takeaways): https://inferenceintel.substack.com/p/build-the-camera-how-measurement Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and causal inference. 🤝 Connect with me on LinkedIn: ⁠https://www.linkedin.com/in/linjia/ If you found this episode valuable, please consider: Following the Podcast: Tap the "+" or "Follow" button on Spotify to stay at the cutting edge of measurement strategy. Sharing the Episode: Know a Data Scientist frustrated by "insignificant" results on successful features? Send this their way. Joining the Conversation: Share your thoughts on today’s topic on LinkedIn—let’s raise the standard of the DS craft together.

• Transcript

No Interference, No Ambiguity: The SUTVA Assumption ｜ EP7: Causal Inference from the Ground Up

No Interference, No Ambiguity: The SUTVA Assumption Your randomized experiment is clean. The groups are balanced and comparable. The p-value is significant. But behind the scenes, the treatment is leaking. User A shared their referral link with User B in the control group, and suddenly your "independent" comparison is contaminated. Welcome to the most common—and most ignored—failure point in experimentation: SUTVA (The Stable Unit Treatment Value Assumption). As Fisher famously warned, consulting a statistician after a broken experiment is just asking for a "post-mortem". If SUTVA breaks, you get confident numbers that mean absolutely nothing. In this episode, we discuss: The Two Pillars of SUTVA: Why valid experiments require both "No Interference" (no spillovers) and "Consistency" (a well-defined treatment). The Three Mechanisms of Interference: From direct network effects to indirect marketplace competition and systemic behavioral redirection. The Consistency Trap: Why "the feature" can't mean different things for different users—and how to avoid "hidden versions" of your intervention. The Experimental Fix: When to move beyond individual randomization toward Cluster Randomization, Switchbacks, or Synthetic Controls. The Interference Diagnostic (Key Takeaways): Contagion Risks? Randomize at the city or region level to keep social interactions within the treated unit. Marketplace Bottlenecks? Use switchback designs to handle units competing for finite supply. Spatial Shifts? Use buffer zones or synthetic controls to ensure you aren't just claiming credit for demand that simply moved elsewhere. Stop running experiments that leak. Start designing for stability. 📖 Read the companion deep dive (with illustrations and takeaways): https://inferenceintel.substack.com/p/no-interference-no-ambiguity-the About the Host Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and causal inference. 🤝 Connect with me on LinkedIn: https://www.linkedin.com/in/linjia/ 🚀 Support the Craft If you found this episode valuable, please consider: Following the Podcast: Tap the "+" or "Follow" button on Spotify to never miss a deep dive into causal inference and GenAI. Sharing the Episode: Know a Data Scientist or Product Leader struggling with "No Overlap"? Send this their way. Joining the Conversation: Share your thoughts on today’s topic on LinkedIn—let’s raise the standard of the DS craft together.

• Transcript

No Overlap, No Answer: The Positivity Assumption | Ep6: Causal Inference from the Ground Up

No Overlap, No Answer: The Positivity Assumption A causal effect can only be estimated where a comparison is actually possible. Imagine evaluating a loyalty program where every enterprise customer is already enrolled—leaving you with no unenrolled counterparts to compare against. This is a violation of Positivity. While exchangeability requires that groups are comparable, positivity requires that the comparison actually exists. In this episode, we discuss: Structural vs. Random Violations: Why business-logic "zeros" cannot be fixed with more data. The Propensity Score Plot: How to visually verify if your treated and untreated groups cover the same territory. The Trimming Trade-off: Why discarding extreme observations to force overlap changes the population your results apply to. The Positivity Audit (Key Takeaways): Verify Overlap: Use propensity scores to ensure groups share common support. Identify Structural Zeros: Recognize when policy or logic makes receiving a treatment impossible for certain subgroups. Watch External Validity: Always report dropped observations to clarify the narrowed scope of your findings. 🚀 Support the Craft If you found this episode valuable, please consider: Following the Podcast: Tap the "+" or "Follow" button on Spotify to never miss a deep dive into causal inference and GenAI. Sharing the Episode: Know a Data Scientist or Product Leader struggling with "No Overlap"? Send this their way. Joining the Conversation: Share your thoughts on today’s topic on LinkedIn—let’s raise the standard of the DS craft together. 📖 Read the companion deep dive (with illustrations and takeaways): https://open.substack.com/pub/inferenceintel/p/no-overlap-no-answer-the-positivity?r=7bs4uy&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true 🤝 Connect with me on LinkedIn: https://www.linkedin.com/in/linjia/ About the Host Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and observational causal inference.

• Transcript

Comparing Apples to Apples: The Exchangeability Assumption | EP5: Causal Inference from the Ground Up

Comparing Apples to Apples: The Exchangeability Assumption Your dashboard flags a troubling trend: users who contacted customer support have a 40% higher churn rate than those who didn’t. The immediate takeaway seems obvious—support is failing. But is it? Or did those customers contact support because something had already gone wrong? In this episode, we tackle the heart of the "Bad Comparison" problem. We dive into Exchangeability—the fundamental assumption that allows us to treat observational data as if it were a randomized experiment. If your groups aren't exchangeable, your model isn't measuring an effect; it's just measuring a pre-existing difference. In this episode, we discuss: The Support Trap: Why correlation often hides the "underlying fire" and leads to backward business decisions. The "Swap" Test: A simple mental framework to determine if your treatment and control groups are truly comparable. Bias Under the Null: How a model can show a massive "effect" even when the treatment does absolutely nothing. Forcing Exchangeability: The role of conditioning and why choosing the right covariates is the most critical decision a Data Scientist makes. Stop settling for bad comparisons. Start ensuring your data is "exchangeable" before you trust the result. 📖 Read the companion deep dive (with illustrations and takeaways): ⁠https://inferenceintel.substack.com/p/comparing-apples-to-apples-the-exchangeability⁠ 🚀 Support the Craft If you found this episode valuable, please consider: Following the Podcast: Tap the "+" or "Follow" button on Spotify to never miss a deep dive into causal inference and GenAI. Sharing the Episode: Know a Data Scientist or Product Leader struggling with the "Data Validity Cliff"? Send this their way. Joining the Conversation: Share your thoughts on today’s topic on LinkedIn—let’s raise the standard of the DS craft together. About the Host Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and causal inference. 🤝 Connect with me on LinkedIn: https://www.linkedin.com/in/linjia/

• Transcript

The Data You'll Never See: Understanding Potential Outcomes | Causal Inference from the Ground up EP4

You can never see the data you need most to make a decision. 📉 It sounds counterintuitive, but the core of Causal Inference isn't just math—it's imagination. 🌌 Most Data Science focuses on predicting the future based on what happened in the past. But Causal Inference asks a much harder question: What would have happened if we had acted differently? In Part 4 of my series, "Causal Inference from the Ground Up," I dive into the Potential Outcomes Framework—the bedrock of how we define "cause" in a world where we can't see parallel universes. The "Discount Trap" 💸 Imagine your retention team sends a 20% discount to at-risk subscribers. Churn drops. Everyone celebrates. But then comes the uncomfortable question: Were those customers actually going to churn, or were they going to renew anyway? If they were going to stay regardless, you just gave away 20% of your revenue for nothing. This is the Fundamental Problem of Causal Inference. We observe the outcome of the treatment, but the counterfactual—whether that same customer would have renewed without the discount is forever hidden from us. In this deep dive, I break down: The 3 Primitives: Units, Treatments, and Potential Outcomes. Individual Treatment Effect (ITE): Why it’s the "holy grail" of decision-making. The Baking Example: A simple way to visualize unobserved realities. The ATE Solution: How we use population averages to "cheat" the fundamental observation gap. If you’re moving beyond simple prediction and into the world of high-stakes decisions, understanding Potential Outcomes is the first step toward true rigor. 📖 Read the companion deep dive : https://inferenceintel.substack.com/p/the-data-youll-never-see-understanding About the Host Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and observational causal inference. Connect with me: Lin on ⁠⁠⁠LinkedIn: ⁠⁠https://www.linkedin.com/in/linjia/⁠⁠ Join Inference & Intelligence Lab biweekly ⁠⁠https://inferenceintel.substack.com/⁠

• Transcript

Ladder of Causation: How to Upgrade from Prediction to Policy | Causal Inference from the Ground up EP3

Headline: Why your "Perfect" Models are failing the Boardroom. For months, the team worked on the model. The AUC was 0.92. The validation sets were clean. But when you shipped it to the real world? Nothing happened. The metrics didn't move. The business didn't grow. Well, you didn't have a "data" problem. You had a Causality problem. You just hit the Data Validity Cliff—the point where your high-tech models are great at "seeing," but completely blind at "doing." In this episode, we break down Judea Pearl’s Ladder of Causation to fix your strategy: The Selection vs. Policy Trap: Why finding users who love you isn't the same as making users love you. The Owl vs. The Scientist: Why "superpowered observation" is a dead end for decision-making. The UK-to-US Simulation: How to "transport" data across markets without a new pilot. Stop being just an observer. Start becoming a simulator. 📖 Read the companion deep dive (with diagrams & checklists) : ⁠⁠https://inferenceintel.substack.com/p/ladder-of-causation-how-to-upgrade About the Host Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and observational causal inference. Connect with me: Lin on ⁠⁠⁠LinkedIn: ⁠https://www.linkedin.com/in/linjia/⁠ Join Inference & Intelligence Lab biweekly ⁠https://inferenceintel.substack.com/⁠

• Transcript

The Bridge to Truth: Why Identification Comes Before Estimation | Causal Inference from the Ground up EP2

The Infinite Data Trap: Why More Data Won't Save Your Causal Models You have petabytes of user data. Your model has 99% validation accuracy. But when you ask it, "What happens if we change our strategy?", it gives you an answer that is confidently wrong. Welcome to the Infinite Data Trap. In this episode, we reveal why "big data" is useless for decision-making without a critical, often neglected step: Identification. We break down the "Bridge to Causal Truth" and explain why identification is the only thing standing between a reliable insight and misinformation masquerading as truth. In this episode, we discuss: Identification vs. Estimation: Why moving from a "what-if" question to a concrete number requires a bridge of assumptions—not just a better algorithm. The E-commerce Blindspot: A real-world look at how unobserved user engagement can make a promotional email look like a success when it’s actually a bias. Beyond Correlation: How to be explicit about your assumptions (like "no interference") and why sensitivity analysis is your best defense against being "confidently wrong." Stop gathering data. Start identifying effects. 📖 Read the companion deep dive : ⁠ https://inferenceintel.substack.com/p/the-bridge-to-truth-why-identification About the Host Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and causal inference. Connect with me: Lin on ⁠⁠⁠LinkedIn: ⁠⁠https://www.linkedin.com/in/linjia/⁠⁠ Join Inference & Intelligence Lab biweekly ⁠⁠https://inferenceintel.substack.com/⁠

• Transcript

The DoubleML Ranking Disaster: Why PLR Fails for Multiple Discrete Treatment | Inference in the Wild EP1

The Ranking Trap: Why PLR Fails with Multiple Treatments You’re testing four different promotional strategies—a discount, free shipping, BOGO, and loyalty points. You run a Partially Linear Regression (PLR), get a clean table of coefficients, and rank them to decide your next big investment. There’s just one problem: Your ranking is likely a lie. In this episode, we dive into a dangerous "trap" in observational causal inference. We explore why regression-style estimators like PLR don't actually estimate the Average Treatment Effect (ATE) when effects are heterogeneous. Instead, they estimate a variance-weighted average that can completely reverse your treatment rankings. In this episode, we discuss: The Overlap Weighting Problem: Why PLR only "learns" where it has leverage, and how that creates a different subpopulation for every treatment you test. PLR vs. IRM: Why the Interactive Regression Model (IRM) is the essential choice for multiple discrete treatments. The Bridge to DoubleML: How to use machine learning to model "nuisance" components while maintaining statistical rigor. RCT vs. Observational Realities: Why PLR works in a randomized trial but fails in the "wild" of marketplace data. Stop letting overlap redefine your estimand. Start using the right model for the job. 📖 Read the companion deep dive : ⁠ ⁠ https://inferenceintel.substack.com/p/launching-the-inference-and-intelligence About the Host Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and observational causal inference. Connect with me: Lin on ⁠⁠⁠LinkedIn: ⁠⁠https://www.linkedin.com/in/linjia/⁠⁠ Join Inference & Intelligence Lab biweekly ⁠⁠https://inferenceintel.substack.com/⁠

• Transcript

The Core Trio of Causal Inference & The Art of Baking a Causal Cake | Causal Inference from the Ground up EP1

The Recipe for Causal Truth: Estimand, Estimator, and Estimate You have a dataset, a model, and a final number. But can you explain—with precision—what that number actually represents? In the world of causal inference, precision is everything. We often use terms like "the result" or "the algorithm," but failing to distinguish between the What, the How, and the Result is where most analytical errors begin. In this episode, we break down the three fundamental pillars of the estimation process: the Estimand, the Estimator, and the Estimate. Using a simple baking analogy, we demystify these academic terms and turn them into a practical framework for your daily data work. In this episode, we discuss: The Estimand (The "What"): Why defining your theoretical "ideal cake" is the most important step before touching any data. The Estimator (The "How"): How your recipe—the algorithm or function—transforms raw inputs into insights. The Estimate (The "Result"): Understanding the concrete number that comes out of the oven, and why it’s only an approximation of the truth. The Estimation Process: How to align these three trios to ensure your business questions are actually being answered by your code. Don't just run models. Understand the ingredients of your causal claims. 📖 Read the companion blogpost : ⁠⁠⁠ ⁠⁠ https://inferenceintel.substack.com/p/causal-inference-from-the-ground About the Host Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and observational causal inference. Connect with me: Lin on ⁠⁠⁠LinkedIn: ⁠⁠⁠https://www.linkedin.com/in/linjia/⁠⁠⁠ Join Inference & Intelligence Lab biweekly ⁠⁠⁠https://inferenceintel.substack.com/⁠