Tail-Likelihood Reinforcement Learning

This paper introduces Tail-Likelihood Reinforcement Learning (TailRL), a novel optimization framework designed to improve how generative policies handle continuous rewards. Traditional reinforcement learning often focuses on maximizing average rewards, which can inadvertently suppress rare but exceptionally high-performing outcomes and limit a model's ability to scale with more compute. TailRL addresses this by maximizing the log-probability of exceeding diverse reward thresholds, effectively treating a continuous signal as a collection of binary success events. This approach places greater mathematical weight on the upper tail of the reward distribution, ensuring that infrequent, high-quality samples are prioritized during training. Empirical tests across tasks like maze navigation and code optimizationdemonstrate that TailRL prevents suboptimal collapse and significantly boosts performance during inference-time sampling. Ultimately, the method provides a simple, critic-free way to align policy training with the goal of finding the best possible solutions rather than just the most common ones.