FlashAttention

FlashAttention is an IO-aware attention mechanism designed to be fast and memory-efficient, especially for long sequences. Its core innovation is tiling, where input sequences are divided into blocks processed within the fast on-chip SRAM, significantly reducing reads and writes to the slower HBM. This contrasts with standard attention, which materializes the entire attention matrix in HBM. By minimizing HBM access and recomputing the attention matrix in the backward pass, FlashAttention achieves faster Transformer training and a linear memory footprint, outperforming many approximate attention methods that overlook memory access costs.