Attention Is Not All You Need: The Case for Hybrid Architectures

Abstract

Transformer attention scales quadratically with context length, and a decade of workarounds has produced diminishing returns. This paper surveys state-space models, linear attention, and hybrid architectures, and argues that the frontier is shifting from pure attention toward mixed designs that route information through cheaper structured operators. We review the empirical evidence and identify where hybrids currently fail.