A Survey on Grokking

The phenomenon of Grokking, first reported by Power et al. [73], has gained significant attention for its unusual characteristics. Grokking is defined by a delay in generalization, during which the training loss reaches near-zero values while the test performance remains poor. This raises a critical question: if not the training loss, what drives generalization? Understanding the mechanisms behind Grokking is essential for gaining insights into the generalization process in broader deep learning contexts. In recent years, this phenomenon has attracted growing interest, with numerous theories and observations proposed to explain it. In this work, we aim to provide a comprehensive review that consolidates the current research and knowledge surrounding Grokking.