Glyph: Efficient ML-Based Detection of Heap Spraying Attacks

Heap spraying is probably the most simple andeffective memory corruption attack, which fills the memory withmalicious payloads and then jumps at a random location inhopes of starting the attacker’s routines. To counter this threat,GRAFFITIhas been recently proposed as the first OS-agnosticframework for monitoring memory allocations of arbitrary appli-cations at runtime; however, the main contributions of GRAFFITIare on the monitoring system, and its detection engine onlyconsiders simple heuristics which are tailored to certain attackvectors and are easily evaded. In this article, we aim to overcomethis limitation and propose GLY P Has the first ML-based heapspraying detection system, which is designed to be effective,efficient, and resilient to evasive attackers. GLY P Hrelies onthe information monitored by GRAFFITI, and we investigate theeffectiveness of different feature spaces based on informationentropy and memory n-grams, and discuss the several engineer-ing challenges we have faced to make GLY P Hefficient withan overhead compatible with that of GRAFFITI.ToevaluateGLY P H, we build a representative dataset with several variantsof heap spraying attacks, and assess GLY P H’s resilience againstevasive attackers through selective hold-out experiments. Resultsshow that GLY P Hachieves high accuracy in detecting sprayingand is able to generalize well, outperforming the state-of-the-artapproach for heap spraying detection, NOZZLE. Finally, we thor-oughly discuss the trade-offs between detection performance andruntime overhead of GLY P H’s different configurations.