HARDWARE-ASSISTED VIRTUALIZATION AND ITS APPLICATIONS TO SYSTEMS SECURITY

In recent years, the number and sophistication of cybercriminals attacks has risen at an alarming pace, and this is not likely to slow down in the near future. To date, security researchers and industry proposed several countermeasures to this phenomenon, and continue to investigate new techniques, in a real arms race against miscreants. Most modern techniques to detect or prevent threats are based on dynamic analysis, that allows to observe the properties and behaviors of software while it runs. Many dynamic approaches are based on virtualization technology. Over the years, indeed, virtualization became the de facto standard environment for the implementation of many dynamic security tools and frameworks. Virtualization has many features that are particularly useful when dealing with systems security. Operating as a hypervisor (i.e., the entity that controls the execution of a system inside a virtual machine), indeed, grants a good degree of transparency and isolation, since the hypervisor is always more privileged than any component running as a guest of a virtual machine. On the contrary, approaches that directly work in the same system of their targets are prone to identification and corruption of their results. Until some years ago, virtualization was uniquely performed via software. Due to the many challenges and intricacies of virtualization, most software hypervisors have lots of prerequisites (e.g., the source code, or binaries, of a system must be modified before it can be run as a guest of a virtual machine). Furthermore, they commonly have bugs, due to the enormous amount of little details that must be handled, and these badly affect transparency and isolation qualities. These pitfalls greatly hinder security systems built on top of software hypervisors. The introduction of an hardware support for virtualization on most commodity CPUs, however, provided a good mean to overcome these limitations. In a strive to contribute to the systems security research field, in this dissertation we show how such hardware support can be leveraged to build tools and frameworks that use dynamic analysis to face some of the many challenges of the field. In more details, we first describe the design and implementation of a generic framework to perform complex dynamic analyses of both user- and kernel-level software, without relying on any native support or any a priori modification of the target. This framework lays the foundation of this dissertation, and on top of it we built the other two contributions: a malware detector and a tool to automatically discover vulnerabilities in Mac OS X kernel modules.