DEALING WITH NEXT-GENERATION MALWARE

Malicious programs are a serious problem that threatens the security of billions of Internet users. Today's malware authors are motivated by the easy financial gain they can obtain by selling on the underground market the information stolen from the infected hosts. To maximize their profit, miscreants continuously improve their creations to make them more and more resilient against anti-malware solutions. This increasing sophistication in malicious code led to next-generation malware, a new class of threats that exploit the limitations of state-of-the-art anti-malware products to bypass security protections and eventually evade detection. Unfortunately, current anti-malware technologies are inadequate to face next-generation malware. For this reason, in this dissertation we propose novel techniques to address the shortcomings of defensive technologies and to enhance current state-of-the-art security solutions. Dynamic behavior-based analysis is a very promising approach to automatically understand the behaviors a malicious program may exhibit at run-time. However, behavior-based solutions still present several limitations. First of all, these techniques may give incomplete results because the execution environments in which they are applied are synthetic and do not faithfully resemble the environments of end-users, the intended targets of the malicious activities. To overcome this problem, we present a new framework for improving behavior-based analysis of suspicious programs, that allows an end-user to delegate security labs the execution and the analysis of a program and to force the program to behave as if it were executed directly in the environment of the former. Our evaluation demonstrated that the proposed framework allows security labs to improve the completeness of the analysis, by analyzing a piece of malware on behalf of multiple end-users simultaneously, while performing a fine-grained analysis of the behavior of the program with no computational cost for the end-users. Another drawback of state-of-the-art defensive solutions is non-transparency: malicious programs are often able to determine that their execution is being monitored, and thus they can tamper with the analysis to avoid detection, or simply behave innocuously to mislead the anti-malware tool. At this aim, we propose a generic framework to perform complex dynamic system-level analyses of deployed production systems. By leveraging hardware support for virtualization available nowadays on all commodity machines, our framework is completely transparent to the system under analysis and it guarantees isolation of the analysis tools running on top of it. The internals of the kernel of the running system need not to be modified and the whole platform runs unaware of the framework. Once the framework has been installed, even kernel-level malware cannot detect it or affect its execution. This is accomplished by installing a minimalistic virtual machine monitor and migrating the system, as it runs, into a virtual machine. To demonstrate the potentials of our framework we developed an interactive kernel debugger, named HyperDbg. As HyperDbg can be used to monitor any critical system component, it is suitable to analyze even malicious programs that include kernel-level modules. Despite all the progress anti-malware technologies can make, perfect malware detection remains an undecidable problem. When it is not possible to prevent a malicious threat from infecting a system, post-infection remediation remains the only viable possibility. However, if the machine has already been compromised, the execution of the remediation tool could be tampered by the malware that is running on the system. To address this problem we present Conqueror, a software-based attestation scheme for tamper-proof code execution on untrusted legacy systems. Besides providing load-time attestation of a piece of code, Conqueror also ensures run-time integrity. Conqueror constitutes a valid alternative to trusted computing platforms, for systems lacking specialized hardware for attestation. We implemented a prototype, specific for the Intel x86 architecture, and evaluated the proposed scheme. Our evaluation showed that, compared to competitors, Conqueror is resistant to both static and dynamic attacks. We believe Conqueror and our transparent dynamic analysis framework constitute important building blocks for creating new security applications. To demonstrate this claim, we leverage the aforementioned solutions to realize HyperSleuth, an infrastructure to securely perform live forensic analysis of potentially compromised production systems. HyperSleuth provides a trusted execution environment that guarantees an attacker controlling the system cannot interfere with the analysis and cannot tamper with the results. The framework can be installed as the system runs, without a reboot and without loosing any volatile data. Moreover, the analysis can be periodically and safely interrupted to resume normal execution of the system. On top of HyperSleuth we implemented three forensic analysis tools: a lazy physical memory dumper, a lie detector, and a system call tracer. The experimental evaluation we conducted demonstrated that even time consuming analyses, such as the dump of the content of the physical memory, can be securely performed without interrupting the services offered by the system.