Computer systems have grown in complexity to the point where manual inspection of system behaviour for purposes of malfunction detection have become unfeasible. As these systems output voluminous logs of their activity, machine led analysis of them is a growing need with already several existing solutions. These largely depend on having hand-crafted features, require raw log preprocessing and feature extraction or use supervised learning necessitating having a labeled log dataset not always easily procurable. We propose a two part deep autoencoder model with LSTM units that requires no hand-crafted features, no preprocessing of data as it works on raw text and outputs an anomaly score for each log entry. This anomaly score represents the rarity of a log event both in terms of its content and temporal context. The model was trained and tested on a dataset of HDFS logs containing 2 million raw lines of which half was used for training and half for testing. While this model cannot match the performance of a supervised binary classifier, it could be a useful tool as a coarse filter for manual inspection of log files where a labeled dataset is unavailable.
Anomaly detection from log files using unsupervised deep learning / S. Bursic, V. Cuculo, A. D'Amelio (LECTURE NOTES IN ARTIFICIAL INTELLIGENCE). - In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)[s.l] : Springer, 2020. - ISBN 978-3-030-54993-0. - pp. 200-207 (( convegno 3rd World Congress on Formal Methods, FM 2019 tenutosi a Porto nel 2019.
|Titolo:||Anomaly detection from log files using unsupervised deep learning|
BURSIC, SATHYA (Corresponding)
|Parole Chiave:||Anomaly detection; Deep learning; Log file|
|Settore Scientifico Disciplinare:||Settore INF/01 - Informatica|
Settore ING-INF/05 - Sistemi di Elaborazione delle Informazioni
|Data di pubblicazione:||2020|
|Digital Object Identifier (DOI):||http://dx.doi.org/10.1007/978-3-030-54994-7_15|
|Tipologia:||Book Part (author)|
|Appare nelle tipologie:||03 - Contributo in volume|
File in questo prodotto:
|Bursic2020_Chapter_AnomalyDetectionFromLogFilesUs.pdf||Publisher's version/PDF||Administrator Richiedi una copia|