Identifying Similar Past Events in a Continuous Monitoring System

Опубликовано 6 сентября 2016, 16:42

Stream processing engines (SPEs) are a new type of data management systems that provide continuous, low latency processing of data streams. These tools are useful in many application domains including computer system monitoring and network monitoring. SPEs focus on processing live data, directly as it arrives into the system. They provide limited or no support for combining live data with historical data. In most monitoring applications, however, when an abnormal event occurs, an administrator must manually examine the current state and the history of the system to understand what happened and diagnose the problem. To facilitate this process, we propose a new technique that automatically compares events on streams and identifies past events similar to newly detected ones.  With our technique, an administrator is shown not only an alert but also past alerts that resemble the current situation. At the heart of our technique is a new similarity measure geared specifically toward the continuous monitoring domain. In this domain, interesting events typically correspond to abnormal situations. Two events are thus most alike when the same monitored objects record similar abnormal values. We show that existing similarity measures do not work well in this environment and we develop a new measure, the Context Distance Measure (CDM, geared specifically toward the monitoring domain. Through experiments with a real dataset from the PlanetLab overlay network, we show that CDM outperforms existing techniques by producing more accurate rankings of similar past events.