Extensible Overlay Networks for Stream Processing and Dissemination

Опубликовано 6 сентября 2016, 17:09

The confluence of ubiquitous, high-performance networking and increased availability of online information has led to the emergence of a new class of large-scale stream processing and dissemination applications. These applications often exhibit diverse logic and performance requirements, yet they all require common facilities, which include construction of an overlay network, routing and processing logic, and membership management. In contrast to existing approaches that provide custom, point solutions to point applications, we introduce XPORT, a general-purpose middleware infrastructure that provides these core functionalities and can be easily extended for a broad spectrum of target applications. Extensibility is the central design consideration for XPORT, which can be customized to support diverse processing logic, stream types, and performance targets through a set of methods that encapsulate application-specific behavior and a cost model for defining the desired QoS and resource-utilization metrics and constraints. Given these specifications, the system automatically creates and optimizes a data-stream acquisition, processing and dissemination overlay network. Its optimization is driven by metric-independent operations, which can refine the structure of the overlay network and the statistics collection process, as well as efficiently distribute processing across the network. In this talk, I will describe the basic concepts and models used by XPORT, discuss its optimization framework and demonstrate its flexibility and effectiveness based on experimental results from its deployment on PlanetLab.