Stream Processing on Bare Metal

Опубликовано 17 августа 2016, 20:47

In line with the insight that 'one size' of databases will not fit all application needs, the database community is currently exploring various alternatives to commodity, CPU-based system designs. One particular candidate in this trend are Field-programmable Gate Arrays (FPGAs), programmable logic chips that allow tailor-made hardware designs optimized for specific systems, applications, or even user queries. In this talk I show how stream processing can be accelerated using FPGAs. I will present a method to translate continuous streaming queries into digital circuits. The circuits can be automatically generated by a compiler by composing library components on an operator-level basis. Our operator library currently contains components for selection, aggregation, grouping, windowing as well as stream joins. We will look at the stream join operator in more detail. We propose new a description of the stream join operation we call 'handshake join'. It provides the traditional window semantics but is more amenable to parallel execution. The handshake stream join also has an excellent scalability behavior on a single FPGA chip, across multiple FPGA chips, and even in traditional multi-core systems. Significant performance improvements can be achieved by inserting the FPGA into the system's data path (e.g., between the network interface and the host CPU). The evaluation shows that queries on the FPGA can process streams at more than one million tuples per second and that they can do this directly from the network, removing much of the overhead of transferring the data to a conventional CPU.