Tech Showcase: Prometheus: Co-Designing Distributed Systems and Datacenter Hardware

Опубликовано 15 августа 2018, 19:57

The modern datacenter fabric increasingly includes programmable devices capable of increasingly sophisticated processing – ranging from programmable switches that support in-network computation to network-attached FPGAs and smart storage devices. This project asks how we can leverage this new capability to build a new generation of storage and data processing systems that achieve dramatically better performance, reliability, and efficiency.

The key idea is to co-design distributed systems with the data center hardware: introducing new low-level primitives and building systems around them. We have already used this approach to achieve order-of-magnitude speedups for replicated services and transaction coordination. Future directions include: a scalable analytics system that handles massive volumes of network telemetry data by moving the first-level filtering and aggregation to programmable network devices; in-network aggregation for faster training of deep neural networks; and replicated remote storage with minimal CPU overhead by using network-attached programmable solid-state drives.

See more at microsoft.com/en-us/research/v...