The Boss: A Petascale DB for Large-Scale Neuroscience Powered by Serverless Advanced Technologies

Опубликовано 19 июня 2017, 21:21

The IARPA Machine Intelligence from Cortical Networks (MICrONS) program is a research endeavor that seeks to improve neurally-plausible machine learning algorithms by developing an understanding of the data representations and learning rules employed by the brain through structurally and functionally interrogating a cubic millimeter of mammalian neocortex. This effort requires the efficient storage, visualization, and processing of petabytes of neuroimaging data. The Johns Hopkins University Applied Physics Laboratory has developed an open-source, highly-available service to manage these data called the Boss.

The Boss leverages Amazon Web Services to provide a cloud-native spatial database with an innovative storage hierarchy and auto-scaling capability to balance cost and performance. The system leverages serverless components extensively to meet both scalability and cost requirements.

In this session we will provide an overview of the Boss, and focus on how JHU/APL leveraged DynamoDB, Lambda, and Step Functions for several high-throughput components of the system. We'll discuss both the challenges faced and successes achieved with serverless technologies.