Skoll: Distributed Continuous Quality Assurance

Опубликовано 6 сентября 2016, 5:35

Quality assurance (QA) tasks, such as testing, profiling, and performance evaluation, have historically been done in-house on developer-generated workloads and regression suites. The shortcomings of in-house QA efforts are well-known and severe, including (1) increased QA cost and schedule and (2) misleading results when the test-cases, input workload, software version and platform at the developer's site differs from those in the field. Consequently, tools and processes are being developed to improve software quality by increasing user participation in the QA process. A central limitation of these approaches, however, is that they focus on isolated mechanisms, not on the coordination and control policies and tools needed to make the global QA process efficient, effective, and scalable. To address these issues, we have initiated the Skoll project, which is developing and validating novel software QA processes and tools that leverage the extensive computing resources of worldwide user communities in a distributed, continuous manner to significantly and rapidly improve software quality. We call this approach distributed, continuous quality assurance (DCQA) and our goal is to enable DCQA processes that conducted around-the-world, around-the-clock on a massive, virtual computing grid provided by numerous remote users. In this talk, I will present an infrastructure, algorithms and tools for developing and executing thorough, transparent, managed, and adaptive DCQA processes. I will also introduce two novel DCQA processes, namely the fault characterization process, which aims at identifying failure-inducing configuration options in complex configuration spaces and the main effects screening process, which rapidly and efficiently detects performance degradations in evolving software systems.