Decision-Theoretic Crowdsourcing

Опубликовано 12 августа 2016, 2:34

Crowdsourcing continues to rise in popularity today and is considered as the future of labor by many. Presently, crowdsourcing marketplaces (e.g., Amazon Mechanical Turk) have enabled the construction of scalable applications for tasks ranging from product categorization and photo tagging to audio transcription and language translation. These vertical applications are typically realized with complex, self-managing workflows that guarantee quality results. But constructing and controlling such workflows is challenging, with a huge number of alternative decisions for the designer to consider. We argue the thesis that artificial intelligence methods can greatly simplify the process of creating and managing complex crowdsourced workflows. We present the design of CLOWDER, which uses decision-theoretic techniques to dynamically optimize the workflows. Preliminary evaluations suggest that these optimized workflows are significantly more economical and return a much higher quality output than those generated by human designers.