Decision-Theoretic Control for Crowdsourcing

Опубликовано 17 августа 2016, 1:49

Crowd-sourcing is a recent framework in which human intelligence tasks are outsourced to a crowd of unknown people (ΓÇ¥workersΓÇ¥) as an open call (e.g., on AmazonΓÇÖs Mechanical Turk). It is also known as ΓÇ£artificial artificial intelligenceΓÇ¥. Crowd-sourcing has become immensely popular with hoards of employers (ΓÇ¥requestersΓÇ¥), who use it to solve a wide variety of jobs, such as dictation transcription, content screening, translation, information extraction, etc. In order to achieve quality results, requesters often subdivide a large task into a chain of bite-sized sub-tasks that are combined into a complex, iterative workflow in which workers check and improve each othersΓÇÖ results. In this talk, I will introduce a planner, TURKONTROL, which formulates workflow control as a decision-theoretic optimization problem, trading off the implicit quality of a solution artifact against the cost for workers to achieve it. We learn the models from real data, and demonstrate that the dynamic workflow, generated by the decision-theoretic agent based on the model, produces (statistically-significant) higher-quality worker outputs compared to the existing static workflow, under equal monetary assumptions. I will also elaborate the mathematical model underneath the planner -- Markov decision processes (MDPs), and briefly introduce a couple of new approaches that solve MDPs optimally: one that significantly speeds up the convergence of planning by using the problemsΓÇÖ graphical information, the other that exploits the availability of external memory to solve much larger problems than previously attempted.