Optimal Inference Algorithms for Crowdsourcing Systems: Classification and Regression.
Crowdsourcing platforms, such as Amazon's Mechanical Turk, emerged as popular venues for purchasing human intelligence at low cost for large volumes of tasks. As many low-paid workers are prone to give noisy answers, one of the fundamental questions is how to identify more reliable workers and exploit this heterogeneity to infer the true answers accurately. In this talk, we present inference algorithms for estimating (i) discrete answers of classification tasks and (ii) continuous answers of regression tasks. In particular, the proposed algorithms asymptotically achieve the fundamental limit of accuracy in tractable complexity under some canonical and mild assumptions, where the optimal algorithms require intractable complexity. We provide experimental results using real datasets. The results suggest that our algorithms improve upon competing state-of-the-art algorithms in terms of accuracy but also achieve the fundamental limit even in the regimes not satisfying assumptions for the theoretical analysis. We further measure the importance of the inference algorithms in a practical application using crowdsourced datasets. To do so, we emulate a crowdsourcing system reproducing PASCAL visual object classes datasets and show that de-noising the crowdsourced data from the proposed scheme can significantly improve the performance for the vision task.
Jungseul Ok received the B.S. in 2011, and finished Ph.D program i n School of Electrical Engineering at KAIST. He is currently working as postdoc in School of Electrical Engineering at KTH, Sweden. In this talk, he will present his work on optimal crowdsourcing system published in ICML 2016 and its recent extension.
Host: Seyoung Yun ( email@example.com)