Cost of Quality in Crowdsourcing


  • Deniz Iren Middle East Technical University
  • Semih Bilgen Middle East Technical University



Crowdsourcing, Human Computation, Cost of Quality, Cost Models, Quality Assurance


Crowdsourcing is a business model which allows practitioners to access a rather cheap and scalable workforce. However, due to loose worker-employer relationships, skill diversity of the crowd and anonymity of participants, it tends to result in lower quality compared to traditional ways of doing work. Thus crowdsourcing practitioners have to apply certain techniques to make sure that the end product complies with the quality requirements. Quality assurance techniques impact project costs and schedules. A well-defined methodology is needed to estimate these impacts in order to manage the crowdsourcing process effectively and efficiently. This paper introduces cost models of common quality assurance techniques that may be applied in crowdsourcing and describes a proposed cost of quality approach for analyzing quality related costs.


Allahbakhsh, M., Benatallah, B., Ignjatovic, A., Motahari-Nezhad, H. R., Bertino, E., & Dustdar, S. (2013). Quality Control in Crowdsourcing Systems: Issues and Directions. Internet Computing, IEEE, 17(2), 76–81. doi:10.1109/MIC.2013.20

Amazon Mechanical Turk. (n.d.). Retrieved from

Arlot, S., & Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4, 40–79.

Conte, S. D., Dunsmore, H. E., & Shen, V. Y. (1985). Software effort estimation and productivity. Advances in Computers, 24, 1–60.

Crosby, P. B. (1979). Quality is free: The art of making quality certain (Vol. 94). McGraw-Hill New York.

CrossRef. (n.d.). Retrieved from

Downs, J., & Holbrook, M. (2010). Are your participants gaming the system?: screening mechanical turk workers. In … in Computing Systems (pp. 0–3). Retrieved from

Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010). Are Your Participants Gaming the System ? Screening Mechanical Turk Workers, 0–3.

Fishman, G. S. (1996). Monte Carlo. Springer.

Geiger, D., & Seedorf, S. (2011). Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes. In … (pp. 1–11). Retrieved from

Grier, D. A. (2011). Foundational Issues in Human Computing and Crowdsourcing. In Position Paper for the CHI 2011 Workshop on Crowdsourcing and Human Computation. CHI.

Hirth, M., Hoßfeld, T., & Tran-Gia, P. (2013). Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Mathematical and Computer Modelling, 57(11-12), 2918–2932. doi:10.1016/j.mcm.2012.01.006

Ho, C.-J., & Vaughan, J. W. (2012). Online Task Assignment in Crowdsourcing Markets. In AAAI.

Hossfeld, T., Hirth, M., & Tran-Gia, P. (2011). Modeling of Crowdsourcing Platforms and Granularity of Work Organization in Future Internet. In Proceedings of the 23rd International Teletraffic Congress (pp. 142–149). International Teletraffic Congress. Retrieved from

Hsueh, M.-C., Tsai, T. K., & Iyer, R. K. (1997). Fault injection techniques and tools. Computer, 30(4), 75–82.

Huang, E., Zhang, H., & Parkes, D. C. (n.d.). Toward Automatic Task Design : A Progress Report Categories and Subject Descriptors, 77–85.

Ipeirotis, P. G., Provost, F., & Wang, J. (2010). Quality management on Amazon Mechanical Turk. Proceedings of the ACM SIGKDD Workshop on Human Computation - HCOMP ’10, 64. doi:10.1145/1837885.1837906

Karger, D. R., Oh, S., & Shah, D. (2011). Budget-optimal crowdsourcing using low-rank matrix approximations. 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), 284–291. doi:10.1109/Allerton.2011.6120180

Kazai, G., Kamps, J., & Milic-Frayling, N. (2011). Worker types and personality traits in crowdsourcing relevance labels. Proceedings of the 20th ACM international conference on Information and knowledge management - CIKM ’11, 1941. doi:10.1145/2063576.2063860

Kern, R., Thies, H., Bauer, C., & Satzger, G. (2010). Quality assurance for human-based electronic services: A decision matrix for choosing the right approach. In Current Trends in Web Engineering (pp. 421–424). Retrieved from

Kern, R., Zirpins, C., & Agarwal, S. (2009). Managing quality of human-based eservices. In Service-Oriented Computing--ICSOC 2008 Workshops (pp. 304–309).

Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., … Horton, J. (2013). The future of crowd work. Proceedings of the 2013 conference on Computer supported cooperative work - CSCW ’13, 1301. doi:10.1145/2441776.2441923

Laporte, C. Y., Berrhouma, N., Doucet, M., & Palza-Vargas, E. (2012). Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation.

Law, E., & Ahn, L. von. (2011). Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, 5(3), 1–121.

Le, J., & Edmonds, A. (2010). Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In … crowdsourcing for search … (pp. 17–20). Retrieved from

Literally Canvas. (n.d.). Retrieved from

Malcolm, D. G., Roseboom, J. H., Clark, C. E., & Fazar, W. (1959). Application of a technique for research and development program evaluation. Operations research, 7(5), 646–669.

McCann, R., Shen, W., & Doan, A. (2008). Matching Schemas in Online Communities: A Web 2.0 Approach. 2008 IEEE 24th International Conference on Data Engineering, 110–119. doi:10.1109/ICDE.2008.4497419

Okubo, Y., Kitasuka, T., & Aritsugi, M. (2013). A Preliminary Study of the Number of Votes under Majority Rule in Crowdsourcing. Procedia Computer Science, 22, 537–543. doi:10.1016/j.procs.2013.09.133

Oleson, D., Sorokin, A., Laughlin, G., & Hester, V. (2011). Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing. Human computation, 43–48. Retrieved from

Quinn, A., & Bederson, B. (2011). Human computation: a survey and taxonomy of a growing field. In … Conference on Human Factors in Computing …. Retrieved from

Quinn, A. J., & Bederson, B. B. (2011). Human Computation: A Survey and Taxonomy of a Growing Field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1403–1412). New York, NY, USA: ACM. doi:10.1145/1978942.1979148

Rao, R. B., Fung, G., & Rosales, R. (2008). On the Dangers of Cross-Validation. An Experimental Evaluation. In SDM (pp. 588–596).

Ribeiro, F., Florencio, D., & Nascimento, V. (2011). Crowdsourcing subjective image quality evaluation. In Image Processing (ICIP), 2011 18th IEEE International Conference on (pp. 3097–3100). doi:10.1109/ICIP.2011.6116320

Ross, J., Irani, L., & Silberman, M. (2010). Who are the crowdworkers?: shifting demographics in mechanical turk. In CHI’10 Extended … (pp. 2863–2872). Retrieved from

Rouse, A. (2010). A preliminary taxonomy of crowdsourcing. Retrieved from

Schenk, E., & Guittard, C. (2011). Towards a characterization of crowdsourcing practices. Journal of Innovation Economics & Management, (1), 93–107.

Schiffauerova, A., & Thomson, V. (2006). A review of research on cost of quality models and best practices. International Journal of Quality & Reliability Management, 23(6), 647–669. doi:10.1108/02656710610672470

Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008). Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 614–622). New York, NY, USA: ACM. doi:10.1145/1401890.1401965

Sorokin, A., & Forsyth, D. (2008). Utility data annotation with Amazon Mechanical Turk. In Computer Vision and Pattern Recognition Workshops, 2008. CVPRW ’08. IEEE Computer Society Conference on (pp. 1–8). doi:10.1109/CVPRW.2008.4562953

Surowiecki, J. (2005). The wisdom of crowds. Random House LLC.

Von Ahn, L., & Dabbish, L. (2008). Designing games with a purpose. Communications of the ACM, 51(8), 57. doi:10.1145/1378704.1378719

Voyer, R., Nygaard, V., Fitzgerald, W., Copperman, H., Suite, B. S., & Francisco, S. (2010). A Hybrid Model for Annotating Named Entity Training Corpora, (July), 243–246.

Vukovic, M. (2009). Crowdsourcing for Enterprises. In Congress on Services - I (pp. 686–692). doi:10.1109/SERVICES-I.2009.56

Vukovic, M., & Bartolini, C. (2010). Towards a research agenda for enterprise crowdsourcing. In Leveraging applications of formal methods, verification, and validation (pp. 425–434). Springer.

Welinder, P., & Perona, P. (2010). Online crowdsourcing: Rating annotators and obtaining cost-effective labels. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, 25–32. doi:10.1109/CVPRW.2010.5543189

Wieringa, R., & Morali, A. (2012). Technical action research as a validation method in information systems design science. In Design Science Research in Information Systems. Advances in Theory and Practice (pp. 220–238). Springer.

Wikipedia. (n.d.). Retrieved from




How to Cite

Iren, D., & Bilgen, S. (2014). Cost of Quality in Crowdsourcing. Human Computation, 1(2).