Architecting Real-Time Crowd-Powered Systems

Authors

  • Walter S Lasecki University of Rochester / Carnegie Mellon University
  • Christopher Homan Rochester Institute of Technology
  • Jeffrey P. Bigham Carnegie Mellon University

DOI:

https://doi.org/10.15346/hc.v1i1.5

Keywords:

real-time crowdsourcing, human computation, system architectures

Abstract

Human computation allows computer systems to leverage human intelligence in computational processes. While it has primarily been used for tasks that are not time-sensitive, recent systems use crowdsourcing to get on-demand, real-time, and even interactive results. In this paper, we present techniques for building real-time crowdsourcing systems, and then discuss how and when to use them. Our goal is to provide system builders with the tools and insights they need to replicate the success of modern systems in order to further explore this new space.

References

Words per minute. http://en.wikipedia.org/wiki/Words_per_minute. (????). Accessed: 2014-06-05.

Archak, N and Sundararajan, A. (2009). Optimal Design of Crowdsourcing Contests.. In ICIS. 200.

Arrow, K. J. (2012). Social choice and individual values. Vol. 12. Yale university press.

Bernstein, D. S, Givan, R, Immerman, N, and Zilberstein, S. (2002). The complexity of decentralized control of Markov decision

processes. Mathematics of operations research 27, 4 (2002), 819–840.

Bernstein, M. S, Brandt, J, Miller, R. C, and Karger, D. R. (2011). Crowds in Two Seconds: Enabling Realtime Crowd-powered

Interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST ’11). ACM, New

York, NY, USA, 33–42. DOI:http://dx.doi.org/10.1145/2047196.2047201

Bernstein, M. S, Karger, D. R, Miller, R. C, and Brandt, J. (2012). Analytic Methods for Optimizing Realtime Crowdsourcing. CoRR

abs/1204.2995 (2012).

Bernstein, M. S, Little, G, Miller, R. C, Hartmann, B, Ackerman, M. S, Karger, D. R, Crowell, D, and Panovich, K. (2010). Soylent:

A Word Processor with a Crowd Inside. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and

Technology (UIST ’10). ACM, New York, NY, USA, 313–322. DOI:http://dx.doi.org/10.1145/1866029.1866078

Bigham, J. P, Jayant, C, Ji, H, Little, G, Miller, A, Miller, R. C, Miller, R, Tatarowicz, A, White, B, White, S, and Yeh, T. (2010).

VizWiz: Nearly Real-time Answers to Visual Questions. In Proceedings of the 23Nd Annual ACM Symposium on User Interface

Software and Technology (UIST ’10). ACM, New York, NY, USA, 333–342. DOI:http://dx.doi.org/10.1145/1866029.1866080

Bigham, J. P and Ladner, R. E. (2011). What the Disability Community Can Teach Us About Interactive Crowdsourcing. interactions

, 4 (July 2011), 78–81. DOI:http://dx.doi.org/10.1145/1978822.1978838

Bigham, J. P and Lasecki, W. S. (2014). Crowd Storage: Storing Information on Existing Memories. In Proceedings of the SIGCHI

Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 601–604. DOI:http://dx.doi.org/10.

/2556288.2557159

Borish, M, Cordar, A, Foster, A, Kim, T, Murphy, J, and Lok, B. (2014). Utilizing Real-time Human-Assisted Virtual Humans to

Increase Real-world Interaction Empathy. In Kansei Engineering & Emotion Research (KEER ’14). 15.

Boudreau, K. J, Lacetera, N, and Lakhani, K. R. (2011). Incentives and problem uncertainty in innovation contests: An empirical

analysis. Management Science 57, 5 (2011), 843–863.

Boutilier, C. (1999). Sequential optimality and coordination in multiagent systems. In IJCAI, Vol. 99. 478–485.

Boutsis, I and Kalogeraki, V. (2013). Crowdsourcing under Real-Time Constraints. In Parallel Distributed Processing (IPDPS), 2013

IEEE 27th International Symposium on. 753–764. DOI:http://dx.doi.org/10.1109/IPDPS.2013.84

Callison-Burch, C. (2009). Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk. In Proceedings

of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1 (EMNLP ’09).

Association for Computational Linguistics, Stroudsburg, PA, USA, 286–295. http://dl.acm.org/citation.cfm?id=1699510.1699548

Carpenter, B. (2011). A Hierarchical Bayesian Model of Crowdsourced Relevance Coding.. In TREC.

Chawla, S, Hartline, J. D, and Sivan, B. (2012). Optimal crowdsourcing contests. In SODA. 856–868.

Chilton, L. B, Sims, C. T, Goldman, M, Little, G, and Miller, R. C. (2009). Seaweed: A Web Application for Designing Economic

Games. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP ’09). ACM, New York, NY, USA,

–35. DOI:http://dx.doi.org/10.1145/1600150.1600162

Cooper, S, Khatib, F, Treuille, A, Barbero, J, Lee, J, Beenen, M, Leaver-Fay, A, Baker, D, Popovi´c, Z, and others, . (2010). Predicting

protein structures with a multiplayer online game. Nature 466, 7307 (2010), 756–760.

Dai Peng, M and Weld, D. S. (2011). Artificial intelligence for artificial artificial intelligence. In Twenty-Fifth AAAI Conference on

Artificial Intelligence (AAAI 2011).

Dalvi, N, Dasgupta, A, Kumar, R, and Rastogi, V. (2013). Aggregating crowdsourced binary ratings. In Proceedings of the 22nd

international conference on World Wide Web. International World Wide Web Conferences Steering Committee, 285–294.

DiPalantino, D and Vojnovic, M. (2009). Crowdsourcing and all-pay auctions. In Proceedings of the 10th ACM conference on Electronic

commerce. ACM, 119–128.

Dow, S, Kulkarni, A, Klemmer, S, and Hartmann, B. (2012). Shepherding the Crowd Yields Better Work. In Proceedings of the

ACM 2012 Conference on Computer Supported Cooperative Work (CSCW ’12). ACM, New York, NY, USA, 1013–1022. DOI:

http://dx.doi.org/10.1145/2145204.2145355

Faliszewski, P, Hemaspaandra, E, and Hemaspaandra, L. A. (2010). Using complexity to protect elections. Commun. ACM 53, 11

(2010), 74–82.

Forlines, C, Miller, S, Guelcher, L, and Bruzzi, R. (2014). Crowdsourcing the Future: Predictions Made with a Social Network. In

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 3655–

DOI:http://dx.doi.org/10.1145/2556288.2556967

Goldman, C. V and Zilberstein, S. (2004). Decentralized control of cooperative systems: Categorization and complexity analysis. J.

Artif. Intell. Res.(JAIR) 22 (2004), 143–174.

Heidari, H and Kearns, M. (2013). Depth-Workload Tradeoffs for Workforce Organization. In First AAAI Conference on Human

Computation and Crowdsourcing.

Hemaspaandra, E, Hemaspaandra, L. A, and Rothe, J. (2012). Online Voter Control in Sequential Elections.. In ECAI. 396–401.

Hemaspaandra, E, Hemaspaandra, L. A, and Rothe, J. (2014). The complexity of online manipulation of sequential elections. J.

Comput. System Sci. 80, 4 (2014), 697–710.

Ho, C.-J, Jabbari, S, and Vaughan, J. W. (2013). Adaptive task assignment for crowdsourced classification. In Proceedings of the 30th

International Conference on Machine Learning (ICML-13). 534–542.

Ho, C.-J and Vaughan, J. W. (2012). Online Task Assignment in Crowdsourcing Markets.. In AAAI.

Horton, J. J and Chilton, L. B. (2010). The Labor Economics of Paid Crowdsourcing. In Proceedings of the 11th ACM Conference on

Electronic Commerce (EC ’10). ACM, New York, NY, USA, 209–218. DOI:http://dx.doi.org/10.1145/1807342.1807376

Ipeirotis, P. G. (2010). Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads, The ACM Magazine for Students 17,

(2010), 16–21.

Jung, H. J and Lease, M. (2013). Crowdsourced Task Routing via Matrix Factorization. CoRR abs/1310.5142 (2013).

Kamar, E, Hacker, S, and Horvitz, E. (2012). Combining Human and Machine Intelligence in Large-scale Crowdsourcing. In Proceedings

of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1 (AAMAS ’12). International

Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 467–474. http://dl.acm.org/citation.cfm?id=2343576.

Kamar, E and Horvitz, E. (2012). Incentives for truthful reporting in crowdsourcing. In Proceedings of the 11th International Conference

on Autonomous Agents and Multiagent Systems-Volume 3. International Foundation for Autonomous Agents and Multiagent

Systems, 1329–1330.

Karger, D. R, Oh, S, and Shah, D. (2011). Iterative learning for reliable crowdsourcing systems. In Advances in neural information

processing systems. 1953–1961.

Kittur, A, Nickerson, J. V, Bernstein, M, Gerber, E, Shaw, A, Zimmerman, J, Lease, M, and Horton, J. (2013). The Future of Crowd

Work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW ’13). ACM, New York, NY, USA,

–1318. DOI:http://dx.doi.org/10.1145/2441776.2441923

Kittur, A, Smus, B, Khamkar, S, and Kraut, R. E. (2011). CrowdForge: Crowdsourcing Complex Work. In Proceedings of the 24th

Annual ACM Symposium on User Interface Software and Technology (UIST ’11). ACM, New York, NY, USA, 43–52. DOI:

http://dx.doi.org/10.1145/2047196.2047202

Kuhn, H. (1953). Extensive Games and the Problem of Information, Contributions to the Theory of Games II, Kuhn, HW and AW

Tucker, eds. 193-216. (1953).

Kulkarni, A, Can, M, and Hartmann, B. (2012). Collaboratively Crowdsourcing Workflows with Turkomatic. In Proceedings of the

ACM 2012 Conference on Computer Supported Cooperative Work (CSCW ’12). ACM, New York, NY, USA, 1003–1012. DOI:

http://dx.doi.org/10.1145/2145204.2145354

Lasecki, Wand Bigham, J. (2012)a. Self-correcting Crowds. In CHI ’12 Extended Abstracts on Human Factors in Computing Systems

(CHI EA ’12). ACM, New York, NY, USA, 2555–2560. DOI:http://dx.doi.org/10.1145/2212776.2223835

Lasecki, W, Miller, C, Sadilek, A, Abumoussa, A, Borrello, D, Kushalnagar, R, and Bigham, J. (2012). Real-time Captioning by

Groups of Non-experts. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST

’12). ACM, New York, NY, USA, 23–34. DOI:http://dx.doi.org/10.1145/2380116.2380122

Lasecki, W. S and Bigham, J. P. (2012)b. Online Quality Control for Real-time Crowd Captioning. In Proceedings of the 14th

International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’12). ACM, New York, NY, USA, 143–150.

DOI:http://dx.doi.org/10.1145/2384916.2384942

Lasecki, W. S, Gordon, M, Koutra, D, Jung, M, Dow, S, and Bigham, J. P. (2014). Glance: Rapidly Coding Behavioral Video with the

Crowd. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST ’14). ACM, New

York, NY, USA, 12.

Lasecki, W. S, Kamar, E, and Bohus, D. (2013)a. Conversations in the Crowd: Collecting Data for Task-Oriented Dialog Learning. In

First AAAI Conference on Human Computation and Crowdsourcing.

Lasecki, W. S, Miller, C. D, and Bigham, J. P. (2013)b. Warping Time for More Effective Real-time Crowdsourcing. In Proceedings

of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). ACM, New York, NY, USA, 2033–2036. DOI:

http://dx.doi.org/10.1145/2470654.2466269

Lasecki, W. S, Murray, K. I, White, S, Miller, R. C, and Bigham, J. P. (2011). Real-time Crowd Control of Existing Interfaces. In

Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST ’11). ACM, New York, NY,

USA, 23–32. DOI:http://dx.doi.org/10.1145/2047196.2047200

Lasecki, W. S, Song, Y. C, Kautz, H, and Bigham, J. P. (2013)a. Real-time Crowd Labeling for Deployable Activity Recognition.

In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW ’13). ACM, New York, NY, USA,

–1212. DOI:http://dx.doi.org/10.1145/2441776.2441912

Lasecki, W. S, Thiha, P, Zhong, Y, Brady, E, and Bigham, J. P. (2013)b. Answering Visual Questions with Conversational Crowd

Assistants. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’13).

ACM, New York, NY, USA, Article 18, 8 pages. DOI:http://dx.doi.org/10.1145/2513383.2517033

Lasecki, W. S, Weingard, L, Ferguson, G, and Bigham, J. P. (2014). Finding Dependencies Between Actions Using the Crowd. In

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 3095–

DOI:http://dx.doi.org/10.1145/2556288.2557176

Lasecki, W. S, Wesley, R, Nichols, J, Kulkarni, A, Allen, J. F, and Bigham, J. P. (2013). Chorus: A Crowd-powered Conversational

Assistant. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST ’13). ACM, New

York, NY, USA, 151–162. DOI:http://dx.doi.org/10.1145/2501988.2502057

Lasecki, W. S, White, S, Murray, K. I, and Bigham, J. P. (2012). Crowd Memory: Learning in the Collective. In Proceedings of

Collective Intelligence. 8.

Lee, J, Kladwang, W, Lee, M, Cantu, D, Azizyan, M, Kim, H, Limpaecher, A, Yoon, S, Treuille, A, and Das, R. (2014). RNA design

rules from a massive open laboratory. Proceedings of the National Academy of Sciences 111, 6 (2014), 2122–2127.

Limpaecher, A, Feltman, N, Treuille, A, and Cohen, M. (2013). Real-time Drawing Assistance Through Crowdsourcing. ACM Trans.

Graph. 32, 4, Article 54 (2013), 8 pages. DOI:http://dx.doi.org/10.1145/2461912.2462016

Lin, C. H, Weld, D. S, and others, . (2012). Dynamically switching between synergistic workflows for crowdsourcing. In Workshops

at the Twenty-Sixth AAAI Conference on Artificial Intelligence.

Little, G, Chilton, L. B, Goldman, M, and Miller, R. C. (2010). TurKit: Human Computation Algorithms on Mechanical Turk. In

Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST ’10). ACM, New York, NY,

USA, 57–66. DOI:http://dx.doi.org/10.1145/1866029.1866040

Liu, Q, Peng, J, and Ihler, A. (2012). Variational inference for crowdsourcing. In Advances in Neural Information Processing Systems.

–700.

Loparev, A, Lasecki, W. S, Murray, K. I, and Bigham, J. P. (2013). Introducing Shared Character Control to Existing Video Games.

(2013).

Mao, A, Chen, Y, Gajos, K. Z, Parkes, D, Procaccia, A. D, and Zhang, H. (2012). TurkServer: Enabling Synchronous and Longitudinal

Online Experiments. In Fourth Workshop on Human Computation (HCOMP 2012).

Mao, A, Kamar, E, and Horvitz, E. (2013). Why Stop Now? Predicting Worker Engagement in Online Crowdsourcing. In First AAAI

Conference on Human Computation and Crowdsourcing.

Mashhadi, A. J and Capra, L. (2011). Quality control for real-time ubiquitous crowdsourcing. In Proceedings of the 2nd international

workshop on Ubiquitous crowdsouring. ACM, 5–8.

Mason, W and Suri, S. (2012). Conducting behavioral research on Amazonâ˘A ´ Zs Mechanical Turk. Behavior research methods 44, 1

(2012), 1–23.

Messias, J. V, Spaan, M, and Lima, P. U. (2011). Efficient offline communication policies for factored multiagent POMDPs. In

Advances in Neural Information Processing Systems. 1917–1925.

Moldovanu, B and Sela, A. (2001). The optimal allocation of prizes in contests. American Economic Review (2001), 542–558.

Moldovanu, B and Sela, A. (2006). Contest architecture. Journal of Economic Theory 126, 1 (2006), 70–96.

Osentoski, S, Crick, C, Jay, G, and Jenkins, O. C. Crowdsourcing for closed loop control. (????).

Parkes, D. C and Procaccia, A. D. (2013). Dynamic Social Choice with Evolving Preferences. (2013), 767–773.

Proper, S and Tadepalli, P. (2009). Solving multiagent assignment markov decision processes. In Proceedings of The 8th International

Conference on Autonomous Agents and Multiagent Systems-Volume 1. International Foundation for Autonomous Agents and

Multiagent Systems, 681–688.

Quinn, A. J and Bederson, B. B. (2011). Human Computation: A Survey and Taxonomy of a Growing Field. In Proceedings of

the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, New York, NY, USA, 1403–1412. DOI:

http://dx.doi.org/10.1145/1978942.1979148

Raykar, V. C, Yu, S, Zhao, L. H, Valadez, G. H, Florin, C, Bogoni, L, and Moy, L. (2010). Learning from crowds. The Journal of

Machine Learning Research 11 (2010), 1297–1322.

Retelny, D, Robaszkiewicz, S, To, A, Lasecki, W, Patel, J, Rahmati, N, Doshi, T, Valentine, M, and Bernstein, M. (2014). Expert

Crowdsourcing with Flash Teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology

(UIST ’14). ACM, New York, NY, USA, 12.

Rossen, B and Lok, B. (2012). A crowdsourcing method to develop virtual human conversational agents. International Journal of

Human-Computer Studies 70, 4 (2012), 301 – 319. DOI:http://dx.doi.org/10.1016/j.ijhcs.2011.11.004

Russell, B. C, Torralba, A, Murphy, K. P, and Freeman,W. T. (2008). LabelMe: A Database andWeb-Based Tool for Image Annotation.

Int. J. Comput. Vision 77, 1-3 (May 2008), 157–173. DOI:http://dx.doi.org/10.1007/s11263-007-0090-8

Sheng, V. S, Provost, F, and Ipeirotis, P. G. (2008). Get another label? improving data quality and data mining using multiple, noisy

labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM,

–622.

Singh, P, Lasecki, W. S, Barelli, P, and Bigham, J. P. (2012). HiveMind: A Framework for Optimizing Open-Ended Responses From

the Crowd. Technical Report. URCS Technical Report.

Snow, R, O’Connor, B, Jurafsky, D, and Ng, A. Y. (2008). Cheap and Fast—but is It Good?: Evaluating Non-expert Annotations for

Natural Language Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP ’08).

Association for Computational Linguistics, Stroudsburg, PA, USA, 254–263. http://dl.acm.org/citation.cfm?id=1613715.1613751

Spaan, M. T and Oliehoek, F. A. Tree-Based Solution Methods for Multiagent POMDPs with Delayed Communication. In Proc. of

th Benelux Conference on Artificial Intelligenceg:

Surowiecki, J. (2005). The wisdom of crowds. Random House LLC.

Van Kleek, M. G, Bernstein, M, Panovich, K, Vargas, G. G, Karger, D. R, and Schraefel, M. (2009). Note to Self: Examining

Personal Information Keeping in a Lightweight Note-taking Tool. In Proceedings of the SIGCHI Conference on Human Factors in

Computing Systems (CHI ’09). ACM, New York, NY, USA, 1477–1480. DOI:http://dx.doi.org/10.1145/1518701.1518924

von Ahn, L. (2005). Human Computation. Ph.D. Dissertation. Carnegie Mellon University, Pittsburgh, PA.

von Ahn, L and Dabbish, L. (2004). Labeling Images with a Computer Game. In Proceedings of the SIGCHI Conference on Human

Factors in Computing Systems (CHI ’04). ACM, New York, NY, USA, 319–326. DOI:http://dx.doi.org/10.1145/985692.985733

Zhang, H, Law, E, Miller, R, Gajos, K, Parkes, D, and Horvitz, E. (2012). Human Computation Tasks with Global Constraints. In

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). ACM, New York, NY, USA, 217–226.

DOI:http://dx.doi.org/10.1145/2207676.2207708

Zhou, D, Basu, S, Mao, Y, and Platt, J. C. (2012). Learning from the wisdom of crowds by minimax entropy. In Advances in Neural

Information Processing Systems. 2195–2203.

Human Computation, Volume 1, Issue 1, October 2014

Downloads

Published

2014-09-07

How to Cite

Lasecki, W. S., Homan, C., & Bigham, J. P. (2014). Architecting Real-Time Crowd-Powered Systems. Human Computation, 1(1). https://doi.org/10.15346/hc.v1i1.5

Issue

Vol. 1 No. 1 (2014)

Section

Research

License

Authors who publish with this journal agree to the following terms:

  1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
  2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
  3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
Crossref
Scopus
Google Scholar
Europe PMC