Exploring the effects of non-monetary reimbursement for participants in HCI research


  • Sarah Wiseman Goldsmiths, University of London
  • Anna L. Cox University College London
  • Sandy J. J. Gould University College London
  • Duncan P. Brumby University College London




Participation, Citizen Science, Quantified Self, Online Experiments, Reimbursement


When running experiments within the field of Human Computer Interaction (HCI) it is common practice to ask participants to come to a specified lab location, and reimburse them monetarily for their time and travel costs. This, however, is not the only means by which to encourage participation in scientific study. Citizen science projects, which encourage the public to become involved in scientific research, have had great success in getting people to act as sensors to collect data or to volunteer their idling computer or brain power to classify large data sets across a broad range of fields including biology, cosmology and physical and environmental science. This is often done without the expectation of payment. Additionally, data collection need not be done on behalf of an external researcher; the Quantified Self (QS) movement allows people to reflect on data they have collected about themselves. This too, then, is a form of non-reimbursed data collection. Here we investigate whether citizen HCI scientists and those interested in personal data produce reliable results compared to participants in more traditional lab-based studies. Through six studies, we explore how participation rates and data quality are affected by recruiting participants without monetary reimbursement: either by providing participants with data about themselves as reward (a QS approach), or by simply requesting help with no extrinsic reward (as in citizen science projects). We show that people are indeed willing to take part in online HCI research in the absence of extrinsic monetary reward, and that the data generated by participants who take part for selfless reasons, rather than for monetary reward, can be as high quality as data gathered in the lab and in addition may be of higher quality than data generated by participants given monetary reimbursement online. This suggests that large HCI experiments could be run online in the future, without having to incur the equally large reimbursement costs alongside the possibility of running experiments in environments outside of the lab.


Amazon Mechanical Turk. (2015). Retrieved from https://www.mturk.com

Bertamini, M., & Munafo, M. R. (2012). Bite-size science and its undesired side effects. Perspectives on Psychological Science, 7(1), 67–71. http://doi.org/10.1177/1745691611429353

Bonney, R., Ballard, H., Jordan, R., McCallie, E., Phillips, T., Shirk, J., & Wilderman, C. C. (2009). Public participation in scientific research: defining the field and assessing its potential for informal science education. A CAISE Inquiry Group Report.

Bonney, R., Cooper, C. B., Dickinson, J., Kelling, S., Phillips, T., Rosenberg, K. V., & Shirk, J. (2009). Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy. BioScience, 59(11), 977–984. http://doi.org/10.1525/bio.2009.59.11.9

Bravata, D. M., Smith-Spangler, C., Sundaram, V., Gienger, A. L., Lin, N., Lewis, R., … Sirard, J. R. (2007). Using pedometers to increase physical activity and improve health: a systematic review. Jama, 298(19), 2296–2304. http://doi.org/10.1001/jama.298.19.2296

Brumby, D. P., Cox, A. L., Back, J., & Gould, S. J. J. (2013). Recovering from an interruption: Investigating speed−accuracy trade-offs in task resumption behavior. Journal of Experimental Psychology: Applied, 19(2), 95–107. http://doi.org/10.1037/a0032696

Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science, 6(1), 3–5. http://doi.org/10.1177/1745691610393980

Crump, M. J. C., McDonnell, J. V, & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PloS One, 8(3), e57410. http://doi.org/10.1371/journal.pone.0057410

Curtis, D. (2014). The man who records all his sneezes. Retrieved from http://www.bbc.co.uk/news/magazine-29525501

Dandurand, F., Shultz, T. R., & Onishi, K. H. (2008). Comparing online and lab methods in a problem-solving experiment. Behavior Research Methods, 40(2), 428–434. http://doi.org/10.3758/BRM.40.2.428

Dunlop, M., Komninos, A., & Durga, N. (2014). Towards High Quality Text Entry on Smartwatches. In CHI’14 Extended Abstracts on Human Factors in Computing Systems (pp. 2365–2370). http://doi.org/10.1145/2559206.2581319

Dunlop, M., & Levine, J. (2012). Multidimensional pareto optimization of touchscreen keyboards for speed, familiarity and improved spell checking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2669–2678). ACM Press. http://doi.org/10.1145/2207676.2208659

Evans, C., Abrams, E., & Reitsma, R. (2005). The Neighborhood Nestwatch Program: Participant Outcomes of a Citizen‐Science Ecological Research Project. Conservation Biology, 19, 589–594. http://doi.org/10.1111/j.1523-1739.2005.00s01.x

Fold It. (2015). Fold It. Retrieved from http://fold.it/

Fritz, T., Huang, E. M., Murphy, G. C., & Zimmermann, T. (2014). Persuasive technology in the real world. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 487–496). http://doi.org/10.1145/2556288.2557383

Galaxy Zoo. (2015). Galaxy Zoo. Retrieved September 18, 2013, from http://www.galaxyzoo.org/

Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012). Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic Bulletin & Review, 19(5), 847–57. http://doi.org/10.3758/s13423-012-0296-9

Google Charts. (2015). Retrieved from https://developers.google.com/chart/

Gould, S. J. J., Cox, A. L., Brumby, D. P., & Wiseman, S. (2015). Home is Where the Lab is: A Comparison of Online and Lab Data From a Time-sensitive Study of Interruption. Human Computation, 45–67. http://doi.org/10.15346/hc.v2i1.4

Halberda, J., Ly, R., Wilmer, J. B., Naiman, D. Q., & Germine, L. (2012). Number sense across the lifespan as revealed by a massive Internet-based sample. Proceedings of the National Academy of Sciences of the United States of America, 109(28), 11116–20. http://doi.org/10.1073/pnas.1200196109

Heer, J., & Bostock, M. (2010). Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 203–212). http://doi.org/10.1145/1753326.1753357

Jennett, C., Eveleigh, A., Mathieu, K., Ajani, Z., & Cox, A. L. (2013). Creativity in citizen science: All for one and one for all. In ACM WebSci 2013, “Creativity and Attention in the Age of the Web” Workshop.

Jennett, C., Furniss, D. J., Iacovides, I., Wiseman, S., Gould, S. J. J., & Cox, A. L. (2014). Exploring Citizen Psych-Science and the Motivations of Errordiary Volunteers. Human Computation, 1(2), 199–218. http://doi.org/10.15346/hc.v1i2.10

Khatib, F., DiMaio, F., Cooper, S., Kazmierczyk, M., Gilski, M., Krzywda, S., … Baker, D. (2011). Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature Structural & Molecular Biology, 18(10), 1175–7. http://doi.org/10.1038/nsmb.2119

Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing user studies with Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 453–456). Florence, Italy: ACM. http://doi.org/10.1145/1357054.1357127

Komarov, S., Reinecke, K., & Gajos, K. Z. (2013). Crowdsourcing Performance Evaluations of User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM. http://doi.org/10.1145/2470654.2470684

Lab In The Wild. (2015). Retrieved from www.labinthewild.org

Li, I., Dey, A., & Forlizzi, J. (2010). A stage-based model of personal informatics systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 557–566). ACM Press. http://doi.org/10.1145/1753326.1753409

MacKenzie, I. S., & Soukoreff, R. W. (2003). Phrase sets for evaluating text entry techniques. In CHI ’03 extended abstracts on Human factors in computer systems - CHI '03 (pp. 754–755). ACM Press. http://doi.org/10.1145/765968.765971

Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44(1), 1–23. http://doi.org/10.3758/s13428-011-0124-6

Mason, W., & Watts, D. (2010). Financial incentives and the performance of crowds. ACM SIGKDD Explorations Newsletter, 11(2), 100–108. http://doi.org/10.1145/1809400.1809422

Munson, S. (2012). Mindfulness, Reflection, and Persuasion in Personal Informatics. In CHI 2012 Personal Informatics in Practice: Improving Quality of Life Through Data Workshop. Retrieved from http://personalinformatics.org/docs/chi2012/munson.pdf

Old Weather. (2015). Old Weather.

Raddick, M. J., Bracey, G., Gay, P. L., Lintott, C. J., Cardamone, C., Murray, P., … Vandenberg, J. (2013). Galaxy zoo: Motivations of citizen scientists. Astronomy Education Review, 12(1), 1–41. http://doi.org/10.3847/AER2011021

Raddick, M. J., Bracey, G., Gay, P. L., Lintott, C. J., Murray, P., Schawinski, K., … Vandenberg, J. (2010). Galaxy Zoo: Exploring the Motivations of Citizen Science Volunteers. Astronomy Education Review, 9. http://doi.org/10.3847/aer2009036

Ramsey, S. R., Thompson, K. L., Mckenzie, M., & Rosenbaum, A. (2016). Psychological research in the internet age: The quality of web-based data. Computers in Human Behavior, 58, 354–360. http://doi.org/10.1016/j.chb.2015.12.049

Reinecke, K., Arbor, A., & Gajos, K. Z. (2015). LabintheWild : Conducting Large-Scale Online Experiments With Uncompensated Samples. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 1364–1378). http://doi.org/10.1145/2675133.2675246

Rogers, Y. (2011). Interaction design gone wild: striving for wild theory. Interactions, 18(4), 58–62. http://doi.org/10.1145/1978822.1978834

Rotman, D., Preece, J., Hammock, J., Procita, K., Hanse, D., Parr, C., … Jacobs, D. (2012). Dynamic Changes in Motivation in Collaborative Citizen- Science Projects. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (pp. 1–10). http://doi.org/10.1145/2145204.2145238

Salthouse, T. A. (1986). Perceptual, cognitive, and motoric aspects of transcription typing. Psychological Bulletin, 99(3), 303–319. http://doi.org/10.1037/0033-2909.99.3.303

Salthouse, T. A., & Saults, J. S. (1987). Multiple spans in transcription typing. The Journal of Applied Psychology, 72(2), 187–96. http://doi.org/10.1037/0021-9010.72.2.187

Swan, M. (2012). Sensor Mania! The Internet of Things, Wearable Computing, Objective Metrics, and the Quantified Self 2.0. Journal of Sensor and Actuator Networks, 1(3), 217–253. http://doi.org/10.3390/jsan1030217

Swan, M. (2013). The Quantified Self: Fundamental Disruption in Big Data Science and Biological Discovery. Big Data, 1(2), 85–99. http://doi.org/10.1089/big.2012.0002

Test My Brain. (2015). Retrieved from www.testmybrain.org

The Royal Society for the Protection of Birds. (2015). The RSPB: Big Garden Birdwatch. Retrieved January 1, 2015, from https://www.rspb.org.uk/discoverandenjoynature/discoverandlearn/birdwatch/index.aspx




How to Cite

Wiseman, S., Cox, A. L., Gould, S. J. J., & Brumby, D. P. (2017). Exploring the effects of non-monetary reimbursement for participants in HCI research. Human Computation, 4(1), 1-24. https://doi.org/10.15346/hc.v4i1.1