Skip to main content

A survey of incentive engineering for crowdsourcing

  • Conor Muldoon (a1), Michael J. O’Grady (a2) and Gregory M. P. O’Hare (a3)

With the growth of the Internet, crowdsourcing has become a popular way to perform intelligence tasks that hitherto would be either performed internally within an organization or not undertaken due to prohibitive costs and the lack of an appropriate communications infrastructure. In crowdsourcing systems, whereby multiple agents are not under the direct control of a system designer, it cannot be assumed that agents will act in a manner that is consistent with the objectives of the system designer or principal agent. In situations whereby agents’ goals are to maximize their return in crowdsourcing systems that offer financial or other rewards, strategies will be adopted by agents to game the system if appropriate mitigating measures are not put in place. The motivational and incentivization research space is quite large; it incorporates diverse techniques from a variety of different disciplines including behavioural economics, incentive theory, and game theory. This paper specifically focusses on game theoretic approaches to the problem in the crowdsourcing domain and places it in the context of the wider research landscape. It provides a survey of incentive engineering techniques that enable the creation of apt incentive structures in a range of different scenarios.

Hide All
Adriaens, T., Sutton-Croft, M., Owen, K., Brosens, D., van Valkenburg, J., Kilbey, D., Groom, Q., Ehmig, C., Thürkow, F., Van Hende, P. & Schneider, K. 2015. Trying to engage the crowd in recording invasive alien species in Europe: experiences from two smartphone applications in northwest Europe. Management of Biological Invasions 6(2), 215225.
Babaioff, M., Dughmi, S., Kleinberg, R. & Slivkins, A. 2015. Dynamic pricing with limited supply. ACM Transactions on Economics and Computation 3(1), 4.
Benkler, Y. & Nissenbaum, H. 2006. Commons-based peer production and virtue. Journal of Political Philosophy 14(4), 394419.
Berg, N. & Gigerenzer, G. 2010. As-if behavioral economics: Neoclassical economics in disguise? History of Economic Ideas 18, 133165.
Blum, A. & Monsour, Y. 2007. Learning, regret minimization, and equilibria. In Algorithmic Game Theory, Nisan, N., Roughgarden, T., Tardos, E. & Vazirani, V. V. (eds). Cambridge University Press, 79102.
Bosha, E., Cilliers, L. & Flowerday, S. 2017. Incentive theory for a participatory crowdsourcing project in a developing country. SA Journal of Information Management 19(1), 7.
Brabham, D. C. 2013. Crowdsourcing. MIT Press.
Camerer, C. F. 2003. Behavioural studies of strategic thinking in games. Trends in Cognitive Sciences 7(5), 225231.
Cameron, J., Banko, K. M & Pierce, W. D. 2001. Pervasive negative effects of rewards on intrinsic motivation: the myth continues. The Behavior Analyst 24(1), 1.
Cooper, H. M. 1988. Organizing knowledge syntheses: a taxonomy of literature reviews. Knowledge in Society 1(1), 104126.
David, G., Michael, R., Erwin, F. & Martin, S. 2012. Crowdsourcing information systems: definition, typology and design. In ICIS 2012 The 33rd International Conference on Information Systems, Joey F George (ed.). Association for Information Systems/AIS Electronic Library (AISeL), 35623572.
Dayama, P., Narayanaswamy, B., Garg, D. & Narahari, Y. 2015. Truthful interval cover mechanisms for crowdsourcing applications. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, 1091–1099. International Foundation for Autonomous Agents and Multiagent Systems.
Drexler, K. E. & Miller, M. S. 1988. Incentive engineering for computational resource management. The Ecology of Computation 2, 231266.
Ebden, M., Huynh, D., Moreau, L. & Roberts, S. 2015. Incentive engineering through subgraph matching with application to task allocation. HAIDM.
Endriss, U., Kraus, S., Lang, J. & Wooldridge, M. 2011. Designing incentives for boolean games. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 1, 79–86. International Foundation for Autonomous Agents and Multiagent Systems.
Ghani, E., Kerr, W. R. & Stanton, C. 2014. Diasporas and outsourcing: evidence from odesk and India. Management Science 60(7), 16771697.
Ghezzi, A., Gabelloni, D., Martini, A. & Natalicchio, A. 2017. Crowdsourcing: a review and suggestions for future research. International Journal of Management Reviews 0, 121.
Groves, T. 1973. Incentives in teams. Econometrica: Journal of the Econometric Society 41, 617631.
Guo, B., Wang, Z., Yu, Z., Wang, Y., Yen, N. Y., Huang, R. & Zhou, X. 2015. Mobile crowd sensing and computing: the review of an emerging human-powered sensing paradigm. ACM Computing Surveys (CSUR) 48(1), 7.
Hamari, J. 2013. Transforming homo economicus into homo ludens: a field experiment on gamification in a utilitarian peer-to-peer trading service. Electronic Commerce Research and Applications 12(4), 236245.
Higgins, C. I., Williams, J., Leibovici, D. G., Simonis, I., Davis, M. J., Muldoon, C., van Genuchten, P., O’Hare, G. M. P. & Wiemann, S. 2016. Citizen OBservatory WEB (COBWEB): a generic infrastructure platform to facilitate the collection of citizen science data for environmental monitoring. International Journal of Spatial Data Infrastructures Research 11(1), 2048.
Howe, J. 2006a. Crowdsourcing: a definition. Retrieved 3 June 2006 from
Howe, J. 2006b. The rise of crowdsourcing. Wired Magazine 14(6), 14.
Kahneman, D. 2003. Maps of bounded rationality: psychology for behavioral economics. The American Economic Review 93(5), 14491475.
Koutsoupias, E. & Papadimitriou, C. 2009. Worst-case equilibria. Computer Science Review 3(2), 6569.
Mao, A., Kamar, E., Chen, Y., Horvitz, E., Schwamb, M. E., Lintott, C. J. & Smith, A. M. 2013. Volunteering versus work for pay: incentives and tradeoffs in crowdsourcing. In First AAAI Conference on Human Computation and Crowdsourcing.
McLeod, S.A. 2007. BF Skinner: Operant conditioning. Retrieved September 9, 2009.
Morschheuser, B., Hamari, J. & Koivisto, J. 2016. Gamification in crowdsourcing: a review. In 2016 49th Hawaii International Conference on System Sciences (HICSS), 4375–4384. IEEE.
Muller, C.L., Chapman, L., Johnston, S., Kidd, C., Illingworth, S., Foody, G., Overeem, A. & Leigh, R.R. 2015. Crowdsourcing for climate and atmospheric sciences: current status and future potential. International Journal of Climatology 35(11), 31853203.
Myerson, R. B. 1981. Optimal auction design. Mathematics of Operations Research 6(1), 5873.
Myerson, R. B. 2013. Game Theory. Analysis of Conflict. Harvard University Press.
Nash, J. 1951. Non-cooperative games. Annals of Mathematics 54, 286295.
Nisan, N., Roughgarden, T., Tardos, E. & Vazirani, V. V. 2007. Algorithmic Game Theory, 1. Cambridge University Press.
O’Grady, M. J., Muldoon, C., Carr, D., Wan, J., Kroon, B. & O’Hare, G. M. P. 2016. Intelligent Sensing for Citizen Science - Challenges and Future Directions. Mobile Networks and Applications 21(2), 375385.
Parkes, D. C. & Wellman, M. P. 2015. Economic reasoning and artificial intelligence. Science 349(6245), 267272.
Petersen, K., Vakkalanka, S. & Kuzniarz, L. 2015. Guidelines for conducting systematic mapping studies in software engineering: an update. Information and Software Technology 64, 118.
Rabin, M. 2002. A perspective on psychology and economics. European Economic Review 46(4), 657685.
Roughgarden, T. 2003. The price of anarchy is independent of the network topology. Journal of Computer and System Sciences 67(2), 341364.
Roughgarden, T. 2010. Algorithmic game theory. Communications of the ACM 53(7), 7886.
See, L., Fritz, S., Dias, E., Hendriks, E., Mijling, B., Snik, F., Stammes, P., Vescovi, F. D., Zeug, G., Mathieu, P.-P., Desnos, Y.-L. & Rast, M. 2016a. Supporting earth-observation calibration and validation: a new generation of tools for crowdsourcing and citizen science. IEEE Geoscience and Remote Sensing Magazine 4(3), 3850.
See, L., Mooney, P., Foody, G., Bastin, L., Comber, A., Estima, J., Fritz, S., Kerle, N., Jiang, B., Laakso, M., Liu, H.-Y., Milčinski, G., Nikšič, M., Painho, M., Pődör, A., Olteanu-Raimond, A.-M. & Rutzinger, M. 2016b. Crowdsourcing, citizen science or volunteered geographic information? The current state of crowdsourced geographic information. ISPRS International Journal of Geo-Information 5(5), 55.
See, L., Schepaschenko, D., Lesiv, M., McCallum, I., Fritz, S., Comber, A., Perger, C., Schill, C., Zhao, Y., Maus, V., Siraj, M.A., Albrecht, F., Cipriani, A., Vakolyuk, M., Garcia, A., Rabia, A.H., Singha, K., Marcarini, A.A., Kattenborn, T., Hazarka, R., Schepaschenko, M., van der Velde, M., Kraxner, F. & Obersteiner, M. 2015. Building a hybrid land cover map with crowdsourcing and geographically weighted regression. ISPRS Journal of Photogrammetry and Remote Sensing 103, 4856.
Senaratne, H., Mobasheri, A., Ali, A. L., Capineri, C. & Haklay, M. 2017. A review of volunteered geographic information quality assessment methods. International Journal of Geographical Information Science 31(1), 139167.
Shaw, A. D., Horton, J. J. & Chen, D. L. 2011. Designing incentives for inexpert human raters. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, 275–284. ACM.
Shah, N. B. & Zhou, D. 2015. Double or nothing: multiplicative incentive mechanisms for crowdsourcing. Advances in Neural Information Processing Systems 1, 19.
Shah-Mansouri, H. & Wong, V. W. S. 2015. Profit maximization in mobile crowdsourcing: a truthful auction mechanism. In2015 IEEE International Conference on Communications (ICC), 3216–3221. IEEE.
Shoham, Y. 2008. Computer science and game theory. Communications of the ACM 51(8), 7479.
Singla, A. & Krause, A. 2013. Truthful incentives in crowdsourcing tasks using regret minimization mechanisms. In Proceedings of the 22nd International Conference on World Wide Web, 1167–1178. International World Wide Web Conferences Steering Committee.
Stewart, O., Lubensky, D. & Huerta, J. M. 2010. Crowdsourcing participation inequality: a SCOUT model for the enterprise domain. In Proceedings of the ACM SIGKDD Workshop on Human Computation, 30–33. ACM.
Tran-Thanh, L., Stein, S., Rogers, A. & Jennings, N. R. 2014. Efficient crowdsourcing of unknown experts using bounded multi-armed bandits. Artificial Intelligence 214, 89111.
Tversky, A. & Kahneman, D. 1992. Advances in prospect theory: cumulative representation of uncertainty. Journal of Risk and uncertainty 5(4), 297323.
Vickrey, W. 1961. Counterspeculation, auctions, and competitive sealed tenders. The Journal of finance 16(1), 837.
Von Neumann, J. & Morgenstern, O. 1944. Theorem 3.1.18, Theory of games and economic behavior, Princeton University Press.
Wang, L., Zhang, D., Wang, Y., Chen, C., Han, X. & M’hamed, A. 2016. Sparse mobile crowdsensing: challenges and opportunities. IEEE Communications Magazine 54(7), 161167.
Yang, D., Xue, G., Fang, X. & Tang, J. 2012. Crowdsourcing to smartphones: incentive mechanism design for mobile phone sensing. In Proceedings of the 18th Annual International Conference on Mobile Computing and Networking, 173–184. ACM.
Yang, J., Adamic, L. A. & Ackerman, M. S. 2008. Crowdsourcing and knowledge sharing: strategic user behavior on taskcn. In Proceedings of the 9th ACM Conference on Electronic Commerce, 246–255. ACM.
Zhang, Y. & Van der Schaar, M. 2012. Reputation-based incentive protocols in crowdsourcing applications. In FOCOM, 2012 Proceedings IEEE, 2140–2148. IEEE.
Zhao, Y. & Han, Q. 2016. Spatial crowdsourcing: current state and future directions. IEEE Communications Magazine 54(7), 102107.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

The Knowledge Engineering Review
  • ISSN: 0269-8889
  • EISSN: 1469-8005
  • URL: /core/journals/knowledge-engineering-review
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed