Hostname: page-component-5d84bcc8dc-2jfv9 Total loading time: 0 Render date: 2026-09-09T15:10:41.465Z Has data issue: false hasContentIssue false
  • English
  • Français

Optimal design of experiments to identify latent behavioral types

Published online by Cambridge University Press:  14 March 2025

Stefano Balietti Open the ORCID record for Stefano Balietti [Opens in a new window] ,
Brennan Klein Open the ORCID record for Brennan Klein [Opens in a new window]  and
Christoph Riedl Open the ORCID record for Christoph Riedl [Opens in a new window]
Stefano Balietti
Affiliation:
Mannheim Center for European Social Science Research (MZES), Mannheim University, A5, 6, 68159 Mannheim, Germany Alfred-Weber Institute of Economics (AWI), Heidelberg University, Bergheimerstraße 20, 69115 Heidelberg, Germany
Brennan Klein
Affiliation:
Network Science Institute, Northeastern University, Boston, MA 02115, USA
Christoph Riedl*
Affiliation:
Network Science Institute, Northeastern University, Boston, MA 02115, USA D’Amore-McKim School of Business, Northeastern University, 360 Huntington Ave, Boston, MA 02115, USA Khoury College of Computer Sciences, Northeastern University, Boston, MA 02115, USA Institute for Quantitative Social Science, Harvard University, Cambridge, MA 02138, USA
*
c.riedl@neu.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Bayesian optimal experiments that maximize the information gained from collected data are critical to efficiently identify behavioral models. We extend a seminal method for designing Bayesian optimal experiments by introducing two computational improvements that make the procedure tractable: (1) a search algorithm from artificial intelligence that efficiently explores the space of possible design parameters, and (2) a sampling procedure which evaluates each design parameter combination more efficiently. We apply our procedure to a game of imperfect information to evaluate and quantify the computational improvements. We then collect data across five different experimental designs to compare the ability of the optimal experimental design to discriminate among competing behavioral models against the experimental designs chosen by a “wisdom of experts” prediction experiment. We find that data from the experiment suggested by the optimal design approach requires significantly less data to distinguish behavioral models (i.e., test hypotheses) than data from the experiment suggested by experts. Substantively, we find that reinforcement learning best explains human decision-making in the imperfect information game and that behavior is not adequately described by the Bayesian Nash equilibrium. Our procedure is general and computationally efficient and can be applied to dynamically optimize online experiments.

Keywords

Optimal experimental designBehavioral typesExpert predictionActive learning

JEL classification

C80: General C90: General C72: Noncooperative Games

Information

Type
Original Paper
Information
Experimental Economics , Volume 24 , Issue 3 , September 2021 , pp. 772 - 799
Copyright
Copyright © 2020 Economic Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Electronic supplementary material The online version of this article (https://doi.org/10.1007/s10683-020-09680-w) contains supplementary material, which is available to authorized users.

References

Azevedo, EM, Deng, A, Olea, JLM, Rao, J, & Weyl, EG (2019). A/B testing with fat tails. A/b testing with fat tails. Journal of Political Economy, 10.1086/710607Google Scholar
Bakshy, E., Dworkin, L., Karrer, B., Kashin, K. Letham, B., Murthy, A. & Singh, S. (2018). AE: A domain-agnostic platform for adaptive experimentation. In Conference on Neural Information Processing Systems (pp. 1–8). http://eytan.github.io/papers/ae_workshop.pdf.Google Scholar
Balietti, S. (2017). nodeGame: Real-time, synchronous, online experiments in the browser Behavior Research Methods(i), (1–31). https://doi.org/10.3758/s13428-016-0824-z.CrossRefGoogle Scholar
Berman, R (2018). Beyond the last touch: Attribution in online advertising. Marketing Science, 37(5), 771–792. 10.1287/mksc.2018.110410.1287/mksc.2018.1104CrossRefGoogle Scholar
Berman, R, Pekelis, L, Scott, A, & Van den Bulte, C (2018). p-Hacking and false discovery in A/B testing. Available at SSRN, 10.2139/ssrn.320479110.2139/ssrn.3204791CrossRefGoogle Scholar
Bramoullé, Y., Djebbari, H., & Fortin, B. (2020). Peer effects in networks: A survey. CEPR Discussion Paper No. DP14260. http://ftp.iza.org/dp12947.pdf.Google Scholar
Camerer, C. F. (2011). Behavioral game theory: Experiments in strategic interaction. Princeton University Press. http://psycnet.apa.org/record/2003-06054-000.Google Scholar
Camerer, CF, & Ho, T-H (1999). Experience-weighted attraction learning in normal form games. Econometrica, 67(4), 27–874. 10.1111/1468-0262.0005410.1111/1468-0262.00054CrossRefGoogle Scholar
Chapman, J., Snowberg, E., Wang, S., & Camerer, C. (2018). Loss attitudes in the US population: Evidence from dynamically optimized sequential experimentation (DOSE). National Bureau of Economic Research, (1–55). https://doi.org/10.3386/w25072.CrossRefGoogle Scholar
Contal, E., Buffoni, D., Robicquet, A., & Vayatis, N. (2013). Parallel Gaussian process optimization with upper confidence bound and pure exploration. In Machine Learning and Knowledge Discovery in Databases (pp. 225–240). Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-40988-2_15.Google Scholar
David, PA (1985). Clio and the economics of QWERTY. American Economic Review, 75(2), 332–337.Google Scholar
De Freitas, N., Smola, A. J., & Zoghi, M. (2012). Exponential regret bounds for Gaussian process bandits with deterministic observations. In Proceedings of the 29th International Coference on International Conference on Machine Learning (pp. 955–962). https://doi.org/10.5555/3042573.3042697.CrossRefGoogle Scholar
DellaVigna, S, & Pope, D (2017). What motivates effort? Evidence and expert forecasts. Review of Economic Studies, 85(2), 1029–1069. 10.1093/restud/rdx03310.1093/restud/rdx033CrossRefGoogle Scholar
Eckles, D., & Kaptein, M. C. (2014). Thompson sampling with the online bootstrap. arXiv, (1–13). arxiv: 1410.4009.Google Scholar
El-Gamal, MA, & Palfrey, TR (1996). Economical experiments: Bayesian efficient experimental design. International Journal of Game Theory, 25, 495–517. 10.1007/BF0180395310.1007/BF01803953CrossRefGoogle Scholar
Erev, I, & Roth, AE (1998). Predicting how people play games: Reinforcement learning in games with unique strategy equilibrium. American Economic Review, 88(4), 848–881.Google Scholar
Erev, I, & Roth, AE (2014). Maximization, learning, and economic behavior. Proceedings of the National Academy of Sciences, 111, 10818–10825. 10.1073/pnas.1402846111CrossRefGoogle ScholarPubMed
Feltovich, N (2000). Reinforcement-based versus belief-based learning models in experimental asymmetric-information games. Econometrica, 68(3), 605–641. 10.1111/1468-0262.00125CrossRefGoogle Scholar
Fershtman, C, & Pakes, A (2012). Dynamic games with asymmetric information: A framework for empirical work. The Quarterly Journal of Economics, 127(4), 1611–1661. 10.1093/qje/qjs02510.1093/qje/qjs025CrossRefGoogle Scholar
Fisher, RA (1936). The design of experiments. American Mathematical Monthly, 43(3), 180 10.2307/2300364Google Scholar
Foley, M, Forber, P, Smead, R, & Riedl, C (2018). Conflict and convention in dynamic networks. Journal of the Royal Society Interface, 15(140), 20170835 10.1098/rsif.2017.083510.1098/rsif.2017.0835CrossRefGoogle ScholarPubMed
Fudenberg, D, & Tirole, J (1991). Game theory, Cambridge: MIT Press.Google Scholar
Gale, J, Binmore, KG, & Samuelson, L (1995). Learning to be imperfect: The ultimatum game. Games and Economic Behavior, 8(1), 56–90. 10.1016/S0899-8256(05)80017-X10.1016/S0899-8256(05)80017-XCrossRefGoogle Scholar
Gilchrist, DS, & Sands, EG (2016). Something to talk about: Social spillovers in movie consumption. Journal of Political Economy, 124(5), 1339–1382. 10.1086/68817710.1086/688177CrossRefGoogle Scholar
Goldman, M, & Rao, J (2016). Experiments as instruments: Heterogeneous position effects in sponsored search auctions. EEAI Endorsed Transactions on Serious Games, 10.4108/eai.8-8-2015.2261043Google Scholar
Görtler, J, Kehlbeck, R, & Deussen, O (2019). A visual exploration of Gaussian processes. Distill, 10.23915/distill.0001710.23915/distill.00017CrossRefGoogle Scholar
Harsanyi, JC (1967). Games with incomplete information played by “Bayesian” players, Part I. The Basic Model. Management Science, 14(3), 159–182. 10.1287/mnsc.1040.0270Google Scholar
Hertwig, R, & Ortmann, A (2001). Experimental practices in economics: A methodological challenge for psychologists?. Behavioral and Brain Sciences, 24(3), 383–403. 10.1037/e683322011-03210.1017/S0140525X01004149CrossRefGoogle ScholarPubMed
Hill, TP (1995). A statistical derivation of the significant-digit law. Statistical Science, 10(4), 354–363. 10.2307/224613410.1214/ss/1177009869CrossRefGoogle Scholar
Ho, T-H, Wang, X, & Camerer, CF (2008). Individual differences in EWA learning with partial payoff information. The Economic Journal, 118(525), 37–59. 10.1111/j.1468-0297.2007.02103.x10.1111/j.1468-0297.2007.02103.xCrossRefGoogle Scholar
Horton, JJ, Rand, DG, & Zeckhauser, RJ (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425. 10.1007/s10683-011-9273-910.1007/s10683-011-9273-9CrossRefGoogle Scholar
Imai, T., & Camerer, C. F. (2018). Estimating time preferences from budget set choices using optimal adaptive design. working paper. http://taisukeimai.com/files/adaptive_ctb.pdf.Google Scholar
Kachelmeier, SJ, & Towry, KL (2005). The limitations of experimental design: A case study involving monetary incentive effects in laboratory markets. Experimental Economics, 8(1), 21–33. 10.1007/s10683-005-0435-510.1007/s10683-005-0435-5CrossRefGoogle Scholar
Katz, ML, & Shapiro, C (1985). Network externalities, competition, and compatibility. American Economic Review, 75(3), 424–440.Google Scholar
Knez, M, & Camerer, CF (1994). Creating expectational assets in the laboratory: Coordination in ‘weakest-link’ games. Strategic Management Journal, 15(1 S), 101–119. 10.1002/smj.425015090810.1002/smj.4250150908CrossRefGoogle Scholar
Kohavi, R, Longbotham, R, Sommerfield, D, & Henne, RM (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140–181. 10.1007/s10618-008-0114-110.1007/s10618-008-0114-1CrossRefGoogle Scholar
Kohavi, R, & Thomke, S (2017). The surprising power of online experiments. Harvard Business Review, 95(5), 2–9.Google Scholar
Kullback, S, & Leibler, RA (1951). On information and sufficiency. The Annals of Mathematical Statistics, 22(1), 79–86. 10.1214/aoms/117772969410.1214/aoms/1177729694CrossRefGoogle Scholar
Letham, B., Karrer, B., Ottoni, G., & Bakshy, E. (2017). Constrained Bayesian optimization with noisy experiments. arXiv, 1–20. arxiv: 1706.07094.Google Scholar
Mason, W, & Suri, S (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44(1), 1–23. 10.3758/s13428-011-0124-610.3758/s13428-011-0124-6CrossRefGoogle ScholarPubMed
McIntyre, DP, & Chintakananda, A (2014). Competing in network markets: Can the winner take all?. Business Horizons, 57(1), 117–125. 10.1016/j.bushor.2013.09.00510.1016/j.bushor.2013.09.005CrossRefGoogle Scholar
Paolacci, G, Chandler, J, & Ipeirotis, PG (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.10.1017/S1930297500002205CrossRefGoogle Scholar
Parker, BM, Gilmour, SG, & Schormans, J (2017). Optimal design of experiments on connected units with application to social networks. Journal of the Royal Statistical Society C, 66(3), 455–480. 10.1111/rssc.12170CrossRefGoogle Scholar
Phan, TQ, & Airoldi, EM (2015). A natural experiment of social network formation and dynamics. Proceedings of the National Academy of Sciences, 112(21), 6595–6600. 10.1073/pnas.140477011210.1073/pnas.1404770112CrossRefGoogle ScholarPubMed
Pooseh, S, Bernhardt, N, Guevara, A, Huys, QJM, & Smolka, MN (2018). Value-based decision-making battery: A Bayesian adaptive approach to assess impulsive and risky behavior. Behavior Research Methods, 50(1), 236–249. 10.3758/s13428-017-0866-x10.3758/s13428-017-0866-xCrossRefGoogle ScholarPubMed
Rasmussen, C. E., & Williams, C. K. I. (2005). Gaussian processes for machine learning The MIT Press. http://gaussianprocess.org/gpml/.10.7551/mitpress/3206.001.0001CrossRefGoogle Scholar
Rzhetsky, A., Foster, J. G., Foster, I. T., & Evans, J. A. (2015). Choosing experiments to accelerate collective discovery. In Proceedings of the National Academy of Sciences (pp. 1–6). https://doi.org/10.1073/pnas.1509757112.CrossRefGoogle Scholar
Salmon, TC (2001). An evaluation of econometric models of adaptive learning. Econometrica, 69(6), 1597–1628. 10.1111/1468-0262.0025810.1111/1468-0262.00258CrossRefGoogle Scholar
Sarin, R, & Vahid, F (2001). Predicting how people play games: A simple dynamic model of choice. Games and Economic Behavior, 34(1), 104–122. 10.1006/game.1999.078310.1006/game.1999.0783CrossRefGoogle Scholar
Schwartz, EM, Bradlow, ET, & Fader, PS (2017). Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Science, 36(4), 500–522. 10.1287/mksc.2016.102310.1287/mksc.2016.1023CrossRefGoogle Scholar
Sobol, IM (1998). On quasi-Monte Carlo integrations. Mathematics and Computers in Simulation, 47(2), 103–112. 10.1016/S0378-4754(98)00096-210.1016/S0378-4754(98)00096-2CrossRefGoogle Scholar
Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. (2010). Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning (pp. 1015–1022). https://doi.org/10.5555/3104322.3104451.CrossRefGoogle Scholar
Stefanakis, TS, Contal, E, Vayatis, N, Dias, F, & Synolakis, CE (2014). Can small islands protect nearby coasts from tsunamis? An active experimental design approach. Proceedings of the Royal Society A, 470(2172), 1–20. 10.1098/rspa.2014.0575Google Scholar
Tauber, EM (1972). Why do people shop?. Journal of Marketing, 36(4), 46–49. 10.2307/1250426Google Scholar
Wang, S. W., Filiba, M., & Camerer, C. F. (2010). Dynamically optimized sequential experimentation (DOSE) for estimating economic preference parameters. arXiv, 1–41. http://pdfs.semanticscholar.org/1707/ded4fdc981aedc2a2f6bab077fcf37acb7d5.pdf.Google Scholar
Zhou, S., Valentine, M., & Bernstein, M. S. (2018). In search of the dream team: Temporally constrained multi-armed bandits for identifying effective team structures. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1–13). https://doi.org/10.1145/3173574.3173682.CrossRefGoogle Scholar
Supplementary material: File

Balietti et al. supplementary material

Appendix for Experimental Economics article: Optimal design of experiments to identify latent behavioral types
Download Balietti et al. supplementary material(File)
File 1.4 MB