Hostname: page-component-76fb5796d-dfsvx Total loading time: 0 Render date: 2024-04-27T07:16:28.065Z Has data issue: false hasContentIssue false
  • English
  • Français

The Theory-Practice Gap in the Evaluation of Agent-Based Social Simulations

Published online by Cambridge University Press:  17 January 2023

David Anzola
David Anzola*
Affiliation:
Innovation Center, School of Management, Universidad del Rosario, Bogotá, Colombia
*
Email: david.anzola@urosario.edu.co
Get access Rights & Permissions [Opens in a new window]

Argument

Agent-based social simulations have historically been evaluated using two criteria: verification and validation. This article questions the adequacy of this dual evaluation scheme. It claims that the scheme does not conform to everyday practices of evaluation, and has, over time, fostered a theory-practice gap in the assessment of social simulations. This gap originates because the dual evaluation scheme, inherited from computer science and software engineering, on one hand, overemphasizes the technical and formal aspects of the implementation process and, on the other hand, misrepresents the connection between the conceptual and the computational model. The mismatch between evaluation theory and practice, it is suggested, might be overcome if practitioners of agent-based social simulation adopt a single criterion evaluation scheme in which: i) the technical/formal issues of the implementation process are tackled as a matter of debugging or instrument calibration, and ii) the epistemological issues surrounding the connection between conceptual and computational models are addressed as a matter of validation.

Keywords

VerificationValidationRepresentationConceptual modelingAgent-based modelingSocial simulation
Type
Research Article
Information
Science in Context , Volume 34 , Issue 3 , September 2021 , pp. 393 - 410
Check for updates
Copyright
© The Author(s), 2023. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Anzola, David. 2019. “Knowledge Transfer in Agent-Based Computational Social Science.” Studies in History and Philosophy of Science Part A 77: 29–38.CrossRefGoogle ScholarPubMed
Anzola, David. 2021a. “Capturing the Representational and the Experimental in the Modelling of Artificial Societies.” European Journal for Philosophy of Science 11. https://doi.org/10.1007/s13194-021-00382-5 CrossRefGoogle Scholar
Anzola, David. 2021b. “Disagreement in Discipline-Building Processes.” Synthese 198: 6201–6224.CrossRefGoogle Scholar
Anzola, David. 2021c. “Social Epistemology and Validation in Agent-Based Social Simulation.” Philosophy & Technology 34: 1333–1361.CrossRefGoogle Scholar
Arabatzis, Theodore. 2013. “Experiment.” In The Routledge Companion to Philosophy of Science, edited by Psillos, Stathis and Curd, Martin, 191–202. New York: Routledge.Google Scholar
Argent, Robert, Richard Sojda, Carlo Guipponi, McIntosh, Brian, Voinov, Alexey, and Maier, Holger. 2016. “Best Practices for Conceptual Modelling in Environmental Planning and Management.” Environmental Modelling and Software 80: 113–21.CrossRefGoogle Scholar
Augusiak, Jacqueline, Van den Brink, Paul, and Grimm, Volker. 2014. “Merging Validation and Evaluation of Ecological Models to ‘Evaludation’: A Review of Terminology and a Practical Approach.” Ecological Modelling 280: 117–28.CrossRefGoogle Scholar
Axelrod, Robert. 1997. “Advancing the Art of Simulation in the Social Sciences.” In Simulating Social Phenomena, edited by Conte, Rosaria, Hegselmann, Rainer, and Terna, Pietro, 21–40. Berlin: Springer.CrossRefGoogle Scholar
Axtell, Robert, Axelrod, Robert, Epstein, Joshua, and Cohen, Michael. 1996. “Aligning Simulation Models: A Case Study and Results.” Computational & Mathematical Organization Theory 1 (2): 123–41.CrossRefGoogle Scholar
Balci, Osman. 1997. “Verification, Validation and Accreditation of Simulation Models.” In Proceedings of the 1997 Winter Simulation Conference, edited by Andradóttir, Sigrun, Healy, Kevin, Withers, David, and Nelson, Barry, 135–41. Atlanta: IEEE.Google Scholar
Balci, Osman. 2003. “Verification, Validation, and Certification of Modeling and Simulation Applications.” In Proceedings of the 2003 Winter Simulation Conference, edited by Chick, Stephen, Sánchez, Paul, Ferrin, David, and Morrice, Douglas. New Orleans: IEEE.Google Scholar
Brooks, Roger, and Wang, Wang. 2015. “Conceptual Modelling and the Project Process in Real Simulation Projects: A Survey of Simulation Modellers.” Journal of the Operational Research Society 66 (10): 1669–85.CrossRefGoogle Scholar
Campbell, Donald. 1957. “Factors Relevant to the Validity of Experiments in Social Settings.” Psychological Bulletin 54 (4): 297–312.CrossRefGoogle Scholar
Cardoso, Rafael. 2010. “Craft Versus Design: Moving Beyond a Tired Dichotomy.” In The Craft Reader, edited by Adamson, Glenn, 321–332. Oxford: Berg Publishers.Google Scholar
Cioffi-Revilla, Claudio. 2014. Introduction to Computational Social Science. Berlin: Springer.CrossRefGoogle Scholar
Colburn, Timothy. 2004. “Methodology of Computer Science.” In The Blackwell Guide to the Philosophy of Computing and Information, edited by Floridi, Luciano, 318–326. New York: Blackwell.Google Scholar
David, Nuno. 2009. “Validation and Verification in Social Simulation: Patterns and Clarification of Terminology.” In Epistemological Aspects of Computer Simulation in the Social Sciences, edited by Squazzoni, Flaminio, 117–129. Berlin: Springer.CrossRefGoogle Scholar
David, Nuno. 2013. “Validating Simulations.” In Simulating Social Complexity, edited by Edmonds, Bruce and Meyer, Ruth, 135–171. Berlin: Springer.CrossRefGoogle Scholar
David, Nuno, Simão, Jaime, and Coelho, Helder. 2005. “The Logic of the Method of Agent-Based Simulation in the Social Sciences: Empirical and Intentional Adequacy of Computer Programs.” Journal of Artificial Societies and Social Simulation 8 (4). http://jasss.soc.surrey.ac.uk/8/4/2.html.Google Scholar
David, Nuno, Simão, Jaime, and Coelho, Helder. 2007. “Simulation as Formal and Generative Social Science: The Very Idea.” In Worldviews, Science and Us, edited by Gershenson, Carlos, Aerts, Diederik, and Edmonds, Bruce, 266–284. Singapore: World Scientific.CrossRefGoogle Scholar
Denning, Peter. 2010. “The Great Principles of Computing.” American Scientist 98 (5): 369–72.Google Scholar
Edmonds, Bruce. 2000. “The Use of Models - Making MABS More Informative.” In Multi-Agent-Based Simulation, edited by Moss, Scott and Davidsson, Paul, 15–32. Berlin: Springer.Google Scholar
Epstein, Joshua. 1999. “Agent-Based Computational Models and Generative Social Science.” Complexity 4 (5): 41–60.3.0.CO;2-F>CrossRefGoogle Scholar
Evans, Michael. 1984. Productive Software Test Management. New York: Wiley.Google Scholar
Fetzer, James. 2001. Computers and Cognition: Why Minds Are Not Machines. Dordrecht: Springer.CrossRefGoogle Scholar
Fraassen, Bas van. 2008. Scientific Representation. Oxford: Oxford University Press.CrossRefGoogle Scholar
Frigg, Roman, and Nguyen, James. 2017. “Models and Representation.” In Springer Handbook of Model-Based Science, edited by Magnani, Lorenzo and Bertolotti, Tommaso, 49–102. Berlin: Springer.CrossRefGoogle Scholar
Galán, José, Luis Izquierdo, Segismundo Izquierdo, José Santos, Ricardo del Olmo, López-Paredes, Adolfo, and Edmonds, Bruce. 2009. “Errors and Artefacts in Agent-Based Modelling.” Journal of Artificial Societies and Social Simulation 12 (1). http://jasss.soc.surrey.ac.uk/12/1/1.html.Google Scholar
Galison, Peter. 1987. How Experiments End. Chicago: University of Chicago Press.Google Scholar
Gilbert, Nigel. 2008. Agent-Based Models. London: Sage.CrossRefGoogle Scholar
Gilbert, Nigel, and Troitzsch, Klaus. 2005. Simulation for the Social Scientist. Glasgow: Open University Press.Google Scholar
Graebner, Claudius. 2018. “How to Relate Models to Reality? An Epistemological Framework for the Validation and Verification of Computational Models.” Journal of Artificial Societies and Social Simulation 21 (3). http://jasss.soc.surrey.ac.uk/21/3/8.html.CrossRefGoogle Scholar
Guala, Francesco. 2002. “Models, Simulations, and Experiments.” In Model-Based Reasoning: Science, Technology, Values, edited by Magnani, Lorenzo and Nersessian, Nancy, 59–74. New York: Kluwer.CrossRefGoogle Scholar
Knuuttila, Tarja. 2021. “Imagination Eextended and Embedded: Artifactual versus Fctional Accounts of Models.” Synthese 198 (21): 5077–5097.CrossRefGoogle Scholar
Louie, Marcus, and Carley, Kathleen. 2008. “Balancing the Criticisms: Validating Multi-Agent Models of Social Systems.” Simulation Modelling Practice and Theory 16 (2): 242–56.CrossRefGoogle Scholar
Mcnamara, Laura, Timothy Trucano, George Backus, Mitchell, Scott, and Slepoy, Alexander. 2008. R&D for Computational Cognitive and Social Models: Foundations for Model Evaluation through Verification and Validation. Albuquerque: Sandia National Laboratories.Google Scholar
Morgan, Mary. 2003. “Experiments Without Material Intervention: Model Experiments, Virtual Experiments and Virtually Experiments.” In The Philosophy of Scientific Experimentation, edited by Radder, Hans, 216–235. Pittsburgh: University of Pittsburgh Press.CrossRefGoogle Scholar
Müller, Peter, and von Storch, Hans. 2004. Computer Modelling in Atmospheric and Oceanic Sciences: Building Knowledge. Berlin: Springer.CrossRefGoogle Scholar
Nardin, Luis, Giulia Andrighetto, Rosaria Conte, Áron Székely, David Anzola, Corinna Elsenbroich, Ulf Lotzmann, Neumann, Martin, Punzo, Valentina, and Troitzsch, Klaus. 2016. “Simulating Protection Rackets: A Case Study of the Sicilian Mafia.” Autonomous Agents and Multi-Agent Systems 30 (6): 1117–1147.CrossRefGoogle Scholar
Nersessian, Nancy, and MacLeod, Miles. 2017. “Models and Simulations.” In Springer Handbook of Model-Based Science, edited by Magnani, Lorenzo and Bertolotti, Tommaso. Berlin: Springer.Google Scholar
Norling, Emma, Edmonds, Bruce, and Meyer, Ruth. 2013. “Informal Approaches to Developing Simulation Models.” In Simulating Social Complexity, edited by Edmonds, Bruce and Meyer, Ruth, 39–56. Berlin: Springer.CrossRefGoogle Scholar
Pool, Robert. 2011. “Modeling Sociocultural Behavior.” In Sociocultural Data to Accomplish Department of Defense Missions. Washington D.C.: National Academies Press.Google Scholar
Powell, Stephen, and Willemain, Thomas. 2007. “How Novices Formulate Models. Part I: Qualitative Insights and Implications for Teaching.” Journal of the Operational Research Society 58 (8): 983–95.CrossRefGoogle Scholar
Quine, Willard. 2013. Word and Object. Cambridge: MIT Press.CrossRefGoogle Scholar
Rand, William, and Rust, Roland. 2011. “Agent-Based Modeling in Marketing: Guidelines for Rigor.” International Journal of Research in Marketing 28 (3): 181–93.CrossRefGoogle Scholar
Robinson, Stewart. 2011. “Conceptual Modeling for Simulation: Definition and Requirements.” In Conceptual Modeling for Discrete-Event Simulation, edited by Robinson, Stewart, Brooks, Roger, Kotiadis, Kathy, and van der Zee, Durk-Jouke, 3–30. London: CRC Press.Google Scholar
Robinson, Stewart. 2020. “Conceptual Modelling for Simulation: Progress and Grand Challenges.” Journal of Simulation 14 (1): 1–20.CrossRefGoogle Scholar
Sargent, Robert. 2013. “Verification and Validation of Simulation Models.” Journal of Simulation 7 (1): 12–24.CrossRefGoogle Scholar
Scherer, Sabrina, Maria Wimmer, Ulf Lotzmann, Moss, Scott, and Pinotti, Daniele. 2015. “Evidence Based and Conceptual Model Driven Approach for Agent-Based Policy Modelling.” Journal of Artificial Societies and Social Simulation 18 (3). http://jasss.soc.surrey.ac.uk/18/3/14.html.CrossRefGoogle Scholar
Schulze, Jule, Müller, Birgit, Groeneveld, Jürgen, and Grimm, Volker. 2017. “Agent-Based Modelling of Social-Ecological Systems: Achievements, Challenges, and a Way Forward.” Journal of Artificial Societies and Social Simulation 20 (2). http://jasss.soc.surrey.ac.uk/20/2/8.html.CrossRefGoogle Scholar
Shadish, William, Cook, Thomas, and Campbell, Donald. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. New York: Houghton Mifflin.Google Scholar
Shiflet, Angela, and Shiflet, George. 2014. Introduction to Computational Science. Princeton: Princeton University Press.Google Scholar
Shiner, Larry. 2012. “‘Blurred Boundaries’? Rethinking the Concept of Craft and Its Relation to Art and Design.” Philosophy Compass 7 (4): 230–44.CrossRefGoogle Scholar
Squazzoni, Flaminio. 2012. Agent-Based Computational Sociology. London: Wiley.CrossRefGoogle Scholar
Tedre, Matti. 2015. The Science of Computing. London: CRC Press.Google Scholar
Troitzsch, Klaus. 2016. “Extortion Rackets: An Event-Oriented Model of Interventions.” In Social Dimensions of Organised Crime, edited by Elsenbroich, Corinna, Anzola, David, and Gilbert, Nigel, 117–132. Berlin: Springer.CrossRefGoogle Scholar
van der Zee, Durk-Jouke, Tolk, Andreas, Pidd, Mike, Kotiadis, Kathy, and Tako, Antuela. 2011. “Education on Conceptual Modeling for Simulation – Beyond the Craft: A Summary of a Recent Expert Panel Discussion.” SCS M&S Magazine 2.Google Scholar
Wang, Wang, and Brooks, Roger. 2011. “Improving the Understanding of Conceptual Modeling.” In Conceptual Modeling for Discrete-Event Simulation, edited by Robinson, Stewart, Brooks, Roger, Kotiadis, Kathy, and van der Zee, Durk-Jouke, 57–70. London: CRC Press.Google Scholar
Willemain, Thomas. 1994. “Insights on Modeling from a Dozen Experts.” Operations Research 42 (2): 213–22.CrossRefGoogle Scholar
Winsberg, Eric. 2010. Science in the Age of Computer Simulation. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Wolfram, Stephen. 1984. “Computer Software in Science and Mathematics.” Scientific American 251 (1): 188–203.CrossRefGoogle Scholar