Book contents
- Frontmatter
- Contents
- Contributors
- Acknowledgments
- Part I Introduction
- Part II Taxonomy of Metacognitive Approaches
- Part III Neuro-Symbolic Models in AI
- Part IV Metacognition with LLMS
- 7 Metacognitive Intervention for Accountable LLMs through Sparsity
- 8 Metacognitive Insights into ChatGPT’s Arithmetic Reasoning
- Part V Metacognition in Learning Agents
- Part VI Assured Machine Learning in High-Stakes Domains
- Part VII Metacognition as a Solution to Handle Failure
- Part VIII Applications of Metacognitive AI
8 - Metacognitive Insights into ChatGPT’s Arithmetic Reasoning
from Part IV - Metacognition with LLMS
Published online by Cambridge University Press: aN Invalid Date NaN
Book contents
- Frontmatter
- Contents
- Contributors
- Acknowledgments
- Part I Introduction
- Part II Taxonomy of Metacognitive Approaches
- Part III Neuro-Symbolic Models in AI
- Part IV Metacognition with LLMS
- 7 Metacognitive Intervention for Accountable LLMs through Sparsity
- 8 Metacognitive Insights into ChatGPT’s Arithmetic Reasoning
- Part V Metacognition in Learning Agents
- Part VI Assured Machine Learning in High-Stakes Domains
- Part VII Metacognition as a Solution to Handle Failure
- Part VIII Applications of Metacognitive AI
Summary
We study the performance of a commercially available large language model (LLM) known as ChatGPT on math word problems (MWPs) from the dataset DRAW-1K. To our knowledge, this is the first independent evaluation of ChatGPT. We found that ChatGPT’s performance changes dramatically based on the requirement to show its work, failing $20\%$ of the time when it provides work compared with $84\%$ when it does not. Further, several factors about MWPs relate to the number of unknowns and number of operations that lead to a higher probability of failure when compared with the prior, specifically noting (across all experiments) that the probability of failure increases linearly with the number of addition and subtraction operations. We also have released the dataset of ChatGPT’s responses to the MWPs to support further work on the characterization of LLM performance and present baseline machine learning models to predict if ChatGPT can correctly answer an MWP.
Information
- Type
- Chapter
- Information
- Metacognitive Artificial Intelligence , pp. 121 - 132Publisher: Cambridge University PressPrint publication year: 2025
Accessibility standard: Unknown
Accessibility compliance for the PDF of this book is currently unknown
and may be updated in the future.