Think-Aloud Reshapes Automated Cognitive Model Discovery Beyond Behavior
Hanbo Xie, Akshay K. Jagadish, Lan Pan, Robert C. Wilson
TLDR
This paper shows that using think-aloud data significantly improves automated cognitive model discovery, revealing mechanisms beyond behavioral data alone.
Key contributions
- Introduces think-aloud traces as a novel data constraint for automated cognitive model discovery.
- Achieves significantly improved predictive performance on held-out data in risky decision-making.
- Discovered models shift structural classes (e.g., from Explicit comparator to Integrated utility).
- Enables identification of cognitive mechanisms not recoverable from behavior alone.
Why it matters
Current cognitive models from LLMs are under-determined by behavior alone. This work demonstrates a novel approach using process-level language data to build more accurate and structurally distinct models. It offers a path to uncover deeper cognitive mechanisms.
Original Abstract
Computational cognitive models discovered using large language models have so far relied solely on behavioral data. However, it is well-known that models produced from the behavioral trajectory alone are typically under-determined. In this work, we explore the use of Think Aloud traces as an additional form of data constraint during automated model discovery. When applied to the domain of risky decision-making, we find that the models discovered with think-aloud achieve significantly improved predictive performance on held-out data. Additionally, we find that the discovered models belong to different structural classes than those discovered from behavior alone for the majority of participants (69.4\%), specifically, it shifts from Explicit comparator towards Integrated utility. These results suggest that process-level language data not only improve model fit, but also systematically reshape the structure of the discovered cognitive models, enabling the identification of mechanisms that are not recoverable from behavior alone.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.