Measuring Choice Difficulty
Chris Chambers, Yusufcan Masatolioglu, Paulo Natenzon, Collin Raymond
TLDR
This paper shows that common measures of choice difficulty (understanding, randomness, confidence) are generally unrelated, providing a framework for their relationship.
Key contributions
- Establishes a theoretical framework for understanding choice difficulty measures.
- Demonstrates that understanding, choice randomness, and confidence are generally unrelated.
- Identifies sufficient conditions under which different difficulty measures align.
- Shows confidence equals understanding in psychophysical tasks paying for correctness.
Why it matters
This paper is crucial for researchers interpreting choice difficulty measures, highlighting that common metrics are often unrelated. It suggests caution in applying these measures and questions their portability between economic and psychophysical experiments.
Original Abstract
We provide a theoretical framework to understand how widely used measures of choice difficulty relate. In a binary-option Bayesian expected-utility framework, we show that three measures of difficulty, (i) understanding (ex-ante value), (ii) choice randomness, and (iii) confidence that the chosen option is ex post correct, are, in general, unrelated, and that this result extends to other potential measures like attenuation. We provide intuitive sufficient conditions which align the orders, using both restrictions on Blackwell experiments that capture well known classes (such as logit) and restrictions on payoffs and demonstrate that in psychophysical tasks that pay only for correctness, confidence coincides with understanding. We show willingness-to-accept to switch, when measured in utils, is equivalent to understanding. Our results suggest caution in interpreting measures of choice difficulty as well as the degree of portability between economics and psychophysics experiments
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.