DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering
Neha Srikanth, Jordan Boyd-Graber, Rachel Rudinger
TLDR
DiscoTrace reveals LLMs lack rhetorical diversity and over-address questions compared to humans, guiding pragmatic LLM development.
Key contributions
- Introduces DiscoTrace, a method to analyze rhetorical strategies in information-seeking QA.
- Reveals human communities exhibit diverse preferences for answer construction strategies.
- Demonstrates LLMs lack rhetorical diversity, even when prompted to mimic specific human styles.
- Highlights LLMs systematically address more question interpretations than human answerers.
Why it matters
This paper is crucial for understanding pragmatic differences between human and LLM answering strategies. It highlights LLMs' lack of rhetorical diversity and tendency to over-address questions, a key limitation. This research guides developing more context-aware and human-like LLM answerers.
Original Abstract
We introduce DiscoTrace, a method to identify the rhetorical strategies that answerers use when responding to information-seeking questions. DiscoTrace represents answers as a sequence of question-related discourse acts paired with interpretations of the original question, annotated on top of rhetorical structure theory parses. Applying DiscoTrace to answers from nine different human communities reveals that communities have diverse preferences for answer construction. In contrast, LLMs do not exhibit rhetorical diversity in their answers, even when prompted to mimic specific human community answering guidelines. LLMs also systematically opt for breadth, addressing interpretations of questions that human answerers choose not to address. Our findings can guide the development of pragmatic LLM answerers that consider a range of strategies informed by context in QA.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.