The Effect of Document Selection on Query-focused Text Analysis
Sandesh S Rangreji, Mian Zhong, Anjalie Field
TLDR
This paper systematically evaluates document selection methods for query-focused text analysis, finding semantic/hybrid retrieval are optimal.
Key contributions
- Systematically evaluated seven document selection methods for query-focused text analysis.
- Tested selection methods on four text analysis techniques (LDA, BERTopic, TopicGPT, HiCode).
- Identified semantic or hybrid retrieval as robust and efficient strategies for document selection.
- Framework establishes data selection as a critical methodological decision, inviting new strategies.
Why it matters
Document selection is a critical, often overlooked step in text analysis. This paper offers practical guidance by identifying robust and efficient selection strategies like semantic or hybrid retrieval. It establishes data selection as a methodological decision, paving the way for future research.
Original Abstract
Analyses of document collections often require selecting what data to analyze, as not all documents are relevant to a particular research question and computational constraints preclude analyzing all documents, yet little work has examined effects of selection strategy choices. We systematically evaluate seven selection methods (from random selection to hybrid retrieval) on outputs from four text analyses methods (LDA, BERTopic, TopicGPT, HiCode) over two datasets with 26 open-ended queries. Our evaluation reveals practice guidance: semantic or hybrid retrieval offer strong go-to approaches that avoid the pitfalls of weaker selection strategies and the unnecessary compute overhead of more complicated ones. Overall, our evaluation framework establishes data selection as a methodological decision, rather than a practical necessity, inviting the development of new strategies.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.