AgenticAI-DialogGen: Topic-Guided Conversation Generation for Fine-Tuning and Evaluating Short- and Long-Term Memories of LLMs
Manoj Madushanka Perera, Adnan Mahmood, Kasun Eranda Wijethilake, Quan Z. Sheng
TLDR
AgenticAI-DialogGen is an agent-based framework that generates topic-guided conversations and a dataset to fine-tune LLMs for improved short- and long-term memory.
Key contributions
- Introduces AgenticAI-DialogGen, an agent-based framework for generating persona-grounded, topic-guided conversations.
- Automatically extracts knowledge graphs, identifies topics, and builds speaker personas from unstructured conversations.
- Generates memory-grounded QA pairs from both short- and long-term conversational histories.
- Creates TopicGuidedChat (TGC) dataset, improving LLM performance on memory-grounded QA tasks.
Why it matters
The paper addresses a critical gap in LLM development: the lack of datasets for fine-tuning and evaluating their short- and long-term memory. By providing an automated generation framework and a new dataset, it enables more effective training and assessment of LLMs' ability to maintain context over extended interactions. This advances conversational AI by improving LLMs' coherence and recall.
Original Abstract
Recent advancements in Large Language Models (LLMs) have improved their ability to process extended conversational contexts, yet fine-tuning and evaluating short- and long-term memories remain difficult due to the absence of datasets that encode both short- and long-term conversational history. Existing conversational datasets lack memory grounding, overlook topic continuity, or rely on costly human annotation. To address these gaps, we introduce AgenticAI-DialogGen, a modular agent-based framework that generates persona-grounded and topic-guided conversations without human supervision. The framework uses LLM agents to extract knowledge graphs, identify topics, build speaker personas, and simulate topic-guided conversations from unstructured conversations. A QA module generates memory-grounded Question Answer (QA) pairs drawn from short- and long-term conversational histories. We also generated a new dataset entitled, TopicGuidedChat (TGC), where long-term memory is encoded as speaker-specific knowledge graphs and short-term memory as newly generated topic-guided conversations. Evaluations depict that AgenticAI-DialogGen yields higher conversational quality and LLMs fine-tuned on TGC dataset achieve improved performance on memory-grounded QA tasks.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.