Unlocking Crowdsourcing for Ontology Matching Validation
TLDR
This paper introduces a novel crowdsourcing system with domain-specific mechanisms to validate ontology matching, addressing challenges from LLMs.
Key contributions
- Novel crowdsourcing system for validating LLM-driven ontology matching.
- Uses differential trustworthiness, coherence pre-filling, and time-dependent beliefs for quality.
- Enables human-in-the-loop validation by integrating with state-of-the-art OM systems.
- Validated effectiveness through two real-world ontology matching use cases.
Why it matters
As LLMs advance ontology matching, validating their outputs becomes a bottleneck for domain experts. This paper offers a scalable, quality-controlled crowdsourcing solution, making human-in-the-loop validation feasible for complex OM tasks. It's crucial for leveraging LLM power in knowledge graph construction.
Original Abstract
Recent advances in large language models (LLMs) pose new challenges for ontology matching (OM). While OM systems built on LLMs have shown remarkable capabilities in discovering more mappings, traditional OM validation that relies on domain experts has become overwhelming. In this study, we explore the use of crowdsourcing for OM validation and introduce a novel crowdsourcing system. We propose three domain-specific mechanisms, namely differential trustworthiness, coherence pre-filling, and time-dependent beliefs, to ensure the quality of crowdsourcing for OM validation. We demonstrate that our crowdsourcing system can be integrated with state-of-the-art OM systems to enable human-in-the-loop validation. Two real-world use cases illustrate the effectiveness of our crowdsourcing system.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.