Pay-Per-Crawl Pricing for AI: The LM-Tree Agent
Richard Archer, Soheil Ghili, Nima Haghpanah
TLDR
The LM-Tree agent proposes a pay-per-crawl model for AI content consumption, adaptively pricing content using LLMs to maximize publisher revenue.
Key contributions
- Proposes "pay-per-crawl" as a new revenue model for publishers facing AI content consumption.
- Introduces the LM-Tree, an adaptive pricing agent using LLMs to segment content and discover value attributes.
- Achieves significant revenue gains (65% over static, 47% over 2-category) on real-world publisher data.
- Outperforms expert-designed taxonomies by 40%, identifying nuanced content value distinctions.
Why it matters
This paper addresses the critical challenge of monetizing content consumed directly by AI. The LM-Tree offers a novel, data-driven solution, significantly boosting publisher revenue via adaptive pricing. Its LLM-powered discovery of hidden content value has major implications for AI-driven content economies.
Original Abstract
As AI systems shift from directing users to content toward consuming it directly, publishers need a new revenue model: charging AI crawlers for content access. This model, called pay-per-crawl, must solve a problem of mechanism selection at scale: content is too heterogeneous for a fixed pricing framework. Different sub-types warrant not only different price levels but different pricing rules based on different unstructured features, and there are too many to enumerate or design by hand. We propose the LM Tree, an adaptive pricing agent that grows a segmentation tree over the content library, using LLMs to discover what distinguishes high-value from low-value items and apply those attributes at scale, from binary purchase feedback alone. We evaluate the LM Tree on real content from a major German technology publisher, using 8,939 articles and 80,451 buyer queries with willingness-to-pay calibrated from actual AI crawler traffic. The LM Tree achieves a 65% revenue gain over a single static price and a 47% gain over two-category pricing, outperforming even the publisher's own 8-segment editorial taxonomy by 40% -- recovering content distinctions the publisher's own categories miss.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.