When Agents Shop for You: Role Coherence in AI-Mediated Markets
TLDR
AI agents' natural language descriptions inadvertently reveal consumer willingness to pay to sellers, a 'preference leakage' not solvable by prompts.
Key contributions
- AI agents' natural language profiles create 'role coherence,' allowing sellers to infer willingness to pay.
- Experiment shows sellers can infer willingness to pay almost perfectly from agent dialogue alone.
- This 'preference leakage' is distinct from instruction-following failures and arises from delegation itself.
- Prompt-level interventions are ineffective; architectural changes are needed to balance privacy and personalization.
Why it matters
This paper highlights a critical privacy risk in AI-mediated markets: consumer preference leakage through agent dialogue. It reveals that current AI delegation methods inherently expose sensitive information, even without explicit disclosure. The findings underscore the urgent need for new architectural designs that prioritize privacy in agent-based commerce.
Original Abstract
Consumers are increasingly delegating purchase decisions to AI agents, providing natural-language descriptions of their preferences and identity. We argue that these representations constitute an information channel, role coherence, through which sellers can infer willingness to pay without explicit disclosure by the buyer agent, leading to preference leakage. In an experiment where a language-model buyer agent shops on behalf of a verbal consumer profile, we show that seller-side inference from dialogue alone recovers willingness to pay nearly one-for-one. Comparing this setting to a numeric-budget condition with confidentiality instructions cleanly isolates role coherence as distinct from instruction-following failure. Because this leakage arises from delegation itself, it cannot be mitigated at the prompt level. Instead, we propose architectural interventions that trade off personalization against preference privacy.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.