Credential Leakage in LLM Agent Skills: A Large-Scale Empirical Study
Zhihao Chen, Ying Zhang, Yi Liu, Gelei Deng, Yuekang Li + 5 more
TLDR
Study reveals widespread credential leakage in LLM agent skills, identifying 520 vulnerable skills and 10 leakage patterns, primarily via debug logs.
Key contributions
- First large-scale empirical study on credential leakage in 17,022 LLM agent skills.
- Identified 520 vulnerable skills with 1,708 issues and a taxonomy of 10 leakage patterns.
- Found debug logging (print/console.log) causes 73.5% of leaks, often cross-modal.
- Leaked credentials are highly exploitable (89.6%) and persistent across skill forks.
Why it matters
This paper uncovers a significant security flaw in LLM agent skills, demonstrating how sensitive credentials are leaked and exploited. It provides a crucial taxonomy and detection pipeline, paving the way for more secure LLM agent development and deployment.
Original Abstract
Third-party skills extend LLM agents with powerful capabilities but often handle sensitive credentials in privileged environments, making leakage risks poorly understood. We present the first large-scale empirical study of this problem, analyzing 17,022 skills (sampled from 170,226 on SkillsMP) using static analysis, sandbox testing, and manual inspection. We identify 520 vulnerable skills with 1,708 issues and derive a taxonomy of 10 leakage patterns (4 accidental and 6 adversarial). We find that (1) leakage is fundamentally cross-modal: 76.3% require joint analysis of code and natural language, while 3.1% arise purely from prompt injection; (2) debug logging is the primary vector, with print and console.log causing 73.5% of leaks due to stdout exposure to LLMs; and (3) leaked credentials are both exploitable (89.6% without privileges) and persistent, as forks retain secrets even after upstream fixes. After disclosure, all malicious skills were removed and 91.6% of hardcoded credentials were fixed. We release our dataset, taxonomy, and detection pipeline to support future research.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.