CrossCommitVuln-Bench: A Dataset of Multi-Commit Python Vulnerabilities Invisible to Per-Commit Static Analysis
TLDR
CrossCommitVuln-Bench introduces a dataset of multi-commit Python vulnerabilities that evade per-commit static analysis, highlighting major SAST limitations.
Key contributions
- Introduces CrossCommitVuln-Bench, a dataset of 15 real-world Python CVEs spanning multiple commits.
- Reveals 87% of these multi-commit vulnerabilities are invisible to per-commit static analysis tools.
- Demonstrates even cumulative SAST only detects 27% of cross-commit vulnerabilities.
- Provides detailed annotations and open-source baselines to foster future research.
Why it matters
Current static analysis tools largely miss vulnerabilities introduced across multiple commits. This paper highlights a critical blind spot, showing 87% of such real-world CVEs are invisible to per-commit SAST. The new dataset and baselines are crucial for developing more effective cross-commit vulnerability detection methods.
Original Abstract
We present CrossCommitVuln-Bench, a curated benchmark of 15 real-world Python vulnerabilities (CVEs) in which the exploitable condition was introduced across multiple commits - each individually benign to per-commit static analysis - but collectively critical. We manually annotate each CVE with its contributing commit chain, a structured rationale for why each commit evades per-commit analysis, and baseline evaluations using Semgrep and Bandit in both per-commit and cumulative scanning modes. Our central finding: the per-commit detection rate (CCDR) is 13% across all 15 vulnerabilities - 87% of chains are invisible to per-commit SAST. Critically, both per-commit detections are qualitatively poor: one occurs on commits framed as security fixes (where developers suppress the alert), and the other detects only the minor hardcoded-key component while completely missing the primary vulnerability (200+ unprotected API endpoints). Even in cumulative mode (full codebase present), the detection rate is only 27%, confirming that snapshot-based SAST tools often miss vulnerabilities whose introduction spans multiple commits. The dataset, annotation schema, evaluation scripts, and reproducible baselines are released under open-source licenses to support research on cross-commit vulnerability detection.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.