PhageBench: Can LLMs Understand Raw Bacteriophage Genomes?
Yusen Hou, Weicai Long, Haitao Hu, Houcheng Su, Junning Feng + 1 more
TLDR
PhageBench evaluates LLMs' ability to understand raw bacteriophage genomes, showing promise but also limitations in complex tasks.
Key contributions
- Introduces PhageBench, the first benchmark for evaluating LLMs' understanding of raw bacteriophage genomes.
- PhageBench comprises 5,600 samples across 5 tasks in Screening, QC, and Phenotype Annotation stages.
- General LLMs show promise in phage contig identification and host prediction, outperforming baselines.
- Reveals LLMs' significant limitations in complex genomic reasoning and long-range dependency tasks.
Why it matters
This paper highlights the potential of LLMs for interpreting complex bacteriophage genomes, crucial for microbial ecosystem regulation and antibiotic alternatives. It also identifies key areas where current models fall short, guiding future research in genomic AI.
Original Abstract
Bacteriophages, often referred to as the dark matter of the biosphere, play a critical role in regulating microbial ecosystems and in antibiotic alternatives. Thus, accurate interpretation of their genomes holds significant scientific and practical value. While general-purpose Large Language Models (LLMs) excel at understanding biological texts, their ability to directly interpret raw nucleotide sequences and perform biological reasoning remains underexplored. To address this, we introduce PhageBench, the first benchmark designed to evaluate phage genome understanding by mirroring the workflow of bioinformatics experts. The dataset contains 5,600 high-quality samples covering five core tasks across three stages: Screening, Quality Control, and Phenotype Annotation. Our evaluation of eight LLMs reveals that general-purpose reasoning models significantly outperform random baselines in phage contig identification and host prediction, demonstrating promising potential for genomic understanding. However, they exhibit significant limitations in complex reasoning tasks involving long-range dependencies and fine-grained functional localization. These findings highlight the necessity of developing next-generation models with enhanced reasoning capabilities for biological sequences.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.