TruncProof: A Guardrail for LLM-based JSON Generation under Token-Length Constraints
TLDR
TruncProof enables LLMs to generate grammatically valid JSON outputs while strictly adhering to predefined token length constraints.
Key contributions
- Addresses LLM JSON generation issues like infinite output or truncation under strict token limits.
- Introduces TruncProof, a grammar-constrained method using LL(1) parsers for valid JSON generation.
- Efficiently approximates minimum tokens for valid completion, ensuring syntactically correct outputs.
- Combines with advanced decoding strategies for both grammatical validity and semantic accuracy.
Why it matters
LLMs often fail to produce valid JSONs under token limits, causing system malfunctions. TruncProof offers a robust solution by guaranteeing grammatically correct outputs within constraints. This is crucial for reliable integration of LLMs with external systems, enhancing their practical applicability.
Original Abstract
The LLM-based generation of machine-readable outputs such as JSON has attracted significant attention for integration with external systems. However, existing approaches cannot strictly enforce the maximum number of tokens to be generated, leading to infinite generation or truncated outputs that cause a system malfunction. To address this limitation, we propose TruncProof, a novel grammar-constrained generation method that enables LLMs to produce grammatically valid JSONs while adhering to a predefined token limit. By leveraging the properties of LL(1) parsers, TruncProof efficiently approximates the minimum number of tokens required to complete a grammatically valid output at each decoding step. Experiments on the Text-to-JSON instruction tasks demonstrate that TruncProof successfully generates syntactically correct outputs even under strict token constraints. Furthermore, we show that TruncProof can be effectively combined with advanced decoding strategies, resulting in outputs that are not only grammatically valid but also semantically accurate.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.