CIDR: A Large-Scale Industrial Source Code Dataset for Software Engineering Research
TLDR
CIDR is a new large-scale dataset of 2,440 proprietary industrial software repositories from 12 partners, designed for diverse software engineering research.
Key contributions
- Introduces CIDR, a dataset of 2,440 proprietary industrial repositories from 12 partners.
- Contains 373 million lines of code across 138 languages with structured metadata.
- Features exclusively proprietary production code, distinct from public open-source corpora.
- Processed via a multi-stage pipeline including quality selection and deterministic anonymization.
Why it matters
This dataset addresses the critical gap in industrial source code for research. It enables new studies in code intelligence, software quality, and language model development, offering real-world insights into proprietary systems. This is crucial for advancing practical software engineering.
Original Abstract
We present Curated Industrial Developer Repository (CIDR), a large-scale dataset of real-world software repositories collected through direct collaboration with 12 industrial partner organizations. The dataset comprises 2,440 repositories spanning 138 programming languages and totalling 373 million lines of code, accompanied by structured per-repository metadata. Unlike existing code corpora derived from public open-source platforms, CIDR consists exclusively of proprietary production codebases contributed under formal data sharing agreements, covering application domains including enterprise web and mobile development, fintech, and custom software consultancy. All repositories were processed through a multi-stage pipeline encompassing structured partner onboarding, two-stage quality selection combining automated metadata filtering with manual code review, and a deterministic anonymization pipeline covering the full version control history. The dataset is intended to support research in code intelligence, software quality analysis, pre-training and fine-tuning of code language models, developer behaviour studies, and construction of agent evaluation benchmarks. Access is provided under a restricted commercial license; details are available at https://fermatix.ai/#Contact.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.