ReproBreak: A Dataset of Reproducible Web Locator Breaks
Thiago Santos de Moura, Leon Adamietz, Samra Mehboob, Yannic Noller
TLDR
ReproBreak is a new dataset of 449 reproducible web locator breaks in Cypress and Playwright tests, addressing the lack of data for evaluating locator fragility.
Key contributions
- Presents ReproBreak, the first dataset of reproducible web locator breaks for Cypress and Playwright.
- Analyzed 359 open-source repositories to identify commits containing locator changes.
- Reproduced and documented 449 locator breaks in top projects, providing automated reproduction scripts.
- Addresses the lack of data for evaluating locator fragility and test robustness at scale.
Why it matters
Locator breaks are a significant challenge in GUI test maintenance, causing tests to fail even when application functionality is stable. ReproBreak provides a crucial resource for researchers to develop and evaluate techniques for more robust web locators and automated test repair, ultimately reducing maintenance overhead.
Original Abstract
Automated GUI testing frameworks such as Cypress and Playwright rely on locators to find and interact with web elements. A locator break occurs when a structural change in the application under test causes a locator to no longer find its target element, resulting in test breakages even when the underlying functionality remains unchanged. Despite its impact on test maintenance, no dataset exists to evaluate locator fragility in Cypress and Playwright at scale. In this paper, we present ReproBreak, a dataset of reproducible locator breaks in web application GUI tests. We analyzed 359 open-source repositories to identify commits that contain locator changes. To confirm whether these changes are indeed locator breaks, we reproduced them in the top 4 projects with the largest number of locator changes and found 449 locator breaks, which are provided in the dataset along with scripts for automated reproduction. We believe ReproBreak serves as a valuable artifact to support research on locator fragility, repair techniques, and test robustness. The video is available at: https://youtu.be/mZByS_TnCvE. The dataset is at https://github.com/rub-sq/ReproBreak.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.