Learning from Change: Predictive Models for Incident Prevention in a Regulated IT Environment
Eileen Kapel, Jan Lennartz, Luis Cruz, Diomidis Spinellis, Arie van Deursen
TLDR
This paper introduces an interpretable machine learning model to predict IT incident risk from changes, outperforming rule-based systems in a regulated environment.
Key contributions
- Developed a predictive incident risk scoring approach for IT changes in a large regulated bank.
- Compared HGBC, LightGBM, and XGBoost models, with LightGBM achieving superior performance.
- Incorporated SHAP values for model explainability and auditability, meeting regulatory needs.
- Demonstrated data-driven, interpretable models outperform rule-based systems for IT incident prevention.
Why it matters
This research is crucial for regulated industries, showing how interpretable ML can proactively prevent IT incidents. It offers a practical framework for improving operational reliability and meeting strict compliance requirements. By moving beyond rule-based systems, it enables more efficient and transparent risk mitigation.
Original Abstract
Effective IT change management is important for businesses that depend on software and services, particularly in highly regulated sectors such as finance, where operational reliability, auditability, and explainability are essential. A significant portion of IT incidents are caused by changes, making it important to identify high-risk changes before deployment. This study presents a predictive incident risk scoring approach at a large international bank. The approach supports engineers during the assessment and planning phases of change deployments by predicting the potential of inducing incidents. To satisfy regulatory constraints, we built the model with auditability and explainability in mind, applying SHAP values to provide feature-level insights and ensure decisions are traceable and transparent. Using a one-year real-world dataset, we compare the existing rule-based process with three machine learning models: HGBC, LightGBM, and XGBoost. LightGBM achieved the best performance, particularly when enriched with aggregated team metrics that capture organisational context. Our results show that data-driven, interpretable models can outperform rule-based approaches while meeting compliance needs, enabling proactive risk mitigation and more reliable IT operations.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.