Training Time Prediction for Mixed Precision-based Distributed Training
Minchul Kang, Changyong Shin, Jinwoo Jeong, Hyunho Lee, Younghun Go + 3 more
TLDR
This paper introduces a precision-aware predictor for distributed deep learning training time, significantly improving accuracy over existing methods.
Key contributions
- Identifies floating-point precision as a critical, overlooked factor causing up to 2.4x training time variation.
- Demonstrates that existing training time predictors, ignoring precision, yield high errors (up to 147.85% MAPE).
- Proposes a novel precision-aware predictor for distributed deep learning training time.
- Achieves robust prediction accuracy with only 9.8% MAPE across diverse precision settings, including mixed precision.
Why it matters
Accurate training time prediction is vital for efficient resource management in distributed deep learning. This paper addresses a critical gap by accounting for precision variations, leading to significantly more reliable predictions. This advancement will optimize resource allocation and reduce costs.
Original Abstract
Accurate prediction of training time in distributed deep learning is crucial for resource allocation, cost estimation, and job scheduling. We observe that the floating-point precision setting is a key determinant of training time, leading to training time variations of ~2.4x over its minimum. However, existing studies on distributed training time prediction rely on static model computation graphs that do not capture precision variations, including mixed precision. According to our experiments, training time prediction without considering precision results in significant prediction errors - reaching up to 147.85% in mean absolute percentage error (MAPE). To address this issue, we propose a precision-aware distributed training time predictor that achieves robust accuracy across diverse precision settings, including mixed precision, with 9.8% MAPE.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.