A Divergence-Based Method for Weighting and Averaging Model Predictions
TLDR
This paper introduces a divergence-based method for weighting and averaging model predictions, outperforming standard methods, especially with small sample sizes.
Key contributions
- Introduces a novel minimum divergence framework for calculating model weights to average probabilistic predictions.
- The method is general, applicable to frequentist, Bayesian, and other statistical/ML model fitting approaches.
- Empirically shown to outperform or match standard model averaging methods, particularly with small datasets.
- Provides theoretical analysis explaining its superior performance in small-sample scenarios.
Why it matters
This paper offers a significant advancement in model averaging, providing a robust method that improves prediction accuracy. Its particular strength in small-sample settings makes it highly valuable for domains with limited data, enhancing reliability where traditional methods struggle.
Original Abstract
This paper uses a minimum divergence framework to introduce a new way of calculating model weights that can be used to average probabilistic predictions from statistical and machine learning models. The method is general and can be applied regardless of whether the models under consideration are fit to data using frequentist, Bayesian, or some other fitting method. The proposed method is motivated in two different ways and is shown empirically to perform better than or on a par with standard model averaging methods, including model stacking and model averaging that relies on Akaike-style negative exponentiated model weighting, especially when the sample size is small. Our theoretical analysis explains why the method has a small-sample advantage.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.