Evaluating Probabilistic Demographic Forecasts

Nico Keilman , University of Oslo

Probabilistic forecasts give results either in terms of predictive probability distributions, or as simulated samples. Sometime after the forecast has been computed, the actual value of the variables in question can be compared with the distributions. With these observations, how can we evaluate the probabilistic forecast? Statisticians have developed so-called scoring functions. A scoring function is a measure for the distance, defined in a specific way, between the distribution and the outcome. However, scoring functions are not widely known among demographers, in spite of the fact that more and more often demographic forecasts are computed as probability forecasts (e.g. the World Population Prospects by the UN Population Division). The aim of the paper is to review scoring functions and their use in demographic applications. It turns out that they have been used in the calibration of some probabilistic demographic forecasts, based on hold-out samples as ex-post data. There are no known applications of scoring functions to probabilistic demographic forecasts computed several years ago. Thus, a second aim is to illustrate scoring functions by evaluating several probabilistic forecasts that became available since the end of the 1990s. For these forecasts, we have two decades of data. We will evaluate probabilistic population forecasts published by Statistics Netherlands in 1998, and the UPE forecasts for selected European countries with jump-off year 2003. We will also evaluate a probabilistic household forecast. To compare scores for forecasts with different jump-off years, we need analyses based on age-period-cohort type of models.

See extended abstract

 Presented in Session 4. Innovations in Demographic Data and Methods