Evaluating probability forecasts for GDP declines using alternative methodologies

Evaluation methodologies for rare events from meteorology, psychology and medical diagnosis are used to examine the value of probabilistic forecasts of real GDP declines during the current quarter (Q0) and each of the next four quarters (Q1-Q4) using data from the Survey of Professional Forecasters. We study the quality of these probability forecasts in terms of their calibration, resolution and odds ratio, as well as the relative operating characteristic (ROC) and alternative variance decompositions. Only the shorter-term forecasts (Q0-Q2) are found to possess significant skill in terms of all measures considered, even though they are characterized by an excess of variability and a lack of calibration.The battery of diagnostic statistics cataloged in this paper should be useful for evaluating regression models with binary dependent variables, particularly when the event of interest is relatively uncommon.