Evaluating algorithmic homeless service allocation

Improving the homelessness system and evaluating the effectiveness of delivered services are critical to achieve optimal usage of limited social resources as well as to improve the outcomes of the homelessness system. In this context, an increasing number of data science and machine learning methods have been recently applied to the domain of homeless service provision. Given the societal impact of this domain, it is critical to understand the limitations of such methods. However, the performance of algorithmic intervention methods is typically evaluated using abstract metrics that have little meaning for the homeless service allocation domain. We show that domain-agnostic measures are insufficient, and propose a set of new, domain-specific evaluation metrics based on hypothetical, yet realistic “what–if” scenarios. Our empirical analysis demonstrates the value of the proposed measures in understanding the outputs of predictive models and the effect of algorithmic interventions for homeless service provision.