Skip to main content
SpringerLink
Log in

Evaluating algorithmic homeless service allocation

  • Research Article
  • Published:
Journal of Computational Social Science Aims and scope Submit manuscript

Abstract

Improving the homelessness system and evaluating the effectiveness of delivered services are critical to achieve optimal usage of limited social resources as well as to improve the outcomes of the homelessness system. In this context, an increasing number of data science and machine learning methods have been recently applied to the domain of homeless service provision. Given the societal impact of this domain, it is critical to understand the limitations of such methods. However, the performance of algorithmic intervention methods is typically evaluated using abstract metrics that have little meaning for the homeless service allocation domain. We show that domain-agnostic measures are insufficient, and propose a set of new, domain-specific evaluation metrics based on hypothetical, yet realistic “what–if” scenarios. Our empirical analysis demonstrates the value of the proposed measures in understanding the outputs of predictive models and the effect of algorithmic interventions for homeless service provision.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The dataset that support the findings of this study is not publicly available, as it has been provided to the authors under a data sharing agreement with CARES of NY, Inc. Information on how to obtain the dataset and reproduce the analysis is available from the corresponding author on request.

Notes

  1. The abbreviations used in this article are summarized in Table 1.

  2. NST is a set of multiple-choice and frequency-type questions, which are designed to measure the vulnerability of youth based on their previous history (e.g., socialization, daily function, homelessness experience).

  3. Collecting necessary data for training an algorithmic model requires human effort.

  4. Even though methods to address the problem of training algorithmic decision-making systems in the presence of untrustworthy training data has recently been explored (e.g., [30]), it remains a challenging open problem.

References

  1. Busch-Geertsema, V., Edgar, W., O’Sullivan, E., & Pleace, N. (2010). Homelessness and homeless policies in Europe: Lessons from research. In Conference on homelessness (vol. 9, p. 10).

  2. NU CEPAL. (2007). Report on the 2010 world programme on population and housing censuses. ECLAC

  3. Henry, M., Watt, R., Mahathey, A., Ouellette, J., & Sitler, A. (2020). The 2020 annual homelessness assessment report (AHAR) to congress: Part 1: Point-in-time estimates of homelessness. The Annals of the American Academy of Political and Social Science.

  4. Paragraphs Page. (2015). Report of the special rapporteur on adequate housing as a component of the right to an adequate standard of living, and on the right to non-discrimination in this context on her mission to Indonesia.

  5. Why are people homeless? National Coalition. (2007). http://nationalhomeless.org/publications/facts/Why.pdf

  6. Azizi, M. J., Vayanos, P., Wilder, B., Rice, E., & Tambe, M. (2018). Designing fair, efficient, and interpretable policies for prioritizing homeless youth for housing resources. In International conference on the integration of constraint programming, artificial intelligence, and operations research (pp. 35–51). Springer.

  7. Kube, A., Das, S., & Fowler, P. J. (2019). Allocating interventions based on predicted outcomes: A case study on homelessness services. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, pp. 622–629).

  8. Chan, H., Rice, E., Vayanos, P., Tambe, M., & Morton, M. (2017). Evidence from the past: Ai decision aids to improve housing systems for homeless youth. In AAAI fall symposia (pp. 149–157).

  9. Gao, Y., Das, S., & Fowler, P. (2017). Homelessness service provision: A data science perspective. In Workshops at the thirty-first AAAI conference on artificial intelligence, 2017.

  10. Chelmis, C., Qi, W., Lee, W., & Duncan, S. (2021). Smart homelessness service provision with machine learning. Procedia Computer Science, 185, 9–18.

    Article  Google Scholar 

  11. Nagendra, C., Satterfield, J., Gillis, L., & Judice, V. (2010). Resource allocation and monitoring strategies. https://files.hudexchange.info/resources/documents/ResourceAllocationandMonitoringStrategies_Presentation.pdf

  12. Dostaler, T., & Nelson, G. (2009). A process and outcome evaluation of a shelter for homeless young women. Canadian Journal of Community Mental Health, 22(1), 99–112.

    Article  Google Scholar 

  13. Wong, Y.-L.I., Culhane, D. P., & Kuhn, R. (1997). Predictors of exit and reentry among family shelter users in New York city. Social Service Review, 71(3), 441–462.

    Article  Google Scholar 

  14. Brown, M., Vaclavik, D., Watson, D. P., & Wilka, E. (2017). Predictors of homeless services re-entry within a sample of adults receiving homelessness prevention and rapid re-housing program (HPRP) assistance. Psychological Services, 14(2), 129.

    Article  Google Scholar 

  15. Asplund, J., Eslami, M., Sundaram, H., Sandvig, C., & Karahalios, K. (2020). Auditing race and gender discrimination in online housing markets. In Proceedings of the international AAAI conference on web and social media (Vol. 14, pp. 24–35).

  16. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6), 1–35.

    Article  Google Scholar 

  17. Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2020). There’s software used across the country to predict future criminals and it’s biased against blacks. 2016.

  18. Pessach, D., & Shmueli, E. (2022). A review on fairness in machine learning. ACM Computing Surveys (CSUR), 55(3), 1–44.

    Article  Google Scholar 

  19. Von Wachter, T., Bertrand, M., Pollack, H., Rountree, J., & Blackwell, B. (2019). Predicting and preventing homelessness in Los Angeles. California Policy Lab and University of Chicago Poverty Lab.

  20. United States Department of Housing and Urban Development. (2019). HMIS guides and tools. Retrieved December 14, 2020, from https://www.hudexchange.info/programs/hmis/hmis-guides/

  21. VanBerlo, B., Ross, M. A. S., Rivard, J., & Booker, R. (2021). Interpretable machine learning approaches to prediction of chronic homelessness. Engineering Applications of Artificial Intelligence, 102, 104243.

    Article  Google Scholar 

  22. Mothilal, R. K., Sharma, A., & Tan, C. (2020). Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 conference on fairness, accountability, and transparency (pp. 607–617).

  23. Ustun, B., Spangher, A., & Liu, Y. (2019). Actionable recourse in linear classification. In Proceedings of the conference on fairness, accountability, and transparency (pp. 10–19).

  24. Dhurandhar, A., Chen, P.-Y., Luss, R., Tu, C.-C., Ting, P., Shanmugam, K., & Das, P. (2018). Explanations based on the missing: Towards contrastive explanations with pertinent negatives. Advances in Neural Information Processing Systems, 31.

  25. Chipman, H. A., George, E. I., & McCulloch, R. E. (2010). Bart: Bayesian additive regression trees. The Annals of Applied Statistics, 4(1), 266–298.

    Article  Google Scholar 

  26. Lahoti, P., Gummadi, K. P., & Weikum, G. (2019). iFair: Learning individually fair data representations for algorithmic decision making. In 2019 IEEE 35th international conference on data engineering (ICDE) (pp. 1334–1345). IEEE.

  27. Hardt, M., Price, E., & Srebro, N. (2016). Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems, 29.

  28. Fry, A., Littlejohns, T. J., Sudlow, C., Doherty, N., Adamska, L., Sprosen, T., Collins, R., & Allen, N. E. (2017). Comparison of sociodemographic and health-related characteristics of UK Biobank participants with those of the general population. American Journal of Epidemiology, 186(9), 1026–1034.

    Article  Google Scholar 

  29. Verma, S., & Rubin, J. (2018). Fairness definitions explained. In 2018 IEEE/ACM international workshop on software fairness (FairWare) (pp. 1–7). IEEE.

  30. Qi, W., & Chelmis, C. (2021). Improving algorithmic decision-making in the presence of untrustworthy training data. In 2021 IEEE international conference on big data (big data) (pp. 1102–1108). IEEE.

  31. Bozdağ, A. (2021). Local-based mapping of carbon footprint variation in turkey using artificial neural networks. Arabian Journal of Geosciences, 14(6), 1–15.

    Article  Google Scholar 

  32. Parcollet, T., & Ravanelli, M. (2021). The energy and carbon footprint of training end-to-end speech recognizers.

  33. Williams, N., Zander, S., & Armitage, G. (2006). A preliminary performance comparison of five machine learning algorithms for practical IP traffic flow classification. ACM SIGCOMM Computer Communication Review, 36(5), 5–16.

    Article  Google Scholar 

  34. Mitchell, S., Potash, E., Barocas, S., D’Amour, A., & Lum, K. (2018). Prediction-based decisions and fairness: A catalogue of choices, assumptions, and definitions. arXiv preprint arXiv:1811.07867.

  35. Vasudevan, S., & Kenthapadi, K. (2020). Lift: A scalable framework for measuring fairness in ml applications. In Proceedings of the 29th ACM international conference on information & knowledge management (pp. 2773–2780).

  36. Kehl, D. L., & Kessler, S. A. (2017). Algorithms in the criminal justice system: Assessing the use of risk assessments in sentencing.

  37. Dressel, J., & Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(1), eaao5580.

    Article  Google Scholar 

  38. Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G., & Chin, M. H. (2018). Ensuring fairness in machine learning to advance health equity. Annals of Internal Medicine, 169(12), 866–872.

    Article  Google Scholar 

  39. Hong, B., Malik, A., Lundquist, J., Bellach, I., & Kontokosta, C. E. (2018). Applications of machine learning methods to predict readmission and length-of-stay for homeless families: The case of win shelters in New York city. Journal of Technology in Human Services, 36(1), 89–104.

    Article  Google Scholar 

  40. Geyer, C. J. (1992). Practical Markov chain Monte Carlo. Statistical Science, 473–483.

  41. Deng, Z., Zhu, X., Cheng, D., Zong, M., & Zhang, S. (2016). Efficient kNN classification algorithm for big data. Neurocomputing, 195, 143–148.

    Article  Google Scholar 

  42. Svetnik, V., Liaw, A., Tong, C., Culberson, J. C., Sheridan, R. P., & Feuston, B. P. (2003). Random forest: A classification and regression tool for compound classification and QSAR modeling. Journal of Chemical Information and Computer Sciences, 43(6), 1947–1958.

    Article  Google Scholar 

  43. Freund, Y., Schapire, R., & Abe, N. (1999). A short introduction to boosting. Journal-Japanese Society for Artificial Intelligence, 14(771–780), 1612.

    Google Scholar 

  44. Hastie, T., Rosset, S., Zhu, J., & Zou, H. (2009). Multi-class adaboost. Statistics and Its Interface, 2(3), 349–360.

    Article  Google Scholar 

  45. United States Department of Housing and Urban Development. (2020). HMIS data standards manual. Retrieved May 26, 2021, from https://www.hudexchange.info/resource/3824/hmis-data-dictionary/

  46. Bi, Y., Xiang, D., Ge, Z., Li, F., Jia, C., & Song, J. (2020). An interpretable prediction model for identifying N7-methylguanosine sites based on XGBoost and SHAP. Molecular Therapy-Nucleic Acids, 22, 362–372.

    Article  Google Scholar 

  47. Parsa, A. B., Movahedi, A., Taghipour, H., Derrible, S., & Mohammadian, A. K. (2020). Toward safer highways, application of XGBoost and SHAP for real-time accident detection and feature analysis. Accident Analysis & Prevention, 136, 105405.

    Article  Google Scholar 

  48. Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 785–794).

  49. Narwane, S. V., & Sawarkar, S. D. (2019). Machine learning and class imbalance: A literature survey. Industrial Engineering Journal, 12(10).

  50. Kapelner, A., & Bleich, J. (2013). bartMachine: Machine learning with Bayesian additive regression trees. arXiv preprint arXiv:1312.2171

Download references

Funding

This material is based upon work supported by the National Science Foundation under Grant no. ECCS-1737443.

Author information

Author notes

  1. Wenting Qi and Charalampos Chelmis contributed equally to this work.

Authors and Affiliations

  1. Department of Computer Science, University at Albany, SUNY, Albany, NY, 12222, USA

    Wenting Qi & Charalampos Chelmis

Authors
  1. Wenting Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charalampos Chelmis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Charalampos Chelmis.

Ethics declarations

Conflict of interest

The authors declare no competing interests regarding the publication of this article.

Ethics approval

This study was conducted in accordance to all relevant policies and procedures set forth by the University at Albany Institutional Review Board for the protection of human subjects.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

See Tables 11 and 12.

Table 11 Tables for experiments statistical results for algorithmic models and Gurobi methods of three domain-specific evaluation metrics 1
Full size table
Table 12 Tables for experiments statistical results for algorithmic models and Gurobi methods of three domain-specific evaluation metrics 2
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qi, W., Chelmis, C. Evaluating algorithmic homeless service allocation. J Comput Soc Sc 6, 59–89 (2023). https://doi.org/10.1007/s42001-022-00190-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42001-022-00190-8

Keywords

Search

Navigation