Analysis of the Impact of Sample Size, Attribute Variance and Within-Sample Choice Distribution on the Estimation Accuracy of Multinomial Logit Models Using Simulated Data

  • Minhui Zeng
  • Ming Zhong
  • John Douglas Hunt


Literature review indicates that sample size, attribute variance and within-sample choice distribution of alternatives are important considerations in the estimation of multinomial logit (MNL) models, but their impacts on the estimation accuracy have not been systematically studied. Therefore, the objective of this paper is to provide an empirical examination to the above issues through a set of simulated discrete choice preference and rank ordered preference datasets. In this paper, the utility coefficients, alternative specific constants (ASCs), and the mean and standard deviation of the four attributes for a set of seven hypothetical alternatives are specified as a priori. Then, synthetic datasets, with varying sample size, attribute variance and within-sample choice distribution are simulated. Based on these datasets, the utility coefficients and ASCs of the specified MNLs are re-estimated and compared with the original values specified as the priori. It is found that (1) the estimation accuracy of utility parameters increases as the sample size increases; (2) the utility coefficients can be re-estimated with reasonable accuracy, but the estimates of the ASCs are confronted with much larger errors; (3) as the variances of the alternative attributes increase, the estimation accuracy improves significantly; and (4) as the distribution of chosen choices becomes more balanced across alternatives within sample datasets, the hit-ratio decreases. The results indicate that (a) under a similar setting presented in this paper, a large sample consisting of a few thousand observations (3000–4000) may be needed in order to provide reasonable estimates for utility coefficients, particularly for ASCs; (b) a larger, but realistic attribute space is preferred in the stated preference survey design; and (c) choice datasets with unbalanced “chosen” choice frequency distribution is preferred, in order to better capture the elasticity between the “perceived utility” associated with alternative’s attributes.


Sample size attribute variance within-sample choice distribution simulated data 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



The authors appreciate the anonymous referees and the editor for their help to improve the quality of the paper. The funding from Hubei Provincial Natural Science Foundation (2015CFB599) and the funding for Top 1% ESI Academic Program from Wuhan University of Technology supported by “the Fundamental Research Funds for the Central Universities” (WUT:2014-VII-036) is appreciated. This study is also supported by the Natural Science and Engineering Research Council (NSERC), Canada and a start-up grant from Wuhan University of Technology. This paper is also partially supported by a grant from the National Natural Science Foundation of China (NSFC No.51778510).


  1. [1]
    Bhat, C.R., & Guo, J. (2004). A mixed spatially correlated logit model: formulation and application to residential choice modeling. Transportation Research Part B: Methodological, 38(2):147–168.CrossRefGoogle Scholar
  2. [2]
    Bhat, C.R., Castro, M. & Khan, M. (2013). A new estimation approach for the multiple discrete-continuous probit (mdcp) choice model. Transportation Research Part B: Methodological, 55:1–22.CrossRefGoogle Scholar
  3. [3]
    Bierlaire, M.C.J. (2006). A theoretical analysis of the cross-nested logit model. Annals of Operations Research, 144(1):287–300.MathSciNetCrossRefzbMATHGoogle Scholar
  4. [4]
    Bliemer, M.C.J. & Rose, J.M. (2008) Construction of experimental designs for mixed logit models allowing for correlation across choice observations. The 87th Annual TRB Meeting, Washington DC, January 2008, USA.Google Scholar
  5. [5]
    Bliemer, M.C.J., Rose, J.M. & Hensher, D.A. (2009) Efficient stated choice experiments for estimating nested logit models. Transportation Research: Part B, 43:19–35.CrossRefGoogle Scholar
  6. [6]
    Brundell-Freij, K. (1997). How good is an estimated logit model? estimation accuracy analyzed by Monte Carlo simulations. Paper presented at the proceedings of seminar F held at European Transport Forum, Brunel University, England, 1–5 September 1997.Google Scholar
  7. [7]
    Crabbe, M., Akinc, D. & Vandebroek, M. (2014). Fast algorithms to generate individualized designs for the mixed logit choice model. Transportation Research Part B: Methodological, 60: 1–15.CrossRefGoogle Scholar
  8. [8]
    Cramer, J.S. (1999). Predictive performance of the binary logit model in unbalanced samples. Journal of the Royal Statistical Society: Series D (The Statistician), 48 (1): 85–94.MathSciNetGoogle Scholar
  9. [9]
    Greene, W.H. & Hensher, D.A. (2013). Revealing additional dimensions of preference heterogeneity in a latent class mixed multinomial logit model. Applied Economics, 45 (14): 1897–1902.CrossRefGoogle Scholar
  10. [10]
    Guan, H.Z. (2004). Disaggregated Model-Analysis Tools for Traffic Behavior (in Chinese). China Communications Press, Beijing.Google Scholar
  11. [11]
    Hensher, D.A. & Greene, W.H. (2002). Specification and estimation of the nested logit model: alternative normalizations. Transportation Research Part B: Methodological, 36 (1): 1–17.CrossRefGoogle Scholar
  12. [12]
    Hunt, J.D., Zhong, M. & Abraham, J.E. (2007). Examining the accuracy of logit modeling with simulated RP and SP data. Presented at the 2007 World Conference of Transportation Research Conference, Berkeley.Google Scholar
  13. [13]
    Koppelman, F.S. & Chu C. (1983). Effect of sample size on disaggregate choice model estimation and prediction. Transportation Research Record: Journal of the Transportation Research Board, 944: 60–69.Google Scholar
  14. [14]
    Lemp, J.D., Kockelman, K.M. & Damien, P. (2012). A bivariate multinomial probit model for trip scheduling: Bayesian analysis of the work tour. Transportation Science, 46 (3): 405–424.CrossRefGoogle Scholar
  15. [15]
    Liang, Y.J. & Yuan, Z.Z. (2014). A logit model for selection of passenger facilities at integrated transport hubs. Journal of Trans-port information and safety, 32 (4):36–40.Google Scholar
  16. [16]
    McFadden, D. (1978). Modeling the choice of residential location. Transportation Research Record: Journal of the Transportation Research Board, 673: 72–77.Google Scholar
  17. [17]
    McFadden, D. (1984) Econometric analysis of qualitative response models. In: Griliches, Z., Intriligator, M.D. (eds.) Handbook of Econometrics II, pp. 1395–1457. Elseviere Science, Amsterdam.CrossRefGoogle Scholar
  18. [18]
    Munizaga, M.A. and Alvarez-Daziano, R. (2005). Testing mixed logit and probit models by simulation. Transportation Research Record: Journal of the Transportation Research Board, 1921: 52–62.CrossRefGoogle Scholar
  19. [19]
    Nerella, S. & Bhat, C.R. (2004). Numerical analysis of effect of sampling of alternatives in discrete choice models. Transportation Research Record: Journal of the Transportation Research Board, 1894: 11–19.CrossRefGoogle Scholar
  20. [20]
    Ortuzar, J. & Willumsen, L.G. (2011). Modeling Transport, 4th Edition, John Wiley & Sons, Great Britain.CrossRefGoogle Scholar
  21. [21]
    Rose, J.M. & Bliemer, M. C.J. (2013). Sample size requirements for stated choice experiments. Transportation, 40(5):1021–1041.CrossRefGoogle Scholar
  22. [22]
    Wang, Y.Q., Li, L., Wang, L., Moore, A., Staley, S. & Li, Z.Z. (2014). Modeling traveler mode choice behavior of a new high-speed rail corridor in China. Transportation Planning and Technology, 37(5):466–483.CrossRefGoogle Scholar
  23. [23]
    Wen, C.H., Wang, W.C. & Fu, C. (2012). Latent class nested logit model for analyzing high-speed rail access mode choice. Transportation Research, Part E: Logistics and Transportation Review, 48 (2):545–554.CrossRefGoogle Scholar
  24. [24]
    Ye, F. & Lord, D. (2014). Comparing three commonly used crash severity models on sample size requirements: multinomial logit, ordered probit and mixed logit models. Analytic Methods in Accident Research, 1: 72–85.CrossRefGoogle Scholar
  25. [25]
    Zhang, Y.L., Liang, F.M. & Xie, Y.C. (2007). Crash injury severity analysis using a Bayesian ordered probit model. Presented at 86th Annual Meeting of the Transportation Research Board (No. 07-2335), Washington, D.C..Google Scholar
  26. [26]
    Zhong, M., & Hunt, J.D. (2006). Sensitivity analysis of logit formulation and estimation. Presented at the 2006 International Conference on Traffic and Transportation Studies, Xi’an, China.Google Scholar
  27. [27]
    Zhou, X., Liu, M., Zhang, D. & Ran, B. (2014). Transfer mode choice of comprehensive passenger transportation terminal based on mixed logit in china. Presented at 93rd Annual Meeting of the Transportation Research Board (No. 14-3968), Washington, D.C..Google Scholar

Copyright information

© Systems Engineering Society of China and Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Minhui Zeng
    • 1
    • 2
    • 3
  • Ming Zhong
    • 1
    • 3
    • 4
  • John Douglas Hunt
    • 1
    • 3
    • 5
  1. 1.Engineering Research Center for Transportation safety of MOEWuhan University of TechnologyWuhanChina
  2. 2.School of Traffic and Transportation EngineeringChangsha University of Science & TechnologyChangshaChina
  3. 3.National Engineering Research Center for Water Transportation SafetyWuhanChina
  4. 4.Department of Civil and Environmental EngineeringUniversity of WaterlooOntarioCanada
  5. 5.Department of Civil EngineeringUniversity of CalgaryAlbertaCanada

Personalised recommendations