Advertisement

Mathematical Programming

, Volume 174, Issue 1–2, pp 41–76 | Cite as

A distributed one-step estimator

  • Cheng Huang
  • Xiaoming HuoEmail author
Full Length Paper Series B
  • 95 Downloads

Abstract

Distributed statistical inference has recently attracted enormous attention. Many existing work focuses on the averaging estimator, e.g., Zhang and Duchi (J Mach Learn Res 14:3321–3363, 2013) together with many others. We propose a one-step approach to enhance a simple-averaging based distributed estimator by utilizing a single Newton–Raphson updating. We derive the corresponding asymptotic properties of the newly proposed estimator. We find that the proposed one-step estimator enjoys the same asymptotic properties as the idealized centralized estimator. In particular, the asymptotic normality was established for the proposed estimator, while other competitors may not enjoy the same property. The proposed one-step approach merely requires one additional round of communication in relative to the averaging estimator; so the extra communication burden is insignificant. The proposed one-step approach leads to a lower upper bound of the mean squared error than other alternatives. In finite sample cases, numerical examples show that the proposed estimator outperforms the simple averaging estimator with a large margin in terms of the sample mean squared error. A potential application of the one-step approach is that one can use multiple machines to speed up large scale statistical inference with little compromise in the quality of estimators. The proposed method becomes more valuable when data can only be available at distributed machines with limited communication bandwidth.

Keywords

De-centralized data Statistical inference One-step method Oracle asymptotic properties M-estimation 

Mathematics Subject Classification

62F10 62F12 

Notes

References

  1. 1.
    Battey, H., Fan, J., Liu, H., Lu, J., Zhu, Z.: Distributed estimation and inference with statistical guarantees (2015). arXiv preprint arXiv:1509.05457
  2. 2.
    Bickel, P.J.: One-step Huber estimates in the linear model. J. Am. Stat. Assoc. 70(350), 428–434 (1975)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends® Mach Learn. 3(1), 1–122 (2011)zbMATHGoogle Scholar
  4. 4.
    Chen, S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Chen, X., Xie, Mg: A split-and-conquer approach for analysis of extraordinarily large data. Stat. Sin. 24, 1655–1684 (2014)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Corbett, J.C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J., Ghemawat, S., Gubarev, A., Heiser, C., Hochschild, P. et al.: Spanner: Googles globally distributed database. In: Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (2012)Google Scholar
  7. 7.
    Fan, J., Chen, J.: One-step local quasi-likelihood estimation. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 61(4), 927–943 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Jaggi, M., Smith, V., Takác, M., Terhorst, J., Krishnan, S., Hofmann, T., Jordan, M.I.: Communication-efficient distributed dual coordinate ascent. In: Advances in Neural Information Processing Systems, pp. 3068–3076 (2014)Google Scholar
  9. 9.
    Jordan, M.I., Lee, J.D., Yang, Y.: Communication-efficient distributed statistical inference. J. Am. Stat. Assoc. (just-accepted) (2018)Google Scholar
  10. 10.
    Kushner, H., Yin, G.G.: Stochastic Approximation and Recursive Algorithms and Applications, vol. 35. Springer, Berlin (2003)zbMATHGoogle Scholar
  11. 11.
    Lang, S.: Real and Functional Analysis, vol. 142. Springer, Berlin (1993)zbMATHGoogle Scholar
  12. 12.
    Lee, J.D., Sun, Y., Liu, Q., Taylor, J.E.: Communication-efficient sparse regression: a one-shot approach (2015). arXiv preprint arXiv:1503.04337
  13. 13.
    Liu, Q., Ihler, A.T.: Distributed estimation, information loss and exponential families. In: Advances in Neural Information Processing Systems, pp. 1098–1106 (2014)Google Scholar
  14. 14.
    Mitra, S., Agrawal, M., Yadav, A., Carlsson, N., Eager, D., Mahanti, A.: Characterizing web-based video sharing workloads. ACM Trans. Web 5(2), 8 (2011)CrossRefGoogle Scholar
  15. 15.
    Rosenblatt, J., Nadler, B.: On the optimality of averaging in distributed statistical learning (2014). arXiv preprint arXiv:1407.2724
  16. 16.
    Rosenthal, H.P.: On the subspaces of \({L}^p\) (\(p>2\)) spanned by sequences of independent random variables. Isr. J. Math. 8(3), 273–303 (1970)CrossRefzbMATHGoogle Scholar
  17. 17.
    Shamir, O., Srebro, N., Zhang, T.: Communication-efficient distributed optimization using an approximate Newton-type method. In: Proceedings of the 31st International Conference on Machine Learning, pp. 1000–1008 (2014)Google Scholar
  18. 18.
    Shapiro, A., Dentcheva, D., Ruszczyński, A.: Lectures on stochastic programming: modeling and theory. SIAM (2009)Google Scholar
  19. 19.
    Tibshirani, R.: Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B 58(1), 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  20. 20.
    van der Vaart, A.W.: Asymptotic Statistics (Cambridge Series in Statistical and Probabilistic Mathematics). Cambridge University Press, Cambridge (2000)Google Scholar
  21. 21.
    Zhang, Y., Duchi, J.C., Wainwright, M.J.: Communication-efficient algorithms for statistical optimization. J. Mach. Learn. Res. 14, 3321–3363 (2013)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Zinkevich, M., Weimer, M., Li, L., Smola, A.J.: Parallelized stochastic gradient descent. In: Advances in Neural Information Processing Systems, pp. 2595–2603 (2010)Google Scholar
  23. 23.
    Zou, H., Li, R.: One-step sparse estimates in nonconcave penalized likelihood models. Ann. Stat. 36(4), 1509–1533 (2008)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature and Mathematical Optimization Society 2019

Authors and Affiliations

  1. 1.Georgia Institute of TechnologyAtlantaUSA

Personalised recommendations