Robust Decentralized Mean Estimation with Limited Communication

  • Gábor Danner
  • Márk Jelasity
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11014)


Mean estimation, also known as average consensus, is an important computational primitive in decentralized systems. When the average of large vectors has to be computed, as in distributed data mining applications, reducing the communication cost becomes a key design goal. One way of reducing communication cost is to add dynamic stateful encoders and decoders to traditional mean estimation protocols. In this approach, each element of a vector message is encoded in a few bits (often only one bit) and decoded by the recipient node. However, due to this encoding and decoding mechanism, these protocols are much more sensitive to benign failure such as message drop and message delay. Properties such as mass conservation are harder to guarantee. Hence, known approaches are formulated under strong assumptions such as reliable communication, atomic non-overlapping transactions or even full synchrony. In this work, we propose a communication efficient algorithm that supports known codecs even if transactions overlap and the nodes are not synchronized. The algorithm is based on push-pull averaging, with novel features to support fault tolerance and compression. As an independent contribution, we also propose a novel codec, called the pivot codec. We demonstrate experimentally that our algorithm improves the performance of existing codecs and the novel pivot codec dominates the competing codecs in the scenarios we studied.


  1. 1.
    Boyd, S., Ghosh, A., Prabhakar, B., Shah, D.: Randomized gossip algorithms. IEEE Trans. Inf. Theory 52(6), 2508–2530 (2006)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Brockett, R.W., Liberzon, D.: Quantized feedback stabilization of linear systems. IEEE Trans. Autom. Control 45(7), 1279–1289 (2000)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Carli, R., Bullo, F., Zampieri, S.: Quantized average consensus via dynamic coding/decoding schemes. Int. J. Robust Nonlinear Control 20(2), 156–175 (2010)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Carli, R., Fagnani, F., Frasca, P., Zampieri, S.: Gossip consensus algorithms via quantized communication. Automatica 46(1), 70–80 (2010)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Fu, M., Xie, L.: Finite-level quantized feedback control for linear systems. IEEE Trans. Automatic Control 54(5), 1165–1170 (2009)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Jelasity, M., Montresor, A., Babaoglu, O.: Gossip-based aggregation in large dynamic networks. ACM Trans. Comput. Syst. 23(3), 219–252 (2005)CrossRefGoogle Scholar
  7. 7.
    Jesus, P., Baquero, C., Almeida, P.S.: Fault-tolerant aggregation for dynamic networks. In: Proceedings of 29th IEEE Symposium on Reliable Distributed Systems (SRDS), pp. 37–43 (2010)Google Scholar
  8. 8.
    Kempe, D., Dobra, A., Gehrke, J.: Gossip-based computation of aggregate information. In: Proceedings of 44th Annual IEEE Symposium on Foundations of Computer Science (FOCS 2003) (2003)Google Scholar
  9. 9.
    Konecný, J., McMahan, H.B., Yu, F.X., Richtárik, P., Suresh, A.T., Bacon, D.: Federated learning: strategies for improving communication efficiency. In: Private Multi-Party Machine Learning (NIPS 2016 Workshop) (2016)Google Scholar
  10. 10.
    Li, T., Fu, M., Xie, L., Zhang, J.F.: Distributed consensus with limited communication data rate. IEEE Trans. Automatic Control 56(2), 279–292 (2011)MathSciNetCrossRefGoogle Scholar
  11. 11.
    McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Singh, A., Zhu, J. (eds.) Proceedings of 20th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 54, pp. 1273–1282 (2017)Google Scholar
  12. 12.
    Montresor, A., Jelasity, M.: Peersim: a scalable P2P simulator (extended abstract). In: Proceedings of 9th IEEE International Conference on Peer-to-Peer Computing (P2P 2009), pp. 99–100 (2009)Google Scholar
  13. 13.
    Nair, G.N., Fagnani, F., Zampieri, S., Evans, R.J.: Feedback control under data rate constraints: an overview. Proc. IEEE 95(1), 108–137 (2007)CrossRefGoogle Scholar
  14. 14.
    Niederbrucker, G., Gansterer, W.N.: Robust gossip-based aggregation: A practical point of view. In: Proceedings of Fifteenth Workshop on Algorithm Engineering and Experiments (ALENEX), pp. 133–147 (2013)CrossRefGoogle Scholar
  15. 15.
    Olfati-Saber, R., Fax, J.A., Murray, R.M.: Consensus and cooperation in networked multi-agent systems. Proc. IEEE 95(1), 215–233 (2007)CrossRefGoogle Scholar
  16. 16.
    Ratanaworabhan, P., Ke, J., Burtscher, M.: Fast lossless compression of scientific floating-point data. In: Data Compression Conference (DCC 2006), pp. 133–142 (2006)Google Scholar
  17. 17.
    van Renesse, R., Birman, K.P., Vogels, W.: Astrolabe: a robust and scalable technology for distributed system monitoring, management, and data mining. ACM Trans. Comput. Syst. 21(2), 164–206 (2003)CrossRefGoogle Scholar
  18. 18.
    Suresh, A.T., Yu, F.X., Kumar, S., McMahan, H.B.: Distributed mean estimation with limited communication. In: Proceedings of 34th International Conference on Machine Learning, (ICML), pp. 3329–3337 (2017)Google Scholar
  19. 19.
    Xiao, L., Boyd, S.: Fast linear iterations for distributed averaging. Syst. Control Lett. 53(1), 65–78 (2004)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Xiao, L., Boyd, S., Lall, S.: A scheme for robust distributed sensor fusion based on average consensus. In: IPSN 2005: Proceedings of 4th International Symposium on Information Processing in Sensor Networks, p. 9 (2005)Google Scholar
  21. 21.
    Zhu, M., Martinez, S.: On the convergence time of asynchronous distributed quantized averaging algorithms. IEEE Trans. Automatic Control 56(2), 386–390 (2011)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.University of SzegedSzegedHungary
  2. 2.MTA-SZTE Research Group on Artificial Intelligence, University of SzegedSzegedHungary

Personalised recommendations