Skip to main content
SpringerLink
Log in

A blockchain datastore for scalable IoT workloads using data decaying

  • Published:
Distributed and Parallel Databases Aims and scope Submit manuscript

Abstract

The Internet of Things (IoT) revolution has introduced sensor-rich devices to an ever growing landscape of smart environments. A key component in the IoT scenarios of the future is the requirement to utilize a shared database that allows all participants to operate collaboratively, transparently, immutably, correctly and with performance guarantees. Blockchain databases have been proposed by the community to alleviate these challenges, however existing blockchain architectures suffer from performance issues. In this paper we introduce Triabase, a novel permissioned blockchain system architecture that applies data decaying concepts to cope with scalability issues in regards to blockchain consensus and storage efficiency. For blockchain consensus, we propose the Proof of Federated Learning (PoFL) algorithm which exploits data decaying models as Proof-of-Work. For storage efficiency, we exploit federated learning to construct data postdiction machine learning models to minimize the storage of bulky data on the blockchain. We present a detailed explanation of our system architecture as well as the implementation in the Hyperledger fabric framework. We use our implementation to carry out an experimental evaluation with telco big data at scale showing that our framework exposes desirable qualities, namely efficient consensus at the blockchain layer while optimizing storage efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 2
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. Triabase. https://triabase.cs.ucy.ac.cy/.

  2. Triabase. https://triabase.cs.ucy.ac.cy/.

  3. CouchDB. https://couchdb.apache.org/.

  4. Vertex AI. https://cloud.google.com/vertex-ai.

References

  1. Yao, L., Sheng, Q.Z., Dustdar, S.: Web-based management of the Internet of Things. IEEE Internet Comput. 19(4), 60–67 (2015)

    Article  Google Scholar 

  2. Fuqaha, A.A., Guizani, M., Mohammadi, M., Aledhari, M., Ayyash, M.: Internet of Things: a survey on enabling technologies, protocols, and applications. IEEE Commun. Surv. Tutor. 17(4), 2347–2376 (2015)

    Article  Google Scholar 

  3. Li, S., Xu, L.D., Zhao, S.: The Internet of Things: a survey. Inf. Syst. Front. 17(2), 243–259 (2015)

    Article  Google Scholar 

  4. Atzori, L., Iera, A., Morabito, G.: The Internet of Things: a survey. Comput. Netw. 54(15), 2787–2805 (2010)

    Article  Google Scholar 

  5. Du, W., Atallah, M.J.: Privacy-preserving cooperative scientific computations. In: Proceedings of 14th IEEE Computer Security Foundations Workshop, pp. 273–282 (2001). https://doi.org/10.1109/CSFW.2001.930152

  6. Billsus, D., Pazzani, M.J.: Learning Collaborative Information Filters. In: Proceedings of the 15th International Conference on Machine Learning. ICML ’98, pp. 46–54. Morgan Kaufmann Publishers, San Francisco (1998)

  7. Atallah, M., Bykova, M., Li, J., Frikken, K., Topkara, M.: Private collaborative forecasting and benchmarking. In: Proceedings of the 2004 ACM Workshop on Privacy in the Electronic Society. WPES ’04, pp. 103–114. Association for Computing Machinery, New York (2004). https://doi.org/10.1145/1029179.1029204

  8. Li, J., Wang, C., Kang, X., Zhao, Q.: Camera localization for augmented reality and indoor positioning: a vision-based 3D feature database approach. Int. J. Digit. Earth 13(6), 727–741 (2020). https://doi.org/10.1080/17538947.2018.1564379

    Article  Google Scholar 

  9. Chen, S., Xu, H., Liu, D., Hu, B., Wang, H.: A vision of IoT: applications, challenges, and opportunities with china perspective. IEEE Internet Things J. 1(4), 349–359 (2014). https://doi.org/10.1109/JIOT.2014.2337336

    Article  Google Scholar 

  10. Costa, C., Zeinalipour-Yazti, D.: Telco big data research and open problems. In: Proceedings of the 35th IEEE International Conference on Data Engineering. ICDE‘19, pp. 2056–2059. IEEE Computer Society, 8–12 April 2019, Macau SAR, China (2019). https://doi.org/10.1109/ICDE.2019.00238

  11. Costa, C., Chatzimilioudis, G., Zeinalipour-Yazti, D., Mokbel, M.F.: Efficient Exploration of Telco big data with compression and decaying. In: 2017 IEEE 33rd International Conference on Data Engineering (ICDE), pp. 1332–1343 (2017). https://doi.org/10.1109/ICDE.2017.175.2375-026X

  12. Costa, C., Konstantinidis, A., Charalampous, A., Zeinalipour-Yazti, D., Mokbel, M.F.: Continuous decaying of telco big data with data postdiction. GeoInformatica 23(4), 533–557 (2019)

    Article  Google Scholar 

  13. Drakatos, P., Demetriou, E., Koumou, S., Konstantinidis, A., Zeinalipour-Yazti, D.: Triastore: A Web 3.0 blockchain datastore for massive IoT workloads. In: 2021 22nd IEEE International Conference on Mobile Data Management (MDM), pp. 187–192 (2021). https://doi.org/10.1109/MDM52706.2021.00038

  14. Drakatos, P., Demetriou, E., Koumou, S., Konstantinidis, A., Zeinalipour-Yazti, D.: Towards a blockchain database for massive IoT workloads. In: 2021 IEEE 37th International Conference on Data Engineering Workshops (ICDEW), pp. 76–79 (2021). https://doi.org/10.1109/ICDEW53142.2021.00021

  15. Li, L., Fan, Y., Tse, M., Lin, K.-Y.: A review of applications in federated learning. Comput. Ind. Eng. 149, 106854 (2020). https://doi.org/10.1016/j.cie.2020.106854

    Article  Google Scholar 

  16. Mammen, P.M.: Federated learning: opportunities and challenges. arXiv Preprint (2021). arXiv. arXiv:2101.05428 [cs]. https://doi.org/10.48550/arXiv.2101.05428. Accessed 17 Aug 2023

  17. Zhu, J., Cao, J., Saxena, D., Jiang, S., Ferradi, H.: Blockchain-empowered federated learning: challenges, solutions, and future directions. ACM Comput. Surv. 55(11), 240–124031 (2023). https://doi.org/10.1145/3570953

    Article  Google Scholar 

  18. Lim, W.Y.B., Luong, N.C., Hoang, D.T., Jiao, Y., Liang, Y.-C., Yang, Q., Niyato, D., Miao, C.: Federated learning in mobile edge networks: a comprehensive survey. IEEE Commun. Surv. Tutor. 22(3), 2031–2063 (2020). https://doi.org/10.1109/COMST.2020.2986024

    Article  Google Scholar 

  19. Khan, L.U., Saad, W., Han, Z., Hossain, E., Hong, C.S.: Federated learning for Internet of Things: recent advances, taxonomy, and open challenges. IEEE Commun. Surv. Tutor. 23(3), 1759–1799 (2021). https://doi.org/10.1109/COMST.2021.3090430

    Article  Google Scholar 

  20. Bravo-Marquez, F., Reeves, S., Ugarte, M.: Proof-of-learning: a blockchain consensus mechanism based on machine learning competitions. In: 2019 IEEE International Conference on Decentralized Applications and Infrastructures (DAPPCON), pp. 119–124 (2019). https://doi.org/10.1109/DAPPCON.2019.00023

  21. Truex, S., Baracaldo, N., Anwar, A., Steinke, T., Ludwig, H., Zhang, R., Zhou, Y.: A hybrid approach to privacy-preserving federated learning. In: Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security. AISec’19, pp. 1–11. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3338501.3357370. Accessed 17 Aug 2023

  22. Yang, Q., Liu, Y., Chen, T., Tong, Y.: Federated machine learning: concept and applications. ACM Trans. Intell. Syst. Technol. 10(2), 12–11219 (2019). https://doi.org/10.1145/3298981

    Article  Google Scholar 

  23. Amiri, M.J., Agrawal, D., Abbadi, A.E.: CAPER: a cross-application permissioned blockchain. Proc. VLDB Endow. 12(11), 1385–1398 (2019). https://doi.org/10.14778/3342263.3342275

    Article  Google Scholar 

  24. Androulaki, E., Barger, A., Bortnikov, V., Cachin, C., Christidis, K., De Caro, A., Enyeart, D., Ferris, C., Laventman, G., Manevich, Y., Muralidharan, S., Murthy, C., Nguyen, B., Sethi, M., Singh, G., Smith, K., Sorniotti, A., Stathakopoulou, C., Vukolić, M., Cocco, S.W., Yellick, J.: Hyperledger fabric: a distributed operating system for permissioned blockchains. In: Proceedings of the 13th EuroSys Conference. EuroSys ’18, pp. 1–15. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3190508.3190538

  25. Wang, J., Wang, H.: Monoxide: scale out blockchains with asynchronous consensus zones. In: Proceedings of 16th USENIX Symp. Netw. Syst. Des. Implement. (NSDI), pp. 95–112 (2019)

  26. Fernández Anta, A., Georgiou, C., Herlihy, M., Potop-Butucaru, M.: Principles of Blockchain Systems. Synthesis Lectures on Computer Science, vol. 9(2), pp. 1–213 Springer, Cham (2021). https://doi.org/10.2200/S01102ED1V01Y202105CSL014

  27. Xu, C., Zhang, C., Xu, J.: vChain: enabling verifiable Boolean range queries over blockchain databases. In: Proceedings of the 2019 International Conference on Management of Data. SIGMOD, pp. 141–158 (2019). https://doi.org/10.1145/3299869.3300083

  28. Peng, Z., Xu, C., Wang, H., Huang, J., Xu, J., Chu, X.: P2B-Trace: privacy-preserving blockchain-based contact tracing to combat pandemics. In: Proceedings of the 2021 ACM SIGMOD International Conference on Management of Data (2021). https://doi.org/10.1145/3448016.3459237

  29. Dai, Y., Xu, D., Maharjan, S., Chen, Z., He, Q., Zhang, Y.: Blockchain and deep reinforcement learning empowered intelligent 5G beyond. IEEE Netw. 33(3), 10–17 (2019). https://doi.org/10.1109/MNET.2019.1800376

    Article  Google Scholar 

  30. Reyna, A., Martín, C., Chen, J., Soler, E., Díaz, M.: On blockchain and its integration with IoT. Challenges and opportunities. Future Gen. Comput. Syst. 88, 173–190 (2018). https://doi.org/10.1016/j.future.2018.05.046

    Article  Google Scholar 

  31. Gilad, Y., Hemo, R., Micali, S., Vlachos, G., Zeldovich, N.: Algorand: Scaling byzantine agreements for cryptocurrencies. In: Proceedings of the 26th Symposium on Operating Systems Principles. SOSP, pp. 51–68 (2017). https://doi.org/10.1145/3132747.3132757

  32. Syta, E., Jovanovic, P., Kogias, E.K., Gailly, N., Gasser, L., Khoffi, I., Fischer, M.J., Ford, B.: Scalable bias-resistant distributed randomness. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 444–460 (2017). https://doi.org/10.1109/SP.2017.45

  33. Li, C., Li, P., Zhou, D., Xu, W., Long, F., Yao, A.: Scaling Nakamoto consensus to thousands of transactions per second. arXiv Preprint (2018). arXiv:1805.03870 [cs]

  34. Dinh, T.T.A., Wang, J., Chen, G., Liu, R., Ooi, B.C., Tan, K.-L.: Blockbench: a framework for analyzing private blockchains. In: Proceedings of the 2017 ACM International Conference on Management of Data. SIGMOD ’17, pp. 1085–1100. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3035918.3064033

  35. Drakatos, P., Koutrouli, E., Tsalgatidou, A.: Rapid blockchain scaling with efficient transaction assignment. In: 2021 6th South-East Europe Design Automation, Computer Engineering, Computer Networks and Social Media Conference (SEEDA-CECNSM), pp. 1–9 (2021). https://doi.org/10.1109/SEEDA-CECNSM53056.2021.9566222

  36. Drakatos, P., Koutrouli, E., Tsalgatidou, A.: Adrestus: secure, scalable blockchain technology in a decentralized ledger via zones. Blockchain Res. Appl. (2022). https://doi.org/10.1016/j.bcra.2022.100093

    Article  Google Scholar 

  37. Sompolinsky, Y., Zohar, A.: Accelerating bitcoin’s transaction processing. Fast money grows on trees, not chains. Citeseer (2013)

  38. Eyal, I., Gencer, A.E., Sirer, E.G., Renesse, R.v.: Bitcoin-NG: a scalable blockchain protocol. In: Proceedings of the 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’16), pp. 45–59 (2016)

  39. Apostolaki, M., Zohar, A., Vanbever, L.: Hijacking bitcoin: routing attacks on cryptocurrencies. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 375–392 (2017). https://doi.org/10.1109/SP.2017.29

  40. Gervais, A., Ritzdorf, H., Karame, G.O., Capkun, S.: Tampering with the delivery of blocks and transactions in bitcoin. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security. CCS ’15, pp. 692–705. Association for Computing Machinery, Denver (2015). https://doi.org/10.1145/2810103.2813655. Accessed 6 June 2020

  41. Kersten, M.L.: Big data space fungus. In: CIDR 2015, 7th Biennial Conference on Innovative Data Systems Research, Asilomar, CA, USA, 4–7 January 2015, Online Proceedings (2015)

  42. Yan, H., Ding, S., Suel, T.: Inverted index compression and query processing with optimized document ordering. In: Proceedings of the 18th International Conference on World Wide Web. WWW ’09, pp. 401–410. Association for Computing Machinery, New York (2009). https://doi.org/10.1145/1526709.1526764. Accessed 26 Sept 2022

  43. Soroush, E., Balazinska, M.: Time travel in a scientific array database. In: 2013 IEEE 29th International Conference on Data Engineering (ICDE), pp. 98–109 (2013). https://doi.org/10.1109/ICDE.2013.6544817

  44. Lakshminarasimhan, S., Shah, N., Ethier, S., Klasky, S., Latham, R., Ross, R., Samatova, N.F.: Compressing the incompressible with ISABELA: in-situ reduction of spatio-temporal data. In: Jeannot, E., Namyst, R., Roman, J. (eds.) Euro-Par 2011 Parallel Processing. Lecture Notes in Computer Science, pp. 366–379. Springer, Berlin (2011). https://doi.org/10.1007/978-3-642-23400-2_34

    Chapter  Google Scholar 

  45. Bhattacherjee, S., Deshpande, A., Sussman, A.: PStore: an efficient storage framework for managing scientific data. In: Proceedings of the 26th International Conference on Scientific and Statistical Database Management. SSDBM ’14, pp. 1–12. Association for Computing Machinery, New York (2014).https://doi.org/10.1145/2618243.2618268. Accessed 26 Sept 2022

  46. You, L.L., Pollack, K.T., Long, D.D.E., Gopinath, K.: PRESIDIO: a framework for efficient archival data storage. ACM Trans. Stor. 7(2), 6–1660 (2011). https://doi.org/10.1145/1970348.1970351

    Article  Google Scholar 

  47. Bhattacherjee, S., Chavan, A., Huang, S., Deshpande, A., Parameswaran, A.: Principles of dataset versioning: exploring the recreation/storage tradeoff. In: Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, vol. 8(12), pp. 1346–1357 (2015). https://doi.org/10.14778/2824032.2824035. Accessed 05 Apr 2021

  48. Chaudhuri, S., Das, G., Narasayya, V.: Optimized stratified sampling for approximate query processing. ACM Trans. Database Syst. 32(2), 9 (2007). https://doi.org/10.1145/1242524.1242526

    Article  Google Scholar 

  49. Zeng, K., Agarwal, S., Dave, A., Armbrust, M., Stoica, I.: G-OLA: generalized on-line aggregation for interactive analysis on big data. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. SIGMOD ’15, pp. 913–918. Association for Computing Machinery, New York (2015). https://doi.org/10.1145/2723372.2735381. Accessed 26 Sept 2022

  50. Agarwal, S., Mozafari, B., Panda, A., Milner, H., Madden, S., Stoica, I.: BlinkDB: queries with bounded errors and bounded response times on very large data. In: Proceedings of the 8th ACM European Conference on Computer Systems. EuroSys ’13, pp. 29–42. Association for Computing Machinery, New York (2013). https://doi.org/10.1145/2465351.2465355. Accessed 2022-09-26

  51. Sidirourgos, L., Kersten, M., Boncz, P.: SciBORQ: scientific data management with bounds on runtime and quality. In: Proceedings of the biennial Conference on Innovative Data Systems Research (2011)

  52. Castro, M., Liskov, B.: Practical Byzantine fault tolerance. In: Proceedings of the 3rd Symposium on Operating Systems Design and Implementation. OSDI ’99, pp. 173–186. USENIX Association, Berkeley (1999)

  53. Lamport, L., Shostak, R., Pease, M.: The Byzantine generals problem. ACM Trans. Program. Lang. Syst. 4(3), 382–401 (1982). https://doi.org/10.1145/357172.357176

    Article  Google Scholar 

  54. Costa, C., Charalampous, A., Konstantinidis, A., Zeinalipour-Yazti, D., Mokbel, M.F.: Decaying Telco big data with data postdiction. In: 2018 19th IEEE International Conference on Mobile Data Management (MDM), pp. 106–115 (2018)

  55. Kosba, A., Miller, A., Shi, E., Wen, Z., Papamanthou, C.: Hawk: The blockchain model of cryptography and privacy-preserving smart contracts. In: 2016 IEEE Symposium on Security and Privacy (SP), pp. 839–858 (2016). https://doi.org/10.1109/SP.2016.55

  56. MARTA hackathon—dataset by brentbrewington. https://data.world/brentbrewington/marta-hackathon Accessed 15 Dec 2022

  57. Sma-RTy—SMArt systems & aRTificial intelligence. https://sma-rty.com/ Accessed 15 Dec 2022

  58. Apache CouchDB: https://couchdb.apache.org/. Accessed 15 Dec 2022

  59. McMahan, B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.y.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282. PMLR (2017)

  60. Yang, T., Andrew, G., Eichner, H., Sun, H., Li, W., Kong, N., Ramage, D., Beaufays, F.: Applied federated learning: improving Google keyboard query suggestions (2018). arXiv:1812.02903 [cs, stat]

  61. Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., Fergus, R.: Regularization of neural networks using DropConnect. In: International Conference on Machine Learning, pp. 1058–1066. PMLR (2013)

Download references

Funding

The authors did not receive support from any organization for the submitted work.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Cyprus, 2109, Nicosia, Cyprus

    Panagiotis Drakatos, Constantinos Costa, Andreas Konstantinidis, Panos K. Chrysanthis & Demetrios Zeinalipour-Yazti

  2. Department of Computer Science, Frederick University, 1036, Nicosia, Cyprus

    Andreas Konstantinidis

  3. Department of Computer Science, University of Pittsburgh, Pittsburgh, 15260, USA

    Constantinos Costa & Panos K. Chrysanthis

Authors
  1. Panagiotis Drakatos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Constantinos Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andreas Konstantinidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Panos K. Chrysanthis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Demetrios Zeinalipour-Yazti
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

A.B.C.E wrote the main manuscript text and A.B. prepared the experimental evaluation results. All authors reviewed the manuscript.

Corresponding author

Correspondence to Demetrios Zeinalipour-Yazti.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose. The pre-last author is a Special Issue Coordinator of the Distributed and Parallel Databases journal. The last author is on the Editorial Board of the Distributed and Parallel Databases journal.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Drakatos, P., Costa, C., Konstantinidis, A. et al. A blockchain datastore for scalable IoT workloads using data decaying. Distrib Parallel Databases (2024). https://doi.org/10.1007/s10619-024-07441-9

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10619-024-07441-9

Keywords

Search

Navigation