Abstract
A common scenario when developing local PC applications such as games, mobile apps, or presentation software is storing many small files or records as application data and needing to retrieve and manipulate those records with some unique ID. In this kind of scenario, a developer has the choice of simply saving the records as files with their unique ID as the filename or using an embedded on-disk key-value database. Many file systems have performance issues when handling large number of small files, but developers may want to avoid a dependency on an embedded database if it offers little benefit or has a detrimental effect on performance for their use case. Despite the need for benchmarks to enable informed answers to this design decision, little research has been done in this area. Our contribution is the comparison and analysis of the performance for the insert, update, get, and remove operations and the space efficiency of storing records as files vs. using key-value embedded databases including SQLite3, LevelDB, RocksDB, and Berkeley DB.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
References
Chandramouli, B., Prasaad, G., Kossmann, D., Levandoski, J., Hunter, J., Barnett, M.: Faster. Proc. VLDB Endowment 11(12), 1930–1933 (2018)
Chen, C., Deng, T., Zhang, J., Zou, Y., Zhu, X., Yin, S.: Optimizing KV-embedded file systems through flat indexing. In: FILT, November 2020
Chen, T.Y., Chang, Y.H., Chen, S.H., Hsu, N.I., Wei, H.W., Shih, W.K.: On space utilization enhancement of file systems for embedded storage systems. ACM Trans. Embed. Comput. Syst. 16(3), 1–28 (2017)
Collberg, C.S., Hartman, J.H., Babu, S., Udupa, S.K.: SLINKY: static linking reloaded. In: Proceedings of the Annual Conference on USENIX Annual Technical Conference, ATEC 2005, USA, pp. 34. USENIX Association (2005)
Gupta, A., Tyagi, S., Panwar, N., Sachdeva, S., Saxena, U.: NoSQL databases: critical analysis and comparison. In: 2017 International Conference on Computing and Communication Technologies for Smart Nation (IC3TSN), pp. 293–299 (2017)
Lutes, K., Patchigolla, V.N.R., Springer, J.: Embedded database management performance. In: Information Technology: New Generations, Third International Conference on, Los Alamitos, CA, USA, pp. 998–1001. IEEE Computer Society, April 2011
Patil, S., Gibson, G.: Scale and concurrency of GIGA+: File system directories with millions of files. In: Proceedings of the 9th USENIX Conference on File and Stroage Technologies, FAST2011, USA, p. 13. USENIX Association (2011)
Puangsaijai, W., Puntheeranurak, S.: A comparative study of relational database and key-value database for big data applications. In: 2017 International Electrical Engineering Congress (iEECON), pp. 1–4 (2017)
Ruan, L., Ding, Y., Dong, B., Li, X., Xiao, L.: Small files problem in parallel file system. In: Network Computing and Information Security, International Conference on, Los Alamitos, CA, USA, May 2011, vol. 2, pp. 227–232. IEEE Computer Society (2011)
Sears, R., Van Ingen, C., Gray, J.: To BLOB or not to BLOB: large object storage in a database or a filesystem? CoRR, arXiv:abs/cs/0701168 (2007)
Stancu-Mara, S., Baumann, P.: A comparative benchmark of large objects in relational databases. In: Proceedings of the 2008 International Symposium on Database Engineering & Applications, IDEAS 2008, New York, NY, USA, pp. 277-284. Association for Computing Machinery (2008)
Techopedia. What is an embedded database? - definition from techopedia, December 2014
Tulkinbekov, K., Kim, D.-H.: CaseDB: lightweight key-value store for edge computing environment. IEEE Access 8, 149775–149786 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hines, J., Cunningham, N., Alférez, G.H. (2023). Performance Comparison of Operations in the File System and in Embedded Key-Value Databases. In: Arai, K. (eds) Intelligent Computing. SAI 2023. Lecture Notes in Networks and Systems, vol 739. Springer, Cham. https://doi.org/10.1007/978-3-031-37963-5_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-37963-5_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-37962-8
Online ISBN: 978-3-031-37963-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)