External sorting on flash storage: reducing cell wearing and increasing efficiency by avoiding intermediate writes
- 358 Downloads
- 1 Citations
Abstract
This paper studies the problem of how to conduct external sorting on flash drives while avoiding intermediate writes to the disk. The focus is on sort in portable electronic devices, where relations are only larger than the main memory by a small factor, and on sort as part of distributed processes where relations are frequently partially sorted. In such cases, sort algorithms that refrain from writing intermediate results to the disk have three advantages over algorithms that perform intermediate writes. First, on devices in which read operations are much faster than writes, such methods are efficient and frequently outperform Merge Sort. Secondly, they reduce flash cell degradation caused by writes. Thirdly, they can be used in cases where there is not enough disk space for the intermediate results. Novel sort algorithms that avoid intermediate writes to the disk are presented. An experimental evaluation, on different flash storage devices, shows that in many cases the new algorithms can extend the lifespan of the devices by avoiding unnecessary writes to the disk, while maintaining efficiency, in comparison with Merge Sort.
Keywords
Sort algorithms External sorting Flash memory Solid-state drive SSD Cell wearing Write endurance Merge SortNotes
Acknowledgments
This research was supported in part by the Israel Science Foundation (Grant 1467/13) and by the Isreali Ministry of Science and Technology (Grant 3-9617). We thank the anonymous reviewers for their insightful comments and suggestions.
References
- 1.Ajwani, D., Malinger, I., Meyer, U., Toledo, S.: Characterizing the performance of flash memory storage devices and its impact on algorithm design. In: Proceedings of the 7th International Conference on Experimental Algorithms, pp. 208–219, Provincetown, MA, USA. Springer (2008)Google Scholar
- 2.Albutiu, M.-C., Kemper, A., Neumann, T.: Massively parallel sort-merge joins in main memory multi-core database systems. Proc. VLDB Endow. 5(10), 1064–1075 (2012)CrossRefGoogle Scholar
- 3.Aldous, D., Diaconis, P.: Longest increasing subsequences: from patience sorting to the Baik–Deift–Johansson theorem. Bull. Am. Math. Soc. 36(4), 413–432 (1999)MathSciNetCrossRefMATHGoogle Scholar
- 4.Balkesen, C., Alonso, G., Teubner, J., Özsu, M.T.: Multi-core, main-memory joins: sort vs. hash revisited. Proc. VLDB Endow. 7(1), 85–96 (2013)CrossRefGoogle Scholar
- 5.Ben-Moshe, S., Fischer, E., Fischer, M., Kanza, Y., Matsliah, A., Staelin, C.: Detecting and exploiting near-sortedness for efficient relational query evaluation. In: Proceedings of the 14th International Conference on Database Theory, pp. 256–267, Uppsala, Sweden. ACM (2011)Google Scholar
- 6.Bender, M.A., Farach-Colton, M., Johnson, R., Kuszmaul, B.C., Medjedovic, D., Montes, P., Shetty, P., Spillane, R.P., Zadok, E.: Don’t thrash: how to cache your hash on flash. In: Proceedings of the 3rd USENIX Conference on Hot Topics in Storage and File Systems, Portland, OR. USENIX Association (2011)Google Scholar
- 7.Bonnet, P., Bouganim, L.: Flash device support for database management. In: Fifth Biennial Conference on Innovative Data Systems Research, pp. 1–8. Asilomar, CA, USA (2011)Google Scholar
- 8.Brähler, S.: Analysis of the android architecture. Master’s thesis, Karlsruhe Institute (2010). http://os.itec.kit.edu/downloads/sa_2010_braehler-stefan_android-architecture.pdf
- 9.Chandramouli, B., Goldstein, J.: Patience is a virtue: revisiting merge and sort on modern processors. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 731–742, Snowbird, Utah, USA (2014)Google Scholar
- 10.Chen, F., Koufaty, D.A., Zhang, X.: Understanding intrinsic characteristics and system implications of flash memory based solid state drives. In: Proceedings of the Eleventh International Joint Conference on Measurement and Modeling of Computer Systems, pp. 181–192, Seattle, WA, USA. ACM (2009)Google Scholar
- 11.Chhugani, J., Nguyen, A.D., Lee, V.W., Macy, W., Hagog, M., Chen, Y.-K., Baransi, A., Kumar, S., Dubey, P.: Efficient implementation of sorting on multi-core simd cpu architecture. Proc. VLDB Endow. 1(2), 1313–1324 (2008)CrossRefGoogle Scholar
- 12.Cormen, T.H., Stein, C., Rivest, R.L., Leiserson, C.E.: Introduction to Algorithms, 2nd edn. McGraw-Hill Higher Education, New York (2001)MATHGoogle Scholar
- 13.Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
- 14.Debnath, B., Sengupta, S., Li, J.: Flashstore: high throughput persistent key-value store. Proc. VLDB Endow. 3(1–2), 1414–1425 (2010)CrossRefGoogle Scholar
- 15.DeWitt, D.J., Do, J., Patel, J.M., Zhang, D.: Fast peak-to-peak behavior with SSD buffer pool. In: Proceedings of the International Conference on Data Engineering, pp. 1129–1140, Washington, DC, USA. IEEE Computer Society (2013)Google Scholar
- 16.Do, J., Kee, Y.-S., Patel, J.M., Park, C., Park, K., DeWitt, D.J.: Query processing on smart SSDs: opportunities and challenges. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pp. 1221–1230, New York, NY, USA (2013)Google Scholar
- 17.Do, J., Patel, J.M.: Join processing for flash SSDs: remembering past lessons. In: Proceedings of the Fifth International Workshop on Data Management on New Hardware, pp. 1–8, Providence, Rhode Island (2009)Google Scholar
- 18.Do, J., Zhang, D., Patel, J.M., DeWitt, D.J., Naughton, J.F., Halverson, A.: Turbocharging DBMS buffer pool using SSDs. In: Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data, pp. 1113–1124, Athens, Greece (2011)Google Scholar
- 19.Fredman, M.L.: On computing the length of longest increasing subsequences. Discrete Math. 11(1), 29–35 (1975)MathSciNetCrossRefMATHGoogle Scholar
- 20.Friedman, R., Hefez, I., Kanza, Y., Levin, R., Safra, E., Sagiv, Y.: Wiser: a web-based interactive route search system for smartphones. In: Proceedings of the 21st International Conference on World Wide Web, pp. 337–340, Lyon, France (2012)Google Scholar
- 21.Garcia-Molina, H., Ullman, J.D., Widom, J.: Database Systems: The Complete Book, 2nd edn. Prentice Hall Press, Upper Saddle River (2008)Google Scholar
- 22.Gotsman, R., Kanza, Y.: A dilution-matching-encoding compaction of trajectories over road networks. GeoInformatica 19(2), 331–364 (2015)CrossRefGoogle Scholar
- 23.Graefe, G.: Implementing sorting in database systems. ACM Comput. Surv. 38(3), 1–37 (2006)CrossRefGoogle Scholar
- 24.Graefe, G.: The five-minute rule 20 years later (and how flash memory changes the rules). Commun. ACM 52(7), 48–59 (2009)CrossRefGoogle Scholar
- 25.Graefe, G., Harizopoulos, S., Kuno, H.A., Shah, M.A., Tsirogiannis, D., Wiener, J.L.: Designing database operators for flash-enabled memory hierarchies. IEEE Data Eng. Bull. 33(4), 21–27 (2010)Google Scholar
- 26.Härder, T.: A scan-driven sort facility for a relational database system. In: Proceedings of the 3rd International Conference on Very Large Data Bases, vol. 3, pp. 236–244, Tokyo, Japan. VLDB Endowment (1977)Google Scholar
- 27.Hu, X.-Y., Eleftheriou, E., Haas, R., Iliadis, I., Pletka, R.: Write amplification analysis in flash-based solid state drives. In: Proceedings of SYSTOR: The Israeli Experimental Systems Conference, pp. 10:1–10:9, Haifa, Israel. ACM (2009)Google Scholar
- 28.Kang, W.-H., Lee, S.-W., Moon, B.: Flash-based extended cache for higher throughput and faster recovery. Proc. VLDB Endow. 5(11), 1615–1626 (2012)CrossRefGoogle Scholar
- 29.Knuth, D.E.: Length of strings for a merge sort. Commun. ACM 6(11), 685–688 (1963)CrossRefMATHGoogle Scholar
- 30.Knuth, D.E.: The Art of Computer Programming: Sorting and Searching, vol. 3, 2nd edn. Addison Wesley Longman Publishing, Redwood City (1998)MATHGoogle Scholar
- 31.Koltsidas, I., Viglas, S.D.: Flashing up the storage layer. Proc. VLDB Endow. 1(1), 514–525 (2008)CrossRefGoogle Scholar
- 32.Koltsidas, I., Viglas, S.D.: Data management over flash memory. In: Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data, pp. 1209–1212, Athens, Greece (2011)Google Scholar
- 33.Kwan, S.C., Baer, J.-L.: The I/O performance of multiway mergesort and tag sort. IEEE Trans. Comput. C–34(4), 383–387 (1985)MathSciNetCrossRefGoogle Scholar
- 34.Larson, P.-A.: External sorting: run formation revisited. IEEE Trans. Knowl. Data Eng. 15(4), 961–972 (2003)CrossRefGoogle Scholar
- 35.Larson, P.-A., Graefe, G.: Memory management during run generation in external sorting. In: Proceedings of the 1998 ACM SIGMOD International Conference on Management of Data, pp. 472–483, Seattle, Washington, USA (1998)Google Scholar
- 36.Lee, S.-W., Moon, B.: Design of flash-based DBMS: an in-page logging approach. In: Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data, pp. 55–66, Beijing, China (2007)Google Scholar
- 37.Lee, S.-W., Moon, B., Park, C., Kim, J.-M., Kim, S.-W.: A case for flash memory SSD in enterprise database applications. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, pp. 1075–1086, Vancouver, Canada. ACM (2008)Google Scholar
- 38.Levin, R., Kanza, Y.: Stratified-sampling over social networks using MapReduce. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 863–874, Snowbird, Utah, USA (2014)Google Scholar
- 39.Liu, X., Salem, K.: Hybrid storage management for database systems. Proc. VLDB Endow. 6(8), 541–552 (2013)CrossRefGoogle Scholar
- 40.Liu, Y., He, Z., Chen, Y.-P.P., Nguyen, T.: External sorting on flash memory via natural page run generation. Comput. J. 54(11), 1882–1990 (2011)CrossRefGoogle Scholar
- 41.Mallows, C.: Problem 62–2, patience sorting. SIAM Rev. 4(2), 148–149 (1962)Google Scholar
- 42.Mann, S.: Wearable computing: a first step toward personal imaging. Computer 30(2), 25–32 (1997)CrossRefGoogle Scholar
- 43.Myers, D.: On the use of NAND flash memory in high-performance relational databases. Master’s thesis, Massachusetts Institute of Technology (MIT) (2008). http://hdl.handle.net/1721.1/43070
- 44.Nyberg, C., Barclay, T., Cvetanovic, Z., Gray, J., Lomet, D.: Alphasort: a cache-sensitive parallel external sort. VLDB J. 4(4), 603–628 (1995)CrossRefGoogle Scholar
- 45.Pang, H., Carey, M.J., Livny, M.: Memory-adaptive external sorting. In: Proceedings of the 19th International Conference on Very Large Data Bases, pp. 618–629, San Francisco, CA, USA. Morgan Kaufmann Publishers (1993)Google Scholar
- 46.Salzberg, B.: Merging sorted runs using large main memory. Acta Inf. 27(3), 195–215 (1989)MathSciNetCrossRefMATHGoogle Scholar
- 47.Soundararajan, G., Prabhakaran, V., Balakrishnan, M., Wobber, T.: Extending SSD lifetimes with disk-based write caches. In: Proceedings of the 8th USENIX Conference on File and Storage Technologies, pp. 8–8, Berkeley, CA, USA. USENIX Association (2010)Google Scholar
- 48.Tsirogiannis, D., Harizopoulos, S., Shah, M.A., Wiener, J.L., Graefe, G.: Query processing techniques for solid state drives. In: Proceedings of the 35th SIGMOD international Conference on Management of Data, pp. 59–72, Providence, Rhode Island, USA (2009)Google Scholar
- 49.Weiser, M.: Some computer science issues in ubiquitous computing. Commun. ACM 36(7), 75–84 (1993)CrossRefGoogle Scholar
- 50.Zhang, N., Tatemura, J., Patel, J., Hacigumus, H.: Re-evaluating designs for multi-tenant OLTP workloads on SSD-based I/O subsystems. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 1383–1394, Snowbird, Utah, USA (2014)Google Scholar
- 51.Zhang, W., Larson, P.-A.: Dynamic memory adjustment for external Mergesort. In: Proceedings of the 23rd International Conference on Very Large Data Bases, pp. 376–385, San Francisco, CA, USA. Morgan Kaufmann Publishers (1997)Google Scholar
- 52.Zhang, W., Larson, P.-A.: Buffering and read-ahead strategies for external Mergesort. In: Proceedings of the 24th International Conference on Very Large Data Bases, pp. 523–533, San Francisco, CA, USA (1998)Google Scholar
- 53.Zheng, L., Larson, P.-A.: Speeding up external Mergesort. IEEE Trans. Knowl. Data Eng. 8(2), 322–332 (1996)CrossRefGoogle Scholar
- 54.Zheng, Y., Xie, X., Ma, W.-Y.: Geolife: a collaborative social networking service among user, location and trajectory. IEEE Data Eng. Bull. 33(2), 32–40 (2010)Google Scholar