A Tickless AMP Distributed Core-Based Microkernel for Big Data

  • Karim SobhEmail author
  • Amr El-Kadi
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1069)


Operating system kernels are designed to manage the underlying hardware resources of an environment and avail them to applications in a unified secure way. General purpose kernels perform average over different diversified application profiles and workloads. Their policies are not designed to be a perfect-fit for Big Data applications. In this paper, we present a special purpose tickless AMP distributed kernel designed with Big Data in mind. Design goals and decisions are presented together with the proposed kernel architecture and main subsystems. A prototype of the proposed kernel was used to conduct experiments for assessing the performance gain in Big Data applications in comparison with the open source Linux kernel. Experiment results show a gain in performance that ranges from 1.52 to 4.89 folds based on the target application profile and workload. The new proposed kernel was able to achieve 2.34 folds speed up over Linux/Hadoop using the TeraSort benchmark. Finally, a factorial ANOVA and regression analysis was conducted to assess the different factors affecting the gain in performance, and we were able to show and quantify the positive influential effect of our proposed new kernel on the performance.


Asymmetric multiprocessing Tickless kernels Microkernel Distributed kernels Multi-core kernels Big data MapReduce Inter-Processor Interrupts (IPI) Network protocols 


  1. 1.
    Altschul, S.F., Gish, W., Miller, W., Myers, E.W., Lipman, D.J.: Basic local alignment search tool. J. Mol. Biol. 215(3), 403–410 (1990)CrossRefGoogle Scholar
  2. 2.
    Borthakur, D.: HDFS architecture guide (2008)Google Scholar
  3. 3.
    Brinkmann, M.: Porting the GNU hurd to the L4 microkernel. Free Software Foundation Inc. (2003)Google Scholar
  4. 4.
    Corbet, J.: (Nearly) full tickless operation in 3.10, May 2013. Accessed on 03 Feb 2019
  5. 5.
    De Witte, N., Vincke, R., Landschoot, S.V., Steegmans, E., Boydens, J.: Comparing dual-core SMP/AMP performance on a telecom architecture (2013)Google Scholar
  6. 6.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. In: OSDI 2004: Sixth Symposium on Operating System Design and Implementation, San Francisco, CA, pp. 137–150 (2004)Google Scholar
  7. 7.
    Engler, D.R., Kaashoek, M.F.: Exterminate all operating system abstractions. In: Proceedings 5th Workshop on Hot Topics in Operating Systems (HotOS-V), May 1995, pp. 78–83 (1995)Google Scholar
  8. 8.
    Giceva, J., Zellweger, G., Alonso, G., Rosco, T.: Customized OS support for data-processing. In: Proceedings of the 12th International Workshop on Data Management on New Hardware, DaMoN 2016, pp. 1–6. ACM, New York (2016)Google Scholar
  9. 9.
    Guarnieri, M.: The unreasonable accuracy of moore’s law [historical]. IEEE Ind. Electron. Mag. 10(1), 40–43 (2016)CrossRefGoogle Scholar
  10. 10.
    Brinch Hansen, P. (ed.): Classic Operating Systems: From Batch Processing to Distributed Systems, 1st edn. Springer Publishing Company, New York (2010)zbMATHGoogle Scholar
  11. 11.
    Heiser, G., Elphinstone, K.: L4 microkernels: the lessons from 20 years of research and deployment. ACM Trans. Comput. Syst. 34(1), 1–29 (2016)CrossRefGoogle Scholar
  12. 12.
    Hunter, C.I., Mitchell, A.L., Jones, P.H., McAnulla, C., Pesseat, S., Scheremetjew, M., Hunter, S.: Metagenomic analysis: the challenge of the data bonanza. Briefings Bioinform. 13(6), 743–746 (2012)CrossRefGoogle Scholar
  13. 13.
    Ionescu, B., Ionescu, D., Gadea, C., Solomon, B., Trifan, M.: An architecture and methods for big data analysis. In: Valentina, E.B., Lakhmi, C.J., Branko, K. (eds.) Soft Computing Applications, pp. 491–514. Springer International Publishing, Cham (2016)Google Scholar
  14. 14.
    Jlassi, A., Martineau, P.: Benchmarking hadoop performance in the cloud - an in depth study of resource management and energy consumption, pp. 192–201 (2016)Google Scholar
  15. 15.
    Khan, N., Yaqoob, I., Abaker, I., Hashem, I.A.T., Zakira, I., Kamaleldin, W.M.A., Alam, M.A., Shiraz, M., Gani, A.: Big data: survey, technologies, opportunities, and challenges. Sci. World J. (2014)Google Scholar
  16. 16.
    Kiefer, T., Schlegel, B., Lehner, W.: Experimental evaluation of NUMA effects on database management systems. In: BTW (2013)Google Scholar
  17. 17.
    Klein, J., Gorton, I.: Runtime performance challenges in big data systems. In: Proceedings of the 2015 Workshop on Challenges in Performance Methods for Software Development, WOSP 2015, pp. 17–22. ACM, New York (2015)Google Scholar
  18. 18.
    Le Mignot, G.: The GNU hurd. In: Extended Abstract of Talk at Libre Software Meeting, Dijon, France (2005)Google Scholar
  19. 19.
    Li, S., Supittayapornpong, S., Maddah-Ali, M.A., Avestimehr, A.S.: Coded terasort. CoRR, abs/1702.04850 (2017)Google Scholar
  20. 20.
    MAPR. Terasort benchmark comparison for yarn. Accessed on 31 Jan 2019
  21. 21.
    Montgomery, D.C.: Design and Analysis of Experiments, 8th edn. Wiley, New York (2012)Google Scholar
  22. 22.
    Moore, G.E.: Cramming more components onto integrated circuits, reprinted from electronics, vol. 38, no. 8, 19 April 1965, pp. 114. IEEE Solid-State Circ. Soc. Newsl. 11(3), 33–35 (2006)Google Scholar
  23. 23.
    Mullender, S.J., van Rossum, G., Tananbaum, A.S., van Renesse, R., van Staveren, H.: Amoeba: a distributed operating system for the 1990s. Computer 23(5), 44–53 (1990). Scholar
  24. 24.
    Murata, Y., Kanda, W., Hanaoka, K., Ishikawa, H., Nakajima, T.: A study on asymmetric operating systems on symmetric multiprocessors. In: Proceedings of the 2007 International Conference on Embedded and Ubiquitous Computing, EUC 2007, pp. 182–195. Springer-Verlag, Heidelberg (2007)Google Scholar
  25. 25.
    Narayanaswamy, G., Balaji, P., Feng, W.: An analysis of 10-gigabit ethernet protocol stacks in multicore environments. In: 15th Annual IEEE Symposium on High-Performance Interconnects (HOTI 2007), Aug 2007, pp. 109–116 (2007)Google Scholar
  26. 26.
    O’Malley, O.: Terabyte sort on apache hadoop (2008)Google Scholar
  27. 27.
    Parziale, L.: Sap on db2 9 for z/os: Implementing application servers on Linux for system z. IBM, International Technical Support Organization (2005)Google Scholar
  28. 28.
    Pavlo, A., Paulson, E., Rasin, A., Daniel, J.A., David, J.D., Samuel, M., Michael, S.: A comparison of approaches to large-scale data analysis. In: Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data, SIGMOD 2009, pp. 165–178. ACM, New York (2009)Google Scholar
  29. 29.
    Psaroudakis, I., Scheuer, T., May, N., Sellami, A., Ailamaki, A.: Scaling up concurrent main-memory column-store scans: towards adaptive NUMA-aware data and task placement. In: The Proceedings of the VLDB Endowment, vol. 8, no. 12, pp. 1442–1453 (2015)CrossRefGoogle Scholar
  30. 30.
    Rabl, T., Jacobsen, H.-A., Mankovskii, S.: Big data challenges in application performance management. In: XLDB 2011, Middleware Systems Research Group.
  31. 31.
    Regehr, J., Stankovic, J.A.: Augmented CPU reservations: towards predictable execution on general-purpose operating systems. In: Proceedings Seventh IEEE Real-Time Technology and Applications Symposium, May 2001, pp. 141–148 (2001)Google Scholar
  32. 32.
    Ritchie, D.M., Thompson, K.: The unix time-sharing system. Commun. ACM 17(7), 365–375 (1974)CrossRefGoogle Scholar
  33. 33.
    Santosa, M.: Choosing the right timer interrupt frequency on Linux. Skripsi Progr. Stud. Sist. Inf. (2010)Google Scholar
  34. 34.
    Scogland, T., Balaji, P., Feng, W., Narayanaswamy, G.: Asymmetric interactions in symmetric multi-core systems: analysis, enhancements and evaluation. In: SC 2008: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, November 2008, pp. 1–12 (2008)Google Scholar
  35. 35.
    Shalf, J.M., Leland, R.: Computing beyond Moore’s law. Computer 48(12), 14–23 (2015)CrossRefGoogle Scholar
  36. 36.
    Siddha, S., Pallipadi, V.: AVD Ven. getting maximum mileage out of tickless. In: Proceedings of the Linux Symposium, vol. 2, pp. 201–207. Citeseer (2007)Google Scholar
  37. 37.
    Slagter, K., Hsu, C.-H., Chung, Y.-C.: An adaptive and memory efficient sampling mechanism for partitioning in mapreduce. Int. J. Parallel Prog. 43(3), 489–507 (2015)CrossRefGoogle Scholar
  38. 38.
    Stonebraker, M.: Operating system support for database management. Commun. ACM 24(7), 412–418 (1981)CrossRefGoogle Scholar
  39. 39.
    Andrew, S.T.: Distributed Operating Systems. Prentice-Hall Inc., Upper Saddle River (1995)Google Scholar
  40. 40.
    Theis, T.N., Wong, H.P.: The end of Moore’s law: a new beginning for information technology. Comput. Sci. Eng. 19(2), 41–50 (2017)CrossRefGoogle Scholar
  41. 41.
    Walter, U., Oberle, V.: \(\mu \)-second precision timer support for the Linux Kernel (2001)Google Scholar
  42. 42.
    Yuan, Q., Zhao, J., Chen, M., Sun, N.: Generos: an asymmetric operating system kernel for multi-core systems. In: 2010 IEEE International Symposium on Parallel Distributed Processing (IPDPS), April 2010, pp. 1–10 (2010)Google Scholar
  43. 43.
    Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M., Shenker, S., Stoica, I.: Fast and interactive analytics over hadoop data with spark. In: USENIX login, vol. 37, pp. 45–51 (2012)Google Scholar
  44. 44.
    Tamer Özsu, M., Valduriez, P.: Distributed data management: unsolved problems and new issues. In: Casavant, T.L. Singhal, M. (eds.) IEEE Computer Readings in Distributed Computing Systems, pp. 512–544. Society Press (1994)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.American University in Cairo AUCCairoEgypt

Personalised recommendations