Bridging Density Functional Theory and Big Data Analytics with Applications

  • Chien-Chang Chen
  • Hung-Hui Juan
  • Meng-Yuan Tsai
  • Henry Horng-Shing LuEmail author
Part of the Springer Handbooks of Computational Statistics book series (SHCS)


The framework of the density functional theory (DFT) reveals both strong suitability and compatibility for investigating large-scale systems in the Big Data regime. By technically mapping the data space into physically meaningful bases, the chapter provides a simple procedure to formulate global Lagrangian and Hamiltonian density functionals to circumvent the emerging challenges on large-scale data analyses. Then, the informative features of mixed datasets and the corresponding clustering morphologies can be visually elucidated by means of the evaluations of global density functionals. Simulation results of data clustering illustrated that the proposed methodology provides an alternative route for analyzing the data characteristics with abundant physical insights. For the comprehensive demonstration in a high dimensional problem without prior ground truth, the developed density functionals were also applied on the post-process of magnetic resonance imaging (MRI) and better tumor recognitions can be achieved on the T1 post-contrast and T2 modes. It is appealing that the post-processing MRI using the proposed DFT-based algorithm would benefit the scientists in the judgment of clinical pathology. Eventually, successful high dimensional data analyses revealed that the proposed DFT-based algorithm has the potential to be used as a framework for investigations of large-scale complex systems and applications of high dimensional biomedical image processing.


Density functional theory (DFT) Big data High dimensional data analysis Image analysis 



We would like to acknowledge the supports from National Science Council, National Center for Theoretical Sciences, Shing-Tung Yau Center, Center of Mathematical Modeling and Scientific Computing at National Chiao Tung University in Taiwan.


  1. Baer R, Livshits E, Salzner U (2010) Tuned range-separated hybrids in density functional theory. Annu Rev Phys Chem 61:85–109CrossRefGoogle Scholar
  2. Bas E, Erdogmus D (2010) Piecewise linear cylinder models for 3-dimensional axon segmentation in Brainbow imagery. In: International symposium on biomedical imaging (ISBI), pp 1297–1300Google Scholar
  3. Bas E, Erdogmus D, Draft RW, Lichtman JW (2012) Local tracing of curvilinear structures in volumetric color images: application to the Brainbow analysis. J Vis Commun Image R 23:1260–1271CrossRefGoogle Scholar
  4. Bullmore ET, Bassett DS (2011) Brain graphs: graphical models of the human brain connectome. Annu Rev Clin Psychol 7:113–140CrossRefGoogle Scholar
  5. Capelle K (2006) A bird’s-eye view of density-functional theory. Braz J Phys 36:1318–1343CrossRefGoogle Scholar
  6. Casagrande D, Sassano M, Astolfi A (2012) Hamiltonian-based clustering: algorithms for static and dynamic clustering in data mining and image processing. IEEE Control Syst 32:74–91Google Scholar
  7. Chen H, Chiang RHL, Storey VC (2012) Business intelligence and analytics: from big data to big impact. MIS Q 36:1165–1188Google Scholar
  8. Chothani P, Mehta V, Stepanyants A (2011) Automated tracing of neurites from light microscopy stacks of images. Neuroinformatics 9:263–278CrossRefGoogle Scholar
  9. Clark MC, Hall LO, Goldgof DB, Velthuizen R, Murtagh FR, Silbiger MS (1998) Automatic tumor segmentation using knowledge-based techniques. IEEE Trans Med Imag 17:187–201CrossRefGoogle Scholar
  10. Clauset A, Newman MEJ, Moore C (2004) Finding community structure in very large networks. Phys Rev E 70:066111Google Scholar
  11. Cramer CJ, Truhlar DG (2009) Density functional theory for transition metals and transition metal chemistry. Phys Chem Chem Phys 11:10757–10816CrossRefGoogle Scholar
  12. Daw MS, Baskes MI (1983) Semiempirical, quantum mechanical calculation of hydrogen embrittlement in metals. Phys Rev Lett 50:1285–1288CrossRefGoogle Scholar
  13. Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc B 39:1–38Google Scholar
  14. Esquivel AV, Rosvall M (2011) Compression of flow can reveal overlapping-module organization in networks. Phys Rev X 1:021025Google Scholar
  15. Finken R, Schmidt M, Löwen H (2001) Freezing transition of hard hyperspheres. Phys Rev E 65:016108Google Scholar
  16. Foulkes WMC, Mitas L, Needs RJ, Rajagopal G (2001) Quantum Monte Carlo simulations of solids. Rev Mod Phys 73:33–83CrossRefGoogle Scholar
  17. Gala R, Chapeton J, Jitesh J, Bhavsar C, Stepanyants A (2014) Active learning of neuron morphology for accurate automated tracing of neurites. Front Neuroanat 8:1–14Google Scholar
  18. Girvan M, Newman MEJ (2002) Community structure in social and biological networks. Proc Natl Acad Sci USA 99:7821–7826MathSciNetCrossRefGoogle Scholar
  19. Gordillo N, Montseny E, Sobrevilla, P (2013) State of the art survey on MRI brain tumor segmentation. Magn Reson Imaging 31:1426–1438CrossRefGoogle Scholar
  20. Grimme S, Antony J, Schwabe T, Mück-Lichtenfeld C (2007) Density functional theory with dispersion corrections for supramolecular structures, aggregates, and complexes of (bio)organic molecules. Org Biomol Chem 5:741–758CrossRefGoogle Scholar
  21. Hampel S, Chung P, McKellar CE, Hall D, Looger LL, Simpson JH (2011) Drosophila Brainbow: a recombinase-based fluorescence labeling technique to subdivide neural expression patterns. Nat Methods 8:253–259CrossRefGoogle Scholar
  22. Hohenberg P, Kohn W (1964) Inhomogeneous electron gas. Phys Rev 136:B864–B871MathSciNetCrossRefGoogle Scholar
  23. Horn D, Gottlieb A (2001) Algorithm for data clustering in pattern recognition problems based on quantum mechanics. Phys Rev Lett 88:018702Google Scholar
  24. Hsu Y, Lu HH-S (2013) Brainbow image segmentation using Bayesian sequential partitioning. Int J Comput Electr Autom Control Inf Eng 7:897–902Google Scholar
  25. Jacobs A (2009) The pathologies of big data. Commun ACM 52:36–44CrossRefGoogle Scholar
  26. Kalal Z, Mikolajczyk K, Matas J (2012) Tracking-learning-detection. IEEE Trans Pattern Anal Mach Intell 34:1409–1422CrossRefGoogle Scholar
  27. Kobiler O, Lipman Y, Therkelsen K, Daubechies I, Enquist LW (2010) Herpesviruses carrying a Brainbow cassette reveal replication and expression of limited numbers of incoming genomes. Nat Commun 1:1–8CrossRefGoogle Scholar
  28. Kohn W (1999) Nobel lecture: electronic structure of matter-wave functions and density functionals. Rev Mod Phys 71:1253–1266CrossRefGoogle Scholar
  29. Kohn W, Sham LJ (1965) Self-consistent equations including exchange and correlation effects. Phys Rev 140:A1133–A1138MathSciNetCrossRefGoogle Scholar
  30. Kreshuk A, Straehle CN, Sommer C, Korthe U, Knott G, Hamprecht FA (2011) Automated segmentation of synapses in 3D EM data. In: International symposium on biomedical imaging (ISBI), pp 220–223Google Scholar
  31. Langreth DC, Perdew JP (1997) Exchange-correlation energy of a metallic surface: wave-vector analysis. Phys Rev B 15:2884–2901CrossRefGoogle Scholar
  32. Lebègue S, Björkman T, Klintenberg M, Nieminen RM, Eriksson O (2013) Two-dimensional materials from data filtering and Ab initio calculations. Phys Rev X 3:031002Google Scholar
  33. Lebowitz JL, Lieb EH (1969) Existence of thermodynamics for real matter with Coulomb forces. Phys Rev Lett 22:631–634CrossRefGoogle Scholar
  34. Lichtman JW, Livet J, Sanes JR (2008) A technicolour approach to the connectome. Nat Rev Neurosci 9:417–422CrossRefGoogle Scholar
  35. Livet J, Weissman TA, Kang H, Draft RW, Lu J, Bennis RA, Sanes JR, Kichtman JW (2007) Transgenic strategies for combinatorial expression of fluorescent proteins in the nervous system. Nature 450:56–62CrossRefGoogle Scholar
  36. Levin A, Rav-Acha A, Lischinski D (2008) Spectral matting. IEEE Trans Pattern Anal Mach Intell 30:1699–1712CrossRefGoogle Scholar
  37. Levy M (1982) Electron densities in search of Hamiltonians. Phys Rev A 26:1200–1208CrossRefGoogle Scholar
  38. Levy M (2010) On the simple constrained-search reformulation of the Hohenberg-Kohn theorem to include degeneracies and more (1964–1979). Int J Quant Chem 110:3140–3144CrossRefGoogle Scholar
  39. Levy M, Perdew JP (1985) Hellmann-Feynman, virial, and scaling requisites for the exact universal density functionals. Shape of the correlation potential and diamagnetic susceptibility for atoms. Phys Rev A 32:2010–2021CrossRefGoogle Scholar
  40. Lu L, Jiang H, Wong WH (2013) Multivariate density estimation by Bayesian Sequential Partitioning. J Am Stat Assoc Theory Methods 108:1402–1410MathSciNetCrossRefGoogle Scholar
  41. March NH (1986) Spatially dependent generalization of Kato’s theorem for atomic closed shells in a bare Coulomb field. Phys Rev A 33:88–89MathSciNetCrossRefGoogle Scholar
  42. McAfee A, Brynjolfsson E (2012) Big data: the management revolution. Harv Bus Rev 90:59–68Google Scholar
  43. Moon N, Bullitt E, van Leemput K, Gerig G (2002) Automatic brain and tumor segmentation. In: MICCAI proceedings. Lecture notes in computer science, vol 2488, pp 372–379CrossRefGoogle Scholar
  44. Needs RJ, Towler MD, Drummond ND, Ríos PL (2010) Continuum variational and diffusion quantum Monte Carlo calculations. J Phys Condens Matter 22:023201Google Scholar
  45. Neese F (2009) Prediction of molecular properties and molecular spectroscopy with density functional theory: from fundamental theory to exchange-coupling. Coord Chem Rev 253:526–563CrossRefGoogle Scholar
  46. Newman MEJ, Girvan M (2004) Finding and evaluating community structure in networks. Phys Rev E 69:026113Google Scholar
  47. Peng H, Long F, Myers G (2011) Automatic 3D neuron tracing using all-path pruning. Bioinformatics 27:i239–i247CrossRefGoogle Scholar
  48. Prastawa M, Bullitt E, Ho S, Gerig G (2004) A brain tumor segmentation framework based on outlier detection. Med Image Anal 8:275–283CrossRefGoogle Scholar
  49. Riley KE, Pitoňák M, Jurečka P, Hobza P (2010) Stabilization and structure calculations for noncovalent interactions in extended molecular systems based on wave function and density functional theories. Chem Rev 110:5023–5063CrossRefGoogle Scholar
  50. Rodriguez A, Ehlenberger DB, Hof PR, Wearne SL (2009) Three-dimensional neuron tracing by voxel scooping. J Neurosci Methods 184:169–175CrossRefGoogle Scholar
  51. Rozas J, Sánchez-DelBarrio JC, Messeguer X, Rozas R (2003) DnaSP, DNA polymorphism analyses by the coalescent and other methods. Bioinformatics 19:2496–2497CrossRefGoogle Scholar
  52. Salasnich L, Toigo F (2008) Extended Thomas-Fermi density functional for the unitary Fermi gas. Phys Rev A 78:053626Google Scholar
  53. Schadt EE, Linderman MD, Sorenson J, Lee L, Nolan GP (2011) Cloud and heterogeneous computing solutions exist today for the emerging big data problems in biology. Nat Rev Genet 12:224–224CrossRefGoogle Scholar
  54. Shao H-C, Cheng W-Y, Chen Y-C, Hwang W-L (2012) Colored multi-neuron image processing for segmenting and tracing neural circuits. In: International conference on image processing (ICIP), pp 2025–2028Google Scholar
  55. Sporns O (2011) The human connectome: a complex network. Ann NY Acad Sci 1224:109–125CrossRefGoogle Scholar
  56. Tóth B, Lempérière Y, Deremble C, de Lataillade J, Kockelkoren J, Bouchaud, J-P (2011) Anomalous price impact and the critical nature of liquidity in financial markets. Phys Rev X 1:021006Google Scholar
  57. Türetken E, González G, Blum C, Fua P (2011) Automated reconstruction of dendritic and axonal trees by global optimization with geometric priors. Neuroinformatics 9:279–302CrossRefGoogle Scholar
  58. Türetken E, Benmansour F, Andres B, Pfister H, Fua P (2013) Reconstructing loopy curvilinear structures using integer programming. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1822–1829Google Scholar
  59. UCI Machine Learning Repository Iris Dataset (1988) Available at
  60. Ullrich CA (2012) Time-dependent density-functional theory. Oxford University Press, New York, pp 21–41Google Scholar
  61. Vasilkoski Z, Stepanyants A (2009) Detection of the optimal neuron traces in confocal microscopy images. J Neurosci Methods 178:197–204CrossRefGoogle Scholar
  62. Voorhis TV, Scuseria GE (1998) A novel form for the exchange-correlation energy functional. J Chem Phys 109:400–410Google Scholar
  63. Wang C, Deng F-G, Li Y-S, Liu X-S, Long GL (2005) Quantum secure direct communication with high-dimension quantum superdense coding. Phys Rev A 71:044305Google Scholar
  64. Wang Y, Narayanaswamy A, Tsai C-L, Roysam B (2011) A broadly applicable 3-D neuron tracing method based on open-curve snake. Neuroinformatics 9:193–217CrossRefGoogle Scholar
  65. Wu J (2006) Density functional theory for chemical engineering: from capillarity to soft materials. AIChE J 52:1169–1193CrossRefGoogle Scholar
  66. Wu T-Y, Juan H-H, Lu HH-S, Chiang A-S (2011) A crosstalk tolerated neural segmentation methodology for Brainbow images. In: Proceedings of the 4th international symposium on applied sciences in biomedical and communication technologies (ISABEL)Google Scholar
  67. Wu T-Y, Juan H-H, Lu, HH-S (2012) Improved spectral matting by iterative K-means clustering and the modularity measure. In: IEEE international conference on acoustics, speech, and signal processing (IEEE ICASSP), pp 1165–1168Google Scholar
  68. Zhang J, Ma K-K, Er M-H, Chong V (2004) Tumor segmentation from magnetic resonance imaging by learning via one-class support vector machine. In: International Workshop on Advanced Image Technology, IWAIT, pp 207–211Google Scholar
  69. Zhang Y, Chen K, Baron M, Teylan MA, Kim Y, Song Z, Greengard P, Wong STC (2010) A neurocomputational method for fully automated 3D dendritic spine detection and segmentation of medium-sized spiny neurons. Neuroimage 50:1472–1484CrossRefGoogle Scholar
  70. Zupan A, Burke K, Ernzerhof M, Perdew JP (1997) Distributions and averages of electron density parameters: explaining the effects of gradient corrections. J Chem Phys 106:10184–10193CrossRefGoogle Scholar
  71. Zupan A, Perdew JP, Burke K, Causá M (1997) Density-gradient analysis for density functional theory: application to atoms. Int J Quant Chem 61:835–845CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Chien-Chang Chen
    • 1
    • 2
  • Hung-Hui Juan
    • 2
  • Meng-Yuan Tsai
    • 3
  • Henry Horng-Shing Lu
    • 3
    • 4
    • 2
    Email author
  1. 1.Bio-MicroSystems Integration Laboratory, Department of Biomedical Sciences and EngineeringNational Central UniversityTaoyuan CityTaiwan
  2. 2.Shing-Tung Yau CenterNational Chiao Tung UniversityHsinchu CityTaiwan
  3. 3.Institute of StatisticsNational Chiao Tung UniversityHsinchu CityTaiwan
  4. 4.Big Data Research CenterNational Chiao Tung UniversityHsinchu CityTaiwan

Personalised recommendations