Pattern Analysis and Applications

, Volume 19, Issue 3, pp 839–855 | Cite as

Approaching the accuracy–cost conflict in embedded classification system design

  • Ulf Jensen
  • Patrick Kugler
  • Matthias Ring
  • Bjoern M. Eskofier
Industrial and Commercial Application

Abstract

Smart embedded systems often run sophisticated pattern recognition algorithms and are found in many areas like automotive, sports and medicine. The developer of such a system is often confronted with the accuracy–cost conflict as the resulting system should be as accurate as possible while being able to run on resource constraint hardware. This article introduces a method to support the solution of this design conflict with accuracy–cost reports. These reports compare classification systems regarding their classification rate (accuracy) and the mathematical operations and parameters of the working phase (cost). Our method is used to deduce the specific cost of various popular pattern recognition algorithms and to derive the overall cost of a classification system. We also show how our analysis can be used to estimate the computational cost for specific hardware architectures. A software toolbox to create accuracy–cost reports was implemented to facilitate the automatic classification system comparison with the presented methodology. The software is available for download and as supplementary material. We performed different experiments on synthetic and real-world data to underline the value of this analysis. Accurate and computationally cheap classification systems were easily identified. We were even able to find a better implementation candidate in an existing embedded classification problem. This work is the first step towards a comprehensive support tool for the design of embedded classification systems.

Keywords

Machine learning Real-time systems Cost estimation Classification system design 

Supplementary material

10044_2015_503_MOESM1_ESM.pdf (641 kb)
Supplementary material 1 (pdf 641 KB)

References

  1. 1.
    Council NR (2001) Embedded, Everywhere. National Academies Press, WashingtonGoogle Scholar
  2. 2.
    Marwedel P (2006) Embedded System Design. Springer, New YorkMATHGoogle Scholar
  3. 3.
    Marwedel P (2010) Embedded and Cyber-Physical Systems in a Nutshell. In: Proc. of the Design Automation ConferenceGoogle Scholar
  4. 4.
    Navet N, Simonot-Lion F (2009) Automotive Embedded Systems Handbook. CRC Press, Boca RatonGoogle Scholar
  5. 5.
    Dounis AI, Caraiscos C (2009) Advanced Control Systems Engineering for Energy and Comfort Management in a Building Environment - A Review. Renew Sust Everg Rev 13(6):1246–1261CrossRefGoogle Scholar
  6. 6.
    Ferrarini L, Martinez Lastra JL, Martel A, Valentini A, Vyatkin V (2008) Embedded Systems Design in Intelligent Industrial Automation. EURASIP J Embed Syst 2008:194697Google Scholar
  7. 7.
    Arshak K, Lyons GM, Cunniffe C, Harris J, Clifford S (2003) A Review of Digital Data Acquisition Hardware and Software for a Portable Electronic Nose. Sens Rev 23(4):332–344CrossRefGoogle Scholar
  8. 8.
    Eskofier BM, Oleson M, DiBenedetto C, Hornegger J (2009) Embedded Surface Classification in Digital Sports. Pattern Recogn Lett 30(16):1448–1456CrossRefGoogle Scholar
  9. 9.
    Ghasemzadeh H, Ostadabbas S, Guenterberg E, Pantelopoulos A (2013) Wireless Medical-Embedded Systems: A Review of Signal-Processing Techniques for Classification. IEEE Sensors J 13(2):423–437CrossRefGoogle Scholar
  10. 10.
    Niemann H (1990) Pattern Analysis and Understanding. Springer, New YorkCrossRefMATHGoogle Scholar
  11. 11.
    Wolf W, Ozer B, Lv T (2002) Smart Cameras as Embedded Systems. Computer 35(9):48–53CrossRefGoogle Scholar
  12. 12.
    Stewart J, Wang CJ (2003) Waveform Recognition of Back-electro Motive Force for Real-time Embedded System Applications Using Support Vector Machines. In: Proc. of the Artificial Neural Networks in Engineering Conference. pp 951–956Google Scholar
  13. 13.
    Benbasat AY, Paradiso JA (2002) An Inertial Measurement Framework for Gesture Recognition and Applications. In: Revised Papers from the International Gesture Workshop on Gesture and Sign Languages in Human-Computer Interaction. pp 9–20Google Scholar
  14. 14.
    Englehart K, Hudgins B (2003) A Robust, Real-time Control Scheme for Multifunction Myoelectric Control. IEEE Trans Biomed Eng 50(7):848–854CrossRefGoogle Scholar
  15. 15.
    Zhang X, Liu Y, Zhang F, Ren J, Sun YL, Yang Q, Huang H (2012) On Design and Implementation of Neural-Machine Interface for Artificial Legs. IEEE Trans Ind Informat 8(2):418–429CrossRefGoogle Scholar
  16. 16.
    Benocci M, Farella E, Benini L (2011) A Context-aware Smart Seat. In: Proc. of the 4th IEEE International Workshop on Advances in Sensors and Interfaces (IWASI 2011). pp 104–109Google Scholar
  17. 17.
    Attenberger A, Buchenrieder K (2014) An Arduino-Simulink-Control System for Modern Hand Protheses. In: Proc. of the 13th International Conference on Artificial Intelligence and Soft Computing (ICAISC 2014). pp 433–444Google Scholar
  18. 18.
    Rahman AS, Yap MM, Shakaff AY, Ahmad MN, Dahari Z, Ismail Z, Hitam MS (2004) A Microcontroller-based Taste Sensing System for the Verification of Eurycoma Longifolia. Sensor Actuat B Chem 101(1–2):191–198CrossRefGoogle Scholar
  19. 19.
    Garcia-Breijo E, Atkinson J, Gil-Sanchez L, Masot R, Ibanez J, Garrigues J, Glanc M, Laguarda-Miro N, Olguin C (2011) A Comparison Study of Pattern Recognition Algorithms Implemented on a Microcontroller for Use in an Electronic Tongue for Monitoring Drinking Waters. Sensor Actuat A Phys 172(2):570–582CrossRefGoogle Scholar
  20. 20.
    Breijo EG, Pinatti CO, Peris RM, Fillol M, Martínez-Máñez R, Camino JS (2013) TNT Detection Using a Voltammetric Electronic Tongue Based on Neural Networks. Sensor Actuat A Phys 192:1–8CrossRefGoogle Scholar
  21. 21.
    Hacker C, Batliner A, Nöth E (2006) Are you looking at me, are you talking with me: Multimodal Classification of the Focus of Attention. In: Proc. of the 9th International Conference on Text, Speech and Dialogue (TSD 2006), vol. 4188. LNCS, pp 581–588Google Scholar
  22. 22.
    Lin CT, Chen YC, Huang TY, Chiu TT, Ko LW, Liang SF, Hsieh HY, Hsu SH, Duann JR (2008) Development of Wireless Brain Computer Interface With Embedded Multitask Scheduling and its Application on Real-Time Driverś Drowsiness Detection and Warning. IEEE Trans Biomed Eng 55(5):1582–1591CrossRefGoogle Scholar
  23. 23.
    Braojos R, Ansaloni G, Atienza D (2013) A Methodology for Embedded Classification of Heartbeats Using Random Projections. In: Proc. of the Design, Automation Test in Europe Conference Exhibition (DATE 2013). pp 899–904Google Scholar
  24. 24.
    Kao T, Lin C, Wang J (2009) Development of a Portable Activity Detector for Daily Activity Recognition. In: IEEE International Symposium on Industrial Electronics (ISIE 2009). pp 115–120Google Scholar
  25. 25.
    Boni A, Pianegiani F, Petri D (2007) Low-Power and Low-Cost Implementation of SVMs for Smart Sensors. IEEE Trans Instrum Meas 56(1):39–44CrossRefGoogle Scholar
  26. 26.
    Anguita D, Carlino L, Ghio A, Ridella S (2011) A FPGA Core Generator for Embedded Classification Systems. J Circuit Syst Comp 20(2):263–282CrossRefGoogle Scholar
  27. 27.
    Pedersen R, Schoeberl M (2006) An Embedded Support Vector Machine. In: Proc. of the International Workshop on Intelligent Solutions in Embedded Systems. pp 1–11Google Scholar
  28. 28.
    Bajaj N, Chiu GT, Allebach JP (2014) Reduction of Memory Footprint and Computation Time for Embedded Support Vector Machine (SVM) by Kernel Expansion and Consolidation. In: Proc. of the IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2014). pp 1–6Google Scholar
  29. 29.
    Charniya NN (2013) Design of near-optimal classifier using multi-layer perceptron neural networks for intelligent sensors. Int J Model Optim 3(1):56–60CrossRefGoogle Scholar
  30. 30.
    Ortega-Zamorano F, Subirats JL, Jerez JM, Molina I, Franco L (2013) Implementation of the C-Mantec Neural Network Constructive Algorithm in an Arduino UNO Microcontroller. In: Advances in Computational Intelligence. Springer, pp 80–87Google Scholar
  31. 31.
    Övünç P, Tülay Y (2010) FPGA implementation of a general regression neural network: an embedded pattern classification system. Digit Signal Process 20(3):881–886CrossRefGoogle Scholar
  32. 32.
    Zwartjes A, Havinga P, Smit G, Hurink J (2012) Using Unsupervised Learning to Improve the Naive Bayes Classifier for Wireless Sensor Networks. In: Proc. of the International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies (UBICOMM 2012). pp 71–76Google Scholar
  33. 33.
    Sohn S (1999) Meta analysis of classification algorithms for pattern recognition. IEEE Trans Pattern Anal Mach Intell 21(11):1137–1144CrossRefGoogle Scholar
  34. 34.
    Reif M, Shafait F, Goldstein M, Breuel T, Dengel A (2014) Automatic classifier selection for non-experts. Pattern Anal Appl 17(1):83–96MathSciNetCrossRefGoogle Scholar
  35. 35.
    Aha DW, Kibler D, Albert MK (1991) Instance-based learning algorithms. Mach Learn 6(1):37–66Google Scholar
  36. 36.
    Jain A, Nikovski D (2008) Incremental exemplar learning schemes for classification on embedded devices. Mach Learn 72(3):189–203MathSciNetCrossRefGoogle Scholar
  37. 37.
    García-Durán R, Fernández F, Borrajo D (2012) A prototype-based method for classification with time constraints: a case study on automated planning. Pattern Anal Appl 15(3):261–277MathSciNetCrossRefGoogle Scholar
  38. 38.
    Buciluǎ C, Caruana R, Niculescu-Mizil A (2006) Model Compression. In: Proc. of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Philadelphia, pp 535–541Google Scholar
  39. 39.
    Tan M (1993) Cost-sensitive learning of classification knowledge and its applications in robotics. Mach Learn 13(1):7–33Google Scholar
  40. 40.
    Iswandy K, Koenig A (2006) Feature selection with acquisition cost for optimizing sensor system design. Adv Radio Sci 4(7):135–141CrossRefGoogle Scholar
  41. 41.
    Weiss Y, Elovici Y, Rokach LL (2013) The CASH algorithm-cost-sensitive attribute selection using histograms. Inform Sci 222:247–268MathSciNetCrossRefGoogle Scholar
  42. 42.
    Ji S, Carin L (2007) Cost-sensitive feature acquisition and classification. Pattern Recogn 40(5):1474–1485CrossRefMATHGoogle Scholar
  43. 43.
    Cebe M, Gunduz-Demir C (2010) Qualitative test-cost sensitive classification. Pattern Recogn Lett 31(13):2043–2051CrossRefGoogle Scholar
  44. 44.
    Paclík P, Duin RP, van Kempen GM, Kohlus R (2002) On Feature Selection with Measurement Cost and Grouped Features. In: Structural, Syntactic, and Statistical Pattern Recognition. Springer, pp 461–469Google Scholar
  45. 45.
    Ahmadi A, Dehzangi O, Jafari R (2012) Brain-Computer Interface Signal Processing Algorithms: A Computational Cost vs. Accuracy Analysis for Wearable Computers. In: Proc. of the Ninth International Conference on Wearable and Implantable Body Sensor Networks. pp 40–45Google Scholar
  46. 46.
    Christ P, Sievers G, Einhaus J, Jungeblut T, Porrmann M, Ruckert U (2013) Pareto-optimal Signal Processing on Low-power Microprocessors. In: Proc. of IEEE Sensors Conference. pp 1–4Google Scholar
  47. 47.
    Witten IH, Frank E, Hall MA (2011) Data Mining: Practical Machine Learning Tools and Techniques, 3rd edn. Morgan Kaufmann, BurlingtonGoogle Scholar
  48. 48.
    Cormen TH, Leiserson CE, Rivest RL, Stein C (2001) Introduction to Algorithms. MIT press, CambridgeMATHGoogle Scholar
  49. 49.
    Puschner P, Burns A (2000) A review of worst-case execution-time analysis. Int J Time Critical Comput Syst 18(2):115–128Google Scholar
  50. 50.
    Wilhelm R, Engblom J, Ermedahl A, Holsti N, Thesing S, Whalley D, Bernat G, Ferdinand C, Heckmann R, Mitra T, Mueller F, Puaut I, Puschner P, Staschulat J, Stenstr-Üm P (2008) The worst-case execution-time problem—overview of methods and survey of tools. ACM T Embed Comput S 7(3):36Google Scholar
  51. 51.
    Duda RO, Hart PE, Stork DG (2000) Pattern Classification, vol 2. Wiley-Interscience, HobokenMATHGoogle Scholar
  52. 52.
    Kaporin I (1999) A practical algorithm for faster matrix multiplication. Numer Linear Algebr 6(8):687–700MathSciNetCrossRefMATHGoogle Scholar
  53. 53.
    Vetterli M, Nussbaumer HJ (1984) Simple FFT and DCT algorithms with reduced number of operations. Signal Process 6(4):267–278MathSciNetCrossRefMATHGoogle Scholar
  54. 54.
    Chau L, Siu W (1995) Direct formulation for the realization of discrete cosine transform using recursive structure. IEEE Trans Circuits Syst II Analog Digit Signal Process 42(1):50–52CrossRefGoogle Scholar
  55. 55.
    Gabutti B (1985) An algorithm for computing generalized Euler’s transformations of series. Computing 34(2):107–116MathSciNetCrossRefMATHGoogle Scholar
  56. 56.
    Hwang S, Kim W (2006) Fast and efficient method for computing ART. IEEE Trans Image Process 15(1):112–117CrossRefGoogle Scholar
  57. 57.
    Knuth D (1998) The art of computer programming. Addison-Wesley, BostonMATHGoogle Scholar
  58. 58.
    Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten I (2009) The WEKA data mining software: An update. SIGKDD Explor 11(1):10–18CrossRefGoogle Scholar
  59. 59.
    Chang C, Lin C (2011) LIBSVM: A library for support vector machines. ACM TIST 2(3):1–27CrossRefGoogle Scholar
  60. 60.
    Polikar R (2007) Bootstrap-inspired techniques in computational intelligence: ensemble of classifiers, incremental learning, data fusion and missing features. IEEE Signal Process Mag 24(4):59–72CrossRefGoogle Scholar
  61. 61.
    Quinlan R (1993) C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, BurlingtonGoogle Scholar
  62. 62.
    Frank E, Wang Y, Inglis S, Holmes G, Witten I (1998) Using model trees for classification. Mach Learn 32(1):63–76CrossRefMATHGoogle Scholar
  63. 63.
    Theodoridis S, Koutroumbas K (2008) Pattern Recognition. Academic Press, WalthamMATHGoogle Scholar
  64. 64.
    Frank E, Witten I (1998) Generating Accurate Rule Sets Without Global Optimization. In: Proc. of the Fifteenth International Conference on Machine Learning. pp 144–151Google Scholar
  65. 65.
    Vapnik VN (1995) The nature of statistical learning theory. Springer, New YorkCrossRefMATHGoogle Scholar
  66. 66.
    Schölkopf B, Smola AJ (2001) Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, CambridgeGoogle Scholar
  67. 67.
    Kressel U (1998) Pairwise classification and support vector machines. In: Schölkopf B, Burges C, Smola A (eds) Advances in Kernel Methods. MIT Press, pp 255–268Google Scholar
  68. 68.
    Microchip Technology Inc (1997) AN660 floating point math functions. http://ww1.microchip.com/downloads/en/AppNotes/00660.pdf. Accessed 09 July 2015
  69. 69.
    Texas Instruments Inc (1999) The MSP430 hardware multiplier. http://www.ti.com/lit/an/slaa042/slaa042.pdf. Accessed 09 July 2015
  70. 70.
    ARM Ltd (2010) Cortex-M3 technical reference manual. http://infocenter.arm.com/help/topic/com.arm.doc.ddi0337e/DDI0337E_cortex_m3_r1p1_trm.pdf. Accessed 09 July 2015
  71. 71.
    Texas Instruments Inc (2000) MSP430 family mixed-signal microcontroller application reports. http://www.ti.com/lit/an/slaa024/slaa024.pdf. Accessed 09 July 2015
  72. 72.
    Iordache C, Tang PT (2003) An Overview of Floating-Point Support and Math Library on the Intel XScale Architecture. In: Proc. of the 16th IEEE Symposium on Computer Arithmetic (ARITH-16 2003). pp 122–128Google Scholar
  73. 73.
    Texas Instruments Inc (2006) Efficient multiplication and division using MSP430. http://www.ti.com.cn/cn/lit/an/slaa329/slaa329.pdf. Accessed 09 July 2015
  74. 74.
    Cypress Semiconductor Corp (2008) ImageCraft assembly language guide. http://www.cypress.com/?docID=29030. Accessed 09 July 2015
  75. 75.
    Cypress Semiconductor Corp (2014) AN2032 PSoC 1 fast unsigned multiplication algorithms. http://www.cypress.com/?docID=50218. Accessed 09 July 2015
  76. 76.
    Micro Digital Inc (2014) GoFast for Thumb-2 and IAR EWARM. http://www.smxrtos.com/ussw/gofast/gofast_arm_iar.html. Accessed 01 Sept 2014
  77. 77.
    Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng C, Stanley HE (2000) PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 101(23):e215–e220CrossRefGoogle Scholar
  78. 78.
    Hausdorff JM, Ladin Z, Wei JY (1995) Footswitch system for measurement of the temporal parameters of gait. J Biomech 28(3):347–351CrossRefGoogle Scholar
  79. 79.
    Hausdorff JM, Mitchell SL, Firtion R, Peng CK, Cudkowicz ME, Wei JY, Goldberger AL (1997) Altered fractal dynamics of gait: reduced stride-interval correlations with aging and Huntington’s Disease. J Appl Physiol 82:262–269Google Scholar
  80. 80.
    Bache K, Lichman M (2013) UCI machine learning repository. http://archive.ics.uci.edu/ml. Accessed 3 July 2013
  81. 81.
    Mangasarian OL, Street WN, Wolberg WH (1995) Breast cancer diagnosis and prognosis via linear programming. Oper Res 43(4):570–577MathSciNetCrossRefMATHGoogle Scholar
  82. 82.
    Chen X (2003) An improved branch and bound algorithm for feature selection. Pattern Recogn Lett 24(12):1925–1933CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London 2015

Authors and Affiliations

  1. 1.Digital Sports Group, Pattern Recognition LabFriedrich-Alexander-Universität Erlangen-Nürnberg (FAU)ErlangenGermany

Personalised recommendations