Journal of Real-Time Image Processing

, Volume 11, Issue 2, pp 349–374 | Cite as

Accelerating embedded image processing for real time: a case study

  • Sol Pedre
  • Tomáš Krajník
  • Elías Todorovich
  • Patricia Borensztejn
Special Issue Paper


Many image processing applications need real-time performance, while having restrictions of size, weight and power consumption. Common solutions, including hardware/software co-designs, are based on Field Programmable Gate Arrays (FPGAs). Their main drawback is long development time. In this work, a co-design methodology for processor-centric embedded systems with hardware acceleration using FPGAs is proposed. The goal of this methodology is to achieve real-time embedded solutions, using hardware acceleration, but achieving development time similar to that of software projects. Well established methodologies, techniques and languages from the software domain—such as Object-Oriented Paradigm design, Unified Modelling Language, and multithreading programming—are applied; and semiautomatic C-to-HDL translation tools and methods are used and compared. The methodology is applied to achieve an embedded implementation of a global vision algorithm for the localization of multiple robots in an e-learning robotic laboratory. The algorithm is specifically developed to work reliably 24/7 and to detect the robot’s positions and headings even in the presence of partial occlusions and varying lighting conditions expectable in a normal classroom. The co-designed implementation of this algorithm processes 1,600 × 1,200 pixel images at a rate of 32 fps with an estimated energy consumption of 17 mJ per frame. It achieves a 16× acceleration and 92 % energy saving, which compares favorably with the most optimized embedded software solutions. This case study shows the usefulness of the proposed methodology for embedded real-time image processing applications.


Real-time image processing Methodology for hardware/software co-design in FPGA  High level synthesis High level modeling Multiple robot localization Multithreaded programming Hardware acceleration 



Xilinx Design Suite was donated by Xilinx University Program. This work has been partially supported by Czech project No. 7AMB12AR022, by EU within ICT - 216240 and Argentinian projects MINCyT RC/11/20, UBACyT 200158 and PICT-2009-0041.


  1. 1.
    Altera.: Altera Introduces SoC FPGAs: Integrating ARM processor system and FPGA into 28-nm single-chip solution. (2011)
  2. 2.
    Arpinen, T., Salminen, E., Hmlinen, TD., Hnnikinen, M.: MARTE profile extension for modeling dynamic power management of embedded systems. J. Syst. Archit. 58(5), 209–219 (2012)CrossRefGoogle Scholar
  3. 3.
    Bailey, B., Martin G.: ESL Models and their Application. Electronic System Level Design and Verification in Practice. Springer, Berlin (2010)Google Scholar
  4. 4.
    Blanza, D., Holland, C.: Embedded market survey. EETimes and Embedded (2012)
  5. 5.
    Bouguet, JY.: Camera calibration toolbox for Matlab. (2008)
  6. 6.
    Brezak, M., Petrović, I., Ivanjko, E.: Robust and accurate global vision system for real time tracking of multiple mobile robots. Robot. Auton. Syst. 56(3), 213–230 (2008)CrossRefGoogle Scholar
  7. 7.
    Bruce, J., Veloso, M.: Fast and accurate vision-based pattern detection and identification. In: IEEE International Conference on Robotics and Automation, ICRA, IEEE, pp. 1277–1282 (2003)Google Scholar
  8. 8.
    ESA European Space Agency VHDL. (2011)
  9. 9.
    Gaisler, J.: A structured VHDL design method. In: Fault-Tolerant Microprocessors for Space Applications, Gaisler Research, pp. 41–50, (2004)
  10. 10.
    Gaisler, J.: A structured VHDL design method. (2011)
  11. 11.
    Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley Longman, Boston (1995)Google Scholar
  12. 12.
    Gunay, N., Dadios, E.: A robust and accurate color-based global vision recognition of highly dynamic objects in real time. In: 8th Asian Control Conference (ASCC), IEEE, pp. 90 –95 (2011)Google Scholar
  13. 13.
    Gupta, S., Dutt, N., Gupta, R., Nicolau, A.: SPARK: A high-level synthesis framework for applying parallelizing compiler transformations. In: 16th Intl. Conf. on VLSI Design, IEEE, pp. 461–467 (2003)Google Scholar
  14. 14.
    Happe, M., Lubbers, E., Platzner, M.: A self-adaptive heterogeneous multi-core architecture for embedded real-time video object tracking. J. Real-Time Image Process pp. 1–16 (2011)Google Scholar
  15. 15.
    Jacquard ROCCC 2.0. (2011)
  16. 16.
    Keskin, O., Uyar, E.: A framework for multi robot guidance control. In: Holonic and Multi-Agent Systems for Manufacturing, Lecture Notes in Computer Science, vol. 5696, Springer, Berlin, pp. 315–323 (2009)Google Scholar
  17. 17.
    Klančar, G., Kristan, M., Kovačič, S., Orqueda, O.: Robust and efficient vision system for group of cooperating mobile robots with application to soccer robots. ISA Trans. 43(3), 329–342 (2004)Google Scholar
  18. 18.
    Klančar, G., Matko, D., Blažič, S.: Wheeled mobile robots control in a linear platoon. J. Intell. Robot. Syst. 54, 709–731 (2009)Google Scholar
  19. 19.
    Kulich, M., Chudoba, J., Kosnar, K., Krajnik, T., Faigl, J., Preucil, L.: SyRoTek—distance teaching of mobile robotics. IEEE Trans. Educ. 56(1), 18–23 (2013)CrossRefGoogle Scholar
  20. 20.
  21. 21.
    Michael, N., Mellinger, D., Lindsey, Q., Kumar, V.: The GRASP multiple micro UAV testbed . IEEE Robotics and Automation Magazine (2010)Google Scholar
  22. 22.
    Mischkalla, F., He, D., Mueller, W.: Closing the gap between UML-based modeling, simulation and synthesis of combined HW/SW systems. In: Design, Automation Test in Europe Conference Exhibition (DATE), 2010, pp. 1201 –1206 (2010)Google Scholar
  23. 23.
    Mueller, W., Rosti, A., Bocchio, S., Riccobene, E., Scandurra, P., Dehaene, W., Vanderperren, Y., Ku, L.: UML for ESL design—Basic principles, tools, and applications. In: IEEE/ACM International Conference on Computer Aided Design, pp. 73–80 (2006)Google Scholar
  24. 24.
  25. 25.
    NVIDIA: CUDA: Parallel Programming. (2012)
  26. 26.
    OMG UML Profile for System on a Chip (SoC) Version 1.0.1 - formal/06-08-01. (2006)
  27. 27.
    OMG.: Systems modeling language (SysML) Version 1.2 - formal/2010-06-01. (2010)
  28. 28.
    OMG UML Profile for MARTE: Modeling and analysis of real-time embedded systems Version 1.1 - formal/2011-06-02. (2011)
  29. 29.
    Paul, J., Laika, A,. Claus, C., Stechele, W., El Sayed Auf, A., Maehle, E.: Real-time motion detection based on SW/HW-codesign for walking rescue robots. J. Real-Time Image Process. 1–16 (2012)Google Scholar
  30. 30.
    Pedre, S., Krajník, T., Todorovich, E., Borensztejn, P.: Hardware/software co-design for real time embedded image processing: A case study. In: Alvarez, L., Mejai,l M., Gomez, L., Jacobo J. (eds): Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, Lecture Notes in Computer Science, vol 7441, Springer Berlin Heidelberg, pp. 599–606 (2012)Google Scholar
  31. 31.
    Pedre, S., Krajník, T., Todorovich, E., Borensztejn, P.: A co-design methodology for processor-centric embedded systems with hardware acceleration using FPGA. In: IEEE 8th Southern Programmable Logic Conference, IEEE, Brazil, pp. 7–14 (2012)Google Scholar
  32. 32.
    Quadri, IR., Gamati, A., Boulet, P., Meftali, S., Dekeyser, JL.: Expressing embedded systems configurations at high abstraction levels with UML MARTE profile: Advantages, limitations and alternatives. J. Syst. Archit. 58(5), 178–194 (2012)CrossRefGoogle Scholar
  33. 33.
    Rao, R., Taylor, C., Kumar, V.: Experiments in robot control from uncalibrated overhead imagery. In: Ang, J., Marcelo, H., Khatib, O. (eds): Experimental Robotics IX, Springer Tracts in Advanced Robotics, vol 21, Springer Berlin Heidelberg, pp. 491–500 (2006)Google Scholar
  34. 34.
    Riccobene, E., Scandurra, P.: Weaving executability into UML class models at PIM level. In: First European Workshop on Behaviour Modelling in Model Driven Architecture (BM-MDA), CTIT Workshop Proceedings Series, Enschede, The Netherlands, pp. 10–28 (2009)Google Scholar
  35. 35.
    Riccobene, E., Scandurra, P., Rosti, A., Bocchio, S.: A SoC design methodology involving a UML 2.0 profile for SystemC. In: Proceedings of the Conference on Design, Automation and Test in Europe, vol. 2, IEEE Computer Society, Washington, DC, USA, DATE 2005, pp. 704–709 (2005)Google Scholar
  36. 36.
    Riccobene, E., Scandurra, P., Rosti, A., Bocchio, S.: A model-driven design environment for embedded systems. In: Proceedings of the 43rd Annual Design Automation Conference, ACM, New York, NY, USA, DAC ’06, pp. 915–918 (2006)Google Scholar
  37. 37.
    Rodriguez-Gomez, R., Fernandez-Sanchez, E., Diaz, J., Ros, E.: Codebook hardware implementation on FPGA for background subtraction. J. Real-Time Image Process. pp. 1–15 (2012)Google Scholar
  38. 38.
    Santarini, M.: Zynq-7000 EPP sets stage for new era of innovations. Xcell J 75, 8–13 (2011)Google Scholar
  39. 39.
    Silva-Filho, A.G., et al.: An ESL approach for energy consumption analysis of cache memories in SoC platforms. Int. J. Reconfigurable Comput. 2011, 1–12 (2011)Google Scholar
  40. 40.
    Sparx Systems (2011) Visual modelling platform.
  41. 41.
    Steggles, P., Gschwind, S.: The UBISENSE smart space platform. In: Third International Conference on Pervasive Computing (2005)Google Scholar
  42. 42.
    Technology, BD.: The AutoESL Auto Pilot High-Level Synthesis. Tool. Tech. Rep. (2010)Google Scholar
  43. 43.
    Thrun, S., Burgard, W., Fox, D.: Probabilistic Robotics (Intelligent Robotics and Autonomous Agents). The MIT Press, Cambridge (2005)Google Scholar
  44. 44.
    Vidal, J., de Lamotte, F., Gogniat, G., Soulard, P., Diguet, JP.: A co-design approach for embedded system modeling and code generation with UML and MARTE. In: Proceedings of the Conference on Design, Automation and Test in Europe, European Design and Automation Association, 3001 Leuven, Belgium, DATE ’09, pp. 226–231 (2009)Google Scholar
  45. 45.
    Virginia, A.J., et al.: An empirical comparison of ANSI-C to VHDL compilers : SPARK, RORCC and DWARV. In: Annual Workshop on Circuits Systems and Signal Processing ProRISC, pp. 388–394 (2007)Google Scholar
  46. 46.
    Xilinx Platform studio and the embedded development kit (EDK). (2011a)
  47. 47.
    Xilinx Xilinx introduces Zynq-7000 family, industry’s first extensible processing platform. (2011b)
  48. 48.
  49. 49.
    Yankova, Y., Kuzmanov, G., Bertels, K., Gaydadjiev, G., Lu, Y., Vassiliadis, S.: DWARV: Delftworkbench automated reconfigurable VHDL generator. In: Intl Conf on Field Programmable Logic and Applications, IEEE, pp. 697–701 (2007)Google Scholar
  50. 50.
    Zhang, Z.: Flexible camera calibration by viewing a plane from unknown orientations. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 1, pp. 666–673 (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Sol Pedre
    • 1
  • Tomáš Krajník
    • 2
  • Elías Todorovich
    • 3
  • Patricia Borensztejn
    • 1
  1. 1.Departamento de ComputaciónFCEN-UBABuenos AiresArgentina
  2. 2.Czech Technical University in PraguePragueCzech Republic
  3. 3.Departamento de Computación y SistemasFCE-UNICENTandilArgentina

Personalised recommendations