Journal of Real-Time Image Processing

, Volume 8, Issue 1, pp 95–110 | Cite as

A self-adaptive heterogeneous multi-core architecture for embedded real-time video object tracking

  • Markus Happe
  • Enno Lübbers
  • Marco Platzner
Special Issue


Sequential Monte Carlo (SMC) represents a principal statistical method for tracking objects in video sequences by on-line estimation of the state of a non-linear dynamic system. The performance of individual stages of the SMC algorithm is usually data-dependent, making the prediction of the performance of a real-time capable system difficult and often leading to grossly overestimated and inefficient system designs. Also, the considerable computational complexity is a major obstacle when implementing SMC methods on purely CPU-based resource constrained embedded systems. In contrast, heterogeneous multi-cores present a more suitable implementation platform. We use hybrid CPU/FPGA systems, as they can efficiently execute both the control-centric sequential as well as the data-parallel parts of an SMC application. However, even with hybrid CPU/FPGA platforms, determining the optimal HW/SW partitioning is challenging in general, and even impossible with a design time approach. Thus, we need self-adaptive architectures and system software layers that are able to react autonomously to varying workloads and changing input data while preserving real-time constraints and area efficiency. In this article, we present a video tracking application modeled on top of a framework for implementing SMC methods on CPU/FPGA-based systems such as modern platform FPGAs. Based on a multithreaded programming model, our framework allows for an easy design space exploration with respect to the HW/SW partitioning. Additionally, the application can adaptively switch between several partitionings during run-time to react to changing input data and performance requirements. Our system utilizes two variants of a add/remove self-adaptation technique for task partitioning inside this framework that achieve soft real-time behavior while trying to minimize the number of active cores. To evaluate its performance and area requirements, we demonstrate the application and the framework on a real-life video tracking case study and show that partial reconfiguration can be effectively and transparently used for realizing adaptive real-time HW/SW systems.


Real-time video object tracking Heterogeneous multi-core architecture Self-adaptation CPU/FPGA systems HW/SW co-design 



The research leading to these results has received funding from the European Union Seventh Framework Programme under grant agreement no. 257906. This work was supported by the International Graduate School of Dynamic Intelligent Systems and by the German Research Foundation under project number PL471/2-1.


  1. 1.
    Ali, U., Malik, M., Munawar, K.: FPGA/soft-processor based real-time object tracking system. 5th Southern Conference on Programmable Logic (2009)Google Scholar
  2. 2.
    Arulampalam, M.S., Maskell, S., Gordon, N., Clapp, T.: A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking. IEEE Trans. Signal Process. 50(2), 174–188 (2002)CrossRefGoogle Scholar
  3. 3.
    Carvalho, E., Calazans, N., Moraes, F.: Heuristics for dynamic task mapping in NoC-based heterogeneous MPSoCs. International Workshop on Rapid System Prototyping (RSP) (2007)Google Scholar
  4. 4.
    Cho, J.U., Jin, S.H., Pham, X.D., Kim, D., Jeon, J.W.: A real-time color feature tracking system using color histograms. International Conference on Control, Automation and Systems (2007)Google Scholar
  5. 5.
    Curtis-Maury, M., Dzierwa, J., Antonopoulos, C.D., Nikolopoulos, D.S.: Online power-performance adaptation of multithreaded programs using hardware event-based prediction. International Conference on Supercomputing (2006)Google Scholar
  6. 6.
    Doucet, A., de Freitas, N., Gordon, N.: Sequential Monte Carlo Methods in Practice. Springer, Berlin (2001)Google Scholar
  7. 7.
    Gilbert, A.L., Giles, M.K., Flachs, G.M., Rogers, R.B., Hsun, U.Y.: A real-time video tracking system. IEEE Trans. Pattern. Anal. Mach. Intell. 2, 47–56 (1980)Google Scholar
  8. 8.
    Happe, M., Lübbers, E., Platzner, M.: A multithreaded framework for sequential Monte Carlo methods on CPU/FPGA platforms. International Workshop on Applied Reconfigurable Computing (ARC) (2009a)Google Scholar
  9. 9.
    Happe, M., Lübbers, E., Platzner, M.: An adaptive sequential Monte Carlo framework with runtime HW/SW partitioning. IEEE Int. Conf. Field Program. Technol. (FPT) (2009b)Google Scholar
  10. 10.
    Hess, R.: Particle filter object tracking. (2006)
  11. 11.
    Huang, C., Vahid, F.: Dynamic coprocessor management for FPGA-enhanced compute platforms. International Conference on Compilers, Architecture and Synthesis for Embedded Systems (CASES) (2008)Google Scholar
  12. 12.
    Kalman, E.R.: A new approach to linear filtering and prediction problems. Trans. ASME—J. Basic Eng. 82(Series D), 35–45 (1960)Google Scholar
  13. 13.
    Kobayashi, T., Nakagawa, K., Imae, J., Zhai, G.: Real Time object tracking on video image sequence using particle swarm optimization. International Conference on Control, Automation and Systems (ICCAS) (2007)Google Scholar
  14. 14.
    Kumar, R., Farkas, K.I., Jouppi, N.P., Ranganathan, P., Tullsen, D.M.: Single-ISA heterogeneous multi-core architectures: the potential for processor power reduction. In: International Symposium on Microarchitecture (2003)Google Scholar
  15. 15.
    Loza, A., Patricio, M., Garcia, J., Molina, J.: Advanced algorithms for real-time video tracking with multiple targets. 10th International Conference on Control, Automation, Robotics and Vision (ICARCV) (2008)Google Scholar
  16. 16.
    Lübbers, E., Platzner, M.: ReconOS: multithreaded programming for reconfigurable computers. ACM Trans. Embed. Comput. Syst. 9(1), 1–33 (2009)Google Scholar
  17. 17.
    Nollet, V., Avasare, P., Eeckhaut, H., Verkest, D., Corporaal, H.: Run-time management of a MPSoC containing FPGA fabric tiles. Trans. Very Large Scale Integr. Syst. (2008)Google Scholar
  18. 18.
    Porikli, F.: Achieving real-time object detection and tracking under extreme conditions. J. Real-time Image Proc. 1(1), 33–40 (2006)Google Scholar
  19. 19.
    Saha, S., Bambha, N.K., Bhattacharyya, S.S.: A parameterized design framework for hardware implementation of particle filters. IEEE International Conference on Acoustics, Speech and Signal Processing (2008)Google Scholar
  20. 20.
    Sankaranarayanan, A.C., Chellappa, R., Srivastava, A.: Algorithmic and architectural design methodology for particle filters in hardware. International Conference on Computer Design (2005)Google Scholar
  21. 21.
    Sigdel, K., Thompson, M., Pimentel, A.D., Galuzzi, C., Bertels, K.: System-level runtime mapping exploration of reconfigurable architectures. Proceedings of the International Symposium on Parallel & Distributed Processing (IPDPS) (2009)Google Scholar
  22. 22.
    Sironi, F., Triverio, M., Hoffmann, H., Maggio, M., Santambrogio, M.: Self-aware adaptation in FPGA-based systems. International Conference on Field Programmable Logic and Applications (FPL) (2010)Google Scholar
  23. 23.
    Smit, L.T., Smit, G.J.M., Hurink, J.L., Broersma, H., Paulusma, D., Wolkotte, P.T.: Run-time assignment of tasks to a heterogeneous processors. In: Embedded Systems Symposium (2004)Google Scholar
  24. 24.
    Smit, L.T., Hurink, J.L., Smit, G.J.M.: Run-time mapping of applications to a heterogeneous SoC. Proceedings of the International Symposium on System-on-Chip (2005)Google Scholar
  25. 25.
    Stitt, G., Lysecky, R., Vahid, F.: Dynamic hardware/software partitioning: a first approach. Proceedings of the Design Automation Conference (DAC) (2003)Google Scholar
  26. 26.
    Visentini, I., Snidaro, L., Foresti, G.L.: Cascaded online boosting. J. Real-time Image Proc. 5(4), 245–257 (2010)Google Scholar
  27. 27.
    Woelk, F., Schiller, I., Koch, R.: An airborne bayesian color tracking system. In: IEEE Intelligent Vehicles Symposium (2005)Google Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  1. 1.Computer Engineering GroupPaderbornGermany
  2. 2.EADS Innovation Works, Technical Capability Center 5MunichGermany

Personalised recommendations