Abstract
In this paper, we present a novel approach for assessing and interacting with surface tracking algorithms targeting video manipulation in post-production. As tracking inaccuracies are unavoidable, we enable the user to provide small hints to the algorithms instead of correcting erroneous results afterwards. Based on 2D mesh warp-based optical flow estimation, we visualize results and provide tools for user feedback in a consistent reference system, texture space. In this space, accurate tracking results are reflected by static appearance, and errors can easily be spotted as apparent change. A variety of established tools can be utilized to visualize and assess the change between frames. User interaction to improve tracking results becomes more intuitive in texture space, as it can focus on a small region rather than a moving object. We show how established tools can be implemented for interaction in texture space to provide a more intuitive interface allowing more effective and accurate user feedback.
Article PDF
Similar content being viewed by others
References
Imagineer Systems. mocha Pro. 2016. Available at http://www.imagineersystems.com/products/mochapro.
Foundry. NUKE. 2016. Available at https://www. foundry.com/products/nuke.
The Pixelfarm. PFTrack. 2016. Available at http://www.thepixelfarm.co.uk/pftrack/.
Klose, F.; Ruhl, K.; Lipski, C.; Magnor, M. Flowlab—An interactive tool for editing dense image correspondences. In: Proceedings of the Conference for Visual Media Production, 59–66, 2011.
Ruhl, K.; Eisemann, M.; Hilsmann, A.; Eisert, P.; Magnor, M. Interactive scene flow editing for improved image-based rendering and virtual spacetime navigation. In: Proceedings of the 23rd ACM International Conference on Multimedia, 631–640, 2015.
Zhang, C.; Price, B.; Cohen, S.; Yang, R. Highquality stereo video matching via user interaction and space–time propagation. In: Proceedings of the International Conference on 3D Vision, 71–78, 2013.
Re:Vision Effects. Twixtor. 2016. Available at http://revisionfx.com/products/twixtor/.
Wilkes, L. The role of ocula in stereo post production. Technical Report. The Foundry, 2009.
Chrysos, G. G.; Antonakos, E.; Zafeiriou, S.; Snape, P. Offline deformable face tracking in arbitrary videos. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 1–9, 2015.
Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics Vol. 23, No. 3, 309–314, 2004.
Liao, M.; Gao, J.; Yang, R.; Gong, M. Video stereolization: Combining motion analysis with user interaction. IEEE Transactions on Visualization & Computer Graphics Vol. 18, No. 7, 1079–1088, 2012.
Wang, O.; Lang, M.; Frei, M.; Hornung, A.; Smolic, A.; Gross, M. Stereobrush: Interactive 2D to 3D conversion using discontinuous warps. In: Proceedings of the 8th Eurographics Symposium on Sketch-Based Interfaces and Modeling, 47–54, 2011.
Doron, Y.; Campbell, N. D. F.; Starck, J.; Kautz, J. User directed multi-view-stereo. In: Computer Vision–ACCV 2014 Workshops. Jawahar, C.; Shan, S. Eds. Springer Cham, 299–313, 2014.
Bartoli, A.; Zisserman, A. Direct estimation of nonrigid registrations. In: Proceedings of the 15th British Machine Vision Conference, Vol. 2, 899–908, 2004.
Zhu, J.; Van Gool, L.; Hoi, S. C. H. Unsupervised face alignment by robust nonrigid mapping. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 1265–1272, 2009.
Hilsmann, A.; Eisert, P. Joint estimation of deformable motion and photometric parameters in single view videos. In: Proceedings of the IEEE 12th International Conference on Computer Vision Workshops, 390–397, 2009.
Gay-Bellile, V.; Bartoli, A.; Sayd, P. Direct estimation of nonrigid registrations with image-based selfocclusion reasoning. IEEE Transactions on Pattern Analysis & Machine Intelligence Vol. 32, No. 1, 87–104, 2010.
Seibold, C.; Hilsmann, A.; Eisert, P. Model-based motion blur estimation for the improvement of motion tracking. Computer Vision and Image Understanding DOI: 10.1016/j.cviu.2017.03.005, 2017.
Hilsmann, A.; Schneider, D. C.; Eisert, P. Image-based tracking of deformable surfaces. In: Object Tracking. InTech, 245–266, 2011.
Pilet, J.; Lepetit, V.; Fua, P. Real-time nonrigid surface detection. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, 822–828, 2005.
Brox, T.; Malik, J. Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 3, 500–513, 2011.
Wedel, A.; Cremers, D.; Pock, T.; Bischof, H. Structure-and motion-adaptive regularization for high accuracy optic flow. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 1663–1668, 2009.
Brox, T.; Bruhn, A.; Papenberg, N.; Weickert, J. High accuracy optical flow estimation based on a theory for warping. In: Computer Vision–ECCV 2004]. Pajdla, T.; Matas, J. Eds. Springer Berlin Heidelberg, 25–36, 2004.
Hollywood Camera Work. Face Capture dataset. 2016. Available at https://www.hollywoodcamerawork.com/tracking-plates.html.
Acknowledgements
This work was partially funded by the German Science Foundation (Grant No. DFG EI524/2-1) and by the European Commission (Grant Nos. FP7-288238 SCENE and H2020-644629 AutoPost).
Author information
Authors and Affiliations
Corresponding author
Additional information
This article is published with open access at Springerlink.com
Johannes Furch is a research associate at Fraunhofer HHI. He received his diploma in computer science from the University of T¨ubingen in 2010. His research focuses on video-based motion analysis for application in visual media production.
Anna Hilsmann received her Dipl.- Ing. degree in electrical engineering and information technology from RWTH Aachen in 2006 and her Dr.-Ing. degree in computer science from HU Berlin in 2014. She joined the Computer Vision and Graphics Group at Fraunhofer HHI in 2007 and the Visual Computing Group at HU Berlin in 2011. Since 2015, she has headed the Computer Vision and Graphics Group at Fraunhofer HHI. Her main research interests cover 3D image and video analysis, such as image registration, model-based deformable tracking and 3D reconstruction, as well as synthesis, image- and video-based rendering, animation, and editing.
Peter Eisert is professor for visual computing at Humboldt University, Berlin and heads the Vision & Imaging Technologies Department of the Fraunhofer HHI, Berlin, Germany. He received his Dipl.-Ing. degree in electrical engineering from TU Karlsruhe and his Dr.-Ing. degree from the University of Erlangen, Germany. In 2001, he worked as a postdoctoral fellow at Stanford University on 3D image analysis and synthesis as well as facial animation and computer graphics. He joined HHI in 2002 and HU Berlin in 2009, where he is coordinating and initiating numerous national and international research projects. He has published more than 150 conference and journal papers and is the associate editor of the International Journal of Image and Video Processing as well as on the editorial board of the Journal of Visual Communication and Image Representation. His research interests include 3D image analysis and synthesis, face processing, image-based rendering, computer vision, and computer graphics.
Rights and permissions
Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Furch, J., Hilsmann, A. & Eisert, P. Surface tracking assessment and interaction in texture space. Comp. Visual Media 4, 3–15 (2018). https://doi.org/10.1007/s41095-017-0089-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-017-0089-1