Skip to main content
SpringerLink
Log in

Flexible three-dimensional modeling of plants using low- resolution cameras and visual odometry

  • Special Issue Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

The three-dimensional reconstruction of plants using computer vision methods is a promising alternative to non-destructive metrology in plant phenotyping. However, diversity in plants form and size, different surrounding environments (laboratory, greenhouse or field), and occlusions impose challenging issues. We propose the use of state-of-the-art methods for visual odometry to accurately recover camera pose and preliminary three-dimensional models on image acquisition time. Specimens of maize and sunflower were imaged using a single free-moving camera and a software tool with visual odometry capabilities. Multiple-view stereo was employed to produce dense point clouds sampling the plant surfaces. The produced three-dimensional models are accurate snapshots of the shoot state and plant measurements can be recovered in a non-invasive way. The results show a free-moving low-resolution camera is able to handle occlusions and variations in plant size and form, allowing the reconstruction of different species, and specimens in different stages of development. It is also a cheap and flexible method, suitable for different phenotyping needs. Plant traits were computed from the point clouds and compared to manually measured reference, showing millimeter accuracy. All data, including images, camera calibration, pose, and three-dimensional models are publicly available.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. The operator can be an human user or a machine, for example, a drone or a rover.

  2. The automatic detection of points corresponding to the soil surface could be performed by different methods: color-based classification, analysis of normal vectors or image segmentation, for example.

  3. Available on GitHub: https://github.com/uzh-rpg/rpg_svo (commit 3ba099).

  4. See http://maker.danforthcenter.org.

  5. Available at https://www.agropediabrasilis.cnptia.embrapa.br/en/web/plantscan/datasets after publication. PLY files are available now as supplementary material.

References

  1. Alenyà, G., Dellen, B., Torras, C.: 3D modelling of leaves from color and ToF data for robotized plant measuring. In: 2011 IEEE International Conference on Robotics and Automation, pp. 3408–3414 (2011). doi:10.1109/ICRA.2011.5980092

  2. Baker, S., Matthews, I.: Lucas–Kanade 20 years on: a unifying framework. Int. J. Computer Vision 56(3), 221–255 (2004). doi:10.1023/B:VISI.0000011205.11775.fd

    Article  Google Scholar 

  3. Bellasio, C., Olejníčková, J., Tesa, R., Sebela, D., Nedbal, L.: Computer reconstruction of plant growth and chlorophyll fluorescence emission in three spatial dimensions. Sensors (Basel, Switzerland) 12(1), 1052–71 (2012). doi:10.3390/s120101052

  4. Bernardini, F., Mittleman, J., Rushmeier, H., Silva, C., Taubin, G.: The ball-pivoting algorithm for surface reconstruction. IEEE Trans. Vis. Computer Graph. 5(4), 349–359 (1999). doi:10.1109/2945.817351

    Article  Google Scholar 

  5. Biskup, B., Scharr, H., Fischbach, A., Wiese-Klinkenberg, A., Schurr, U., Walter, A.: Diel growth cycle of isolated leaf discs analyzed with a novel, high-throughput three-dimensional imaging method is identical to that of intact leaves. Plant Physiol. 149(3), 1452–1461 (2009). doi:10.1104/pp.108.134486

    Article  Google Scholar 

  6. Biskup, B., Scharr, H., Schurr, U., Rascher, U.: A stereo imaging system for measuring structural parameters of plant canopies. Plant Cell Environ. 30(10), 1299–1308 (2007). doi:10.1111/j.1365-3040.2007.01702.x

    Article  Google Scholar 

  7. Biskup, B., Scharr, H., Schurr, U., Rascher, U.: A stereo imaging system for measuring structural parameters of plant canopies. Plant Cell Environ. 30(10), 1299–1308 (2007). doi:10.1111/j.1365-3040.2007.01702.x

    Article  Google Scholar 

  8. Brandner, M.: Bayesian uncertainty evaluation in vision-based metrology. In: Gallegos-Funes, F. (ed.) Vision Sensors and Edge Detection, 1993. INTECH Open Access Publisher (2010)

  9. Calakli, F., Taubin, G.: SSD: smooth signed distance surface reconstruction. Computer Graph. Forum 30(7), 1993–2002 (2011). doi:10.1111/j.1467-8659.2011.02058.x

    Article  Google Scholar 

  10. Chéné, Y., Rousseau, D., Lucidarme, P., Bertheloot, J., Caffier, V., Morel, P., Belin, E., Chapeau-Blondeau, F.: On the use of depth camera for 3D phenotyping of entire plants. Computers Electron. Agric. 82, 122–127 (2012). doi:10.1016/j.compag.2011.12.007

    Article  Google Scholar 

  11. Cobb, J.N., Declerck, G., Greenberg, A., Clark, R., McCouch, S.: Next-generation phenotyping: requirements and strategies for enhancing our understanding of genotype–phenotype relationships and its relevance to crop improvement. TAG. Theoretical and applied genetics. Theoretische und angewandte Genetik 126(4), 867–87 (2013). doi:10.1007/s00122-013-2066-0

  12. Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 3rd edn. The MIT Press, Cambridge (2009)

    MATH  Google Scholar 

  13. Engels, C., Stewénius, H., Nistér, D.: Bundle adjustment rules. Photogramm. Computer Vision 2, 266–271 (2006). ISSN: 1682-1750

  14. Fahlgren, N., Gehan, M.A., Baxter, I.: Lights, camera, action: high-throughput plant phenotyping is ready for a close-up. Curr. Opin. Plant Biol. 24, 93–99 (2015). doi:10.1016/j.pbi.2015.02.006

    Article  Google Scholar 

  15. Fiorani, F., Schurr, U.: Future scenarios for plant phenotyping. Ann. Rev. Plant Biol. 64, 267–291 (2013). doi:10.1146/annurev-arplant-050312-120137

    Article  Google Scholar 

  16. Forster, C., Pizzoli, M., Scaramuzza, D.: SVO: fast semi-direct monocular visual odometry. In: IEEE International Conference on Robotics and Automation (ICRA) (2014)

  17. Fraundorfer, F., Scaramuzza, D.: Visual odometry. Part II: matching, robustness, optimization, and applications. IEEE Robot. Autom. Mag. 19(2), 78–90 (2012). doi:10.1109/MRA.2012.2182810

    Article  Google Scholar 

  18. Furbank, R.T., Tester, M.: Phenomics-technologies to relieve the phenotyping bottleneck. Trends Plant Sci. 16(12), 635–644 (2011). doi:10.1016/j.tplants.2011.09.005

    Article  Google Scholar 

  19. Furukawa, Y., Curless, B., Seitz, S., Szeliski, R.: Towards internet-scale multi-view stereo. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1434–1441. IEEE, San Francisco (2010). doi:10.1109/CVPR.2010.5539802

  20. Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. 32(8), 1362–1376 (2010). doi:10.1109/TPAMI.2009.161

    Article  Google Scholar 

  21. Golub, G.H., Van Loan, C.F.: Matrix Computations (Johns Hopkins Studies in Mathematical Sciences), 3rd edn. The Johns Hopkins University Press, Baltimore (1996)

    Google Scholar 

  22. Granier, C., Aguirrezabal, L., Chenu, K., Cookson, S.J., Dauzat, M., Hamard, P., Thioux, J.J., Rolland, G., Bouchier-Combaud, S., Lebaudy, A., Muller, B., Simonneau, T., Tardieu, F.: PHENOPSIS, an automated platform for reproducible phenotyping of plant responses to soil water deficit in Arabidopsis thaliana permitted the identification of an accession with low sensitivity to soil water deficit. New Phytol. 169(3), 623–635 (2006). doi:10.1111/j.1469-8137.2005.01609.x

    Article  Google Scholar 

  23. Green, J., Appel, H., Rehrig, E., Harnsomburana, J., Chang, J.F., Balint-Kurti, P., Shyu, C.R.: Phenophyte: a flexible affordable method to quantify 2d phenotypes from imagery. Plant Methods 8(1), 45 (2012). doi:10.1186/1746-4811-8-45

    Article  Google Scholar 

  24. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004)

    Book  MATH  Google Scholar 

  25. Hartmann, A., Czauderna, T., Hoffmann, R., Stein, N., Schreiber, F.: HTPheno: an image analysis pipeline for high-throughput plant phenotyping. BMC Bioinform. 12(1), 148 (2011). doi:10.1186/1471-2105-12-148

    Article  Google Scholar 

  26. Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328–341 (2008). doi:10.1109/TPAMI.2007.1166

    Article  Google Scholar 

  27. Houle, D., Govindaraju, D.R., Omholt, S.: Phenomics: the next challenge. Nat. Rev. Genet. 11(12), 855–866 (2010). doi:10.1038/nrg2897

    Article  Google Scholar 

  28. Irani, M., Anandan, P.: About direct methods. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) Vision Algorithms: Theory and Practice. Lecture Notes in Computer Science, vol. 1883, pp. 267–277. Springer, Berlin Heidelberg (2000). doi:10.1007/3-540-44480-7_18

  29. Jay, S., Rabatel, G., Hadoux, X., Moura, D., Gorretta, N.: In-field crop row phenotyping from 3d modeling performed using structure from motion. Computers Electron. Agric. 110, 70–77 (2015). doi:10.1016/j.compag.2014.09.021

    Article  Google Scholar 

  30. JCGM: Evaluation of Measurement Data. An Introduction to the Guide to the Expression of Uncertainty in Measurement and Related Documents (2009)

  31. Kang, S.B., Quan, L.: Image-Based Modeling of Plants and Trees. Morgan & Claypool Publishers, San Rafael (2010)

    Google Scholar 

  32. Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proceedings of the fourth Eurographics symposium on Geometry processing, vol. 7 (2006)

  33. Kazmi, W., Foix, S., Alenyà, G., Andersen, H.J.: Indoor and outdoor depth imaging of leaves with time-of-flight and stereo vision sensors: analysis and comparison. ISPRS J. Photogramm. Remote Sensing 88, 128–146 (2014). doi:10.1016/j.isprsjprs.2013.11.012

    Article  Google Scholar 

  34. Klein, G., Murray, D.: Parallel tracking and mapping on a camera phone. In: Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality, ISMAR ’09, pp. 83–86. IEEE Computer Society, Washington, DC (2009). doi:10.1109/ISMAR.2009.5336495

  35. Leung, C., Appleton, B., Buckley, M., Sun, C.: Embedded voxel colouring with adaptive threshold selection using globally minimal surfaces. Int. J. Computer Vision 99(2), 215–231 (2012). doi:10.1007/s11263-012-0525-8

    Article  MathSciNet  Google Scholar 

  36. Lhuillier, M., Quan, L.: Match propagation for image-based modeling and rendering. IEEE Trans. Pattern Anal. Mach. Intell. 24(8), 1140–1146 (2002). doi:10.1109/TPAMI.2002.1023810

    Article  Google Scholar 

  37. Lou, L., Liu, Y., Sheng, M., Han, J., Doonan, J.H.: A cost-effective automatic 3D reconstruction pipeline for plants using multi-view images. In: Mistry, M., Leonardis, A., Witkowski, M., Melhuish, C. (eds.) Advances in Autonomous Robotics Systems. Lecture Notes in Computer Science, vol. 8717, pp. 221–230. Springer International Publishing, Cham (2014). doi:10.1007/978-3-319-10401-0

  38. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Computer Vision 60(2), 91–110 (2004). doi:10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  39. Ma, W., Kruth, J.P.: Nurbs curve and surface fitting for reverse engineering. Int. J. Adv. Manuf. Technol. 14(12), 918–927 (1998). doi:10.1007/BF01179082

    Article  Google Scholar 

  40. Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., Kadir, T., Gool, L.V.: A comparison of affine region detectors. Int. J. Computer Vision 65(1–2), 43–72 (2005). doi:10.1007/s11263-005-3848-x

    Article  Google Scholar 

  41. Nagel, K.A., Putz, A., Gilmer, F., Heinz, K., Fischbach, A., Pfeifer, J., Faget, M., Blossfeld, S., Ernst, M., Dimaki, C., Kastenholz, B., Kleinert, A.K., Galinski, A., Scharr, H., Fiorani, F., Schurr, U.: GROWSCREEN-Rhizo is a novel phenotyping robot enabling simultaneous measurements of root and shoot growth for plants grown in soil-filled rhizotrons. Funct. Plant Biol. 39, 891–904 (2012). doi:10.1071/FP12023

    Article  Google Scholar 

  42. Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: Dtam: dense tracking and mapping in real-time. In: Proceedings of the 2011 International Conference on Computer Vision, ICCV ’11, pp. 2320–2327. IEEE Computer Society, Washington, DC (2011). doi:10.1109/ICCV.2011.6126513

  43. Nister, D., Naroditsky, O., Bergen, J.: Visual odometry. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004, vol. 1, pp. I–652–I–659 (2004). doi:10.1109/CVPR.2004.1315094

  44. Paulus, S., Schumann, H., Kuhlmann, H., Léon, J.: High-precision laser scanning system for capturing 3D plant architecture and analysing growth of cereal plants. Biosyst. Eng. 121, 1–11 (2014). doi:10.1016/j.biosystemseng.2014.01.010

    Article  Google Scholar 

  45. Pereyra-Irujo, G.A., Gasco, E.D., Peirone, L.S., Aguirrezábal, L.A.N.: GlyPh: a low-cost platform for phenotyping plant growth and water use. Funct. Plant Biol. 39(11), 905 (2012). doi:10.1071/FP12052

    Article  Google Scholar 

  46. Pizzoli, M., Forster, C., Scaramuzza, D.: REMODE: probabilistic, monocular dense reconstruction in real time. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA) (2014)

  47. Pollefeys, M., Nistér, D., Frahm, J.M., Akbarzadeh, A., Mordohai, P., Clipp, B., Engels, C., Gallup, D., Kim, S.J., Merrell, P., Salmi, C., Sinha, S., Talton, B., Wang, L., Yang, Q., Stewénius, H., Yang, R., Welch, G., Towles, H.: Detailed real-time urban 3D reconstruction from video. Int. J. Computer Vision 78(2), 143–167 (2008). doi:10.1007/s11263-007-0086-4

    Article  Google Scholar 

  48. Pound, M.P., French, A.P., Murchie, E.H., Pridmore, T.P.: Automated recovery of three-dimensional models of plant shoots from multiple color images. Plant Physiol. 166(December), 1688–1698 (2014). doi:10.1104/pp.114.248971

    Article  Google Scholar 

  49. Rascher, U., Blossfeld, S., Fiorani, F., Jahnke, S., Jansen, M., Kuhn, A.J., Matsubara, S., Märtin, L.L.A., Merchant, A., Metzner, R., Müller-Linow, M., Nagel, K.A., Pieruschka, R., Pinto, F., Schreiber, C.M., Temperton, V.M., Thorpe, M.R., Dusschoten, D.V., Van Volkenburgh, E., Windt, C.W., Schurr, U.: Non-invasive approaches for phenotyping of enhanced performance traits in bean. Funct. Plant Biol. 38(12), 968 (2011). doi:10.1071/FP11164

    Article  Google Scholar 

  50. Reuzeau, C., Frankard, V., Hatzfeld, Y., Sanz, A., Camp, W.V., Lejeune, P., Wilde, C.D., Lievens, K., de Wolf, J., Vranken, E., Peerbolte, R., Broekaert, W.: Traitmill? A functional genomics platform for the phenotypic analysis of cereals. Plant Genetic Resour. 4(01), 20–24 (2006). doi:10.1079/PGR2005104

    Article  Google Scholar 

  51. Rusu, R., Cousins, S.: 3d is here: point cloud library (pcl). In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp. 1–4 (2011). doi:10.1109/ICRA.2011.5980567

  52. Santos, T., Koenigkan, L., Barbedo, J., Rodrigues, G.: 3D plant modeling: localization, mapping and segmentation for plant phenotyping using a single hand-held camera. In: Agapito, L., Bronstein, M.M., Rother C. (eds.) Computer Vision—ECCV 2014 Workshops. Lecture Notes in Computer Science, vol. 8928, pp. 247–263. Springer International Publishing (2015). doi:10.1007/978-3-319-16220-1_18

  53. Santos, T., Ueda, J.: Automatic 3D plant reconstruction from photographies, segmentation and classification of leaves and internodes using clustering 1. In: Risto Sievänen, P.N., Nikinmaa, E., Godin, C., Lintunen, A. (ed.) Proceedings of the 7th International Conference on Functional-Structural Plant Models, pp. 95–97. Saariselkä, Finland (2013)

  54. Santos, T.T., de Oliveira, A.A.: Image-based 3D digitizing for plant architecture analysis and phenotyping. In: Saúde, A.V., Guimarães, S.J.F. (eds.) Workshop on Industry Applications (WGARI) in SIBGRAPI 2012 (XXV Conference on Graphics, Patterns and Images). Ouro Preto (2012)

  55. Scaramuzza, D., Fraundorfer, F.: Visual odometry (Tutorial). IEEE Robot. Autom. Mag. 18(4), 80–92 (2011). doi:10.1109/MRA.2011.943233

    Article  Google Scholar 

  56. Seitz, S., Curless, B., Diebel, J.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2006)

  57. Shakarji, C.: Least-squares fitting algorithms of the NIST algorithm testing system. J. Res. Natl. Inst. Stand. Technol. 103(6), 633 (1998). doi:10.6028/jres.103.043

    Article  Google Scholar 

  58. Sirault, X., Fripp, J., Paproki, A., Kuffner, P., Nguyen, C., Li, R., Daily, H., Guo, J., Furbank, R.: PlantScan: a three-dimensional phenotyping platform for capturing the structural dynamic of plant development and growth. In: Proceedings of the 7th International Conference on FunctionalStructural Plant Models, pp. 45–48. Saariselkä, Finland (2013)

  59. Snavely, N., Seitz, S., Szeliski, R.: Modeling the world from internet photo collections. Int. J. Computer Vision 80(2), 189–210 (2008). doi:10.1007/s11263-007-0107-3

    Article  Google Scholar 

  60. Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment a modern synthesis vision algorithms: theory and practice. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) Vision Algorithms: Theory and Practice. Lecture Notes in Computer Science, vol. 1883, book part (with own title) 21, pp. 153–177. Springer, Berlin/Heidelberg (2000). doi:10.1007/3-540-44480-7_21

  61. Tsaftaris, S.A., Noutsos, C.: Plant phenotyping with low cost digital cameras and image analytics. In: Environmental Science and Engineering (Subseries: Environmental Science), vol. 3, pp. 238–251. Springer, Berlin Heidelberg (2009). doi:10.1007/978-3-540-88351-7-18

  62. van der Heijden, G., Song, Y., Horgan, G., Polder, G., Dieleman, A., Bink, M., Palloix, A., van Eeuwijk, F., Glasbey, C.: SPICY: towards automated phenotyping of large pepper plants in the greenhouse. Funct. Plant Biol. 39(11), 870 (2012). doi:10.1071/FP12019

    Article  Google Scholar 

  63. Vadez, V., Kholova, J., Hummel, G., Zhokhavets, U., Gupta, S.K., Hash, C.T.: LeasyScan: a novel concept combining 3D imaging and lysimetry for high-throughput phenotyping of traits controlling plant water budget. J. Exp. Bot. 1–13 (2015). doi:10.1093/jxb/erv251

  64. Vogiatzis, G., Hernandez, C.: Video-based, real-time multi view stereo. Image Vision Comput. 29(7), 434–441 (2011). doi:10.1016/j.imavis.2011.01.006

    Article  Google Scholar 

  65. Wu, C., Agarwal, S., Curless, B., Seitz, S.M.: Multicore bundle adjustment. In: CVPR 2011, x, pp. 3057–3064. IEEE (2011). doi:10.1109/CVPR.2011.5995552

  66. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000). doi:10.1109/34.888718

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by Brazilian Agricultural Research Corporation (Embrapa) under grants 03.11.07.007.00.00 (PlantScan) and 05.12.12.001.00.02 (PhenoCorn). We would like to thank Dra. Juliana E. de C. T. Yassitepe and the Center for Molecular Biology and Genetic Engineering (CBMEG-Unicamp) for providing the greenhouse facilities. We also thank the reviewers who provided us with invaluable feedback that greatly contributed to this final version.

Author information

Authors and Affiliations

  1. Embrapa Agricultural Informatics, Av. André Tosello, 209, Campinas, SP, 13083-886, Brazil

    Thiago T. Santos & Gustavo C. Rodrigues

Authors
  1. Thiago T. Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gustavo C. Rodrigues
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thiago T. Santos.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Santos, T.T., Rodrigues, G.C. Flexible three-dimensional modeling of plants using low- resolution cameras and visual odometry. Machine Vision and Applications 27, 695–707 (2016). https://doi.org/10.1007/s00138-015-0729-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-015-0729-3

Keywords

Search

Navigation