Advertisement

Machine Vision and Applications

, Volume 27, Issue 5, pp 695–707 | Cite as

Flexible three-dimensional modeling of plants using low- resolution cameras and visual odometry

  • Thiago T. Santos
  • Gustavo C. Rodrigues
Special Issue Paper

Abstract

The three-dimensional reconstruction of plants using computer vision methods is a promising alternative to non-destructive metrology in plant phenotyping. However, diversity in plants form and size, different surrounding environments (laboratory, greenhouse or field), and occlusions impose challenging issues. We propose the use of state-of-the-art methods for visual odometry to accurately recover camera pose and preliminary three-dimensional models on image acquisition time. Specimens of maize and sunflower were imaged using a single free-moving camera and a software tool with visual odometry capabilities. Multiple-view stereo was employed to produce dense point clouds sampling the plant surfaces. The produced three-dimensional models are accurate snapshots of the shoot state and plant measurements can be recovered in a non-invasive way. The results show a free-moving low-resolution camera is able to handle occlusions and variations in plant size and form, allowing the reconstruction of different species, and specimens in different stages of development. It is also a cheap and flexible method, suitable for different phenotyping needs. Plant traits were computed from the point clouds and compared to manually measured reference, showing millimeter accuracy. All data, including images, camera calibration, pose, and three-dimensional models are publicly available.

Keywords

Image-based phenotyping Plant digitizing 3-D reconstruction 

Notes

Acknowledgments

This work was supported by Brazilian Agricultural Research Corporation (Embrapa) under grants 03.11.07.007.00.00 (PlantScan) and 05.12.12.001.00.02 (PhenoCorn). We would like to thank Dra. Juliana E. de C. T. Yassitepe and the Center for Molecular Biology and Genetic Engineering (CBMEG-Unicamp) for providing the greenhouse facilities. We also thank the reviewers who provided us with invaluable feedback that greatly contributed to this final version.

Supplementary material

138_2015_729_MOESM1_ESM.ply (8.8 mb)
Supplementary material 1 (ply 9049 KB)
138_2015_729_MOESM2_ESM.ply (1.1 mb)
Supplementary material 2 (ply 1103 KB)
138_2015_729_MOESM3_ESM.ply (13.2 mb)
Supplementary material 3 (ply 13548 KB)
138_2015_729_MOESM4_ESM.ply (7.5 mb)
Supplementary material 4 (ply 7714 KB)
138_2015_729_MOESM5_ESM.ply (6.8 mb)
Supplementary material 5 (ply 6927 KB)
138_2015_729_MOESM6_ESM.ply (7.7 mb)
Supplementary material 6 (ply 7906 KB)
138_2015_729_MOESM7_ESM.ply (19.9 mb)
Supplementary material 7 (ply 20339 KB)
138_2015_729_MOESM8_ESM.ply (13 mb)
Supplementary material 8 (ply 13331 KB)

References

  1. 1.
    Alenyà, G., Dellen, B., Torras, C.: 3D modelling of leaves from color and ToF data for robotized plant measuring. In: 2011 IEEE International Conference on Robotics and Automation, pp. 3408–3414 (2011). doi: 10.1109/ICRA.2011.5980092
  2. 2.
    Baker, S., Matthews, I.: Lucas–Kanade 20 years on: a unifying framework. Int. J. Computer Vision 56(3), 221–255 (2004). doi: 10.1023/B:VISI.0000011205.11775.fd CrossRefGoogle Scholar
  3. 3.
    Bellasio, C., Olejníčková, J., Tesa, R., Sebela, D., Nedbal, L.: Computer reconstruction of plant growth and chlorophyll fluorescence emission in three spatial dimensions. Sensors (Basel, Switzerland) 12(1), 1052–71 (2012). doi: 10.3390/s120101052
  4. 4.
    Bernardini, F., Mittleman, J., Rushmeier, H., Silva, C., Taubin, G.: The ball-pivoting algorithm for surface reconstruction. IEEE Trans. Vis. Computer Graph. 5(4), 349–359 (1999). doi: 10.1109/2945.817351 CrossRefGoogle Scholar
  5. 5.
    Biskup, B., Scharr, H., Fischbach, A., Wiese-Klinkenberg, A., Schurr, U., Walter, A.: Diel growth cycle of isolated leaf discs analyzed with a novel, high-throughput three-dimensional imaging method is identical to that of intact leaves. Plant Physiol. 149(3), 1452–1461 (2009). doi: 10.1104/pp.108.134486 CrossRefGoogle Scholar
  6. 6.
    Biskup, B., Scharr, H., Schurr, U., Rascher, U.: A stereo imaging system for measuring structural parameters of plant canopies. Plant Cell Environ. 30(10), 1299–1308 (2007). doi: 10.1111/j.1365-3040.2007.01702.x CrossRefGoogle Scholar
  7. 7.
    Biskup, B., Scharr, H., Schurr, U., Rascher, U.: A stereo imaging system for measuring structural parameters of plant canopies. Plant Cell Environ. 30(10), 1299–1308 (2007). doi: 10.1111/j.1365-3040.2007.01702.x CrossRefGoogle Scholar
  8. 8.
    Brandner, M.: Bayesian uncertainty evaluation in vision-based metrology. In: Gallegos-Funes, F. (ed.) Vision Sensors and Edge Detection, 1993. INTECH Open Access Publisher (2010)Google Scholar
  9. 9.
    Calakli, F., Taubin, G.: SSD: smooth signed distance surface reconstruction. Computer Graph. Forum 30(7), 1993–2002 (2011). doi: 10.1111/j.1467-8659.2011.02058.x CrossRefGoogle Scholar
  10. 10.
    Chéné, Y., Rousseau, D., Lucidarme, P., Bertheloot, J., Caffier, V., Morel, P., Belin, E., Chapeau-Blondeau, F.: On the use of depth camera for 3D phenotyping of entire plants. Computers Electron. Agric. 82, 122–127 (2012). doi: 10.1016/j.compag.2011.12.007 CrossRefGoogle Scholar
  11. 11.
    Cobb, J.N., Declerck, G., Greenberg, A., Clark, R., McCouch, S.: Next-generation phenotyping: requirements and strategies for enhancing our understanding of genotype–phenotype relationships and its relevance to crop improvement. TAG. Theoretical and applied genetics. Theoretische und angewandte Genetik 126(4), 867–87 (2013). doi: 10.1007/s00122-013-2066-0
  12. 12.
    Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 3rd edn. The MIT Press, Cambridge (2009)zbMATHGoogle Scholar
  13. 13.
    Engels, C., Stewénius, H., Nistér, D.: Bundle adjustment rules. Photogramm. Computer Vision 2, 266–271 (2006). ISSN: 1682-1750Google Scholar
  14. 14.
    Fahlgren, N., Gehan, M.A., Baxter, I.: Lights, camera, action: high-throughput plant phenotyping is ready for a close-up. Curr. Opin. Plant Biol. 24, 93–99 (2015). doi: 10.1016/j.pbi.2015.02.006 CrossRefGoogle Scholar
  15. 15.
    Fiorani, F., Schurr, U.: Future scenarios for plant phenotyping. Ann. Rev. Plant Biol. 64, 267–291 (2013). doi: 10.1146/annurev-arplant-050312-120137 CrossRefGoogle Scholar
  16. 16.
    Forster, C., Pizzoli, M., Scaramuzza, D.: SVO: fast semi-direct monocular visual odometry. In: IEEE International Conference on Robotics and Automation (ICRA) (2014)Google Scholar
  17. 17.
    Fraundorfer, F., Scaramuzza, D.: Visual odometry. Part II: matching, robustness, optimization, and applications. IEEE Robot. Autom. Mag. 19(2), 78–90 (2012). doi: 10.1109/MRA.2012.2182810 CrossRefGoogle Scholar
  18. 18.
    Furbank, R.T., Tester, M.: Phenomics-technologies to relieve the phenotyping bottleneck. Trends Plant Sci. 16(12), 635–644 (2011). doi: 10.1016/j.tplants.2011.09.005 CrossRefGoogle Scholar
  19. 19.
    Furukawa, Y., Curless, B., Seitz, S., Szeliski, R.: Towards internet-scale multi-view stereo. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1434–1441. IEEE, San Francisco (2010). doi: 10.1109/CVPR.2010.5539802
  20. 20.
    Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. 32(8), 1362–1376 (2010). doi: 10.1109/TPAMI.2009.161 CrossRefGoogle Scholar
  21. 21.
    Golub, G.H., Van Loan, C.F.: Matrix Computations (Johns Hopkins Studies in Mathematical Sciences), 3rd edn. The Johns Hopkins University Press, Baltimore (1996)Google Scholar
  22. 22.
    Granier, C., Aguirrezabal, L., Chenu, K., Cookson, S.J., Dauzat, M., Hamard, P., Thioux, J.J., Rolland, G., Bouchier-Combaud, S., Lebaudy, A., Muller, B., Simonneau, T., Tardieu, F.: PHENOPSIS, an automated platform for reproducible phenotyping of plant responses to soil water deficit in Arabidopsis thaliana permitted the identification of an accession with low sensitivity to soil water deficit. New Phytol. 169(3), 623–635 (2006). doi: 10.1111/j.1469-8137.2005.01609.x CrossRefGoogle Scholar
  23. 23.
    Green, J., Appel, H., Rehrig, E., Harnsomburana, J., Chang, J.F., Balint-Kurti, P., Shyu, C.R.: Phenophyte: a flexible affordable method to quantify 2d phenotypes from imagery. Plant Methods 8(1), 45 (2012). doi: 10.1186/1746-4811-8-45 CrossRefGoogle Scholar
  24. 24.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004)CrossRefzbMATHGoogle Scholar
  25. 25.
    Hartmann, A., Czauderna, T., Hoffmann, R., Stein, N., Schreiber, F.: HTPheno: an image analysis pipeline for high-throughput plant phenotyping. BMC Bioinform. 12(1), 148 (2011). doi: 10.1186/1471-2105-12-148 CrossRefGoogle Scholar
  26. 26.
    Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328–341 (2008). doi: 10.1109/TPAMI.2007.1166 CrossRefGoogle Scholar
  27. 27.
    Houle, D., Govindaraju, D.R., Omholt, S.: Phenomics: the next challenge. Nat. Rev. Genet. 11(12), 855–866 (2010). doi: 10.1038/nrg2897 CrossRefGoogle Scholar
  28. 28.
    Irani, M., Anandan, P.: About direct methods. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) Vision Algorithms: Theory and Practice. Lecture Notes in Computer Science, vol. 1883, pp. 267–277. Springer, Berlin Heidelberg (2000). doi: 10.1007/3-540-44480-7_18
  29. 29.
    Jay, S., Rabatel, G., Hadoux, X., Moura, D., Gorretta, N.: In-field crop row phenotyping from 3d modeling performed using structure from motion. Computers Electron. Agric. 110, 70–77 (2015). doi: 10.1016/j.compag.2014.09.021 CrossRefGoogle Scholar
  30. 30.
    JCGM: Evaluation of Measurement Data. An Introduction to the Guide to the Expression of Uncertainty in Measurement and Related Documents (2009)Google Scholar
  31. 31.
    Kang, S.B., Quan, L.: Image-Based Modeling of Plants and Trees. Morgan & Claypool Publishers, San Rafael (2010)Google Scholar
  32. 32.
    Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proceedings of the fourth Eurographics symposium on Geometry processing, vol. 7 (2006)Google Scholar
  33. 33.
    Kazmi, W., Foix, S., Alenyà, G., Andersen, H.J.: Indoor and outdoor depth imaging of leaves with time-of-flight and stereo vision sensors: analysis and comparison. ISPRS J. Photogramm. Remote Sensing 88, 128–146 (2014). doi: 10.1016/j.isprsjprs.2013.11.012 CrossRefGoogle Scholar
  34. 34.
    Klein, G., Murray, D.: Parallel tracking and mapping on a camera phone. In: Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality, ISMAR ’09, pp. 83–86. IEEE Computer Society, Washington, DC (2009). doi: 10.1109/ISMAR.2009.5336495
  35. 35.
    Leung, C., Appleton, B., Buckley, M., Sun, C.: Embedded voxel colouring with adaptive threshold selection using globally minimal surfaces. Int. J. Computer Vision 99(2), 215–231 (2012). doi: 10.1007/s11263-012-0525-8 MathSciNetCrossRefGoogle Scholar
  36. 36.
    Lhuillier, M., Quan, L.: Match propagation for image-based modeling and rendering. IEEE Trans. Pattern Anal. Mach. Intell. 24(8), 1140–1146 (2002). doi: 10.1109/TPAMI.2002.1023810 CrossRefGoogle Scholar
  37. 37.
    Lou, L., Liu, Y., Sheng, M., Han, J., Doonan, J.H.: A cost-effective automatic 3D reconstruction pipeline for plants using multi-view images. In: Mistry, M., Leonardis, A., Witkowski, M., Melhuish, C. (eds.) Advances in Autonomous Robotics Systems. Lecture Notes in Computer Science, vol. 8717, pp. 221–230. Springer International Publishing, Cham (2014). doi: 10.1007/978-3-319-10401-0
  38. 38.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Computer Vision 60(2), 91–110 (2004). doi: 10.1023/B:VISI.0000029664.99615.94 CrossRefGoogle Scholar
  39. 39.
    Ma, W., Kruth, J.P.: Nurbs curve and surface fitting for reverse engineering. Int. J. Adv. Manuf. Technol. 14(12), 918–927 (1998). doi: 10.1007/BF01179082 CrossRefGoogle Scholar
  40. 40.
    Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., Kadir, T., Gool, L.V.: A comparison of affine region detectors. Int. J. Computer Vision 65(1–2), 43–72 (2005). doi: 10.1007/s11263-005-3848-x CrossRefGoogle Scholar
  41. 41.
    Nagel, K.A., Putz, A., Gilmer, F., Heinz, K., Fischbach, A., Pfeifer, J., Faget, M., Blossfeld, S., Ernst, M., Dimaki, C., Kastenholz, B., Kleinert, A.K., Galinski, A., Scharr, H., Fiorani, F., Schurr, U.: GROWSCREEN-Rhizo is a novel phenotyping robot enabling simultaneous measurements of root and shoot growth for plants grown in soil-filled rhizotrons. Funct. Plant Biol. 39, 891–904 (2012). doi: 10.1071/FP12023 CrossRefGoogle Scholar
  42. 42.
    Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: Dtam: dense tracking and mapping in real-time. In: Proceedings of the 2011 International Conference on Computer Vision, ICCV ’11, pp. 2320–2327. IEEE Computer Society, Washington, DC (2011). doi: 10.1109/ICCV.2011.6126513
  43. 43.
    Nister, D., Naroditsky, O., Bergen, J.: Visual odometry. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004, vol. 1, pp. I–652–I–659 (2004). doi: 10.1109/CVPR.2004.1315094
  44. 44.
    Paulus, S., Schumann, H., Kuhlmann, H., Léon, J.: High-precision laser scanning system for capturing 3D plant architecture and analysing growth of cereal plants. Biosyst. Eng. 121, 1–11 (2014). doi: 10.1016/j.biosystemseng.2014.01.010 CrossRefGoogle Scholar
  45. 45.
    Pereyra-Irujo, G.A., Gasco, E.D., Peirone, L.S., Aguirrezábal, L.A.N.: GlyPh: a low-cost platform for phenotyping plant growth and water use. Funct. Plant Biol. 39(11), 905 (2012). doi: 10.1071/FP12052 CrossRefGoogle Scholar
  46. 46.
    Pizzoli, M., Forster, C., Scaramuzza, D.: REMODE: probabilistic, monocular dense reconstruction in real time. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA) (2014)Google Scholar
  47. 47.
    Pollefeys, M., Nistér, D., Frahm, J.M., Akbarzadeh, A., Mordohai, P., Clipp, B., Engels, C., Gallup, D., Kim, S.J., Merrell, P., Salmi, C., Sinha, S., Talton, B., Wang, L., Yang, Q., Stewénius, H., Yang, R., Welch, G., Towles, H.: Detailed real-time urban 3D reconstruction from video. Int. J. Computer Vision 78(2), 143–167 (2008). doi: 10.1007/s11263-007-0086-4 CrossRefGoogle Scholar
  48. 48.
    Pound, M.P., French, A.P., Murchie, E.H., Pridmore, T.P.: Automated recovery of three-dimensional models of plant shoots from multiple color images. Plant Physiol. 166(December), 1688–1698 (2014). doi: 10.1104/pp.114.248971 CrossRefGoogle Scholar
  49. 49.
    Rascher, U., Blossfeld, S., Fiorani, F., Jahnke, S., Jansen, M., Kuhn, A.J., Matsubara, S., Märtin, L.L.A., Merchant, A., Metzner, R., Müller-Linow, M., Nagel, K.A., Pieruschka, R., Pinto, F., Schreiber, C.M., Temperton, V.M., Thorpe, M.R., Dusschoten, D.V., Van Volkenburgh, E., Windt, C.W., Schurr, U.: Non-invasive approaches for phenotyping of enhanced performance traits in bean. Funct. Plant Biol. 38(12), 968 (2011). doi: 10.1071/FP11164 CrossRefGoogle Scholar
  50. 50.
    Reuzeau, C., Frankard, V., Hatzfeld, Y., Sanz, A., Camp, W.V., Lejeune, P., Wilde, C.D., Lievens, K., de Wolf, J., Vranken, E., Peerbolte, R., Broekaert, W.: Traitmill? A functional genomics platform for the phenotypic analysis of cereals. Plant Genetic Resour. 4(01), 20–24 (2006). doi: 10.1079/PGR2005104 CrossRefGoogle Scholar
  51. 51.
    Rusu, R., Cousins, S.: 3d is here: point cloud library (pcl). In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp. 1–4 (2011). doi: 10.1109/ICRA.2011.5980567
  52. 52.
    Santos, T., Koenigkan, L., Barbedo, J., Rodrigues, G.: 3D plant modeling: localization, mapping and segmentation for plant phenotyping using a single hand-held camera. In: Agapito, L., Bronstein, M.M., Rother C. (eds.) Computer Vision—ECCV 2014 Workshops. Lecture Notes in Computer Science, vol. 8928, pp. 247–263. Springer International Publishing (2015). doi: 10.1007/978-3-319-16220-1_18
  53. 53.
    Santos, T., Ueda, J.: Automatic 3D plant reconstruction from photographies, segmentation and classification of leaves and internodes using clustering 1. In: Risto Sievänen, P.N., Nikinmaa, E., Godin, C., Lintunen, A. (ed.) Proceedings of the 7th International Conference on Functional-Structural Plant Models, pp. 95–97. Saariselkä, Finland (2013)Google Scholar
  54. 54.
    Santos, T.T., de Oliveira, A.A.: Image-based 3D digitizing for plant architecture analysis and phenotyping. In: Saúde, A.V., Guimarães, S.J.F. (eds.) Workshop on Industry Applications (WGARI) in SIBGRAPI 2012 (XXV Conference on Graphics, Patterns and Images). Ouro Preto (2012)Google Scholar
  55. 55.
    Scaramuzza, D., Fraundorfer, F.: Visual odometry (Tutorial). IEEE Robot. Autom. Mag. 18(4), 80–92 (2011). doi: 10.1109/MRA.2011.943233 CrossRefGoogle Scholar
  56. 56.
    Seitz, S., Curless, B., Diebel, J.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2006)Google Scholar
  57. 57.
    Shakarji, C.: Least-squares fitting algorithms of the NIST algorithm testing system. J. Res. Natl. Inst. Stand. Technol. 103(6), 633 (1998). doi: 10.6028/jres.103.043 CrossRefGoogle Scholar
  58. 58.
    Sirault, X., Fripp, J., Paproki, A., Kuffner, P., Nguyen, C., Li, R., Daily, H., Guo, J., Furbank, R.: PlantScan: a three-dimensional phenotyping platform for capturing the structural dynamic of plant development and growth. In: Proceedings of the 7th International Conference on FunctionalStructural Plant Models, pp. 45–48. Saariselkä, Finland (2013)Google Scholar
  59. 59.
    Snavely, N., Seitz, S., Szeliski, R.: Modeling the world from internet photo collections. Int. J. Computer Vision 80(2), 189–210 (2008). doi: 10.1007/s11263-007-0107-3 CrossRefGoogle Scholar
  60. 60.
    Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment a modern synthesis vision algorithms: theory and practice. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) Vision Algorithms: Theory and Practice. Lecture Notes in Computer Science, vol. 1883, book part (with own title) 21, pp. 153–177. Springer, Berlin/Heidelberg (2000). doi: 10.1007/3-540-44480-7_21
  61. 61.
    Tsaftaris, S.A., Noutsos, C.: Plant phenotyping with low cost digital cameras and image analytics. In: Environmental Science and Engineering (Subseries: Environmental Science), vol. 3, pp. 238–251. Springer, Berlin Heidelberg (2009). doi: 10.1007/978-3-540-88351-7-18
  62. 62.
    van der Heijden, G., Song, Y., Horgan, G., Polder, G., Dieleman, A., Bink, M., Palloix, A., van Eeuwijk, F., Glasbey, C.: SPICY: towards automated phenotyping of large pepper plants in the greenhouse. Funct. Plant Biol. 39(11), 870 (2012). doi: 10.1071/FP12019 CrossRefGoogle Scholar
  63. 63.
    Vadez, V., Kholova, J., Hummel, G., Zhokhavets, U., Gupta, S.K., Hash, C.T.: LeasyScan: a novel concept combining 3D imaging and lysimetry for high-throughput phenotyping of traits controlling plant water budget. J. Exp. Bot. 1–13 (2015). doi: 10.1093/jxb/erv251
  64. 64.
    Vogiatzis, G., Hernandez, C.: Video-based, real-time multi view stereo. Image Vision Comput. 29(7), 434–441 (2011). doi: 10.1016/j.imavis.2011.01.006 CrossRefGoogle Scholar
  65. 65.
    Wu, C., Agarwal, S., Curless, B., Seitz, S.M.: Multicore bundle adjustment. In: CVPR 2011, x, pp. 3057–3064. IEEE (2011). doi: 10.1109/CVPR.2011.5995552
  66. 66.
    Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000). doi: 10.1109/34.888718 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.Embrapa Agricultural InformaticsCampinasBrazil

Personalised recommendations