Introduction

The morphological characteristics of tree stems (or simply stem form or shape), including stem height, diameter, straightness, taper, and volume, are important consideration in timber forest management (Prendes et al. 2022). The straightness of standing tree stems is a critical component of the economic potential of young trees and of timber quality for mature trees. In particular, stem straightness is an important quality trait for selection during tree breeding research (Alcorn et al. 2007; Hai et al. 2008; Cameron et al. 2012; Ballesta et al. 2019; Woeste et al. 2021). Despite its importance, the measurement of tree stem straightness with visual techniques is subjective (Carino et al. 2006; Hamner et al. 2007) and locally biased because practical tools to assign absolute straightness values are not available. For example, Cameron et al. (2012) and Mora et al. (2019) divided stem straightness into six classes, Hai et al. (2008) and Luechanimitchit et al. (2017) used five classes, and Jensen and Löf (2017) and Woeste et al. (2021) used a binary system. Aside from bias caused by an individual rater’s interpretation of straightness categories, visual straightness ratings cannot effectively compare ratings across sites and over time.

Advances in ground-based Lidar (also referred to as Terrestrial Laser Scanning or TLS) and Structure-from-Motion (SfM) photogrammetry enable digital characterization of tree stem diameters and heights (Bienert et al. 2021; Dong et al. 2021; Prendes et al. 2022). Both Lidar and SfM techniques generate data that are subsequently analyzed using methods that generally include 3D point cloud formation, preprocessing, segmentation, stem cylinder reconstruction, interpolation (smoothing), and quantification (Raumonen et al. 2013). Some early-stage Lidar applications resulted in stem taper measurements with root mean squared error (RMSE) between 1.5 and 7.0 cm and diameter at breast height (DBH) RMSE ranging from 1.9 to 9.2 cm (Liang et al. 2014). Liang et al. (2014) compared Lidar and photogrammetric methods in measuring tree DBH; SfM was considered low-cost and low-weight, requiring only easy-to-use equipment. The accuracy of tree attribute estimates using SfM was acceptable (within practical tolerances) though lower than the tested Lidar techniques. Several types of platforms are used for Lidar measurements, including backpacks, handhelds, and unmanned aircraft vehicles (UAV) under or above tree canopy. Ground-based and above-canopy UAV Lidar data has shown promise for measuring stem DBH and taper, comparing with the measurements from above-canopy UAV Lidar (Hyyppä et al. 2020). Despite considerable interest in the use of ground-based Lidar for tree stem measurement, quantifying stem straightness has received little attention (Erasmus et al. 2018; McTague and Weiskittel 2021; Salekin et al. 2021; Prendes et al. 2022).

One quantitative definition of stem straightness is the maximum deviation from the stem’s center line perpendicular to a straight line (chord) joining the two center points at the two ends of the stem. Erasmus et al. (2018) adopted this definition to quantify straightness using ground-based Lidar to measure 18-year-old plantation grown Pinus patula. They used 3D coordinate-based vector equations to derive perpendicular deviations along a tree stem, and then calculated stem straightness based on the maximum deviation of any tree after multiple scans.

Prendes et al. (2022) assessed stem straightness by calculating the maximum sagitta, a value similar to deviation as described by Erasmus et al. (2018), and sinuosity (the actual path length divided by the shortest path length of a tree stem). Prendes et al. (2022) computed the stem straightness using two steps. They first divided a stem into 20 cm sections from 0.5 m above ground to 4.9 m height, and then determined the maximum deviation for each section in the same way as Erasmus et al. (2018). The second step was to standardize the maximum sagitta for the entire stem because scanned data may be incomplete for the upper part of a stem. Prendes et al. (2022) divided the sum of the lengths of the rectilinear segments that connect the centers of the stem sections by the straight-line segment that joined the end points of the stem. The minimum value for sinuosity is 1. Based on 385 sample trees [15-year-old pine (Pinus pinaster) in Northern Spain], the maximum sagitta was between 0.02 and 0.39 m and sinuosity ranged from 1.00 to 1.25. A comparison of their Lidar-based, maximum sagitta measurements with a three-class visual rating scale assessment showed high agreement. Sinuosity values averaged near to 1 (no sinuosity) with little variation, possibly indicating that sinuosity has relatively low power to discriminate stem straightness.

Both Lidar and SfM techniques involve sophisticated data processing. Eliopoulos et al. (2020) presented a cheaper, simpler, and faster approach to measure tree DBH using stereoscopic photogrammetry. The goal of this paper is to demonstrate how stereoscopic photogrammetry is integrated with depth-image computer vision to estimate stem straightness and to compare the digital output from this approach with a traditional visual rating.

Materials and methods

Surveying procedure

An Intel Realsense D455 camera was used to record video samples of the trees. The camera outputted multiple video streams: of importance is the depth video stream which was used to calculate various tree metrics. Our ForSense software suite, used to process tree measurements, was programmed primarily in C+ + . The Robot Operating System (ROS) was used to coordinate information sharing between ForSense and the depth camera and to record depth videos for archival or off-site processing. The ForSense software was developed independently of the D455 camera, so any stereoscopic depth camera can be used to capture images for analysis by ForSense after minor configuration changes.

A total of 160 black walnut trees from Purdue University were surveyed, including 106 trees from the Richard G. Lugar Forestry Farm and 54 trees from the John S. Wright Center, both located near West Lafayette, Tippecanoe County, in north-central Indiana, USA. The trees were growing on level ground and separated by about 5 m in row (Lugar) or 6 m (Martell). Trees at Lugar were growing with a competing understory while the trees at Martell were in a park-like setting where the grass was mowed. Manual measurements of tree diameters at the heights of 0.3 m (1 ft) (D1F), breast-height of 1.4 m (4.5 ft) (DBH), and 2.7 m (9 ft) (D9F) were taken as the baseline reference for stem diameters. Diameters were hand-measured using a caliper and a height pole with lines demarcating 0.3, 1.4 and 2.7 m heights (non-SE units were chosen because they are standard in US forestry). Stem quality and straightness data, previously rated visually by trained foresters, were used as the baseline reference for stem straightness. The stem quality data rated visually were the mean of two independent observers rating each tree stem on an ordinal scale of 1.0 = poor to 5.0 = excellent, resulting in nine possible categories (1.0, 1.5, 2.0, …, 5.0).

Depth video of the trees was taken using the D455 depth camera connected to a laptop. The camera was held at breast height and 2 − 3 m away from the tree stem. To capture as much of the tree stem as possible, the camera was oriented vertically, with the two stereoscopic infrared sensors located above/below each other rather than side by side. The surveyor held the camera and walked past the trees row by row at a comfortable speed of about 1 − 2 m s−1. The camera used ambient infrared light to automatically generate the image frames in the depth video stream. The outputted depth stream had a resolution of 720 × 1080 px at 30 frames per second; each pixel in a depth frame represented the distance between that point on the image and the camera in mm. During processing by ForSense each image was further cropped, limiting the area of interest to the middle 50% of pixel columns to prevent neighboring trees from appearing in frame and interfering with measurements. A dynamic depth cutoff was implemented to discard (set to 0) pixels further/closer than + / − 25% of the distance from the camera to stem at breast height.

Diameter algorithm

To calculate stem diameter, we slightly modified the method described by Eliopoulos et al. (2020). Within a depth frame, the diameter of the tree stem at a particular row in the depth image was calculated using the width of the stem and the distance from the camera to either side of the stem. For each row of pixels in the frame, stems were identified as a continuous segment of non-zero pixels, with the first and last pixel on the segment corresponding to the edges of the stem. If there were multiple segments in a row indicating a stem, the segment closest to the middle pixel row was selected as the correct stem segment.

Tree diameter computations were based on the following geometric model (Fig. 1): assuming tree stem cross sections at each pixel row are perfect circles, let A denote the center of the circle, C denote the camera’s location, D and E denote the edges of the stem, and F be the intersection between AC and DE. Let \(a=\frac{\mathrm{CD}+\mathrm{CE}}{2}\) be the average distance from the camera to the visible edges of the stem, \(b=\frac{\mathrm{DE}}{2}\) be half the distance between the detected stem edges D and E, and \(r=\mathrm{AD}=\mathrm{AE}\) be the radius of the stem (Fig. 1).

Fig. 1
figure 1

Geometric relationships between a tree diameter and depth image parameters (Eliopoulos et al. 2020)

Full size image

Because ΔCFD and ΔCDA are similar triangles, \(\frac{a}{b}=\frac{\mathrm{AC}}{r}\), \(\frac{{a}^{2}}{{b}^{2}}=\frac{{\mathrm{AC}}^{2}}{{r}^{2}}\), \(\frac{{a}^{2}}{{b}^{2}}=\frac{{r}^{2}+{a}^{2}}{{r}^{2}}\) (Pythagorean theorem on ΔCDA), \({a}^{2}{r}^{2}=\left({r}^{2}+{a}^{2}\right){b}^{2}\), \({r}^{2}=\frac{{a}^{2}{b}^{2}}{{a}^{2}-{b}^{2}}\), thus

$$d=2r=2\sqrt{\frac{{a}^{2}{b}^{2}}{{a}^{2}-{b}^{2}}}$$
(1)

where, d is stem diameter.

The middle row of a depth image was assumed to correspond to the 1.4 m mark, and DBH calculations were made using that row. An approximate correspondence between depth pixels and physical distance was calculated by averaging depth values of the middle row of pixels in the stem. This correspondence was then used to calculate 1.1 m below and 1.3 m above the middle row to find the rows corresponding the D1F and D9F, respectively. Each depth frame produces one D1F, one DBH, and one D9F estimate. For each tree stem detected, depth-derived measurements for each of D1F, DBH, and D9F were averaged across multiple frames containing the same tree after outliers were discarded using the 1.5 IQR rule.

Straightness algorithm

The straightness of the stem section between 0.3 m (1 ft) and 2.7 m (9 ft) was measured from the depth frames (Fig. 2). These values were selected because a 2.4 m (8 ft) log meets a common standard in the US, but any heights could be stipulated. At the 0.3 and 2.7 m pixel rows, pixels corresponding to the edge of the stem were identified in a manner identical to that used during diameter measurement. Their x-axis values were averaged to obtain the stem midpoints at 0.3 and 2.7 m, respectively. A “straight line” connects these 2 midpoints. For each pixel row between 0.3 and 2.7 m, the midpoints of the stem edges were similarly calculated and connected to form a “center line”. The straightness of a tree was defined as one minus the ratio of the largest horizontal deviation from the (ideal) straight line to the (actual) center line-called maximum deviation (MD) and the diameter of the stem at the pixel row in which the maximum deviation occurred (DMD). Stem straightness was computed as

Fig. 2
figure 2

Illustration of tree stem straightness computation

Full size image
$$\mathrm{Straightness}=1-\frac{\mathrm{MD}}{\mathrm{DMD}}$$
(2)

The straightness values of a tree stem fell within the range (− inf, 1), where 1 denoted a perfectly straight stem and values < 0 denoted a crooked stem of no lumber value. Each depth frame produced one straightness value. Because the depth images were captured as the camera moved past the trees, each stem’s straightness was computed over multiple perspectives. The straightness of a tree was defined as the lowest straightness value observed across multiple depth frames of the same tree after outliers were discarded.

Consistency and repeatability

To determine if the method above produced repeatable outcomes across different measurement events, approximately half of the trees were chosen at random to be resurveyed and their measurements recomputed. Subsequently, the mean absolute error (MAE) and RMSE between the 1st and 2nd measurements were calculated.

Results

Of the 160 trees surveyed, 136 trees were successfully processed using ForSense to run on a regular laptop computer, requiring 18 min and 38 s of footage. Of the 24 trees we could not process, some were inaccessible to the camera operator and others were obscured by underbrush or the canopy of neighboring trees. The diameter and straightness values for individual trees were computed in real time.

The black walnut trees at Lugar Farm had an average diameter of 15.1 cm as measured using a caliper (Table 1) while ForSense DBH estimates averaged 14.3 cm. When the ForSense-based estimates of D1F, DBH, and D9F were compared to the caliper derived measures, MAE values ranged between 1.2 cm and 1.6 cm depending on the height at which the estimate was made, while RMSE values ranged between 1.5 cm and 2.7 cm. The depth-derived DBH measurements had the smallest MAE and RMSE. D9F estimation using the depth camera was less accurate than D1F or DBH estimation (Table 1) (Fig. 3).

Table 1 Absolute and relative differences in diameter measurements between depth imagery and manual methods for trees in Lugar Farm (n = 106)
Full size table
Fig. 3
figure 3

Comparisons of the values of D1H (a), DBH (b), and D9H (c) estimated using depth imagery (y-axis) and manual methods (x-axis)

Full size image

Stem straightness estimated using ForSense at both Lugar Farm and Martell Forest was positively correlated with the visual rating (Figs. 4 and 5). Most of the trees surveyed showed straightness values of 0.70 − 0.89 (Fig. 5). In general, straightness values from 0.90 − 1.00 denote (near) perfectly straight trees, 0.80 − 0.89 denote moderately straight trees with some imperfections, 0.70 − 0.79 denote trees with a slight curvature, and 0.60 − 0.70 denote crooked trees. Straightness values of 0.59 or less denote highly crooked stems, usually with at least one visible bend in the stem (Fig. 6).

Fig. 4
figure 4

Comparison of tree stem straightness based on analysis of stereo depth images (straightness value) and visual ratings of foresters for trees at Lugar Farm (a) and Martell Forest (b)

Full size image
Fig. 5
figure 5

Frequency of trees in straightness classes as determined using maximum deviation derived from stereo depth images at Lugar Farm (a) and Martell Forest (b) as well as straightness rating classes by human at Lugar Farm (c) and Martell Forest (d)

Full size image
Fig. 6
figure 6

Paired illustrations of tree stem straightness. The left picture is a color image for human reference. The right picture is a ForSense output of a depth image. Shown are trees with straightness values of 0.42 (A), 0.58 (B), 0.70 (C), 0.83 (D), 0.88 (E) and 0.94 (F). White outlines on either side of the stems trace what the software has identified as the stem edges. The two lines going down the stem are the center line of the stem and a straight (ideal) line. The horizontal line represents the row with the greatest difference between the straight line and the center line

Full size image

The ForSense estimates of D1H, DBH, and D9H were highly consistent between the 1st and 2nd runs (Fig. 7). We observed a sub-centimeter difference in diameter when measuring the same tree multiple times (Table 2). Similarly, the straightness computations were highly consistent between runs; straightness values from different runs differed by + / − 0.03 on average (Table 2).

Fig. 7
figure 7

Comparisons of the values of D1H (a), DBH (b), D9H (c), and stem straightness (d) between two measurements using ForSense

Full size image
Table 2 Difference between first and second replication of using depth imagery to measure diameter at three heights and to calculate straightness (N = 43)
Full size table

Discussion

Limitations of the ForSense system

This research demonstrated that a simple, inexpensive depth camera and the ForSense software can produce accurate, consistent estimates of stem diameter and stem straightness, comparable to those produced using SfM photogrammetry and ground-based Lidar, at much less cost, and in real time. The estimates of tree diameters and straightness were not only comparable to manual measurements but also repeatable. We believe this system has immediate application for monitoring tree development over time and comparing tree attributes across sites. The initial implementation of this system demonstrates both its potential and its current inadequacies.

The accuracy of any camera-based measurement system is reduced by factors that alter its sensors’ presumed position. If the camera’s orientation is not parallel to the tree stems or if the recorded footage is shaky, the estimated diameter is biased upward because the measured segment will correspond to an elliptical stem cross section rather than an idealized circle (Fig. 1). Error of this kind can be mitigated by attaching the camera to a gimbal mount that helps stabilize the camera in a vertical position relative to the ground. Camera misalignment error can be removed digitally by employing an additional algorithm that measures and accounts for the stem angle relative to the camera.

Because we estimated diameter and straightness from depth frames, the camera requires a clear line of sight to the tree stem. ForSense makes errors in measuring stem diameters or straightness when the software is unable to identify the stem outline. Bushes, undergrowth, canopy cover, and other visual obstructions can significantly degrade the algorithm’s accuracy, particularly with regards to D1F and D9F. Foresters rating straightness visually are confronted with the same limitation. ForSense will produce better data if brush or undergrowth is cleared from around the trees before they are measured (Fig. 8a). Because Intel Realsense depth cameras (to which the D455 belongs) uses stereoscopic infrared (IR) sensors for depth detection, conditions that interfere with the sensing of the natural infrared signal (sunlight reflecting from objects) will also degrade the data stream and the quality of the data. As a result, the system may produce poor data in low-light situations due to dense canopy, inside growth chambers or greenhouses, at dawn, dusk, or at night. Conversely, if the sun is behind an object being measured and is within the line of sight, the intense blast of infrared will oversaturate the sensor, obscuring the stem outline and the data for those frames will be degraded (Fig. 8b).

Fig. 8
figure 8

Illustration of sources of error when using stereo cameras and ForSense software. A tree with nearby undergrowth, leaves, and branches that are in a similar depth plane as the target (a); A tree stem partially obscured by IR saturation due to the sensor being pointed directly at the sun (b)

Full size image

Our experiments showed clear agreement between a tree’s straightness value and human visual rating (Fig. 4), but the correlation between the two measures was not perfect, especially for visual ratings of 4.0 or higher. We believe this finding implies limitations in the efficacy of subjective visual ratings. Visual raters may not be able to reliably distinguish mostly straight from perfectly straight trees.

Absolute versus relative measures

Although the trees at Lugar Farms and Martell Forest shared similar straightness value ranges, the Martell trees received a lower human visual rating score than their Lugar counterparts (the Martell straightness boxplots are left shifted compared to Lugar’s). This outcome points to another advantage of the straightness computation over human visual rating-consistency across time and space. Even if human visual ratings by one rater are consistent within a plantation over time, and even if all the trees are evaluated by the same raters, there is a high risk that a rater will be biased by covariates such as the difficulty of terrain, weather, or the time of day. It is difficult for a rater to avoid a central tendency in assigning ratings, resulting in kurtosis, and it is difficult to avoid using other trees in the vicinity for comparison, adding spatial autocorrelation that may not be removable by analysis. On the other hand, the straightness values, while no doubt subject to errors, is likely to produce data that permits consistent comparisons of trees in different plantations over time. The ForSense system also retains a standard, RGB and a depth image of every tree, which simplifies examination of outliers and individual data points that do not appear to agree with previous measurements.

Stem volume

Because of the algorithm we used to estimate the straightness values is calculated using the maximum deviation from the center line and straight line (Figs. 2 and 9), it can be applied to estimate harvestable lumber volume (\({V}_{H}\)) as a percentage of the stem’s gross volume (\({V}_{G}\)). A straightness value of 1.0 means that (nearly) all the measured stem can be converted to lumber, while a straightness value equal to or less than 0 would indicate it is impossible to cut a straight plank across the length of the stem. A linear interpolation may be applicable to obtain \({V}_{H}\) as follows:

Fig. 9
figure 9

Illustration of harvestable lumber volume (green) as a function of stem volume (brown). As the curvature of a tree’s stem increases, the fraction of harvestable lumber (indicated by the straightness ratio) decreases

Full size image
$${V}_{H}={V}_{G}\times \mathrm{Straightness}$$
(3)

An empirical examination of the relationship between straightness values and true harvestable log volume is currently beyond the scope of this paper; an experiment of this type would involve destructive sampling.

Because the straightness value is sensitive to the diameter of the tree being measured (the denominator of the straightness, Fig. 2), it is positively correlated with tree diameter. From the perspective of forest mensuration, this artifact captures the biological and commercial reality that log value does not scale linearly with diameter. Among trees of similar degrees of curvature, thinner (often younger) trees will be penalized more than thicker ones. Imperfections on the tree stem such as irregular bulges or branch stubs will also cause the centerline to deviate from the straight line and reduce a tree’s straightness score even if these bumps have little effect on volume (Fig. 10). Imperfections of this type, which are more common on younger trees, can reduce the effectiveness of the straightness value as an estimator of harvestable lumber. However, since irregularities in a stem almost always decrease the value of a log, the ForSense straightness model may still serve as a realistic estimator of overall log value.

Fig. 10
figure 10

A tree’s straightness ratio is reduced due to a branch stub on its stem

Full size image

In the future, a recursive extension of the straightness algorithm may be implemented to calculate the maximum harvestable lumber from a crooked stem that would otherwise hold little or no lumber value. Rather than computing the straightness value for the entire stem from top to bottom, the algorithm could segment the stem into smaller yet straighter segments. Using a minimum segment size of some user-supplied value, it would be possible to treat the harvestable log value as an optimization problem (in this case, maximization) with the objective function being the recursive application of the straightness algorithm on the smaller stem segments (Fig. 11).

Fig. 11
figure 11

By segmenting the stem into smaller, straighter sections, it is possible to increase the percentage of harvestable lumber from a highly curved log

Full size image

Similar developments in Lidar and SfM

Digital approaches to stem straightness measurement have only begun to appear in literature (Erasmus et al. 2018; Prendes et al. 2022). These studies utilized 3D point clouds to determine stem curvature. We could not find similar research based on SfM point clouds.

The main advantage of a 3D-point-cloud-based approach to stem curvature estimation is its ability to detect curvature in the direction towards/away from the sensor. In contrast, our current stem straightness algorithm only analyzes curvature 2-dimensionally. Curvature towards/away from the sensor is accounted for by analysis of the stem from multiple perspectives as the camera moves toward and then past a stem. Given that our ForSense works on depth images, a future iteration of the algorithm may also include the stem pixels’ depth values to achieve a true 3D analysis.

The other advantage to point-cloud-based stem analysis is its ability to filter out non-stem datapoints. For example, Liang et al. (2014) recorded point clouds of an entire tree, including branches and leaves in the canopy. The stem points were filtered out from the point cloud from which a stem model was reconstructed, free of non-stem elements. In contrast, our system's performance degrades unusably when non-stem elements of a tree are recorded, as discussed above in Fig. 8a.

A drawback to point clouds is their high computation cost, both in memory and runtime. Point clouds’ runtime and required space increase cubically with resolution, versus depth images, for which computational costs increase with the square of resolution. Once a dense, filtered point cloud is achieved, however, straightness extraction is relatively straightforward.

In view of the methods employed by Erasmus et al. (2018) and Prendes et al. (2022), our ForSense algorithm is comparatively straightforward. More sophisticated algorithms for straightness and log volume estimation using stereo depth imagery are on the horizon, but require additional research.

Conclusion

We demonstrated the use of a computer vision system to automate the measurement of tree stem diameter and straightness in two black walnut plantations, although the system need not be limited to this context. The system uses a stereo camera to capture depth images or depth videos with embedded pixel values that indicate the distance from the camera to various points in a depth frame. The system is highly accurate (less than 2 cm MAE) and its estimates of straightness showed a strong correlation with human visual ratings of stem quality. The system is also consistent, achieving a sub-centimeter difference between repeated measurements of diameter, and + / − 0.03 when calculating straightness values. The system is time-efficient compared to hand measurements, and it produces repeatable data in real time, supporting on-site decision-making by forest managers. The system is not computationally intensive and it can be run on a regular laptop. The next step for this system is to pair a tree’s automated measurements with its geo-spatial data and to automate data entry into a database.