Plant Leaf Segmentation and Phenotyping Based on 3D Point Cloud

II. MATERIALS AND METHODS

A. Platform and Experimental Subjects

The processing unit is a desktop with an Intel Core i7-4770 CPU, 16 GB RAM, and an NVIDIA GTX 1080Ti GPU. The software environment includes Microsoft VS2015 with a PCL library [45] and VisualSFM [46], [47], which are all operated under Windows 10. In experiments, we used the camera (IMX378, Sony, Tokyo, Japan) from a cell phone (MI 5s, MI, China) to capture images of our experiment subjects for generating point clouds.

Two types of potted greenhouse ornamentals, Maranta arundinacea and Dieffenbachia picta (Fig. 1), are adopted as research subjects in this article. A total of four samples with two in each plant species are experimented. The leaves of the two types differ a lot in texture, color, and shape, so that they are good samples to test the generality of the proposed leaf segmentation and phenotyping method. Comparing to Dieffenbachia picta, samples of Maranta arundinacea are taller, and have denser foliages.

B. Framework

The proposed method can be divided into five stages, and the diagram of the framework is shown in Fig. 2. The first stage is to construct accurate 3-D point clouds of plants by utilizing the MVS technique. In this stage, images of the subject are first captured from various directions. Then, the scale-invariant feature transform (SIFT) descriptor [48] is used to detect key matching points and to search the correspondences among images. Later, a sparse point cloud can be generated via bundle adjustment [49]. At last, clustering views for multiview stereo (CMVS) [50] is applied to the sparse model for producing a dense point cloud of the subject plant. The output point cloud of the first stage contains a lot of background information and noise that may interfere with the leaf segmentation step that follows. Therefore, in the second stage, we preprocess the point clouds by removing noncanopy areas (e.g., pot and ground) and suppressing noise points.

In the third stage, we design a stems removal process which employs DoN [42] and Euclidean clustering to remove the stem system in the canopy. And after DoN, a neighborhood point search is applied to fill back some leaf points that are falsely removed together with the stems. In the fourth stage, the curvature feature of each point in the point cloud is computed first, and the points whose curvature values are higher than a threshold are removed from the point cloud because they are very likely to belong to the overlapping areas among different leaves. Afterwards, a region growing algorithm based on multifeatures is applied to segment individual leaves in the canopy. In the last stage, we measure phenotypic features such as the leaf area, leaf length, width, and leaf inclination angle for each segmented leaf of the sample plants in a fully automatic way.

For calculating the leaf area, we apply the facet oversegmentation algorithm [51] on each single leaflet; then each leaf surface is decomposed into many 3-D facets that are spatially flat and smooth. PCA [52] is applied for calculating the normal of each facet, and by rotating each facet to align its normal with the z-axis (the gravity direction) in the 3-D coordinate system of the total plant, we can project the 3-D facet onto the XOY horizontal plane as a 2-D manifold. After applying Delaunay triangulation [53] on the projected 2-D manifold of each facet, the area of a facet is approximated by the sum of all triangle areas. Furthermore, the area of a leaf is then estimated by summing up all its facet areas.

For calculating the length, width, and inclination angle of a leaf, we construct a 3-D minimum bounding box for each leaf. The orientation of the bounding box reveals the leaf inclination angle, and the dimension of the box packages its leaf length and width.