AI-based soft-sensor for shelf life prediction of ‘Kesar’ mango

This paper presents prediction of shelf-life of ‘Kesar’ cultivar of mangoes stored under specified conditions based on their respiration rate and ripeness levels. A deep-CNN was fine-tuned on 1524 image data of mangoes stored under different conditions to classify the ripeness levels of mangoes as ‘unripe’, ‘early-ripe’, ‘partially-ripe’ and ‘ideally-ripe’. CO2 respiration rate (RRCO2) was further calculated using principle of enzyme kinetics to establish a correlation between RRCO2 and ripeness levels. A Support Vector Regression model was employed to predict the shelf life and ripeness levels of mangoes under different storage conditions, thereby creating an AI based soft-sensor. The developed methodology can be used for other climacteric fruits besides mangoes. This solution can be used by producers and distributors for post-harvest handling of climacteric fruits like mango. It will also aid retailers in taking dynamic decisions with respect to pricing, logistics and storage conditions to be maintained to get the desired ripening rate, thus, contributing to reduction of wastage of fruits and subsequent economic losses. Article highlights Variation in CO2 respiration rate of ‘Kesar’ mangoes over different maturity stages were observed under different supply chain scenarios simulated in lab environment AI models were developed based on respiration rate and ripeness levels for prediction of shelf life of mangoes under different supply chain scenarios. These models once deployed helps all stake holders in fruit supply chain to take dynamic decisions such as repricing, recycling and repurposing based on the predicted shelf life thus minimizing wastage and maximizing profit. Variation in CO2 respiration rate of ‘Kesar’ mangoes over different maturity stages were observed under different supply chain scenarios simulated in lab environment AI models were developed based on respiration rate and ripeness levels for prediction of shelf life of mangoes under different supply chain scenarios. These models once deployed helps all stake holders in fruit supply chain to take dynamic decisions such as repricing, recycling and repurposing based on the predicted shelf life thus minimizing wastage and maximizing profit.


Introduction
Climacteric fruits like mangoes, papaya, banana etc. are plucked in unripe and allowed to ripen post-harvest [1]. Ripening of climacteric fruits is an irreversible phenomenon and is guided by a series of physiological and biochemical changes which lead to the transformation of a hard, unripe fruit to edible, ripe fruit with soft pulp and desirable taste and flavour [2]. Moreover, peel colour of climacteric fruits has a direct correlation with the fruit quality which also plays an important role in consumer acceptability [3][4][5]. For example, ripening of most of the mango varieties including 'Kesar' cultivar leads to the change in colour of peel from green to yellow. Ripe mangoes are soft to touch, with a pleasant aroma and exhibit a flavour often described as a combination of peach and pineapple [6]. However, non-climacteric fruits such as oranges, grapes, watermelons, litchis, strawberries, raspberries, cherries etc. are harvested only when they are fully ripe and do not demonstrate significant changes in peel colour [1].
India is the largest producer of mangoes in the world accounting to about 50% of worldwide production [7]. The leading mango producing states in India are Uttar Pradesh, Telangana, Maharashtra and Karnataka. Mangoes grown in India are widely exported to countries like UAE, Bangladesh, UK, Saudi Arabia, and Nepal (FAOSTAT). Delicious 'Kesar' cultivar of mangoes are grown in the states of Gujrat and Maharashtra and exported to different parts of the world owing to their excellent taste and flavour.
The most important factors determining the quality of mangoes are harvest conditions, optimum temperatures maintained during post-harvest handling and storage. Mangoes are very sensitive to cold temperatures and green unripe mangoes, when stored at temperatures below 10 °C suffer from chilling injuries like discoloration on the skin, pitting, uneven ripening, and poor flavour and colour development. Chilling injuries causes delay in the natural ripening process [4,6]. Ideally, mangoes should be stored at lower temperature only when they are fully ripe to retain their freshness for longer duration [3,6].
Owing to this ripening phenomenon, post-harvest handling becomes very important in mangoes [7,8]. Export of naturally ripened juicy mangoes to distant places pose difficulties as mangoes tend to become overripe and inedible when shipped over large distances, resulting in huge economic losses. Thus, to minimize economic losses due to over-ripening under harsh transportation conditions, fruit sellers prefer artificial ripening of unripe mangoes at destination before selling them to consumers [1]. The, producers, distributors, retailers, commercial packaging houses as well as pulping industries need to assess the ripeness level, predict the duration until they reach the ideally ripe stage and take dynamic decisions in real-time so as to reprice, repurpose, reroute, and recycle mangoes to reduce wastage and economic losses.
This process of dynamic decision making when enabled will further reduce the need of artificial ripening of mangoes by using harmful chemicals and assist key stake holders in judicious planning of logistics thus ensuring that the consumers receive the best quality mangoes in stipulated time.
Research has been conducted in the past to study the changes over time in the quality of mangoes [9,10]. Researches have also been done on compositional changes of mangoes as it degrades over time [11,12]. However, to the best of our knowledge, no specific study was found on the prediction of shelf life and ripening duration of 'Kesar' cultivar of mangoes under different storage conditions. This paper illustrates prediction of shelf-life of 'Kesar' cultivar of mangoes based on their respiration rate and ripeness level under different storage conditions. An inhouse sensing system and temperature controlled enclosure was developed to record the variation in released carbon di-oxide (CO 2 ), oxygen (O 2 ) intake and emitted ethylene (C 2 H 4 ), during respiration of mangoes along with the changes in their peel colour from green to yellow through different ripening stages under different storage conditions. Different storage conditions replicated were-(1) the conditions within a household refrigerator (4 °C), (2) ambient conditions (18 °C, 20 °C) and (3) elevated temperatures (25 °C, 30 °C) as experienced in large storage houses. Deep Convolutional Neural Networks (CNNs) 'VGG-16' , 'ResNet-50' and 'Inception-v3' pre-trained on natural image dataset 'Imagenet' was further fine-tuned on mango images to classify ripeness level of mangoes as 'unripe' , 'early-ripe' , 'partially-ripe' and 'ideally-ripe' . Respiration rate (RR CO2 ) was calculated based on the variation in released CO 2 and consumed O 2 using principles of Enzyme Kinetics and Michaelis-Menten (MM) equation. A correlation was established between RR CO2 and ripeness level obtained for different storage conditions. A Support Vector Regression model (SVM) was further employed to predict the shelf life of 'Kesar' mangoes based on its current RR CO2 and ripeness level when stored under different conditions. The models thus developed can work as soft sensor in prediction of shelf life of mangoes traversing through any node in supply chains.
The outline of the paper on use of AI based soft sensor for prediction of shelf life of 'Kesar' cultivar mangoes is as follows. Section 2 depicts the 'Material and Methods' . Sections 2.1, 2.2 and 2.3 presents the 'Experimental Methods' , 'Theoretical Considerations' and 'Computation Methods' respectively that were used for developing the AI based soft sensor for the prediction of shelf life. Section 3 depicts the Experimental and Computational 'Results' obtained on the predicted shelf life of mangoes under different storage conditions. Section 4 is the 'Discussion' section which presents the analysis of the results obtained and relevance of this research to the stake holders involved in fruit supply chain. Section 5 presents the 'Conclusion' .

Material and methods
'Kesar' cultivar of mangoes were classified into four ripening stages namely 'unripe'; 'early-ripe'; 'partially-ripe' and 'ideally-ripe' based on previously published study [13,14]. 'Unripe' mangoes refer to hard, green and unripe mangoes, 'early-ripe' were the predominantly green mangoes with yellow patches at early stage of ripening, 'partially-ripe' were predominantly yellow mangoes with green patches ripened partially and 'ideally-ripe' mangoes refer to soft yellow mangoes which are ideally ripe for consumption. Mangoes at different ripening stages were kept at 18 °C, 20 °C, 25 °C, 30 °C storage temperatures for calculating the shelf life. The 'ideally-ripe' mangoes were further kept at 4 °C, similar to the temperature observed in household refrigerators to calculate freshness duration at 'ideally-ripe' stage.

Experimental methods
An in-house internet of things (IoT) and Cloud enabled sensing system and temperature controlled custom enclosure were developed for the quantitative sensing and synchronized recording of the variations in consumed oxygen, released CO 2 and emitted ethylene along with the pictorial variations in terms of colour, texture and firmness. A customized storage enclosure of 40 L ( Fig. 1a) with an integrated sensor suite consisting of CO 2 , oxygen, ethylene, temperature and humidity sensors along with high resolution cameras, installed in multiple directions, were used for data collection. The temperature, humidity, light intensity and ventilation were controlled and maintained at different storage conditions. Mangoes were kept in this enclosure and variation in sensory parameters were recorded over time at a given storage condition.
The sensed voltage signals from the select sensors (CO 2 and O 2 ) recorded at periodic intervals were passed through antialiasing filters to remove unwanted high frequency noise signals and pre-processed by signal conditioning unit. Analog to Digital Converter (ADC) then converted these analog signals to digital signals which are then processed by a microcontroller unit and interfaced with laptop/iPad. Alarms are generated by the microcontroller in case of system malfunctions like temperature overload and improper ventilation which in-turn triggers the actuators to bring the system back to equilibrium conditions. The time series sensory data and images collected at periodic intervals were annotated with respect to number of days. The robust models were trained on these annotated data under different storage conditions to predict shelf life.
The schematic of the overall framework designed to predict shelf life of 'Kesar' cultivar of mangoes is presented in Fig. 1b.

Theoretical considerations
CO 2 respiration rate was calculated (RR CO2 ) by following the principles of enzyme kinetics and Michaelis-Menten (MM) equation which can be used to find RR CO2 of any fresh produce like climacteric fruits and vegetables. There are various types of respiration rate models presented in literature such as linear, polynomial, exponential and MM [15][16][17]. MM equation is formulated based on inhibition role of CO 2 which can be classified into four categories namely (1) no inhibition (2) competitive inhibition (3) uncompetitive inhibition (4) non-competitive inhibition are represented below [16] R is the role of CO 2 in respiration mediated via inhibition mechanisms, y O2 and y CO2 are the oxygen and   [15,18].
where R m represents the MM equation coefficients, R p is pre-exponential factor, E a is the activation energy, T is the storage temperature in Kelvin and R is the Universal gas constant (8.314 kJ/Kg mole K). The variation in RR CO2 was calculated over a duration of 24 h and was normalized by the volume of the custom enclosure and weight of the mango. The overall Eq. (6) used for computing RR CO2 is shown below [12] RR CO2 is the CO 2 respiration rate calculated in ml Kg −1 h −1 , R(t) and R(t-1) are the role of CO 2 in respiration mediated via inhibition mechanisms at time instance t and t − 1 respectively, ∆t is the periodic interval at which R is measured, V is the volume of the custom enclosure (40 L) and W is the weight of the mango which was calculated to be 257 g.

Computational methods
The coefficients of MM equation for uncompetitive inhibition namely V m , K m and K iu were calculated by fitting a nonlinear regression equation on the experimentally recorded gas concentrations [14]. These MM equation coefficients were further correlated to different storage temperatures using Arrhenius Eq. (5) [15,18]. MM equation and Arrhenius equation were combinedly used to calculate RR CO2 under different storage temperatures, consumed O 2 and released CO 2 .
Weights of different deep Convolutional Neural Networks (CNN) namely, ResNet-50 [19], Inception-v3 [20,21] and VGG-16 [22] without the top fully connected layer were considered for further fine tuning. A dense fully connected layer was added and trained on four ripeness levels of 'Kesar' mangoes namely 'unripe'; 'early-ripe'; 'partiallyripe'; 'ideally-ripe' using 'softmax' function to generate a fine tuned CNN model. The computational architecture for fine-tuning of CNNs is presented in Fig. 2a. All these CNN networks are pretrained on natural image dataset named 'Imagenet' [23]. Transfer learning and Data Augmentation were employed to overcome the lack of huge annotated dataset. Data augmentation dramatically expanded training and validation image dataset and improved robustness of CNN network for classification of ripeness level of 'Kesar' mangoes. The open source Tensor Flow [24] was used to implement and train the models on GPU. During training, the original input images were cropped and resized to the target input size of [224 × 224 × 3] with scale and aspect ratio augmentation [20,21]. The network was trained using 'SGD' optimizer [25] with a momentum of 0.9, learning rate of 0.0004 and batch size of 8. Other classification methods were tried to classify the ripeness level of 'Kesar' mangoes. However the pretrained CNN networks namely ResNet-50 [19], VGG-16 [20,21] and Inception-v3 [22] performed significantly better with improved classification accuracy as compared to other classification Research Article methods as they exploit hand-crafted features resulting in limited performance [26,27]. The fine-tuned CNN model predicted the ripeness level of an unknown mango image. A correlation was established between ripeness level and CO 2 respiration rate (RR CO2 ) under specified storage environment. Different ML based regression methods were employed to predict the shelf life of 'Kesar' cultivar of mangoes based on current ripeness level and RR CO2 under different storage conditions. The corresponding computational architecture for prediction of shelf life is presented in Fig. 2b. The Support Vector Regression (SVR) method performed better than other regression methods like decision tree. SVR was used instead of Support Vector Machine (SVM) Classifier to minimize error. The entire annotated dataset of 1524 samples was divided into training, validation and test data with 70%: 20%: 10% split, respectively.
The model was trained over training dataset for 20 epochs until validation accuracy and training accuracy converged and validation and training loss converged. The model was further used to calculate the prediction accuracies over training, validation and test dataset.

Results
'Kesar' cultivar of mangoes, classified into four ripening stages namely 'unripe'; 'early-ripe'; 'partially-ripe'; 'ideallyripe' were kept at 18 °C, 20 °C, 25 °C and 30 °C storage temperatures for calculating the shelf life. The custom enclosure with the integrated sensor suite recorded the synchronized variations in the intake of O 2 and release of CO 2 at periodic intervals under different storage temperatures mentioned above. The coefficients of MM equation computed for uncompetitive inhibition namely, V m , K m and K iu at different storage temperatures are presented in Table 1.
RR CO2 was finally calculated under different storage conditions with respect to the consumed O 2 and released CO 2 using the Eq. 6. The variation of CO 2 respiration rate over time (in mL kg −1 h −1 ) under different storage conditions is presented in Fig. 3.
The 'ideally-ripe' mangoes were further kept at 4 °C, similar to the temperature observed in household refrigerators and the same is presented in Fig. 4.
Deep CNN networks namely 'VGG-16' , 'Inception-v3' and 'ResNEt-50' were fine-tuned and trained for classification of 'Kesar' cultivar of mangoes into four different ripeness levels and the same is presented in Fig. 5a. 1524 images of 'Kesar' mangoes were collected and annotated for four ripeness levels 'unripe'; 'early-ripe'; 'partially-ripe' and 'unripe' based on average RGB scores of 84:102:24; 139:154:36; 179:154:29 and 225:145:7, respectively. CNN networks were trained on these annotated images. To avoid data over-fitting original dataset was extended by data augmentation including translation and reflection in vertical and horizontal directions. After data augmentation images were resized to 224 × 224 pixels for 'VGG-16' and 'ResNet-50' and to 299 × 299 pixels for 'Inception-v3' network. The training mechanism involved redefining of the proposed networks by back-propagation mechanism [24] which calculates the minimized square difference between the classified ground truth and predicted output iteratively. The fine-grained classification performance on different CNN networks is presented in Table 2.
The variation in training and validation loss and training and validation accuracy over 20 epochs on using 'VGG-16' is graphically represented in Fig. 5b.
The CO 2 respiration rate (RR CO2 ) of 'Kesar' cultivar of mangoes were further correlated to the CNN predicted ripeness level under storage temperatures of 18 °C, 20 °C, 25 °C and 30 °C and the same is presented in Fig. 6. Support Vector Regression (SVR) methods were further employed to predict the shelf life of 'Kesar' cultivar of mangoes based on current ripeness level and RR CO2 under different storage conditions and the same is presented in Fig. 7

Discussion
The observed release rate of CO 2 as shown in Fig. 3, increases up to a certain point and then decreases as the fruit respires. The rate of release of CO 2 increases with the increase of temperature, thus increasing the natural ripening rate of mangoes [15]. The CO 2 respiration rate increases until the unripe mangoes naturally ripen to 'ideally ripe' stage, beyond which the rate decreases indicating over-ripening of mangoes. At the peak of CO 2 release rate the mangoes are ideally ripe for consumption and most optimal in taste and flavour [16]. Also, as indicated by Fig. 3 the CO 2 respiration rate is maximum at 30 °C and minimum at 18 °C and it takes 4 days and 10 days respectively, for hard green unripe mangoes to reach 'ideally-ripe' stage. Hence, it is clear that the mangoes take sufficiently longer time to ripen when kept at lower temperature as compared to mangoes kept at elevated temperatures. So, based on the necessity of the stake holder, storage temperature can be altered to get the desired ripening rate.  As depicted in Fig. 4, it was observed that the mangoes maintained their freshness at 'ideally-ripe' stage for 4-5 days before being wrinkled and over-ripened. It can be noticed, that mangoes when preserved at 18 °C, 20 °C, 25 °C and 30 °C take 10, 8, 6 and 4 days, respectively to reach from 'unripe' to 'ideally-ripe' stage and beyond this duration they start to over ripen. However, when 'ideallyripe' mangoes are kept at 4 °C they retain their freshness with ideal taste, flavour and nutritional content for additional 4-5 days. For example, 'unripe' mango when maintained at a constant temperature of 18 °C, becomes over ripe after 10 days. Instead, if these mangoes are stored at 4 °C on the 10th day when they are 'ideally ripe' , the mangoes remain fresh for additional 5 days at the 'ideally-ripe' stage before becoming over ripe. This is due to the reduction in the CO 2 respiration rate and emission of ethylene at lower temperature which decelerates the natural ripening process.
As presented in Table 2 'VGG-16' showed the best performance as compared to the other networks. The graph in Fig. 5b confirms that there is no data over-fitting. Training accuracy increase from 8.41 to 98.31% and training loss reduces from 4.3593 to 0.1339 after 20 epochs. Validation accuracy increase from 30.41 to 98.96% following similar trend of training accuracy and converges with the training accuracy after 15th epoch. Validation loss also reduces from 3.6693 to 0.1187 after 20 epochs. The prediction accuracy using 'VGG-16' on training dataset was 99%, validation dataset was 99% and test data set was 96.2%.
As depicted in Fig. 6, the ripeness levels are numbered from 1 to 4 representing four ripeness levels 'unripe' , 'earlyripe' , 'partially-ripe' and 'ideally-ripe' , respectively. It is evident that with the increase in CO 2 respiration rate, ripeness level also increases which further corresponds to the increase in natural ripening rate. Avoiding the boundary conditions, it can be observed that change in CO 2 respiration rate (in mL/kg/h) between 20 to 30; 45 to 55; and 60 to 70 changes the ripeness level from 'unripe' to 'early-ripe'; 'early-ripe' to 'partially-ripe'; and 'partially-ripe' to 'ideallyripe' respectively. Further, when the CO 2 respiration rate is beyond 80 ml/kg/hr mangoes can be considered to be over-ripe.
As depicted in Fig. 7, the prediction accuracy of SVR model on test data was maximum at 20 °C (94.5%) and minimum at 30 °C (78.3%) and the training losses in the methods were 0.1123 and 0.6584, respectively. The prediction accuracy at 18 °C and 25 °C were 83.6% and 90.3% with a training loss of 0.4573 and 0.2219, respectively. The developed SVR model has better prediction accuracy in the temperature range of 18-25 °C as compared to higher temperature of 30 °C.
From the graphs in Fig. 7 it is evident that an increase in temperature proportionally increases the release rate of CO 2 from climacteric fruits like mango which in turn increases the natural ripening rate of climacteric fruits and in turn decreases the shelf life. The graphs in Fig. 7 thus presents the shelf life until which the 'Kesar' mangoes can be consumed with respect to its current ripeness level when maintained at a specified storage temperature. Beyond this predicted duration, the mangoes tend to become over ripe, wrinkled and unfit for consumption when maintained at the same storage temperature. For example, as shown in Fig. 7a, the shelf life of 'unripe' 'Kesar' mango releasing 18.6 mL/kg/h CO 2 at ripeness level '1' was experimentally found to be 12 days and predicted shelf life was 11 days when kept at 18 °C temperature. The predicted shelf life compares well in accordance with the experimentally observed shelf-life under all the four storage conditions. Most importantly, as observed in Fig. 7, the predicted shelf life by SVR model is less than or equal to the experimentally observed shelf life under storage temperatures of 18 °C, 20 °C, 25 °C and 30 °C.
Our work helps retailers, farmers, and distributors to take dynamic decisions on repricing and rerouting of mangoes based on the remaining ripening duration such that the stock can be cleared in time and wastage can be minimized. Further, this work assists in controlling and altering the modified atmospheric packaging (MAP) conditions of the packaged mangoes in bulk to get the desired ripening rate and accordingly enhance shelf life of packaged fruits. Moreover, in case of large consignments when bulk of mangoes are kept together it is extremely difficult to get images of individual mango especially when they are transported in trucks. However, in such scenarios inexpensive CO 2 and O 2 sensors can be easily deployed and CO 2 respiration rate can be measured. Once the respiration rate is measured in real-time, it can be correlated to the ripeness levels and dynamic decisions can be taken on the consignment to maximize profit and reduce wastage based on the ripeness level of mangoes. In order to use these soft sensors for field applications, a handheld device is further being developed to facilitate end users to easily ascertain the storage temperature, released CO 2 and consumed O 2 by the fruits. This portable device will be IoT enabled and will thus facilitate real-time remote monitoring of shelf life which in turn will assist the stake holders to take dynamic decision well in advance.

Conclusion
In this paper we have presented the prediction of shelf-life of 'Kesar' cultivar of mangoes based on their respiration rate and ripeness level under different storage conditions. The methodology presented for prediction can be used for prediction of shelf life of any climacteric fruits. Prediction of shelf-life involved recording of time series variation in emission of CO 2 and consumption of O 2 along with the variation of images using the custom designed infrastructure developed in-house. AI models (CNN and SVR) were trained on these collected data and images to predict shelf life of mangoes. These cloud based AI models acts as soft sensors for real life supply chain scenarios. Thus, it is implied that the predicted shelf life being less than or equal to the actual shelf life under different storage temperatures, will assist stake holders to take dynamic decisions like rerouting, repurposing, recycling and repricing in advance or at correct time thus contributing to reduction of wastage of mangoes and corresponding economic losses.

Conflict of interest Authors declare there exist no conflict of interests.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.