Determination of Multi-Component Failure in Automotive System Using Deep Learning

The introduction of high-bandwidth communication systems for vehicle-to-vehicle and vehicle-to-infrastructure connections is enabling new technologies and services including optimal traffic control, car-sharing service, and connected and automated vehicles [1–5]. An area of interest that has not been explored much, however, is utilizing the data collected from these vehicles for predictive diagnostics or prognostics of the vehicle components [6]. Being able to predict component failure of a vehicle would be of great benefit as it could allow for reduction of maintenance time and effort, lower operation costs, and improved user safety [7].

The standard approach for handling component failure in automotive systems has been to employ a preventive maintenance, or scheduled maintenance program. Utilizing statistical data, preventative maintenance can be implemented such that critical components are replaced before failures typically occur [7]. However, due to the variation in each individual vehicle, it is difficult to apply this approach to a fleet of vehicles, nor is it economically viable to replace every component on a staggered schedule. For such cases, reactive maintenance can be employed [7]. In reactive maintenance, the failure of vehicle components is detected using an onboard diagnostic system [8]. For example, electronic control units (ECU) utilize heuristic and model-based methods to determine what constitutes a failure [9]. Then, techniques such as failure mode and effects analysis are applied utilizing sensor data and analytical models to determine potential failure modes of the components. The resulting failure modes and their related parameters are used in rule-based algorithms in onboard diagnostic systems [10,11]. This is a general approach without utilizing any particular data unique to a specific vehicle to generate appropriate predictions.

Although preventive and reactive maintenance approaches are currently prevalent in the automotive industry, predictive maintenance (PdM) is anticipated to offer considerable advantages, including reduced machine downtime and maintenance costs [12]. For this reason, PdM has attracted significant interest from researchers in the field of automotive systems. However, PdM has not seen any real application on the individual vehicle scale due to the difficulties associated with data collection [12,13]. Obtaining the necessary data for automotive systems requires the following at minimum: numerous sensors to measure physical signals, a means of transmitting and storing this data, expert knowledge of the system, and sufficient vehicle operation time to cover the progression of a failure mode from beginning to end [7]. Obtaining this data is further complicated by the fact that the lifecycle of any set of vehicles can involve dissimilar drive cycles and environments, which can have a significant effect on the progression of failure. For example, it has been shown that adding geographical data can provide more accurate maintenance scheduling for fleet management [14]. At the component level, active failure detection using machine learning methods has been applied in some studies [6,7]. Among various automotive components, batteries have been the primary focus for hybrid-electric and fully electric vehicles [15–17]. Failures of other mechanical components such as bearings, brakes, gearboxes, and suspension systems have also been investigated. However, the aforementioned data limitations remain unresolved, and these limitations have led to a reliance on simplified experiments or models as a workaround [18–21]. Due to these challenges, the implementation of predictive maintenance for individual vehicles has not been extensively researched, making it the primary focus of this paper.

There have been many different data-driven approaches for component diagnostics and prognostics to date. These include physics-based models, knowledge-based approaches, statistical methods, machine learning approaches, and deep learning approaches [22–28]. However, most of the approaches found in the literature involve either binary or multi-class problems focused on isolated failure modes. Such binary and multi-class problems operate under the assumption of mutual exclusion, which restricts their ability to classify concurrently occurring component failures effectively. The research problems that do include multiple failure modes have treated them as a multi-class problem, which is not a viable approach as the number of components, failure modes of each component, and combinations of the two increase to the full scale on a real-world vehicle. In this paper, it is hypothesized that a multi-label approach has the potential to resolve this issue. While the multi-class classification picks only one output class requiring 2ⁿ different output nodes for n different failing components, the multi-label classification will generate multiple output classes using just n different output nodes. This is because, in a multi-label approach, a single data case can correspond to multiple labels representing different failed components. However, a general approach of applying multi-label methods to time-series data, particularly automotive data, has not been studied to date. In addition, the approach of reshaping such time-series data into images for use in convolutional neural networks is lacking in the literature. Therefore, this paper demonstrates the feasibility of such a general approach by evaluating the performance of various pretrained learning models using automotive system data. This serves as an example for potential diverse applications in future work.

The goal of this paper is to propose a new approach to determine the binary failure states of multiple components in an automotive system by employing transfer learning with time-series data. To accomplish this goal, an automotive engine system with at least one existing component failure is considered in this paper. To obtain the necessary system data, a high-fidelity diesel engine simulation model employing an exhaust gas recirculation (EGR) system and a variable-geometry turbocharger was developed in commercial software, gt-suite, as shown in Fig. 1. Within this model, four subsystems are assumed to have a single potential failure mode: the EGR, the compressor (Comp), the injectors (Inj1, Inj2, Inj3, Inj4), and the intercooler (Inter). The EGR is assumed to fail due to accumulation of soot along the EGR inner wall. For the modeling purpose, this soot accumulation is assumed to be of uniform thickness along the EGR pipe and remains constant throughout each simulation. The compressor is assumed to fail due to progressive buildup of obstructions in the intake filter, represented as a decrease in the intake pipe diameter. The intercooler is assumed to fail due to the accumulation of debris and other material, represented as an obstruction on the intake side of the intercooler channel. Finally, the injectors are assumed to fail over time due to clogging from impurities in the fuel and exhaust gas particulates, represented by a uniform restriction in the diameter of the fuel injector output channels. Each individual injector is considered to have its own unique failure mode, meaning that there are seven total individual failure modes considered in this study.

In this study, the steady-state behavior of the system is of primary interest since it is regarded that steady-state behavior would provide the best means of isolating the effect of each failure mode. This means that the possible effects of transient behavior during drive-cycle simulation need to be minimized. To accomplish this, the engine simulation model is gradually brought to a target speed over 2 s and then held constant thereafter. Each specific steady-state engine operation with a combination of failure modes is then simulated for a minimum of 15 s, with a step size of 0.01 s, until steady-state is reached or a maximum simulation time of 2 min has passed, whichever occurs first. In the case that steady-state cannot be reached in this time frame, it is assumed that it is not possible to reach a true steady-state. Ten representative cases for steady-state engine operation, shown in Table 1, are chosen to cover the entire operational range of the engine. For the simulation study, the EGR failure due to the internal soot accumulation is modeled as decreasing diameter of the EGR pipe from valve to outlet as described in Table 2. To prevent the simulated gas flow from reaching supersonic speed, the EGR pipe clogging is limited to a maximum of 80%, which is regarded as the component failure point.

To include the thermal effect of accumulated soot in the simulation, the overall thermal conductivity in the pipe is modeled based on a pseudo-composite “material” consisting of deposited soot and stainless-steel pipe. Utilizing a lumped parameter modeling approach, the overall thermal conductivity is calculated as the weighted mean of the stainless steel’s thermal conductivity and the soot’s thermal conductivity, which is assumed to be 1 W/mK [29]. The density of the composite material is calculated by dividing the total mass by the combined volume of the materials. Utilizing these two material effects, the engine model is run for each case shown in Table 2 for the ten operational conditions specified in Table 1. Similarly, the diameter and respective failure percentage of the compressor, intercooler, and injector are shown in Tables 3–5, respectively. The thermal characteristics of these components are ignored as the failure modes involve only clogging obstructions at specific points in the channel.

Utilizing the failure conditions shown in Tables 2–5, various combinations of multiple component failures can be simulated. Each combination of failing components, with varying health conditions (HC), simulated under each specific operational conditions specified in Table 1 represents a specific simulation. These simulations produce time-series data of 672 unique signals representing sensor output and other information from the simulation model. For example, these signals include the engine speed, engine torque, spark timing, and EGR valve opening, which can be obtained from the ECU and other control units in a vehicle. As previously mentioned, the cases where no component failure occurred are not included in the data as the no-failure case represents a mutually-exclusive case.

As the goal of this study is to predict the failure modes, independent of time scale, a convolutional neural network (CNN) is chosen to classify the aforementioned time-series data. To utilize this data in a CNN, a process has been developed to transform each simulation dataset into an image. While transforming time-series data into images has been reported in the literature, the approach presented here deviates from conventional approaches [30,31]. To accomplish this, the global maximum and minimum values of each signal are obtained and utilized to normalize each respective time-series data signal to the range of 0–255. Then, a sliding window average is used to compress each time-series signal to 224 instances in time regardless of the overall signal length in time. From the 672 normalized and compressed time-series data signals, 224 signals are randomly selected to form a gray-scale image of 224 × 224 pixels. Then, an additional 224 signals are randomly selected to form the second gray-scale image, and the remaining 224 signals are used to form the third gray-scale image. In this manner, three gray-scale images of square size are formed and they are designated as red, green, and blue (RGB) color images. This approach is adopted to harness all the information encapsulated in the 672 signals, and present it in a format typically recognized as input by CNNs. With the three images combined, each simulation data set can be completely represented by an RGB image of 224 × 224 pixels as shown in Fig. 2. The image data generated in this manner contain diverse time scales from 15 s to 120 s depending on how quickly steady-state is reached in each simulation. Since CNNs can extract hidden characteristics in the data regardless of the time scale that the image represents, the data compression does not deteriorate CNNs’ classification capability.

A drawback of such a method, however, is that it is challenging to obtain sufficient amount of data for the training of machine learning-based prediction models. The deterministic simulation model shown in Fig. 1 does not provide sufficient randomness between simulations, and traditional image transformation techniques such as rotation are not applicable as they inherently mischaracterize the dataset. To mitigate the issue of the limited and homogeneous data size, white Gaussian noise is added to each data signal with a signal-to-noise ratio randomly selected between 40 dB and 60 dB. This is done ten times for each original image, increasing the total number of images from 1660 to 16,600.

One characteristic of this RGB image formation method, as shown in Fig. 2, is the lack of large variations in the data. Most parts of the image are white due to the combination of signals near a steady-state maximum value. As a consequence, the areas of meaningful information on a single image are relatively small and narrow, making it difficult to develop a prediction model with the optimal hyperparameters and well-tuned model parameters using the image data only. Therefore, a multi-label transfer learning approach utilizing pretrained CNNs is applied to develop failure modes classification models in this research. Since this data/image type represents a unique input for publicly available CNNs pretrained on photographs of real-life situations, a comparative analysis to determine how such data would perform is conducted by retraining the following 14 CNNs with the data: AlexNet (Alex), VGG, ResNet (Res), SqueezeNet (Squeeze), DenseNet (Dense), InceptionNet V3 (Incept), GoogLeNet (GoogLe), ShuffleNet (Shuffle), MobileNet (Mobile), ResNeXt (NeXt), Wide ResNet (Wide), MNASNet (MNAS), EfficientNet (Efficient), and RegNet (Reg) [32–45]. For each network, the first half of the layers have their weights frozen, the final half of the layers are unfrozen, and the classification output layer is adjusted to fit the new output space. If a network cannot be divided evenly, the extra layers are frozen. Then, each pretrained network is retrained over 30 epochs with the image data set from the engine model with a distribution of 70% for training (11,620), 20% for validation (3320), and 10% for testing (1660). This process is repeated five times with different validation and training sets each time, with the results averaged to account for overfitting. To center the pixel values in the images, and reduce the possible range of the network features, the pixel values in the images are normalized by subtracting the image mean and dividing by the standard deviation obtained from the ImageNet dataset. The mean and standard deviation of the RGB images are [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225], respectively. The batch size for each network is chosen to be 32, while the learning rate is selected as 1 × 10⁻⁴. An adam optimizer is employed for training each network.

Due to the nature of multi-label classification, various complications exist in data preparation including inter-class imbalances due to label co-occurrence and the positive-negative label imbalance [46,47]. As shown in Fig. 3, there are significant imbalances in the occurrence of labels. However, unlike multi-class imbalance, labels cannot be guaranteed to be balanced during preprocessing without causing imbalance in the classes due to the uneven representation of labels within the classes themselves, as shown by the 20 representative multi-mode failure classes in Table 6. For example, while Class 1 has one label representing the Intercooler failure mode, Class 17 also shares this label with the addition of the Compressor label. As a result, upsampling for balancing labels will likely lead to greater skew in the underlying classes within the data. Similarly, common techniques such as SMOTE to upsample minority classes, can lead to greater label imbalance due to the uneven number of labels contained within individual classes [48]. This is further complicated within this data as two additional class structures, the 166 health condition cases and the ten times operational range multiplier, are also hidden within the data as shown in Fig. 4. As the health condition classes and operational range condition are balanced within the original data, attempting to balance either labels or failure mode classes will cause all other conditions to become imbalanced, as demonstrated in Fig. 4.

For these reasons, it is a common practice to manage label imbalance within the loss function during training, such as with weighting or similar processes. If the focus of training is on the label-level utilizing such a process, the imbalance in the classes can be ignored if there is sufficient class representation within the data to allow an approximately even split between the training, validation, and test data. As shown in Figs. 5–7, this distribution of combinations of labels is relatively consistent within the three sets for training, validation, and test. Therefore, this inter-class imbalance is ignored for this study and label-based weighting is applied. However, weighting the loss function only is considered insufficient to account for the label imbalance given the dependency of the labels and the class distribution. This is demonstrated by the positive-negative label imbalance, as the data are biased toward negative labels (62.3% of labels are 0 representing the healthy cases). Therefore, the optimized asymmetric loss function developed by Ben-Baruch et al. is also utilized to reduce the effect of imbalance during training [47]. This loss function utilizes the ratio of positive and negative labels within an image to dynamically down-weight trivial classifications to improve the probability of learning minority label cases.

For the evaluation of the performance of different CNNs, the following metrics are used: the precision-recall area under the curve (PR AUC), the receiver operator curve area under the curve (ROC AUC), the mean average precision (mAP), the exact match ratio (EMR), the Hamming Loss, and the F1-Score [49–52]. The PR AUC and ROC AUC are determined from numerical integration of their respective curves via a trapezoidal method, while the mAP is defined as the average of the PR AUC of each individual label for a particular CNN. The EMR, Hamming Loss, and F1-Score are presented as well for the respective precision and recall values to describe the nuances in behavior between the different CNN models.

The primary metric used for describing classification performance in this research is the PR AUC, which is commonly referred to as average precision (AP). However, as there is imbalance in the labels, it is necessary to normalize the PR AUC for each label to compare the relative performance of training. This is accomplished by determining the unachievable region of each PR curve based on the proportion of positive labels within the label set, as shown in Table 7 [53]. From this table, considering that a perfectly balanced label has a value of 0.5, it is clear that the injectors have higher average skew than the other components, meaning the minimum PR AUC for these components is lower. Normalized PR AUC results from the testing set, shown in Table 8, correspond well with the label distribution in Fig. 3, with the models having the highest by-label performance placed in the order of intercooler, compressor, EGR, and then injectors. The average ROC AUC results, shown in Table 9, also correspond well with the label distribution. In general, the CNNs are more proficient in properly classifying intercooler and compressor failures, having normalized AP values above 0.7 for most of the trained models. Notably, the InceptionNet and MNAS show particularly poor performance, likely attributed to an inability to converge on a solution within the number of training epochs. When comparing the normalized AP to the minimum AP, it is clear that the MNAS is no better than random guessing in most cases, with the InceptionNet having much lower performance compared to other CNNs. The EGR appears to be one of the primary labels that differentiate the performance of the CNNs as shown by comparing the overall mAP in Table 10 to the by-label PR AUC in Table 6. In particular, the RegNet, ResNet152, ResNeXt, and Wide ResNet have normalized mAP values over 0.65 as a result of EGR PR AUCs of approximately 0.7. EfficientNet and GoogLeNet have similar proficiency with EGR compared to the aforementioned CNNs, but have poorer classification performance with the Injectors. The injectors, on average, are appeared to be classified with lower precision than the other system components. This is an expected result as the injectors have high linear dependence among themselves, and share similar signal characteristics with EGR failure. Notably, the pattern shown by the biased-label distribution in Fig. 3 is less consistent among the injectors, sometimes favoring injectors with less sample data over others. However, comparing the normalized AP values to the minimum AP values shows that this effect is significantly lower than the proportional amount of skew, confirming that the loss function effectively handles the label imbalance.

The average ROC AUC for each classification model shows relatively good performance for each label. The EGR and intercooler appear to have the most distinct failure mode among all types of failures simulated for the purposes of classification. The compressor and injectors appear to have a similar level of distinctiveness among the classifiers, with Injector 3 standing out as less distinct on average compared to the other labels. Most of the previously mentioned CNNs with high mAP values show high ROC AUC performance in distinguishing between labels. Others, such as the InceptionNet and DenseNet also show relatively higher ROC AUC performance although their mAP is noticeably lower than the aforementioned CNNs. This can primarily be attributed to a lower EGR AP and Intercooler AP for the InceptionNet and a lower EGR AP for the DenseNet, which implies that these models have poor recall for higher precision values.

For determining the optimal threshold value for classifying a binary health condition of each component, the Maximum F1-Score, the EMR, and the Hamming Loss were investigated. The calculated mAP, EMR, and Hamming Loss based on the optimal thresholds can be found in Table 10. The micro (m), macro (M), weighted (W), and samples (S) averages of the thresholds associated with each metric are also presented in Figs. 8 and 9 [54]. The thresholds based on the F1-Score appear to be more balanced, with values between 0.5 and 0.6 for most cases. The EMR and Hamming Loss thresholds appear to be more restrictive, having values in the range of 0.6–0.7. For maximizing the F1-Score, the samples average is considered to be the most representative for the multi-label case. For most CNNs, excluding the InceptionNet and MNAS network, the F1-Score is around 0.7 as shown in Figs. 10 and 11. As such, all CNNs are shown to have significantly improved performances over a baseline classifier. The F1-Scores corresponding to the maximum EMRs and minimum Hamming Losses, are presented in Figs. 10 and 11, also show improvement beyond the baseline. However, they show poorer performance when compared to maximizing the F1-Score, with the EMR showing better results as compared to the Hamming Loss.

The precision and recall for each metric, shown in Figs. 12–15, provide more detailed characterization of each classifier. From the figures, maximizing the F1-Score reveals that the CNNs are generally biased toward high recall in terms of performance. This is expected behavior for data that are imbalanced and biased toward negative values, showing that they are more cost-efficient for the classifiers to accept wrong predictions in certain situations as opposed to learning to differentiate between the positive and negative cases. The EMR thresholding produced a more balanced approach between precision and recall as compared to maximizing the F1-Score. As the recall is reduced significantly to achieve the higher precision, this approach confirms that the recall bias is present within the classifiers. Minimizing the Hamming Loss shows that improving precision leads to a disproportionately greater reduction in recall compared to the cost of reducing precision when choosing to increase recall, as observed in the results from the maximum F1-Scores and EMRs.

Finally, it is of interest to know which the CNN has the highest amount of classification variance. To accomplish this, the mean and coefficient of variation of all pretrained CNNs’ confusion matrices normalized with ground truth for each thresholding approach are presented in Figs. 16–18. The true positive distribution percentage for the mean values is represented by the diagonal of the confusion matrix. In Fig. 16, it can be seen that the predictions by the maximum F1-Score thresholding best match the ground truth, with a relatively uniform offset corresponding to incorrect predictions. Fig. 17 shows that the EMR approach sacrifices EGR and injector accuracies for improved classification accuracies of the compressor and intercooler. The Hamming Loss thresholding, presented in Fig. 18, shows a similar trend to Fig. 17 without the trade-off in EGR prediction accuracy.

The coefficient of variation matrices in Figs. 16–18 shows where the CNNs have the largest variance in prediction distribution. From Fig. 16, a threshold chosen by maximizing the F1-Score reveals that most variation in the correct answers involves failure modes related to the compressor, Injector 1, and Injector 4. For the compressor, the variance is associated with EGR failure. For Injector 1, most differentiation involves Injectors 2 and 3, while Injector 4 is related to variance involving Injector 3. For thresholding based on the maximum F1-Score, the variance shows that the models are biased toward predicting Intercooler failure even when it is not present. This behavior can also be shown in Figs. 17 and 18. For the EMR thresholding, there is greater variance in the true positive predictions, implying that the CNNs tend to prioritize learning distinct labels for classification. From this approach, it can be seen that most variation in erroneous predictions appears to involve Injector 1, which is associated with all failure modes except the EGR. The Hamming Loss shows similar behavior to the EMR although its true positive predictions are more stable.

This paper presented a multi-label transfer learning approach, which involves retraining fourteen different pretrained CNNs with multi-mode component failure data to predict the binary failure states of targeted components. In this approach, a preprocessing procedure was proposed to represent non-mutually exclusive multi-mode component failure data from an automotive system, in a suitable form for retraining the pretrained CNNs. The retrained CNNs were designed such that the failure modes of an EGR, compressor, injectors, and intercooler of a diesel engine can be identified. Although there have been many studies to determine a single specific component failure using machine learning and deep learning algorithms, to the best of the authors’ knowledge, this is the first attempt to predict concurrent failures of multiple components utilizing a multi-label deep learning approach.

When the retrained classifier models based on CNNs were applied to the preprocessed simulation data, the test results showed good overall classification performance. The normalized mAP varied from 0.6 to 0.65 for most CNNs, which is comparable to performance with conventional image datasets. To characterize the prediction results, three thresholding procedures were presented: maximum F1-Score, maximum EMR, and minimum Hamming Loss. The results revealed a preference of higher recall over precision with the CNNs, which is in agreement with previous reports concerning biased multi-label data. However, the CNNs showed significant improvement over baseline classification models for the expected unachievable regions. Finally, the mean and coefficient of variation of the normalized confusion matrices were presented to characterize the differences in learning performance within the CNNs for each label. The results showed a true positive distribution in line with the expected distribution, with a bias toward predicting Intercooler failure. Injectors 1 and 4 were found to have the highest labeling variance among CNNs, with most differentiation involving the relation to the EGR. Therefore, for accurate prediction of component failure, it is important that the training dataset includes a balanced label distribution as much as possible. However, as the performance metrics between the injectors do deviate from this trend, it is suggested that the dependencies between similar components play a much more significant role than label imbalance for the purposes of classifying failure modes in this regime.

The process presented in this paper is for detecting concurrent failure of multiple components that are inherently correlated and can have different health conditions over time within the operational range of a vehicle. This paper has demonstrated that it is possible to determine which components are failing regardless of their severity in the vehicle when the health conditions are hidden, though this predictive capability decreases as failure modes become increasingly ambiguous such as seen with the Injectors. Based on the results of this research, more thorough analyses on the effects of the health condition, signal randomness, and RGB image randomness as well as increasing the dataset size and improving the label and class balance of the training and validation datasets will be conducted in the future. Also, the studied CNNs will be extended in future work to improve the classification performance by combining the CNNs into a consensus learning model. Finally, the effectiveness of different preprocessing techniques will also be examined.

This work was supported by the National Science Foundation under Grant No. 1650564.

There are no conflicts of interest.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.