Variable Selection for Correlated High-Dimensional Data with Infrequent Categorical Variables: Based on Sparse Sample Regression and Anomaly Detection Technology

Abstract

We devised a new variable selection method to extract infrequent event variables (binary data) that have one or more interaction effects, particularly for industrial application. The method is a combination of sparse sample regression (SSR), a latest type of just-in-time modeling (JIT), and T² and Q statistics, which are typical anomaly detection methods. JIT is based on locally weighted regression to cope with changes in process characteristics as well as nonlinearity. In the proposed method, modeling is performed multiple times using variables whose effects on the objective variable are known. The sample to be used is selected by SSR automatically and weighted as well. By evaluating how far the estimated value of the coefficient of the created model is from the value of the reference model with T² and Q statistics, models that contain more (or less) samples in which events that affect the objective variable occur are detected. Variable selection is performed by ranking the ratio of events to the sample used by the model detected as an abnormal value. Synthetic data was used to verify the method. In the verification, we succeeded in extracting one of the correct answers from a total of 5000 variables including six variables that are correct effects, and the summary of the verification with actual data was shown.