Heuristic Rank Selection with Progressively Searching Tensor Ring Network

Recently, Tensor Ring Networks (TRNs) have been applied in deep networks, achieving remarkable successes in compression ratio and accuracy. Although highly related to the performance of TRNs, rank selection is seldom studied in previous works and usually set to equal in experiments. Meanwhile, there is not any heuristic method to choose the rank, and an enumerating way to find appropriate rank is extremely time-consuming. Interestingly, we discover that part of the rank elements is sensitive and usually aggregate in a narrow region, namely an interest region. Therefore, based on the above phenomenon, we propose a novel progressive genetic algorithm named Progressively Searching Tensor Ring Network Search (PSTRN), which has the ability to find optimal rank precisely and efficiently. Through the evolutionary phase and progressive phase, PSTRN can converge to the interest region quickly and harvest good performance. Experimental results show that PSTRN can significantly reduce the complexity of seeking rank, compared with the enumerating method. Furthermore, our method is validated on public benchmarks like MNIST, CIFAR10/100, UCF11 and HMDB51, achieving the state-of-the-art performance.

: The overview of Progressive Searching Tensor Ring Network (PSTRN), where different color represents different rank element candidate. PSTRN initializes search space by sampling from state space, then alternately executes evolutionary phase and progressive phase for P times to derive optimal TRN.

Introduction
Deep neural networks have made great successes in various areas, such as image classification [6,21,15], autonomous driving [1,30,14], game artificial intelligence [20,16] and so on [13,25,24]. However, parameters redundancy leads to two major drawbacks for deep neural networks: 1) difficult training, and 2) poor ability to run on resource-constrained devices (e.g., mobile phones [7] and Internet of Things (IoT) devices [11]). To address these problems, Tensor Ring (TR) has been introduced to deep neural networks. With a ring-like structure as shown in Fig. 3, TR can significantly reduce the parameters of Convolutional Neural Network (CNN) [26] and Recurrent Neural Network (RNN) [17], and even can achieve better results than uncompressed models in some tasks. Thus tensor ring is increasingly being researched.
However, as the crucial component of tensor ring, setting of rank (e.g. R 0 ∼ R 3 in Fig. 3) is seldom investigated. In most of the existing works, it merely sets to be equal in whole network [26]. Such an equal setting requires multiple manual attempts for a feasible rank value and often leads to a weak result. Fortunately, as shown in our synthetic experiment, we discover the relationship between the rank distribution and its performance. Experimental results demonstrate the link that part of rank elements with good performance will gather to the interest region. Then we extend this phenomenon to build our Hypothesis 1. Utilizing the hypothesis, we design a heuristic algorithm to explore the potential power of tensor ring.
Specifically, we propose Progressive Searching Tensor Ring Network (PSTRN) inspired by Neural Architecture Search (NAS) [31]. Similarly, our approach is divided into three parts, search space: combinations of rank element candidates for TRN in evolutionary phase; search strategy: the Non-dominated Sorted Genetic Algorithm-II (NSGA-II) [3] to search rank; performance estimation strategy: stochastic gradient descent to train TRN.
The overall framework of PSTRN is illustrated in Fig. 1. In the searching process, we initialize search space first. Then through evolutionary phase, we derive optimized rank within search space. Next, in order to draw near interest region, the proposed approach shrinks the bound of search space to the around of optimized rank during progressive phase. By alternately executing evolutionary phase and progressive phase, our algorithm can find rank with high performance. Additionally, on large-scale models (i.e. ResNet20/32 [6] and WideResNet28-10 [29]), the performance estimation is time-consuming, which is harmful to search speed. So we employ a weight inheritance strategy [18] to accelerate the evaluation of rank.
Experimental results prove that PSTRN can obtain optimal rank of TRN according to the Hypothesis 1. And our algorithm can compress LeNet5 [12] with compression ratio as 16x and 0.49% error rate in MNIST [4] image classification task. In TR-ResNets, our approach can achieve state-of-the-art performance on CIFAR10 and CIFAR100 [9]. PSTRN also exceeds TR-LSTM models that set rank elements equal on HMDB51 and UCF11. Furthermore, compared with the enumerating method, our work can greatly reduce the complexity of seeking rank. Overall, our contributions can be summarized as follows: 1. PSTRN can search rank automatically instead of manual setting. At the meantime, The time cost is reduced significantly by progressively searching, compared with an enumerating method. 2. To speed up the search on large-scale model, our proposed method adopts weight inheritance into the search process. And the proposed method achieves about 200× speed-up ratio on classification tasks of CIFAR10/100 datasets. 3. As a heuristic approach based on the Hypothesis 1, our algorithm can achieve better performance with fewer parameters than existing works. All the experimental results demonstrate the rationality of the Hypothesis that is first found by us.

Background
In this section, we will introduce the tensor background and some related works that consist of rank fixed method and rank selection method. The rank fixed method is the work that sets rank manually, while rank selection method means the work of learning the rank.

Tensor Background
In this part, we would like to introduce the background of tensor.

Notation
A tensor is a high-order array. In this paper, a d-order tensor T ∈ R L 1 ×L 2 ···×L d is denoted by a boldface Euler script letter. With all subscripts fixed, each element of a tensor is expressed as: T l 1 ,l 2 ,...l d ∈ R. Given a subset of subscripts, we can get a sub-tensor. For example, given a subset {L 1 = l 1 , L 2 = l 2 }, we can obtain a sub-tensor T l 1 ,l 2 ∈ R L 3 ···×L d . Fig. 2 draws the tensor diagrams that present the graphical notations and the essential operations. Fig. 2: Tensor diagrams. (a) presents the graphical notation of a tensor T ∈ R L 1 ×L 2 ×L 3 . (b) demonstrates the contraction between two 4-order tensors, which is the contraction between A and B.

Tensor Contraction
Tensor contraction can be performed between two tensors if some of their dimensions are matched. As shown in Fig. 2b, given two 4-order tensors A ∈ R I 1 ×I 2 ×I 3 ×I 4 and B ∈ R J 1 ×J 2 ×J 3 ×J 4 , when I 3 = D 1 = J 1 and I 4 = D 2 = J 2 , the contraction between these two tensors results in a tensor with the size of I 1 × I 2 × J 3 × J 4 , where the matching dimension is reduced, as shown in equation: (1)

Tensorization
Given a matrix M ∈ R I×O , we transfer it into a new tensor satisfying the equation: where M, N are the number of the input nodes and output nodes respectively. Therefore, a corresponding element of W

Tensor Ring Format (TRF)
TRF is constructed with a series of 3-order nodes linked one by one, forming a ring-like structure. The TRF of a d-order tensor can be formulated as where R = {R i |i ∈ {0, 1, . . . , d − 1}} denotes the rank of TRF, and the symbol Z represents the tensor ring node. Fig. 3 shows a graph structure of a simple TRF. Through replacing layers(e.g. convolutional layer, fully-connected layer) of a network with TRF, we can derive a TRN.

Rank Fixed
Tensor ring decomposition has been successfully applied to the compression of deep neural networks. Wenqi et al. [26] compress both the fully connected layers and the convolutional layers of CNN with the equal rank elements for whole network. Yu et al. [17] replace the over-parametric input-to-hidden layer of LSTM with TRF, when dealing with high-dimensional input data. Rank of these models are determined via multiple manual attempts by manipulation, which requires much time.

Rank Selection
In this part, we would like to introduce the works of rank selection. Yerlan et al. [28] formulate the low-rank compression problem as a mixed discrete-continuous optimization jointly over the rank elements and over the matrix elements. Zhiyu et al. [2] propose a novel rank selection scheme for tensor ring, which apply deep deterministic policy gradient to control the selection of rank. Their algorithms calculate the optimal rank directly from the trained weight matrix without the analysis of rank. Different from them, our approach is inspired by the relevance between the rank distribution and performance of Hypothesis 1 , towards a better result.

Methodology
To verify the optimization of PSTRN on TRN, we choose two most commonly used deep neural networks for evaluation, i.e. Tensor Ring CNN (TR-CNN) and Tensor Ring LSTM (TR-LSTM). In the section, we first present preliminaries of TR-CNN and TR-LSTM, including graphical illustrations of the two TR-based models. Then we elaborate on evolutionary phase and progressive phase of PSTRN. The implementation of weight inheritance will be given in final.

TR-LSTM
By replacing each matrix of the affine matrices W * ∈ R I×O of input vector x ∈ R I with TRF in LSTM, we implement the TR-LSTM model as introduced by Yu et al. [17]. Similar to TR-CNN, the nodes are combined by input nodes U (i) and output nodes V (j) , and the decomposition needs to follow A 6-node example is shown in Fig. 4b. Compression ratio of TR-LSTM can be computed as

Progressive Searching Tensor Ring Network
In our search process, the rank R of a TRN is formulated as where d is the number of rank elements, r * is a rank element candidate , and m is the quantity of rank element candidates. Full combinations of the rank elements (i.e. state space) can be calculated as Next, we would like to introduce the Hypothesis 1, the extension of the aforementioned gathering phenomenon.
Hypothesis 1 When a shape-fixed TRN performs well, part or all of its rank elements are sensitive and each of them will tend to aggregate in a narrow region, which is called interest region.
According to Hypothesis 1, the optimal rank can be found in the interest region. It is a more efficient and accurate way to find a optimal rank in interest region rather than a much wider range of the whole rank element candidates. Thus, we build an pipeline of PSTRN to achieve the purpose by two alternative procedures: -Evolutionary phase: finding good models in the search space and locating the interest region through well-performed models. -Progressive phase: calculating the width of a rough approximation of interest region and defining search space within this region.
Through these two procedures, the rank of a TRN will approach the interest region and be optimized. Additionally, we apply weight inheritance to accelerate the training process. The pseudocode of the algorithm is shown as below, where P is the number of progressive phase, and G is the generations of each evolutionary phase.

Algorithm 1 Progressive Searching Tensor Ring Network
Input: Datasets D, generations of evolutionary phase G, number of progressive phase P. Initialize the search space if Large-scale model (TR-ResNets) then Do warm-up of weights end if Generate a set of randomized ranks following Eq. (8) Compute their accuracy and compression ratios via Eq. (5) and Eq. (7) for g = 1, 2, ..., G do Do selection, mutation and crossover Obtain the promising rank end for Determine the search space for next phase by Eq. (14) end for Output: A set of well-performed tensor ring rank elements

Evolutionary Phase
As described in Hypothesis 1 that well-performed models aggregate in interest region, good models keep a high probability of appearing in interest region. Therefore, we determine interest region around the models with high performance.
In PSTRN, we adopt multi-objectives genetic algorithm NSGA-II [3] to search for TR-based models with high performance and few parameters.
A typical genetic algorithm requires two prerequisites, the genetic representation of solution domain (i.e. search space), and the fitness functions (i.e. classification accuracy and compression ratio) that is used to evaluate each individual. In the process, an individual means the rank R and each rank element R * is in {r 1 ,r 2 , . . . ,r n } that is sampled from the whole rank element candidates. The search space is a sub-space of the state space and can be calculated as The method of choosing the search space will be introduced in the progressive phase. Classification accuracy is obtained by testing the model on the test dataset. And compression ratio of TR-CNN and TR-LSTM are calculated by Eq. (5) and Eq. (7) respectively.
The key idea of the genetic algorithm is to evolve individuals via some genetic operations. At each evolutionary generation, the selection process preserves strong individuals as a population and then sorts them according to their fitness function, while eliminating weak ones. The retained strong individuals reproduce new children through mutation and crossover with a certain probability. After this, we obtain the new population consists of the new children and the retained strong individuals. The new population executes the evolution to derive next generation. When the termination condition is met, evolutionary phase stops and optimization of the rank will be completed. Eventually, taking the top-k individuals into consider, we derive the most promising rank elementR * bŷ where R * ,i is a rank element of the i-th individual and f loor denotes the rounding down operation. The interest region is around theR * .

Progressive Phase
Progressive phase is used to determine the next search space as shown in Fig. 5. At the begining of the PSTRN, we first obtain initial search space via sampling from the state space at equal intervals as below: where R min is the minimum of rank element candidates, and b 1 is the initial sampling interval. Then through carrying out evolutionary phase within initial search space, we derive the promising rank whereR i,j , i ∈ {0, 1, 2, . . . , d−1}, j ∈ {2, 3, . . . , P } denotes the i-th promising rank element in j-th evolutionary phase. Based on the optimized rank, our PSTRN shrinks bound of search space to: where R max is the maximum of rank element candidates, and {s j |j ∈ {2, 3, . . . , P }} is the offset coefficient and usually sets to b j−1 . Thus the rank element candidates of the next search space can be expressed as: where b j is the sampling interval of the j-th progressive phase, satisfying The interval b j is gradually reduced, and when b j decreases to 1, the progressive phase will stop.
In addition, considering the above Hypothesis 1 cannot be proved by theory, the progressive genetic algorithm may fall into local optima. Therefore, we adds an exploration mechanism to the algorithm. Concretely speaking, except for the initial phase, the algorithm has a 10% probability to choose rank within the search space in the previous evolutionary phase.
In the above evolutionary phase, the solution domain is one of the key components. Generally speaking, it will try to cover all possible states. Such an excessive solution domain may lead to the divergence of search algorithm. Compared with full state space, our algorithm can improve the search process in computational complexity significantly.

Weight Inheritance
During evolutionary phase, to validate the performance, the searched TRN needs to be fully trained, which is the most time-consuming part of the search process. On MNIST, we can train searched TR-LeNet5 from scratch because of its fast convergence. But the training speed is slow on ResNets. Thus we employ weight inheritance as a performance estimation acceleration strategy, which is inspired by the architecture evolution [18].
In our algorithm, to inherit trained weight directly, the rank R = {R k i |i ∈ [0, 1, . . . , d − 1]} of k-th layer needs to follow Obviously, the number of rank elements to be searched is reduced to k from d. For the k-th layer, we will load the checkpoint whenever possible. Namely, if the k-th layer matches V k , the weights are preserved. Such a method is called warm-up.
During search process, we directly inherit the weights trained in warm-up stage and fine-tune the weights for each searched TRN. Instead of training from scratch, fine-tuning the trained weights can greatly resolve the time-consuming problem. For example, training ResNet20 on CIFAR10 from scratch requires about 200 epochs. On the contrary, our training with fine-tuning only needs 1 epoch, which brings the acceleration of 200×.

Experiments
In this section, we conduct experiments to verify the effectiveness of PSTRN. First, to display the relation between the rank elements and performance of TRbased models, we design the synthetic experiment. Then we estimate the effect of the searched TR-based models on prevalent benchmarks. The optimization objectives of NSGA-II [3] are classification performance and compression ratio, namely PSTRN-M. In addition, to gain the TR-based model with high performance, we also conduct optimization algorithm PSTRN-S that only consider classification accuracy. All the experiments are implemented on Nvidia Tesla V100 GPUs. 2 Fig. 6: Rank distribution between R 1 and other rank elements R 0 , R 2 , R 3 of top-100 models. The size of the circle denotes the number of models that has the same two rank elements, and the circle color represents ranking. The blue line is the border of the interest region.

Synthetic Experiment
Previous works of rank search lack of heuristic method, so they derive rank elements depending on decomposition, which limits the exploration of searching rank. Hypothesis 1 would bring a promising way to solve this problem, and we would like show the phenomenon of interest region in a synthetic experiment.
Experimental Setting Given a low-rank weight matrix W ∈ R 144×144 . We generate 5000 samples, and each dimension follows the normal distribution, i.e.
x ∼ N (0, 0.05I), where I ∈ R 144 is the identity matrix. Then we generate the label y according to y = W(x + ) for each x, where ∼ N (0, 0.05I) is a random Gaussian noise. Data pairs of {x, y} constitute the dataset. We divide it into 4000 samples as the training set and 1000 samples as the testing set. For the model, we constructed the TR-linear model by replacing the W ∈ R 144×144 with a TRF ∈ R 12×12×12×12 . Then we train the TR-linear model with different ranks to completion, and validate the performance on the testing set with Mean-Square Error (MSE) between the predictionŷ and label y. The rank is denoted as R = {R 0 , R 1 , R 2 , R 3 |R * ∈ {3, 4, . . . , 15}}.
In the experiment, optimizer sets to Adam with a learning rate 1e − 2, MSE is adopted as loss function and batch size is 128. The total epoch is 100, and the learning rate decreases 90% every 30 epoch. For a comparison, we run the enumerating results as the baseline, which needs 13 4 = 28561 times training.
Experimental Results Fig. 6 shows the distribution of top-100 rank elements sorted by the value of loss. The size of the circle denotes the number of models who has the same two rank elements. And the circle color represents ranking. It shows that it is not ideal to set each rank element the same arbitrarily. We calculate the mean µ(3.6) and standard variance δ(0.96) of top-100 models, and derive the interest region [µ − δ, µ + δ]([2.64, 4.56]). Obviously, R 1 mostly distributes in the interest region. Should be noted that other rank elements do not show an apparent phenomenon, for the reason that they do not play a critical role in the performance. Our model can find the interest region that is important to the ability of models and achieve good results. on HMDB51), the accuracy in search process was obtained on the test dataset instead of the validation dataset which is widely used in works of NAS [31]. Thus we supplemented the relevant experiments and shown them in the Appendix.  As shown in Fig. 7a, the approximation of interest region gradually approaches groudtruth, which demonstrates that PSTRN can locate interest region precisely. As illustrated in Fig. 7b, our model can find the best rank in the second phase, which proves the powerful capacity of PSTRN algorithm. Compared with 28561 enumerating results, we only need n gen × pop size × P = 10 × 20 × 3 = 600 times training, which is much smaller. pop size and n gen are the population size and the number of generations respectively. Undoubtedly, our PSTRN can find the optimal rank efficiently and precisely.
To prove the performance of the progressive search, we also conduct an ablation experiment to compare PSTRN and NSGA-II. pop size of NSGA-II is set to 20, which is the same as PSTRN. And n gen is set to 30. The experimental results are shown in Table 1. The ranking of the searched model among all 28561 possible models is shown in the last column. It can be seen that our proposed progressive evolutionary algorithm can converge faster.

Experiments on MNIST and Fashion MNIST
MNIST dataset has 60k grayscale images with 10 object classes, and the resolution of each data is 28 × 28. Fashion MNIST is more complicated and easy to replace MNIST in experiments.
Experimental Setting We evaluate our PSTRN on MNIST and Fashion MNIST by searching TR-LeNet5 that is proposed by Wenqi et al. [26]. As shown in Table 2, TR-LeNet5 is constructed with two tensor ring convolutional layers and two tensor ring fully-connected layers. Thus, the total rank is R = {R 0 , R 1 , . . . , R 19 |R * ∈ {2, 3, . . . , 30}}. Accordingly, the computational complexity size is 29 20 ≈ 1.77 × 10 29 for enumeration. TR-LeNet5 is trained by Adam [8] on mini-batches of size 128. The random seed is set to 233. The loss function is cross entropy. Models are trained for 20 epochs with an initial learning rate as 0.002 that is decreased by 0.9 every 5 epochs. PSTRN runs 40 generations at each evolutionary phase with population size 30. The number of rank elements searched is 20. The number of progressive phase is 3. The interval b * of each phase is 5, 2 and 1 respectively. Thus complexity of our PSTRN is n gen × pop size × P = 30 × 40 × 3 = 3600.

LeNet5
TR-LeNet5 layer shape shape searching rank conv1  Table 3 and Table 4, where original LeNet5 is proposed by LeCun et al. [12], Bayesian Automatic Model Compression(BAMC) [23] leverages Dirichlet process mixture models to explore the layer-wise quantization policy, LR-L [28] learn the rank of each layer for SVD, and TR-Nets [26] compresses deep neural network via tensor ring decomposition with equal rank elements. The superscript ri represents there are results of re-implement, and r is the rank of works that set rank elements to equal. These settings would be retained in subsequent experiments. Both of the search processes for MNIST and Fashion MNIST cost about 5 GPU days. In Table 3, the first block shows the results of rank-fixed methods, which manually set rank elements to equal. The second block is the work that automatically compresses the model. As expected, both PSTRN-M and PSTRN-S achieve best performance on MNIST. Our algorithm compress LeNet5 with compression ratio as 6.5x and 0.49% error rate. And our PSTRN can also exceed models that set rank manually on Fashion MNIST as shown in Table 4. Further, Fig. 8 demonstrates that our proposed approach outperforms rank-fixed works on MNIST. Obviously, when fixed rank r is bigger than 20, the TR-Nets will be over-fitting. And our proposed work can find the suitable rank with best performance.
The ranks of searched TR-LeNet5 are shown in Table 5. The symbol // indicates the different layers.

Experiments on CIFAR10 and CIFAR100
Both CIFAR10 and CIFAR100 datasets consist of 50,000 train images and 10,000 test images with size as 32 × 32 × 3. CIFAR10 has 10 object classes and CIFAR100 has 100 categories. Experimental Setting The dimension of TR-RseNet is shown in Table6, Ψ is the number of ResBlock. TR-ResNet32 is built as introduced by Wenqi et al. [26] with Ψ as 5, and TR-ResNet20 is constructed as proposed by Zhiyu et al. [2] with Ψ as 3. First, we apply the PSTRN-M/S to search TR-ResNet20/32 on CIFAR10. Further, we transfer TR-ResNet20/32 searched by PSTRN-M/S on CIFAR10 into CIFAR100 to evaluate the transferability of PSTRN. Considering that training TR-ResNet20/32 on CIFAR10 is time-consuming, we apply the weight inheritance to accelerate the process. Specifically, we pre-train weight of the model in warm-up stage. Then we load pre-trained weights directly. The training epoch of warm-up is set to 30. The rank R = {R 0 , R 1 , . . . , R 6 |R * ∈ {2, 3, . . . , 20}}.
TR-ResNets are trained via SGD [19] with momentum 0.9 and a weight decay of 5 × 10 −4 on mini-batches of size 128. The random seed is set to 233. The loss function is cross entropy. TR-ResNets are trained for 200 epochs with an initial learning rate as 0.02 that is decreased by 0.8 after every 60 epochs. Our approach runs 20 generations at each evolutionary phase with population size 30. The number of rank elements searched is 7. The number of progressive phase is 3. The interval b * of each phase is 3, 2 and 1 respectively. The complexity of our  approach is n gen × pop size × P = 30 × 40 × 3 = 3600, which is much smaller than computational complexity 19 7 ≈ 8.9 × 10 8 of enumeration method.
Experimental Results The results for ResNet20 and ResNet32 are given in Table 8 and Table 9. The search process for ResNet20/32 and WideResNet28- 10 cost about 2.5/3.2 GPU days and 3.8 GPU days respectively. In the Tables  8 and 9, original ResNet20/32 are the model proposed by Kaiming et al. [6]. Tucker [7] and TT [5] are works that compress neural networks by other tensor decomposition methods. TR-RL [2] search the rank of TR-based model based on reinforcement learning. The first block compares PSTRN-M with the results of low rank decomposition works that have fewer parameters. Obviously, PSTRN-M surpasses other methods in both classification accuracy and compression ratio. The second block reports the performance of PSTRN-S and models that are poor on compression. Results tell that our algorithm achieve high performance and beyonds works with 0.10+M parameters.
The results for TR-WideResNet28-10 are given in Table 10. Apparently, PSTRN-M exceeds other methods in both classification accuracy and compression ratio. The experimental results show that PSTRN-M can exceed the manually designed models in terms of classification accuracy and compression ratio. In the case that the number of parameters is smaller than the manually designed model with r as 15, the accuracy of PSTRN-M is higher. In addition, through transferring the searched PSTRN-M/S on CIFAR10 into CIFAR100, PSTRN obtain excellent results as well. This proves that our proposed PSTRN can not only find well-performed models, but also possesses transferability.
The ranks of searched TR-ResNet20 and TR-ResNet32 are shown in Table 11, 12, and Table 13.  Experimental Setting In this experiment, we sample 12 frames from each video clip randomly. And then we extract features from the frames via Inception-V3 [22] input vectors and reshape it into 64 × 32. The shape of hidden layer tensor is set as 32 × 64 = 2048. For TR-LSTM, as shown in Table 14, the rank is denoted as R = {R 0 , R 1 , R 2 , R 3 |R * ∈ {15, 16, . . . , 60}}. The complexity of enumerating approach is 46 4 ≈ 4.5 × 10 6 .
TR-LSTM is trained via Adam with a weight decay of 1.7 × 10 −4 on minibatches of size 32. The random seed is set to 233. The loss function is the crossentropy. In the search phase, searched models are trained for 100 epochs with an initial learning rate of 1e-5. Our approach runs 20 generations at each evolutionary phase with a population size of 20. The number of rank elements searched is 4. The number of progressive phases is 3. The interval b * of each phase is 8, 3 and 1 respectively. The computational complexity of our PSTRN is n gen × pop size × P = 20 × 20 × 3 = 1200 that is much smaller.
Experimental Results The comparisons between our approach and manuallydesigned method are shown in Table 15 and Table 16. The search process for HMDB51 and UCF11 cost about 1.4 GPU days and 0.5 GPU days respectively. Experimental results on Table 15 show that our searched rank exceed in others    Table 16. As can be seen, with the compression ratio greater than twice that of the manually designed model with r as 40, the classification accuracy of PSTRN-S is 1.26% higher. PSTRN-M also achieves higher accuracy with fewer parameters. Table 17 demonstrates the ranks for TR-LSTM that searched via PSTRN. Remark Unlike the Pytorch, Keras is a high level package with many tricks under the table, e.g. hard sigmoid in RNNs. Thus, for fairer comparison and validation of search results, we implement this experiment in Pytorch and remove the tricks in the Keras package. Additionally, through Keras implement, our searched TR-LSTM achieve 64.5% accuracy with a compression ratio 48, which is better than 63.8% with a compression ratio 25 [17].
Another important component is the shape of a tensor ring decomposition. The method of choosing the shape is notorious. And actually there are almost not any way to select the shape efficiently. Therefore, our PSTRN simply chooses a shape with a similar size by manipulation. The effect of shape on TR-based model is unknown and waits to be solved in the future.
Generally, the rank has similar attribution in different kinds of tensor decomposition like Tucker, Tensor Train and so on. It is reasonable to assume that the Hypothesis 1 is suitable for them. Therefore, it is promising to employ PSTRN on them to explore their potential power.

Conclusion
In this paper, we propose a novel algorithm PSTRN based on Hypothesis 1 to search optimal rank. As a result, our algorithm compresses LeNet5 with 16× compression ratio and 0.22% accuracy improvement on MNIST. And on CIFAR10 and CIFAR100, our work achieves state-of-the-art performance with a high compression ratio for ResNet20, ResNet32 and WideResNet28-10. Not only the CNN, we also show excellent performance of LSTM on HMDB51 and UCF11. Further, for large-scale datasets, we will explore more performance evaluation acceleration strategies to optimize rank elements more efficiently.

Conflict of Interest Statement
On behalf of all authors, the corresponding author states that there is no conflict of interest.