3DMeT: 3D Medical Image Transformer for Knee Cartilage Defect Assessment

Abstract

While convolutional neural networks (CNNs) are dominating the area of computer-aided 3D medical image diagnosis, they are incapable of capturing global information due to the intrinsic locality of convolution. Transformers, another type of neural network empowered with self-attention mechanism, are good at representing global relations, yet computationally expensive and do not generalize well on small datasets. Applying Transformers on 3D medical images has two major problems: 1) medical 3D volumes are bigger in size than natural images which makes training process computationally impractical, 2) and 3D medical image datasets are usually smaller than natural image datasets since medical images are expensive to collect. In this paper, we propose the 3D Medical image Transformer (3DMeT) to address these two issues. 3DMeT introduces 3D convolutional layers to perform block embedding instead of the original linear embedding to cut the computational cost. Additionally, we propose a teacher-student training strategy to address the data-hungry issue by adapting convolutional layers’ weights from a CNN teacher. We conduct experiments on knee images, results demonstrate that the 3DMeT (70.2) confidently outperforms the 3DCNNs (65.3) and Vision Transformer (58.7).

S. Wang and Z. Zhuang—Contributed equally.

This work was supported by the National Key Research and Development Program of China (2018YFC0116400), National Natural Science Foundation of China (NSFC) grants (62001292), Shanghai Pujiang Program(19PJ1406800), and Interdisciplinary Program of Shanghai Jiao Tong University.