Dimension Estimation Using Autoencoders and Application

Abstract

Dimension Estimation (DE) and Dimension Reduction (DR) are two closely related topics, but with quite different goals. In DR, one attempts to project a random vector, either linearly or nonlinearly, to a lower-dimensional space that preserves the information contained in the original higher-dimensional space. However, in DE, one attempts to estimate the intrinsic dimensionality or number of latent variables in a set of measurements of a random vector. DE and DR are closely linked because reducing the dimension to a smaller value than suggested by DE will likely lead to information loss. In particular, when considering linear methods such as Principal Component Analysis (PCA), DE and DR are often accomplished simultaneously. However, in this paper, we will focus on a particular class of deep neural networks called autoencoders (AEs), which are used extensively for DR but are less well studied for DE. We show that several important questions arise when using AEs for DE, above and beyond those that arise for more classic DR/DE techniques such as PCA. We address AE architectural choices and regularization techniques that allow one to transform AE latent layer representations into estimates of intrinsic dimension. We demonstrate the effectiveness of our techniques on synthetic, image processing benchmark problems, and most importantly, diverse applications such as the analysis of financial markets and network security.

This chapter is an extension of the conference paper with additional results (This document does not contain technology or Technical Data controlled under either the U.S. International Traffic Arms Regulations or the U.S. Export Administration Regulations).