Optimizing Dictionary Size

Abstract

Until now the number of atoms was an input parameter of the DL algorithms. Its choice was left to the user, leading usually to a trial and error approach. We discuss here possible ways to optimize the number of atoms. The most common way to pose the DL problem is to impose a certain representation error and attempt to find the smallest dictionary that can ensure that error. The algorithms solving this problem use the sparse coding and dictionary update ideas of the standard algorithms, but add and remove atoms during the DL iterations. They start either with a small number of atoms, then try to add new atoms that are able to significantly reduce the error, or with a large number of atoms, then remove the less useful ones; the growing strategy seems more successful and is likely to have the lowest complexity. Working on a general DL structure for designing dictionaries with variable size, we present some of the algorithms with best results, in particular Stagewise K-SVD and DLENE (DL with efficient number of elements); the first serves also as basis for an initialization algorithm that leads to better results than the typical random initializations. We present the main ideas of a few other methods, insisting on those based on clustering, in particular on the mean shift algorithm. Finally, we discuss how OMP can be modified to reduce the number of atoms without impeding too much on the quality of the representation.