Skip to main content
SpringerLink
Log in

Compressive Classification: Where Wireless Communications Meets Machine Learning

  • Chapter
Compressed Sensing and its Applications

Part of the book series: Applied and Numerical Harmonic Analysis ((ANHA))

  • 3055 Accesses

Abstract

This chapter introduces Shannon-inspired performance limits associated with the classification of low-dimensional subspaces embedded in a high-dimensional ambient space from compressive and noisy measurements. In particular, it introduces the diversity-discrimination tradeoff that describes the interplay between the number of classes that can be separated by a compressive classifier—measured via the discrimination gain—and the performance of such a classifier—measured via the diversity gain—and the relation of such an interplay to the underlying problem geometry, including the ambient space dimension, the subspaces dimension, and the number of compressive measurements. Such a fundamental limit on performance is derived from a syntactic equivalence between the compressive classification problem and certain wireless communications problems. This equivalence provides an opportunity to cross-pollinate ideas between the wireless information theory domain and the compressive classification domain. This chapter also demonstrates how theory aligns with practice in a concrete application: face recognition from a set of noisy compressive measurements.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    We constrain the random measurement kernel to be full row rank. We also constrain the distribution of the random measurement kernel to be invariant to rotations. These constraints are obeyed by the standard Gaussian i.i.d. random kernels in compressive sensing.

  2. 2.

    The final inequalities hold element-wise.

  3. 3.

    We assume that the channel model is real rather than complex as in previous treatments in the wireless communications literature (e.g., [30, 57]). This distinction is unimportant, as it is straightforward to adapt arguments over complex-valued channels to real-valued ones and vice-versa.

  4. 4.

    The DMT was introduced in the context of wireless communications to characterize the high-SNR performance of fading coherent MIMO channels [57, 58]. It shows that the spatial flexibility provided by multiple antennas can simultaneously increase the achievable rate and decrease the probability of error in a wireless communications channel, but only according to a tradeoff that is tightly characterized at high SNR.

  5. 5.

    By working with the upper bound to the misclassification probability rather than the true one, we obtain a lower bound to the diversity gain rather than the exact one.

  6. 6.

    Note that this classifier is mismatched in view of the fact that the class conditioned distributions are not necessarily Gaussian. In fact, it is immediate to demonstrate that face samples within each class do not pass the Royston’s multivariate normality test [41], as they return p-values below 10−3 for all classes.

  7. 7.

    Note that this suggests that nature may tend to approximately distribute the subspaces uniformly on the Grassmann manifold.

References

  1. Adini, Y., Moses, Y., Ullman, S.: Face recognition: the problem of compensating for changes in illumination direction. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 721–732 (1997)

    Article  Google Scholar 

  2. Aggarwal, V., Ashikhmin, A., Calderbank, R.: A Grassmannian packing based on the Nordstrom-Robinson code. In: IEEE Information Theory Workshop, Chengdu, China, pp. 1–5 (2006)

    Google Scholar 

  3. Alon, U., Barkai, N., Notterman, D.A., Gish, K., Ybarra, S., Mack, D., Levine, A.J.: Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proc. Natl. Acad. Sci. 96(12), 6745–6750 (1999)

    Article  Google Scholar 

  4. Ashikhmin, A., Calderbank, R.: Space-time Reed-Muller codes for noncoherent MIMO transmission. In: IEEE International Symposium on Information Theory, Adelaide, Australia, 1952–1956 (2005)

    Google Scholar 

  5. Ashikhmin, A., Calderbank, R., Kewlin, W.: Multidimensional second order Reed-Muller codes as Grassmannian packings. In: IEEE International Symposium on Information Theory, Seattle, WA, USA, pp. 1001–1005 (2006)

    Google Scholar 

  6. Ashikhmin, A., Calderbank, R.: Grassmannian packings from operator Reed-Muller codes. IEEE Trans. Inf. Theory 56(10), 5689–5714 (2010)

    Article  MathSciNet  Google Scholar 

  7. Basri, R., Jacobs, D.W.: Lambertian reflectance and linear subspaces. IEEE Trans. Pattern Anal. Mach. Intell. 25(2), 218–233 (2003)

    Article  Google Scholar 

  8. Calderbank, R., Hardin, R.H., Rains, E.M., Shor, P.W., Sloane, N.J.A.: A group-theoretic framework for the construction of packings in Grassmannian spaces. J. Algebraic Comb. 9, 129–140 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  9. Candès, E., Romberg, J., Tao, T.: Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)

    Article  Google Scholar 

  10. Candès, E., Romberg, J., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8), 1207–1223, 2006.

    Article  MATH  Google Scholar 

  11. Candès, E., Tao, T.: Near-optimal signal recovery from random projections: universal encoding strategies? IEEE Trans. Inf. Theory 52(12), 5406–5425 (2006)

    Article  Google Scholar 

  12. Carson, W.R., Chen, M., Rodrigues, M.R.D., Calderbank, R., Carin, L.: Communications-inspired projection design with application to compressive sensing. SIAM J. Imaging Sci. 5(4), 1185–1212 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  13. Chen, M., Silva, J., Paisley, J., Wang, C., Dunson, D., Carin, L.: Compressive sensing on manifolds using a nonparametric mixture of factor analyzers: algorithm and performance bounds. IEEE Trans. Signal Process. 58(12), 6140–6155 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  14. Chen, M., Carson, W.R., Rodrigues, M.R.D., Calderbank, R., Carin, L.: Communication-inspired linear discriminant analysis. In: International Conference on Machine Learning, Edinburgh, UK, pp. 919–926 (2012)

    Google Scholar 

  15. Davenport, M., Boufounos, P., Wakin, M., Baraniuk, R.: Signal processing with compressive measurements. IEEE J. Sel. Top. Sign. Process. 4(2), 445–460 (2010)

    Article  Google Scholar 

  16. Donoho, D.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  17. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. Wiley-Interscience, New York, NY (2000)

    Google Scholar 

  18. Elhamifar, Vidal, R.: Sparse subspace clustering: algorithm, theory, and applications. IEEE Trans. Pattern Anal. Mach. Intell. 35(11), 2765–2781 (2013)

    Google Scholar 

  19. Georghiades, A.S., Belhumeur, P.N., Kriegman, D.J.: From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 643–660 (2001)

    Article  Google Scholar 

  20. Hastie, T., Simard, P.Y.: Metrics and models for handwritten character recognition. Stat. Sci. 13(1), 54–65 (1998)

    Article  MATH  Google Scholar 

  21. Hild, K., Erdogmus, D., Torkkola, K., Principe, J.: Feature extraction using information-theoretic learning. IEEE Trans. Pattern Anal. Mach. Intell. 28(9), 1385–1392 (2006)

    Article  Google Scholar 

  22. Hochwald, B.M., Marzetta, T.L.: Unitary space-time modulation for multiple-antenna communications in Rayleigh flat fading. IEEE Trans. Inf. Theory 46(2), 543–564 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  23. Hochwald, B.M., Marzetta, T.L., Richardson, T.J., Sweldens, W., Urbanke, R.: Systematic design of unitary spce-time constellations. IEEE Trans. Inf. Theory 46(6), 1962–1973 (2000)

    Article  Google Scholar 

  24. Hull, J.J.: A database for handwritten text recognition research. IEEE Trans. Pattern Anal. Mach. Intell. 16(5), 550–554 (1994)

    Article  Google Scholar 

  25. Kaski, S., Peltonen, J.: Informative discriminant analysis. In: International Conference on Machine Learning, Washington, DC, USA, pp. 329–336 (2003)

    Google Scholar 

  26. LeCun, Y., Jackel, L., Bottou, L., Brunot, A., Cortes, C., Denker, J., Drucker, H., Guyon, I., Muller, U., Sackinger, E., Simard, P., Vapnik, V.: Comparison of learning algorithms for handwritten digit recognition. In: International Conference on Artificial Neural Networks, Warsaw, Poland, pp. 53–60 (1995)

    Google Scholar 

  27. Lee, K.-C., Ho, J., Kriegman, D.J.: Acquiring linear subspaces for face recognition under variable lighting. IEEE Trans. Pattern Anal. Mach. Intell. 27(5), 684–698 (2005)

    Article  Google Scholar 

  28. Liu, G., Lin, Z., Yu, Y.: Robust subspace segmentation by low-rank representation. In: International Conference on Machine Learning, Haifa, Israel, pp. 663–670 (2010)

    Google Scholar 

  29. Liu, L., Fieguth, P.: Texture classification from random features. IEEE Trans. Pattern Anal. Mach. Intell. 34(3), 574–586 (2012)

    Article  Google Scholar 

  30. Marzetta, T.L., Hochwald, B.M.: Capacity of a mobile multiple-antenna communication link in Rayleigh flat fading. IEEE Trans. Inf. Theory 45(1), 139–157 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  31. Nenadic, Z.: Information discriminant analysis: feature extraction with an information-theoretic objective. IEEE Trans. Pattern Anal. Mach. Intell. 29(8), 1394–1407 (2007)

    Article  Google Scholar 

  32. Nokleby, M., Rodrigues, M.R.D., Calderbank, R.: Discrimination on the Grassmann manifold: fundamental limits of subspace classifiers. Available at http://arxiv.org/abs/1404.5187

  33. Qiu, Q., Sapiro, G.: Learning robust subspace clustering. Available at http://arxiv.org/abs/1308.0273

  34. Qiu, Q., Sapiro, G.: Learning transformations for clustering and classification. Available at http://arxiv.org/abs/1309.2074

  35. Qiu, Q., Sapiro, G.: Learning transformations for classification forests. Available at http://arxiv.org/abs/1312.5604

  36. Reboredo, H., Renna, F., Calderbank, R., Rodrigues, M.R.D.: Compressive classification of a mixture of Gaussians: analysis, designs and geometrical interpretation. Available at http://arxiv.org/abs/1401.6962

  37. Reeves, G., Gastpar, M.: The sampling rate distortion tradeoff for sparsity pattern recovery in compressed sensing. IEEE Trans. Inf. Theory 58(5), 3065–3092 (2012)

    Article  MathSciNet  Google Scholar 

  38. Reeves, G., Gastpar, M.: Approximate sparsity pattern recovery: information-theoretic lower bounds. IEEE Trans. Inf. Theory 59(6), 3451–3465 (2013)

    Article  MathSciNet  Google Scholar 

  39. Renna, F., Calderbank, R., Carin, L., Rodrigues, M.R.D.: Reconstruction of signals drawn from a Gaussian mixture from noisy compressive measurements. IEEE Trans. Signal Process. 62(9), 2265–2277 (2014)

    Article  MathSciNet  Google Scholar 

  40. Ross, D.T., et al.: Systematic variation in gene expression patterns in human cancer cell lines. Nat. Genet. 24(3), 227–235 (2000)

    Article  Google Scholar 

  41. Royston, J.P.: Some techniques for assessing multivariate normality based on the Shapiro-Wilk W. Appl. Stat. 32(2), 121–133 (1983)

    Article  Google Scholar 

  42. Soltanolkotabi, M., Candès, E.: A geometric analysis of subspace clustering with outliers. Ann. Stat. 40(4), 2195–2238 (2012)

    Article  MATH  Google Scholar 

  43. Soltanolkotabi, M., Elhamifar, E., Candès, E.: Robust subspace clustering. Available at arxiv.org/abs/1301.2603

    Google Scholar 

  44. Tao, D., Li, X., Wu, X., Maybank, S.: Geometric mean for subspace selection. IEEE Trans. Pattern Anal. Mach. Intell. 31(2), 260–274 (2009)

    Article  Google Scholar 

  45. Torkkola, K.: Learning discriminative feature transforms to low dimensions in low dimensions. In: Advances in Neural Information Processing Systems, Vancouver, Canada, pp. 969–976 (2001)

    Google Scholar 

  46. Torkkola, K.: Feature extraction by non-parametric mutual information maximization. J. Mach. Learn. Res. 3, 1415–1438 (2003)

    MathSciNet  MATH  Google Scholar 

  47. Tenenbaum, J., de Silva, V., Langford, J.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000)

    Article  Google Scholar 

  48. Tulino, A., Caire, G., Verdú, S., Shamai, S.: Support recovery with sparsely sampled free random matrices. IEEE Trans. Inf. Theory 59(7), 4243–4271 (2013)

    Article  Google Scholar 

  49. Wainwright, M.: Sharp thresholds for high-dimensional and noisy sparsity recovery using l1-constrained quadratic programming (lasso). IEEE Trans. Inf. Theory 55(5), 2183–2202 (2009)

    Article  MathSciNet  Google Scholar 

  50. Wainwright, M.: Information-theoretic limits on sparsity recovery in the high dimensional and noisy setting. IEEE Trans. Inf. Theory 55(12), 5728–5741 (2009)

    Article  MathSciNet  Google Scholar 

  51. Wang, W., Wainwright, M., Ramchandran, K.: Information-theoretic limits on sparse signal recovery: Dense versus sparse measurement matrices. IEEE Trans. Inf. Theory 56(6), 2967–2979 (2010)

    Article  MathSciNet  Google Scholar 

  52. Wang, L., Carlson, D., Rodrigues, M.R.D., Wilcox, D., Calderbank, R., Carin, L.: Designed measurements for vector count data. In: Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, pp. 1142–1150 (2013)

    Google Scholar 

  53. Wang, L., Razi, A., Rodrigues, M.R.D., Calderbank, R., Carin, L.: Nonlinear information-theoretic compressive measurement design. In: International Conference on Machine Learning, Beijing, China, pp. 1161–1169 (2014)

    Google Scholar 

  54. Wang, Y., Xu, H.: Noisy sparse subspace clustering. In: International Conference on Machine Learning, Atlanta, GA, USA, pp. 89–97 (2013)

    Google Scholar 

  55. Wright, J., Yang, M., Ganesh, A., Sastry, S., Ma, Y.: Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31(2), 210–227 (2009)

    Article  Google Scholar 

  56. Zheng, L., Tse, D.: The diversity-multiplexing tradeoff for non-coherent multiple antenna channels. In: The Annual Allerton Conference on Communication, Control and Computing, Monticello, IL, USA, pp. 1011–1020 (2002)

    Google Scholar 

  57. Zheng, L., Tse, D.: Communication on the Grassmann manifold: a geometric approach to the noncoherent multiple-antenna channel. IEEE Trans. Inf. Theory 48(2), 359–383 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  58. Zheng, L., Tse, D.: Diversity and multiplexing: a fundamental tradeoff in multiple-antenna channels. IEEE Trans. Inf. Theory 49(5), 1073–1096 (2003)

    Article  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by the Royal Society International Exchanges Scheme IE120996. The work of Robert Calderbank and Matthew Nokleby is also supported in part by the Air Force Office of Scientific Research under the Complex Networks Program.

Author information

Authors and Affiliations

  1. Department of Electronic and Electrical Engineering, University College London, London, UK

    Miguel Rodrigues

  2. Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA

    Matthew Nokleby & Robert Calderbank

  3. Department of Computer Science, Instituto de Telecomunicações, University of Porto, Porto, Portugal

    Francesco Renna

Authors
  1. Miguel Rodrigues
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matthew Nokleby
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Francesco Renna
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Robert Calderbank
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miguel Rodrigues .

Editor information

Editors and Affiliations

  1. Lehrstuhl für Theoretische Informationstechnik, Technische Universität München, München, Bayern, Germany

    Holger Boche

  2. Department of Electrical and Computer Engineering, Duke University, Durham, North Carolina, USA

    Robert Calderbank

  3. Institut für Mathematik, Technische Universität Berlin, Berlin, Berlin, Germany

    Gitta Kutyniok

  4. Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic

    Jan Vybíral

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Rodrigues, M., Nokleby, M., Renna, F., Calderbank, R. (2015). Compressive Classification: Where Wireless Communications Meets Machine Learning. In: Boche, H., Calderbank, R., Kutyniok, G., Vybíral, J. (eds) Compressed Sensing and its Applications. Applied and Numerical Harmonic Analysis. Birkhäuser, Cham. https://doi.org/10.1007/978-3-319-16042-9_15

Download citation

Publish with us

Policies and ethics

Search

Navigation