Labeled Compression Schemes for Extremal Classes

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9925)


It is a long-standing open problem whether there exists a compression scheme whose size is of the order of the Vapnik-Chervonienkis (VC) dimension d. Recently compression schemes of size exponential in d have been found for any concept class of VC dimension d. Previously, compression schemes of size d have been given for maximum classes, which are special concept classes whose size equals an upper bound due to Sauer-Shelah. We consider a generalization of maximum classes called extremal classes. Their definition is based on a powerful generalization of the Sauer-Shelah bound called the Sandwich Theorem, which has been studied in several areas of combinatorics and computer science. The key result of the paper is a construction of a sample compression scheme for extremal classes of size equal to their VC dimension. We also give a number of open problems concerning the combinatorial structure of extremal classes and the existence of unlabeled compression schemes for them.


  1. 1.
    Anstee, R., Rónyai, L., Sali, A.: Shattering news. Graphs Comb. 18(1), 59–73 (2002)MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Bandelt, H., Chepoi, V., Dress, A., Koolen, J.: Combinatorics of lopsided sets. Eur. J. Comb. 27(5), 669–689 (2006)MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Ben-David, S., Litman, A.: Combinatorial variability of Vapnik-Chervonenkis classes with applications to sample compression schemes. Discret. Appl. Math. 86(1), 3–25 (1998)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Blumer, A., Ehrenfeucht, A., Haussler, D., Warmuth, M.K.: Learnability and the Vapnik-Chervonenkis dimension. J. Assoc. Comput. Mach. 36(4), 929–965 (1989)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Blumer, A., Ehrenfeucht, A., Haussler, D., Warmuth, M.K.: Occam’s razor. Inf. Process. Lett. 24(6), 377–380 (1987)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Bollobás, B., Radcliffe, A.J.: Defect Sauer results. J. Comb. Theory Ser. A 72(2), 189–208 (1995)MathSciNetCrossRefMATHGoogle Scholar
  7. 7.
    Bollobás, B., Radcliffe, A.J., Leader, I.: Reverse Kleitman inequalities. Proc. Lond. Math. Soc. Ser. A (3) 58, 153–168 (1989)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Chernikov, A., Simon, P.: Externally definable sets and dependent pairs. Isr. J. Math. 194(1), 409–425 (2013)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Daniely, A., Shalev-Shwartz, S.: Optimal learners for multiclass problems. In: COLT, pp. 287–316 (2014)Google Scholar
  10. 10.
    Doliwa, T., Simon, H.U., Zilles, S.: Recursive teaching dimension, learning complexity, and maximum classes. In: Hutter, M., Stephan, F., Vovk, V., Zeugmann, T. (eds.) ALT 2015. Lecture Notes in Artificial Intelligence (LNAI), vol. 6331, pp. 209–223. Springer, Heidelberg (2010). doi:10.1007/978-3-642-16108-7_19 CrossRefGoogle Scholar
  11. 11.
    Dress, A.: Towards a theory of holistic clustering. DIMACS Ser. Discret. Math. Theoret. Comput. Sci. 37, 271–289 (1997). (Amer. Math. Soc.)MathSciNetMATHGoogle Scholar
  12. 12.
    Floyd, S., Warmuth, M.K.: Sample compression, learnability, and the Vapnik-Chervonenkis dimension. Mach. Learn. 21(3), 269–304 (1995)Google Scholar
  13. 13.
    Freund, Y., Schapire, R.E.: Boosting: Foundations and Algorithms. Adaptive Computation and Machine Learning. MIT Press, Cambridge (2012)MATHGoogle Scholar
  14. 14.
    Gartner, B., Welzl, E.: Vapnik-Chervonenkis dimension and (pseudo-)hyperplane arrangements. Discret. Comput. Geom. (DCG) 12, 399–432 (1994)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Greco, G.: Embeddings and the trace of finite sets. Inf. Process. Lett. 67(4), 199–203 (1998)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Kozma, L., Moran, S.: Shattering, graph orientations, and connectivity. Electron. J. Comb. 20(3), P44 (2013)MathSciNetMATHGoogle Scholar
  17. 17.
    Kuzmin, D., Warmuth, M.K.: Unlabeled compression schemes for maximum classes. J. Mach. Learn. Res. 8, 2047–2081 (2007)MathSciNetMATHGoogle Scholar
  18. 18.
    Lawrence, J.: Lopsided sets and orthant-intersection by convex sets. Pac. J. Math. 104(1), 155–173 (1983)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Litman, A., Moran, S.: Unpublished results (2012)Google Scholar
  20. 20.
    Littlestone, N., Warmuth, M.: Relating data compression and learnability (1986, Unpublished)Google Scholar
  21. 21.
    Livni, R., Simon, P.: Honest compressions and their application to compression schemes. In: COLT, pp. 77–92 (2013)Google Scholar
  22. 22.
    Mészáros, T., Rónyai, L.: Shattering-extremal set systems of VC dimension at most 2. Electron. J. Comb. 21(4), P4.30 (2014)MathSciNetMATHGoogle Scholar
  23. 23.
    Moran, S.: Shattering-extremal systems (2012). CoRR abs/1211.2980Google Scholar
  24. 24.
    Moran, S., Shpilka, A., Wigderson, A., Yehudayoff, A.: Teaching and compressing for low VC-dimension. In: ECCC TR15-025 (2015)Google Scholar
  25. 25.
    Moran, S., Warmuth, M.K.: Labeled compression schemes for extremal classes (2015). CoRR abs/1506.00165.
  26. 26.
    Moran, S., Yehudayoff, A.: Sample compression schemes for VC classes. J. ACM 63(3), 21:1–21:10 (2016)CrossRefGoogle Scholar
  27. 27.
    Pajor, A.: Sous-espaces \(l^{n}_{1}\) des espaces de banach. Travaux en Cours. Hermann, Paris (1985)Google Scholar
  28. 28.
    Rubinstein, B.I.P., Rubinstein, J.H.: A geometric approach to sample compression. J. Mach. Learn. Res. 13, 1221–1261 (2012)MathSciNetMATHGoogle Scholar
  29. 29.
    Samei, R., Yang, B., Zilles, S.: Generalizing labeled and unlabeled sample compression to multi-label concept classes. In: Auer, P., Clark, A., Zeugmann, T., Zilles, S. (eds.) ALT 2015. Lecture Notes in Artificial Intelligence (LNAI), vol. 8776, pp. 275–290. Springer, Heidelberg (2014). doi:10.1007/978-3-319-11662-4_20 Google Scholar
  30. 30.
    Sauer, N.: On the density of families of sets. J. Comb. Theory Ser. A 13, 145–147 (1972)MathSciNetCrossRefMATHGoogle Scholar
  31. 31.
    Shelah, S.: A combinatorial problem; stability and order for models and theories in infinitary languages. Pac. J. Math. 41, 247–261 (1972)MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Valiant, L.: A theory of the learnable. Commun. ACM 27, 1134–1142 (1984)CrossRefMATHGoogle Scholar
  33. 33.
    Warmuth, M.K.: Compressing to VC dimension many points. In: Schölkopf, B., Warmuth, M.K. (eds.) COLT/Kernel 2003. LNCS (LNAI), vol. 2777, pp. 743–744. Springer, Heidelberg (2003)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Technion, Israel Institute of TechnologyHaifaIsrael
  2. 2.Microsoft ResearchHerzliyaIsrael
  3. 3.Max Planck Institute for InformaticsSaarbrückenGermany
  4. 4.Computer Science DepartmentUniversity of CaliforniaSanta CruzUSA

Personalised recommendations