When Does Greedy Learning of Relevant Attributes Succeed?

— A Fourier-Based Characterization —
  • Jan Arpe
  • Rüdiger Reischuk
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4598)


We introduce a new notion called Fourier-accessibility that allows us to precisely characterize the class of Boolean functions for which a standard greedy learning algorithm successfully learns all relevant attributes. If the target function is Fourier-accessible, then the success probability of the greedy algorithm can be made arbitrarily close to one. On the other hand, if the target function is not Fourier-accessible, then the error probability tends to one. Finally, we extend these results to the situation where the input data are corrupted by random attribute and classification noise and prove that greedy learning is quite robust against such errors.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Akutsu, T., Bao, F.: Approximating Minimum Keys and Optimal Substructure Screens. In: Cai, J.-Y., Wong, C.K. (eds.) COCOON 1996. LNCS, vol. 1090, pp. 290–299. Springer, Heidelberg (1996)Google Scholar
  2. 2.
    Akutsu, T., Miyano, S., Kuhara, S.: Algorithms for Identifying Boolean Networks and Related Biological Networks Based on Matrix Multiplication and Fingerprint Function. J. Comput. Biology 7(3-4), 331–343 (2000)CrossRefGoogle Scholar
  3. 3.
    Akutsu, T., Miyano, S., Kuhara, S.: A Simple Greedy Algorithm for Finding Functional Relations: Efficient Implementation and Average Case Analysis. Theoret. Comput. Sci. 292(2), 481–495 (2003)MATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Almuallim, H., Dietterich, T.G.: Learning Boolean Concepts in the Presence of Many Irrelevant Features. Artificial Intelligence 69(1-2), 279–305 (1994)MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Alon, N., Spencer, J.: The Probabilistic Method. Wiley-Intersci. Ser. Discrete Math. Optim. John Wiley and Sons, Chichester (1992)Google Scholar
  6. 6.
    Arpe, J.: Learning Concepts with Few Unknown Relevant Attributes from Noisy Data. PhD thesis, Institut für Theoretische Informatik, Universität zu Lübeck (2006)Google Scholar
  7. 7.
    Arpe, J., Reischuk, R.: Robust Inference of Relevant Attributes. In: Gavaldá, R., Jantke, K.P., Takimoto, E. (eds.) ALT 2003. LNCS (LNAI), vol. 2842, pp. 99–113. Springer, Heidelberg (2003)Google Scholar
  8. 8.
    Arpe, J., Reischuk, R.: Learning Juntas in the Presence of Noise. In: Cai, J.-Y., Cooper, S.B., Li, A. (eds.) TAMC 2006. LNCS, vol. 3959, pp. 387–398. Springer, Heidelberg, Invited to appear in special issue of TAMC 2006 in Theoret. Comput. Sci., Series A (2006)CrossRefGoogle Scholar
  9. 9.
    Arpe, J., Reischuk, R.: When Does Greedy Learning of Relevant Attributes Succeed?—A Fourier-based Characterization. Technical Report ECCC TR06-065, Electronic Colloquium on Computational Complexity (2006)Google Scholar
  10. 10.
    Bahadur, R.R.: A Representation of the Joint Distribution of Responses to n Dichotomous Items. In: Solomon, H. (ed.) Studies in Item Analysis and Prediction, pp. 158–168. Stanford University Press, Stanford (1961)Google Scholar
  11. 11.
    Bernasconi, A.: Mathematical Techniques for the Analysis of Boolean Functions. PhD thesis, Università degli Studi di Pisa, Dipartimento di Ricerca in Informatica (1998)Google Scholar
  12. 12.
    Blum, A., Furst, M., Jackson, J.C., Kearns, M., Mansour, Y., Rudich, S.: Weakly Learning DNF and Characterizing Statistical Query Learning Using Fourier Analysis. In: Proc. 26th STOC 1994, pp. 253–262 (1994)Google Scholar
  13. 13.
    Blum, A., Langley, P.: Selection of Relevant Features and Examples in Machine Learning. Artificial Intelligence 97(1-2), 245–271 (1997)MATHCrossRefMathSciNetGoogle Scholar
  14. 14.
    Blumer, A., Ehrenfeucht, A., Haussler, D., Warmuth, M.K.: Occam’s Razor. Inform. Process. Lett. 24(6), 377–380 (1987)MATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    Boros, E., Horiyama, T., Ibaraki, T., Makino, K., Yagiura, M.: Finding Essential Attributes from Binary Data. Ann. Math. Artif. Intell. 39(3), 223–257 (2003)MATHCrossRefMathSciNetGoogle Scholar
  16. 16.
    Chvátal, V.: A Greedy Heuristic for the Set Covering Problem. Math. Oper. Res. 4(3), 233–235 (1979)MATHMathSciNetCrossRefGoogle Scholar
  17. 17.
    Feige, U.: A Threshold of ln n for Approximating Set Cover. J. ACM 45(4), 634–652 (1998)MATHCrossRefMathSciNetGoogle Scholar
  18. 18.
    Fukagawa, D., Akutsu, T.: Performance Analysis of a Greedy Algorithm for Inferring Boolean Functions. Inform. Process. Lett. 93(1), 7–12 (2005)CrossRefMathSciNetMATHGoogle Scholar
  19. 19.
    Furst, M.L., Jackson, J.C., Smith, S.W.: Improved Learning of AC 0 Functions. In: Proc. 4th COLT 1991, pp. 317–325Google Scholar
  20. 20.
    Johnson, D.S.: Approximation Algorithms for Combinatorial Problems. J. Comput. System Sci. 9(3), 256–278 (1974)MATHMathSciNetCrossRefGoogle Scholar
  21. 21.
    Kleinberg, J., Tardos, É.: Algorithm Design. Addison-Wesley, Reading (2005)Google Scholar
  22. 22.
    Linial, N., Mansour, Y., Nisan, N.: Constant Depth Circuits, Fourier Transform, and Learnability. J. ACM 40(3), 607–620 (1993)MATHCrossRefMathSciNetGoogle Scholar
  23. 23.
    Mansour, Y.: Learning Boolean Functions via the Fourier Transform. In: Roychodhury, V., Siu, K.-Y., Orlitsky, A. (eds.) Theoretical Advances in Neural Computation and Learning, pp. 391–424. Kluwer Academic Publishers, Dordrecht (1994)Google Scholar
  24. 24.
    Mossel, E., O’Donnell, R.W., Servedio, R.A.: Learning functions of k relevant variables. J. Comput. System Sci. 69(3), 421–434 (2004)MATHCrossRefMathSciNetGoogle Scholar
  25. 25.
    Reischuk, R.: Can Large Fanin Circuits Perform Reliable Computations in the Presence of Noise? Theoretical Comput. Sci. 240(4), 319–335 (2000)MATHCrossRefMathSciNetGoogle Scholar
  26. 26.
    Shamir, R., Dietrich, B.: Characterization and Algorithms for Greedily Solvable Transportation Problems. In: Proc. 1st SODA 1990, pp. 358–366 (1990)Google Scholar
  27. 27.
    Slavík, P.: A Tight Analysis of the Greedy Algorithm for Set Cover. In: Proc. 28th STOC 1996, pp. 435–441 (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Jan Arpe
    • 1
  • Rüdiger Reischuk
    • 1
  1. 1.Institut für Theoretische Informatik, Universität zu Lübeck, Ratzeburger Allee 160, 23538 LübeckGermany

Personalised recommendations