Statistical Learning Theory

Abstract

A machine learning system, in general, learns from the environment, but statistical machine learning programs (systems) learn from the data. This chapter presents techniques for statistical machine learning using Support Vector Machines (SVM) to recognize the patterns and classify them, predicting structured objects using SVM, k-nearest neighbor method for classification, and Naive Bayes classifiers. The artificial neural networks are presented with brief introduction to error-correction rules, Boltzmann learning, Hebbian rule, competitive learning rule, and deep learning. The instance-based learning is treated in details with its algorithm and learning task. The chapter concludes with a summary, and a set of practice exercises.

Exercises

• 1. “The task of text categorization is to assign a given document to one of the categories out of a fixed set of categories. This is done on the basis text contents. The Naive Bayes model is often used for this purpose, where a query variable is the document category and the “effect” variables are presence/ absence of each word in the language. It is assumed that words occur independently in the documents, and their frequencies determine the document category.“ For this statement,

  • a. Explain how such a models can be constructed, given a set of “training data” in the form of documents that have been already assigned to categories.

  • b. Explain how to categorize a new document.

  • c. Is the independence assumption reasonable? Justify your answer.

• 2. What linear or nonlinear function is used by an SVM for performing classification? How is an input vector $\mathbf {x}_i$ (instance) assigned to the positive or negative classes.

• 3. Consider the SVM for the training data given in $\mathfrak {R}^2$, in Fig. 14.12a, b; find out the separating hyperplanes in both the cases.

• 4. Why is the Naive Bayes classification called Naive? What are the main goals behind this classification?

• 5. Consider the data given in Table 14.1, and use these to train a Naive Bayes classifier with designation attribute as the class label and all the remaining attributes regarded as input. Once you have your Naive Bayesian classifier, test the following unseen instances to find out the class:

  • a. Marketing, $36\ldots 41$, $51K \ldots 55K$

  • b. Sale, $36\ldots 41$, $71K \ldots 75K$

Fig. 14.12

Training data for SVMs

Table 14.1 Data set for machine learning