Abstract
The raw format of real data is usually widely variable. Many values may be missing, inconsistent across different data sources, and erroneous. For the analyst, this leads to numerous challenges in using the data effectively. For example, consider the case of evaluating the interests of consumers from their activity on a social media site. The analyst may first need to determine the types of activity that are valuable to the mining process. The activity might correspond to the interests entered by the user, the comments entered by the user, and the set of friendships of the user along with their interests. All these pieces of information are diverse and need to be collected from different databases within the social media site. Furthermore, some forms of data, such as raw logs, are often not directly usable because of their unstructured nature. In other words, useful features need to be extracted from these data sources. Therefore, a data preparation phase is needed.
“Success depends upon previous preparation, and without such preparation there is sure to be failure.”—Confucius
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Setting the gradient of the Lagrangian relaxation \(\overline{v}^T C \overline{v} - \lambda (||\overline{v}||^2 -1)\) to 0 is equivalent to the eigenvector condition \(C \overline{v} - \lambda \overline{v}=0\). The variance along an eigenvector is \(\overline{v}^T C \overline{v}= \overline{v}^T \lambda \overline{v} = \lambda\). Therefore, one should include the orthonormal eigenvectors in decreasing order of eigenvalue λ to maximize preserved variance in reduced subspace.
- 2.
The means of the remaining columns also need be stored if the data set is not mean centered.
- 3.
The squared error is the sum of squares of the entries in the error matrix \(D - Q_k \Sigma_k P_k^T\).
- 4.
Concepts that are not present predominantly in the collection will be ignored by truncation. Therefore, alternative meanings reflecting infrequent concepts in the collection will be ignored. While this has a robust effect on the average, it may not always be the correct or complete disambiguation of polysemous words.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Aggarwal, C. (2015). Data Preparation. In: Data Mining. Springer, Cham. https://doi.org/10.1007/978-3-319-14142-8_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-14142-8_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-14141-1
Online ISBN: 978-3-319-14142-8
eBook Packages: Computer ScienceComputer Science (R0)