Getting to Know Thy Data

Abstract

This chapter discusses and demonstrates the importance of getting to know the data that we use to conduct higher education policy analysis and evaluation. More specifically, this chapter addresses the need to know the structure of datasets. The identification and exploration of missing data are also discussed in this chapter.

5.6 Appendix

*Chapter 5 Syntax *use time series data from Chap. 4. cd "C:\Users\Marvin\Dropbox\Manuscripts\Book\Chapter 4\Stata" use "Percent of US high school graduates in PSE, 1960 to 2016.dta" *examine structure of the dataset describe *reduce the amount of memory required by float by compressing the data using /// compress *compare after compressing, show structure describe *open panel dataset cd "C:\Users\Marvin\Dropbox\Manuscripts\Book\Chapter 5\Stata files" use "Example 5.0.dta" *compress the data and show structure compress describe * recast int id describe *save save "Example 5.0.dta", replace *clear all *using a large amount of data from secondary data sources such as the /// National Center for Education Statistics’ NCES /// public-use High School Longitudinal Study of 2009 (HSLS:09) student dataset *set the maximum variables to 10,0000 set maxvar 10000 *download all student data from the HSLS:09 dataset in Stata *examine a shortened version of the HSLS:09 dataset’s structure describe, short *look at how much memory this dataset uses memory *try to see if we can compress the data compress *close dataset clear all *import an Excel file (SHEEO finance data) cd "C:\Users\Marvin\Dropbox\Manuscripts\Book\Chapter 5\Excel files" import excel /// "SHEEO_SHEF_FY18_Nominal_Data.xlsx", sheet("State and U.S. Nominal Data (2") firstrow *Because we want to use only post-Great Recession data, we drop observations /// if they are prior to fiscal year (FY) 2010 or if less than FY 2010. drop if FY<2010 *We use the list command to take a quick look at the data, particularly /// with respect to FY 2010. We will also make the command /// conditional by using list if FY==2010 *Because we want only states in our dataset, we drop all observations for /// the U.S. total and Washington DC. drop if State=="US" drop if State=="Washington DC" *we employ the user-created statastates (Schpero 2018) program to create /// fips codes and other state identifiers; include nogenerate option to /// prevent the generation the variable _merge statastates, name(State) nogenerate *To create a variable, stateid, based on state names, we use egen. egen stateid = group(State) *We use the compress command to save computer memory. compress *we use stateid and FY to declare the dataset to be a panel xtset stateid FY, yearly *data are saved to a file with a new name. cd "C:\Users\Marvin\Dropbox\Manuscripts\Book\Chapter 5\Stata files" save "Example 5.2.dta" clear all *Using selected variables from the public-use version of the HSLS:09 /// at we saved in a Stata file (i.e., Example 5.3) use "Example 5.3.dta" * determine if and how missing data are coded of the variable S3CLGPELL Codebook S3CLGPELL *before conducting missing data analysis, we would need to /// change to change -9 to “. “ for all variables mvdecode _all, mv(-9=.) *Missing data analysis *install Stata user-created program, mdesc (Medeiros and Blanchette 2011) install ssc mdesc, replace *produce a table with the number of missing values, total number of cases, /// and percent missing for each variable in our file. mdesc *use the Stata command misstable tree, with various options, to show the /// pattern of “missingness” in the data misstable tree *use Stata command, misstable misstable patterns *use another option, misstable tree, frequency *conduct missing data analysis employing Stata user-created /// routine "xtmis" (Nguyen 2008). The program must be installed by typing: ssc install tomata ssc install xtmis * use the Stata command "tostring" to create a string variable (unitid_s), /// based on the numeric IPEDS variable (unitid). Then we invoke xtmis. tostring unitid, generate(unitid_s) xtmis grantlow , id(unitid_s) * Stata user-created "missings" command; install most recent version /// net install dm0085_1.pkg, replace *examine missing data by SES quartiles bysort X1SESQ5 : missings table *same command can be repeated by racial-ethnic groups. bysort X1RACE : missings table *Missing Data - Missing Completely at Random *install the Stata user-written program, mcartest (Li 2013) net install st0318.pkg, replace *set maximum variables to 10,000 and Open a large dataset – public /// use version of the HSLS09 set maxvar 10000 use "HSLS09.dta" *keep selected variables keep STU_ID X1SEX X1RACE X1SES X1SESQ5 X4ATPRLVLA S3CLGPELL P1TUITION *convert code (-9) for missing data to “.” mvdecode _all, mv(-9=.) *test assumption of missing completely at random (MCAR) of two variables mcartest S3CLGPELL P1TUITION *add covariates to test the covariate-dependent missingness (CDM) mcartest S3CLGPELL P1TUITION = i.X1RACE if X1RACE !=. , unequal emoutput nolog *exit Stata exit *end