Decomposition of Inequality of Opportunity in India: An Application of Data-Driven Machine Learning Approach

Abstract

This paper introduces a novel measure of inequality of opportunity (IOp) in India, by comparing both ex-ante and ex-post results, which aligns with Roemer’s (1998) equality of opportunity, theory. The study utilizes data-driven machine learning algorithms, namely conditional inference tree and conditional inference forest, to measure ex-ante IOp, and a transformation tree to estimate ex-post IOp. The findings indicate that, according to the ex-ante approach, approximately 58–61 percent of the overall income inequality can be attributed to variations in circumstances, while around 46 percent of the overall income inequality is explained by differences in the degree of efforts. The results from the tree-based analysis reveal that parents’ occupation, sector (rural–urban areas), and geographical regions are the primary circumstances contributing to IOp, which is further confirmed by the Shapley decomposition exercise. Specifically, individuals residing in rural areas in the eastern and central parts of the country, whose parents are employed in low-skilled and unskilled occupations, and have below secondary and no formal education, and who belong to marginalized social groups, exhibit significantly lower average income. Consequently, it is crucial to implement regional-level development policies that specifically target marginalized groups in order to foster a more equitable society and mitigate overall income inequality.

Appendices

Appendix 1: Sample Selection and Construction of the Variables

Variable Selection

From the PLFS 2018–19, six variables have been selected, three variables namely sector, caste, and gender are used in the existing form, while another three variables namely states, parents’ education, and parents’ occupations are modified, and created in the new form. The sector is categorized as rural, and urban; gender as male, female, and caste as General Caste (GEN), Scheduled Caste (SC), Scheduled Tribe (ST), and Other Backward Classes (OBC). In the gender variable, transgender has been dropped before the analysis.

The state variable is categorized into 36 states/union territories of India, which have been modified and classified into six broad following geographical regions as follows:

• 1. North: Jammu and Kashmir, Himachal Pradesh, Punjab, and Haryana

• 2. East: Bihar, Jharkhand, Orissa, and West Bengal

• 3. Central: Uttar Pradesh, Rajasthan, Madhya Pradesh, Uttarakhand, and Chhattisgarh

• 4. North-East: Sikkim, Arunachal Pradesh, Assam, Nagaland, Meghalaya, Manipur, Mizoram, and Tripura

• 5. South: Karnataka, Andhra Pradesh, Tamil Nadu, Pondicherry, Kerala, and Lakshadweep

• 6. West: Gujrat, Daman and Diu, Dadra and Nagar Haveli, Maharashtra, and Goa.

The education variable is classified into following four broad categories as follows:

• 1. Illiterate or no education: (code 1: Illiterate)

• 2. Below secondary: (code 2–7, literate to up to middle school)

• 3. Secondary and above secondary: (code 8–10, secondary to higher secondary)

• 4. Graduate and above: (code 12–13, graduate and post-graduate)

The occupation/skill level is classified into following four broad categories using NCO (National Classification of Occupations) at one digit: (as per.OECD Employment Outlook 2014; NCO, 2015, Ministry of Labour and Employment, Government of India).

1.Unskilled or routine manual task: Typically involves the performance of simple and routine physical or manual tasks (NCO code 9: Elementary Occupations or unskilled such as domestic helpers, cleaners, street vendors and garbage collectors)

2. Low-skilled or non-routine manual task: Typically involves the performance of tasks such as operating machinery and electronic equipment, driving vehicles, maintenance and repair of electrical and mechanical equipment and manipulation, ordering and storage information (NCO code 4–8, low skilled as clerical jobs, service workers, shop and market sales workers, craft and related trade workers, etc.).

3. Medium skill or non-routine cognitive task: Typically involves the performance of complex technical and practical tasks that require an extensive body of factual, technical and procedural knowledge in a specialized field (NCO code 3, as professional and technical associates); and.

4.High skilled or cognitive task: Typically involves the performance of tasks that require complex problem solving, decision making and creativity based on an extensive body of theoretical and factual knowledge in a specialized field (NCO 2 as professional and Technicians).

The concept of skill level is not applied in the case of NCO code 1 as legislators, managers, etc., as skills for executing tasks and duties of these occupations varied to such an extent that it was not feasible to link them with any of the four, broad skill levels.

Sample Selection

For the selection of sample following multi-stage procedure has been adopted.

In the first stage, the parent of each respondent has been identified using the relation to the head variable in the data. For an individual identified as self (code 1), the household member with code 7 (labeled Father/Mother/Father-in-Law/Mother-in-Law) was treated as the parents and prepared the first set of data with children and parents.

In the second stage, the individuals as unmarried children (code 5) and married children (code 3) have been identified, and further, the parents of these children are identified as household heads labeled as self (code 1) in the data. T the respondent labeled self was identified as the parent and prepared the second set of data with children and parents.

In case of duplicate records (or multiple parental information), we have deleted the duplicate case by carefully looking at the unit records. Once both the files were cleaned, we merged both the files along with key variables in the data as discussed above.

Appendix 2: Grid Search CV Process for Conditional Inference Tree and Conditional Inference Forest

In the Grid Search CV process, the data are divided into training and test sets. Different combinations of min-split (minimum number of observations required to perform a split) and alpha values are tested, and the combination that yields the lowest root mean squared error (RMSE) for the test set is selected. The RMSE is a measure of the model's prediction accuracy. For Conditional inference tree model with MPCI as dependent variables, the Grid Search CV has been conducted. After evaluating various combinations, an alpha value of 0.07 and a min-split value of 10,000 provide the lowest RMSE. The robustness of the endogenously chosen alpha is examined by comparing the results with alpha values of 0.01 and 0.05 as given in Table

Table A.2.1 Ctree results MPCI

This comparison is done following the approach outlined by Salas-Rojo and Rodriguez (2022).

Similarly, after evaluating various combinations, an alpha value of 0.06 and number of tree at 200 provide the lowest RMSE for conditional inference forest model. The robustness of the endogenously chosen alpha is examined by comparing the results with alpha values of 0.01 and 0.05 as given

Table A.2.2 Cforest results MPCI

Appendix 3: Plots for MPCI