Student Learning and Teacher Competence

Abstract

In this chapter, we analyze individual primary school children learning outcomes, using a unique dataset, the SDI dataset that comprises information about learning achievement for students, schools, and teachers. The novelty of the dataset is the collection of information from teachers, testing their knowledge in math, French, and non-verbal reasoning. Our results show that enrolling in a private school and living in an urban area are associated with better learning outcomes, even when the characteristics of the schools and teachers are controlled for. The results also show a worryingly low level of competency for teachers. Overall, the results point to a deep learning crisis, so deep that serious public action is needed to address it: changes at the margin are unlikely to suffice.

4.1 Introduction to the Chapter

Our goal is to investigate individual learning outcome drivers. This is carried out by using pupils’ test scores obtained from an independent learning assessment survey: the Survey and Delivery Indicators (SDI) survey. This is opposed to the way student performance was measured in the previous chapter, where we relied on the results of the end-of-school exam. The SDI dataset captures students and teachers’ knowledge data for students in grade four. The data suggests that while enrollment rates have reached unprecedented levels, suggesting enormous progress in human capital accumulation, the picture is less rosy than it may seem at first sight. Many students, it turns out, fail to acquire basic skills, despite attending school. Moreover, many teachers show skill deficiencies.

Key results are illustrated in Table 4.1, which presents summary statistics for public and private schools, with an additional breakdown between urban and rural public schools. Out of a maximum score of 100, students score on average 45.1. Since the SDI test is a grade specific test, one would expect the vast majority of students to score highly, 70–80% or higher—on average! So, these results point toward insufficient learning by pupils who attend primary school. The results hold for both French and mathematics and hold for both private and public institutions, with private schools doing better and rural public schools doing worse.

Table 4.1 SDI key results

One of the unique features of the SDI approach is that teacher qualifications are also tested. Since pupils cannot learn more from their teachers than what the teachers know, assessing teacher knowledge is evidently important. Worryingly the average score of teachers was not different than that from their students: 43.0. Among the teachers, only 2.7% scored 80% or higher. In rural public schools, even less than 1% managed to get a score of 80%. Evidently, teacher qualifications are an issue.

Other aspects of the learning environment are not encouraging either. Overall, 21.6% of teachers were not in school during an unannounced visit (a further 15% were at school, but not in the classroom). When in the classroom, teachers taught 79.2% of the time, meaning that nearly one‐fifth of the time was devoted to other activities. Cumulating the sources of lost teaching time, pupils have roughly 48.9% of the scheduled teaching time.

In Chapter 2 it was noted how the average student teacher ratio has remained constant at 42–1 since the introduction of free primary education, suggesting that number of teachers increased to accommodate the massive entrance of new pupils since school fees were abolished.

It is not only teachers who are missing from school, so are students. Though the official student teacher ratio is 42–1, the student teacher ratio that was observed was much lower: 29–1. With that, the number of teachers to students actually exceeds the norm of 40–1.

Scholastic inputs are also lacking. This was already pointed out in the previous chapter. There are important input deficiencies that make teaching more difficult. Roughly one‐quarter (22.3%) of schools had the minimum infrastructure, primarily because only one school in four has functional, private, and accessible latrines. 14.6% of public schools have the minimum teaching equipment, and textbooks are only available half the time.

Togo’s outcomes are not unlike those of other African countries (see Table 4.10) which also show inadequate learning outcomes by pupils, a high fraction of unqualified teachers, a lack of teacher motivation as evidenced by frequent absenteeism and inadequate scholastic and infrastructural inputs. In the remainder of this chapter we use the SDI data to unpack which of the various factors, teacher presence, scholastic materials or teacher knowledge is the key driver of student learning outcomes. The literature suggests a somewhat weak relationship between resources and student performance, which, in turn, has been associated with deficiencies in the incentive structure of school and education systems. Indeed, in Togo incentives for teaching staff are not great, as evidenced by the reliance on temporary staff and teaching assistants/volunteers and the frequent strikes that marred the SDI data collection (http://datatopics.worldbank.org/sdi/). Therefore, scholastic materials alone may have a limited impact on the quality of education, yet it is possible that inputs are complementary to staff motivation, so coupling improvements in both may have significant impacts (see Hanushek 2006). As noted by Duflo et al. (2011), the fact that budgets have not kept pace with enrollment, leading to large pupil–teacher ratios, overstretched physical infrastructure, and insufficient number of textbooks, etc., is problematic. For Togo this is not entirely the case as budgets did keep pace with increases in enrollment. Yet while student teacher ratios did not increase, infrastructure deficits remain huge. However, simply increasing the level of resources might not address the quality deficit in education without also taking teachers’ incentives and qualifications into account. In fact, we find that if only one thing can be addressed through public policy making, teacher knowledge should be given the priority. Before turning to the analysis, however, we first present the SDI data.

4.2 SDI Data

To date, there is no standardized set of indicators, at least not for Africa, to measure the quality of education. The SDI surveys attempt to fill this gap by providing a homogenous set of indicators for several countries, on learning outcomes as well as facilities and teachers characteristics. The datasets are collected over time, enabling governments and service provider to track progress within and across countries over time.¹

The SDI data for Togo were collected in 2013 from a representative sample of 200 primary schools, 1141 teachers, and 1938 grade four pupils. To get a reliable measure of teacher presence, schools were visited twice, each time unannounced. During the period of data collection, there were a number of strikes, both by civil servants and employees of faith‐based schools that were felt differently in rural and urban areas and the SDI results reflect these realities. First visits were reprogrammed to the extent possible so that teams conducted the first visits on days when there were no strikes. However, second visits were allowed to fall on strike days to reflect the reality faced by pupils (Rockmore 2016).

The results provide a representative snapshot of the quality of service delivery and the physical environment within which services are delivered in public primary schools. In addition to information about student knowledge, the SDI provides information on three levels of service delivery: measures of (i) teacher effort; (ii) teacher knowledge and ability; and (iii) the availability of key inputs, such as textbooks, basic teaching equipment, and infrastructure (such as sanitation, quality of lighting etc.). The test used to assess the pupils’ level of knowledge is a standard, grade specific test. The SDI survey instrument was adapted to the Togolese context through a participatory process involving technical discussions, training, and piloting with the Ministry of Education’s National (Education) Evaluation Commission (Commission nationale d’évaluation; CNE). The tests assess the ability to do basic reading and arithmetic through a test articulated in three main sections: French, mathematics, and non-verbal reasoning. In the French vocabulary task, for instance, the pupils have to know the correct French words for four subjects drawn as pictures (a tree, an elephant, a pair of shoes, a t-shirt); in the ordering number task pupils should rank in ascending order a series of six number below 1000; finally, the non-verbal reasoning section is composed by four simple exercises where the pupils are asked to choose among a series of geometrical figures or patterns having different shape, color, and texture which one among the six options available would fit with those stated as in the questions. More detail about the SDI test can be found in Box 4.1.

Box 4.1 SDI test

Pupils tests
French	Reading–decoding exercises: Pupils’ reading and decoding skills, as well as the knowledge of French vocabulary, are tested through exercises based on graphological and iconographical identification: • Letter recognition: pupils are asked to be able to identify a specific letter among a set of nine (the exercise is repeated three times) • Word recognition: pupils are asked to be able to identify a specific French word among a set of nine (the exercise is repeated three times) • Vocabulary: pupils are asked to be able to state the correct French words for four subjects drawn as pictures (a tree, an elephant, a pair of shoes, a t-shirt)
	Reading comprehension exercises : • Reading Short Sentence: pupils are asked to be able to correctly read aloud a short sentence eight words long • Reading Long Sentence: pupils are asked to be able to correctly read aloud a long sentence seventy words long • Reading Comprehension: pupils are asked to answer to three simple questions about the long sentence previously read
Mathematics	Numbers : Pupils’ basic arithmetic skills are tested through exercises based on counting, numbers recognition, and ranking: • Number Recognition: pupils are asked to be able to identify a specific number among a set of nine (the exercise is repeated three times) • Ordering Numbers: pupils are asked to be able to rank in ascending order a series of six number below 1000
	Four basic operations : Pupils’ basic arithmetic skills are tested through exercises based on the four basic operations (addition, subtraction, multiplication, division) • Single-digit Addition • Double-digit Addition • Triple-digit Addition • Single-digit Subtraction • Double-digit Subtraction • Single-digit Multiplication • Double-digit Multiplication •Triple-digit Multiplication • Single-digit Division A • Single-digit Division B
	Analytical skills : Pupils’ analytical skills are tested through exercises which require a combination of critical thinking and basic arithmetic skills • Comparing Ratios: pupils are asked to be able to compare the results of three ratios • Problem Solving A: pupils are asked to be able to solve a problem which requires the use of multiplication • Problem Solving B: pupils are asked to be able to identify the number that will follow in a sequence of four numbers
Non-verbal reasoning	Logic: Pupils’ reasoning abilities are tested through a series of four puzzles • Shape Recognition A: pupils are asked to be able to choose among six geometrical figures or signs having different shape, color, and texture which figure or sign would be fit to complete the sequence of four • Shape Recognition B: pupils are asked to be able to choose among six geometrical figures or signs having different shape, color, and texture which figure or sign would be fit to complete the sequence of four • Pattern Recognition A: pupils are asked to be able to choose among six geometrical figures or signs having different shape, color, and texture which figure or sign would be fit to complete the sequence of four • Pattern Recognition B: pupils are asked to be able to choose among six geometrical figures or signs having different shape, color, and texture which figure or sign would be fit to complete the sequence of four

4.3 Descriptive Evidence

4.3.1 Pupils’ Knowledge

To provide a meaningful but compact snapshot of the pupils’ general knowledge, we decided to select, as four key indicators, the average score to the vocabulary question, the joint non-verbal reasoning exercises, the single-digit multiplication, and the double-digit multiplication questions. Table 4.2 summarizes the average score in percentage points for the specific questions asked in each section. Table 4.4 illustrates the ratio of pupils with correct answers by region and rural areas.

Table 4.2 Descriptive SDI test results

Looking at the mathematical skills (Table 4.2) the poor performance by pupils is striking. Ordering number is known by only half the students. Results are (somewhat surprising) better for the percentage of pupils able to do single-digit addition. The lowest and dramatic score is on double-digit multiplication, which shows that pupils do not know the method to solve the operation, when the operation cannot be solved using one’s own fingers.

Pupils’ average test score in non-verbal reasoning is particularly homogeneous both across regions and area if compared to the other key indicators. Table 4.4 shows that there is no relevant difference in average score computed in specific regions or area with the only exceptions of Savanes, Kara and Plateaux where, on average, children perform slightly worse than their peers who live in other regions.

A look at the distribution (Fig. 4.1) of the pupils’ score reveals a distribution across private schools mirroring that in public schools, with private schools showing a much higher ratio of pupils with higher scores. The differences in pupils’ test score between public and private schools are largely driven by the results of the French test: a possible explanation could be the fact that children going to private school are more likely to speak (or be exposed to) French at home given that may come from wealthier and more educated family backgrounds (Figs. 4.2, 4.3 and 5.2) and most likely they live in Lomé. This explanation is reinforced by the finding that teachers in public and private schools do about equally well on the French test (Table 4.5).

Fig. 4.1

(Source Authors’ calculations using SDI data)

Distribution of pupils’ average scores across public (0) and private (1)

Fig. 4.2

(Source Authors’ calculations using SDI data)

Distribution of pupils’ French test scores

Fig. 4.3

(Source Authors’ calculations using SDI data)

Distribution of pupils’ math test scores

The distribution of math scores (Fig. 4.3) is less skewed than the total one, showing a bigger mass at central values, with a vast majority whose math score is below 50% in public schools, the picture is again mirrored in private schools, albeit more centered toward the middle of the distribution. The average score distribution shows a marked difference in the distribution, as well as in the average values as previously shown. Public schools show an alarmingly large number of pupils with around zero scores while this evidence is not present at this extent in the private schools. This evidence shows that there is a significant group of pupils who are not learning at all.

Finally, Table 4.3 provides a snapshot of the correlation existing among the scores achieved by pupils in the three subjects. Although statistically significant different from zero the correlations among test scores widely differ across areas, showing the highest correlation between French and math (around 60%), while the correlation is lower between non-verbal reasoning and math (and French too), not reaching 20%. It suggests that non-verbal reasoning measures something that is very different from math and French.

Table 4.3 Test score correlations

4.3.2 Regional Differences

Irrespective of test score, urban areas always do better than rural areas and pupils in the Golfe-Lomé region always outperform those in other regions (note however, that this region is also almost entirely urban and with a high prevalence of private schools). With regard to the French vocabulary question (Table 4.4), which tested the ability of pupils to write the French word for four different pictures, pupils attending a school located in an urban area perform better than the other ones. In terms of regions, on average, children located in the mostly urban Golfe Lomé outperform the national average (88.2 percentage points against 65.2 percentage points). With respect to math (single or double-digit multiplication) pupils in Golfe-Lomé score higher (16.4% for single-digit multiplication; 15.6% of double-digit multiplication) than those in any other region. Particularly for double-digit multiplication the difference is stark as there is no region where more than 5% manage to successfully complete double-digit multiplication. That said, even the score for Lomé is dismal. The difference in score for non-verbal reasoning is relatively small, with all regions scoring between 47 and 57 percentage points. The superior performance of Golfe-Lomé is much stronger for vocabulary than for maths, and least for non-verbal reasoning, suggesting that the main difference with other regions (and between urban and rural areas) is language related rather than differences in analytic skills or reasoning.

Table 4.4 SDI test scores of pupils by region and rural and urban areas

4.3.3 Teachers Knowledge

The unique feature of this dataset allows researchers to know the scores for teachers as well as for students. Teachers were tested with similar questions administered to students, at a slightly higher level of difficulty. The average scores show an impressively low level, with mathematical knowledge standing slightly below of 35% of correct responses. Expecting that a teacher should master the taught subject, an average level of 80% (70% if you would like to be less ambitious) could safely be considered a lower threshold. The level of teachers’ knowledge is general poor showing particularly low level in mathematical and pedagogical skills (see Filmer, Molina, et al. 2017). In Table 4.5 we show the average values by urban/rural and private/public schools. Difference in knowledge of French does not show up overall, suggesting that the level of knowledge of teachers does not differ widely. A remarkably low ratio of teachers has more than 70% of answers correct, a percentage of 13% in French and 15% in math. These numbers vary across schools, ranging from 17% in French in urban schools to 11% in rural schools.

The SDI dataset allows to further break down the urban/rural dimension across different regions (Table 4.6). As was the case for pupil test scores, teachers in Golfe-Lomé and teachers in urban areas score better than teachers in all other regions or those in rural areas. However, unlike the pupil test score, the French knowledge of teachers is substantially uniform (around 50 percentage points) across regions, and only slightly better in urban areas than in rural areas.

Table 4.5 Distribution of teachers’ test scores

As previously demonstrated, teachers’ mathematical skills are quite poor. By looking at regional differences it can be seen how teachers located in the Golfe-Lomé region outperform their colleagues living in other regions of the country (43.1 percentage points against a national average of 33.4 percentage points). Also, in this case, almost all urban areas perform better than the rural ones.

The teachers’ questionnaire contains a section dedicated to investigate their pedagogical skills; the range of exercises and questions asked help providing a snapshot of teachers’ ability to prepare and organize lessons, assess children’s ability, and evaluate students’ progress. Results from these questions are not encouraging. Teachers’ pedagogical knowledge is disturbingly low: the average ratio of correct answers (expressed in percentage points) varies from 15.8 of Plateaux region to 22.1 of Savanes with the national average reaching only 19.6 (Table 4.6).

Table 4.6 SDI test scores for teachers by region and rural and urban areas

4.4 Analysis of Variance

It might be interesting to investigate to which degree the variation in pupils’ test scores is driven by school level (including teacher qualification) and individual level-characteristics. To explore this, we use a standard Analysis of Variance (ANOVA) model. The ANOVA model let us understand whether it is possible to claim that there is no significant difference in the means of pupils’ test score at school level, which makes the use of multivariate analysis important to detect the relevance of different drivers. It must be noticed that, since to the SDI dataset is based on information collected from 1938 grade four pupils randomly selected among 200 randomly selected school in Togo, the model takes form of One-way random effects ANOVA and then the results can be generalized back to the entire population (Table 4.7).

Table 4.7 Analysis of variance of SDI test scores

For each of the three tests the “Between Groups” line refers to the variation of the school mean around the population mean, the “Within Groups” line refers to the variation of the pupils’ scores around their school mean, and the “Total” line refers to the variation of the pupils’ scores around the population mean. The F-statistics are given by the ratio between the groups Mean Square (MS) and within groups MS. In all the three case the null hypothesis that the average value of the dependent variable is the same for all groups is rejected; the within group MS is definitively smaller than the between group MS. This suggests that the differences between test scores are not solely explained by differences between students and confirms the relevance of school levels characteristics (including teacher qualifications) in explaining pupils’ test scores.

4.5 Regression Analysis

In this section we present the results of our regression analysis where we investigate how each factor affects pupil scores in math, French, and Non-Verbal Reasoning. We consider as dependent variable the scores in these topics (100% is the maximum score with all correct answers).

Multivariate analysis allows us to control for each factor, keeping other determinants constant, which allows us to purge our results by possible contamination with other inter-related factors.

Results are presented in Table 4.9.

Table 4.8 Summary of variables used in the regression analysis

Starting with the gender of the student, results show that scores are lower for girls, despite being significant in mathematics only. Pupil teacher ratios do not have the same impact on the three subjects. Larger classes have a negative effect on performance in French while it has a small and positive effect on non-verbal reasoning. Class size is not significant in mathematics. Remarkably, absenteeism of teachers does not affect the final score results nor does teacher experience; having an assistant/volunteer teacher, on the other hand, is a critical driver, particularly in French. The presence of assistant teachers shows the highest (negative) impact, by decreasing French marks by 60 percentage points versus five in mathematics. The availability of books does matter with the exception of non-verbal reasoning where it is not significant. Interestingly, book availability has a higher impact for French than for math, suggesting that improving performance by simple providing additional text books is more difficult in conceptual topics like mathematics. In fact, the results suggest that the way to address improvements in test scores may vary by subject. For a subject like math, having a text book is as important as having a qualified teacher. For French, having a qualified teacher is much more critical than the availability of textbooks.

Multivariate analysis confirms what was reported earlier that better learning outcomes are found in private schools, with private schools exhibiting the highest coefficients: students at private schools score 9 percentage points more in their math test than their public-school peers; in French the gap is the highest with 20 percentage points of difference. Let us remind the reader that our dataset does not collect information on parental background, including wealth. Attending a private school versus a public one could be endogenously driven by the wealth of the family. Given these features of the data, the private school effect is likely to be driven by a wealth effect, rather than a school effect. Students coming from richer families are more likely to go to private schools (Fig. 5.2). Controlling for the quality of teachers, private schools perform better, which can be easily explained by the demand side rather than by the quality of supply (which is controlled for). The gap is larger in French, which is also a better proxy for belonging to richer families, more exposed to French language.

It is worth remarking that the ratio of female teachers positively affects the performance, with the exception of math, this probably suggesting additional effort, which cannot be controlled by regression analysis being unobserved. Since the ratio of female staff in public schools is only 14%, this is another area for attention.

Some of the school inputs that traditionally get quite some attention from donors, such as toilets or whether a child had breakfast (school feeding) do not seem to affect test scores. Surprisingly, parental associations matter in a negative way, even after controlling for the frequency of the meetings. This could be explained by the endogeneity of the variable. Parental associations are a choice variable, and they could be more likely to occur when student results are poor, or in isolated (rural) areas where the only schools available are (former) local initiative schools (EDIL) which were started and run by parents (Table 4.9).

Table 4.9 Regression analysis on pupils’ scores

4.6 Concluding Remarks

The SDI results for Togo are not out of line of those for other countries in the region. These countries too, face inadequate student learning, unqualified teachers, low teacher motivation, and insufficient scholastic and infrastructural resources. Clearly there are differences attributable to country specificities though. Learning outcomes are better in Kenya, for instance, even though the average time spent by teachers in Kenya is comparable to that in other countries (see Table 4.10).

One of the consequences of the free education program in Togo seems to have been that many ill-qualified assistant teachers entered the school system. And as Table 2.2 demonstrates, the hiring of temporary staff and teaching assistants has continued since. After all, within the public school system there is a strong reliance on teaching assistants and temporary staff, staff getting paid half, or less than half, compared to what their civil servant colleagues make and who make up 60% of the staff complement. Yet this is not the entire story. When only 2.7% of teachers can satisfactorily complete their test, the 40% of teachers who are civil servants and who presumable completed their training at teacher colleges are deficient in their knowledge too!

The evidence shown in this chapter points at a weakness of the system that goes beyond the marginal impact of some factors. Indeed, average values of pupils’ learning outcomes shown in this chapter point toward such an insufficiency in the level of learning, such a deep learning crisis, that it needs massive public action to address, and not by changes at the margin. Looking at the most effective forces behind learning outcomes, two factors stand out as pivotal: private school attendance and the low presence of teaching assistants or volunteers. Whereas the former can plausibly be attributed to demand forces i.e. family background, the latter relates to supply dynamics. Teachers qualifications mirror, indeed, school public investment into human capital. Private school attendance, conversely, reveals a parental choice, which hides the family welfare, better off households more likely to send their kids to private schools. This is particularly consistent across all outcomes, while other factors, when relevant, are often limited to having an impact on outcome. Private school attendance can offset the negative factors such as higher presence of voluntary teachers, by suggesting that wealth effect of the parent can neutralize the negative supply side effect.

Appendix

4.1.1 Regression Variables Description

Dependent Variables

• Pupils’ mathematics score (total_math_score): variable which defines the pupils’ total score in the math section, it is expressed in percentage points

• Pupils’ French score (total_lang_score): variable which defines the pupils’ total score in the French section, it is expressed in percentage points

• Pupils’ non-verbal reasoning (total_nvr_score): variable which defines the pupils’ total score in the non-verbal reasoning section, it is expressed in percentage points.

Independent Variables

• Female (female): dummy variable which takes value 1 if the pupil is girl and 0 otherwise

• Urban (urban): dummy variable which takes value 1 if the school is located in an urban area and 0 otherwise

• Private (private): dummy variable which takes value 1 if the school is private and 0 otherwise

• Pupil–Teacher Ratio (pupil teacher ratio): variable which defines the number of students per teacher (values from 0 to 113)

• Toilets (toilet): dummy variable which takes value 1 if the school has toilets and 0 otherwise

• Presence rate non-assistant teachers: variable which defines the percentage of the non-assistant teacher is present at school, values from 0 to 1

• Multi-grade classes: dummy variable taking value 1 if in the school there are multi-grade classes (our elaboration)

• Student had Breakfast: dummy variable taking value 1 if the child had breakfast and 0 otherwise (our elaboration)

• Number of classroom: number of classrooms in the school has (values from 0 to 20)

• Share of pupils with pencils: variable which defines the percentage of pupils having a pencil in a given school, values from 0 to 1

• Share of female teachers: variable which defines the percentage of female teachers within each school, values from 0 to 1 (our elaboration)

• Share of teachers: variable which defines the percentage of volunteer teachers within each school, values from 0 to 1 (our elaboration)

• Average years of education of teachers: variable which defines how many years of education have on average the teachers in a given school, values from 4.33 to 9 (our elaboration)

• Average years of teaching: variable which defines by how many years on average the teachers teach in a given school, values from 2.25 to 25.375 (our elaboration)

• Parents’ association: dummy variable which takes value 1 if in the school is active a parents’ association and 0 otherwise

• Number of meetings: variable (in logarithm) which describes the number of meetings held by parents’ association during the year

• Share of pupils with mathematics textbook: variable which defines the percentage of students having a mathematics textbook in a given school, values from 0 to 1

• Share of pupils with French textbook: variable defining the percentage of students having a French textbook in a given school, values from 0 to 1

• Share of pupils with textbook: variable defining the percentage of students having a textbook in a given school, values from 0 to 1.

Table 4.10 Key SDI results for schools in Togo and selected African countries