Starting in 1999, schools in Quebec implemented an ambitious curricular & instructional program at all schools in the province. Broadly speaking, this program can be considered “constructivist” and the math program in particular seems to have been of the “reform math” variety. To get a sense for what the reformers had in mind, they described wanting students to increasingly

find answers to questions arising out of everyday experience, to develop a personal and social value system, and to adopt responsible and increasingly autonomous behaviors

and

Instead of passively listening to teachers, students will take in active, hands-on learning. They will spend more time working on projects, doing research and solving problems based on their areas of interest and their concerns. They will more often take part in workshops or team learning to develop a broad range of competencies.

A little over a decade later, a team of economists went in to see how these reforms were going. (An older, ungated version of the paper can be found here.)

Apparently it did not go well.

Catherine Johnson has a good rundown, but I wanted to highlight a few things in particular.

First and foremost, the overall results in terms of student learning appear to have been quite bad:

Our data set allows us to differentiate impacts according to the number of years of treatment and the timing of treatment. Using the changes-in-changes model, we find that the reform had negative effects on students’ scores at all points on the skills distribution and that the effects were larger the longer the exposure to the reform.

This study provides support for my pedagogy of privilege hypothesis, namely that “progressive” teaching *may* be acceptable for the strongest students, even if most students, and especially the weakest students, are likely to flounder:

In grade 2, only students in the 75th percentile appear to be significantly impacted by the reform. However as we move from grade 2 to grades 9–10, the effect also becomes significant for lower and average performing students. In grades 9–10, the magnitude of the coefficients is the largest for students in the 25th percentile, and slowly decreases as one moves toward the upper tail of the distribution. Looking at the top of the distribution (90th percentile), we also find negative effects across all grades, but the estimates are generally not significant. It is possible that the reform did not harm top performers. It is also possible that the reform did impact top performers, but that the number of observations at this mass point is too small to obtain precise estimates…

Lower performing students were impacted more severely, and the effects grew larger as students progressed from primary to secondary school. These large negative effects are worrying, and suggest that the reform may have harmed those most in need.

Notably – and ominously – the reforms in Quebec seem to be aligned with reforms that have been advocated in other places, including the United States:

Evidence…suggests that most OECD countries are moving away (or have long moved away) from the traditional (more academic) teaching approach. More specifically, the teaching approach promoted by the Quebec reform is comparable to the reform-oriented teaching approach in the United States. As of 2006, this approach was widely spread across the United States (although more traditional approaches remained dominant) and it was supported by leading organizations such as the National Council of Teachers of Mathematics, the National Research Council, and the American Association for the Advancement of Science.

Katharine Beals predicted – correctly, in my experience – that advocates of reform math would respond by claiming that there must have been “implementation” problems. This is not an inherently unreasonable argument to make, but it has a whiff of wishful thinking about it and is not clearly supported by the evidence.

Consider, for example, that these changes in Quebec were rolled out in an extraordinarily cautious fashion by the standards of education reform. From the paper:

click to enlarge

The implementation timeline spanned more than a decade and was rolled out in only one or two grades per year, always preceded by a year of planning, training, and professional development:

Extensive training was provided to support the new program. The year prior to the implementation in Elementary Cycle 1, teachers, principals and government officials began the task of preparing the implementation of the reform. Sixteen pilot schools along with several other Lead schools in the English sector experimented with the key concepts of the program of study, as well as school organizational approaches that could be best suited to the strategies required to maximize the effectiveness of the learning environment.

In June 2000, principals in conjunction with teachers began developing their implementation plans for September 2000. Each school was allowed to develop its own approach to deal with the implementation since no single approach was believed to meet the needs of each school across Quebec…In 2000, all schools, both elementary and secondary, participated in some way to the development of the implementation of the reform despite the fact that it did not affect all levels of schooling at the time. Guides for teachers were produced. The implementation was staggered over many years (grades), giving time for teachers to adapt to the new programs.

It may be that the reforms could have been implemented more effectively, but this is nevertheless a *lot* of preparation. If a decade of gradual phase-in with elaborate supports is inadequate for effective reform math implementation, arguably the problem is not with “implementation” per se.

The strongest support for the “poor implementation” hypothesis is the fact that the researchers found that as time went on, younger students seemed to experience less of a negative effect from the instructional reform. This could suggest that implementation was getting better over time, but as the authors note this finding is neither unambiguous nor encouraging:

We find that grade 2 students, 8 years after the implementation of the reform, no longer seem to experience a significant negative effect…The reform being ambitious, it is possible that it took a fair number of years for teachers to develop the necessary skills to fully deploy all aspects of the reform. It may also be the case that, observing the decline in students’ academic performance, teachers informally decided to reintroduce some of their pre-reform teaching approaches, and set aside in part or in totality the reform approach…[W]e are unable to identify which of these two explanations is dominant. In any case, this finding implies that at best the provincial reform had no long run effects on the development of procedural mathematics skills.

In other words, teachers may have gotten better at teaching math using constructivist techniques or they may have given up on trying. In either case, the authors were unable to find any significant positive effect of the reform for young students even after their schools were 8 years into implementation.

To see if their findings are limited by the particular math test taken by students in Quebec, the authors also look at TIMSS and PISA results. Those international tests assess a broader range of skills and allow them to compare trends in Quebec to trends in neighboring Ontario. The patterns are similar. For TIMSS:

Grade 8 students’ performance shows a similar pattern when results from 2007 and 2011 are compared with results from all previous years: Quebec’s performance in both mathematics and sciences is trending downwards, while the performance in Ontario is increasing or stable…

Estimated effects are large and negative in all cases. They are significant in mathematics in both grades, but only in grade 8 in science.

And PISA:

The ERES project has recently produced two reports comparing the math knowledge and the French proficiency of grade 11 students exposed to the reform to that of pre-reform ERES. The math test uses 25 questions from exercises administered during the 2003 and 2006 PISA assessments…In sum, students in the reform group scored slightly lower on average, with a larger difference in geometry and algebra. As for the French proficiency test, they do not find any significant differences, but almost 30% of grade 11 students in the two groups did not complete the assessment.

Overall, the evidence from TIMSS and PISA suggests a worsening of Quebec’s students performance post reform in mathematics and at best a stand still in science and French.

The authors also did not limit themselves to academic indicators, and looked at the effect of reform on (self-reported) indicators of student behavior. Again, they found some negative effects and – at best – some null effects:

We find that the vast majority of coefficients across all grades and outcomes suggest a negative impact of the reform on students’

behavior. With a few exceptions, these effects are rarely significant. In grades 5–6 and 7–8, we find a significant worsening of the situation for the following measures: hyperactivity, anxiety, physical aggression, interpersonal competencies and emotional quotient. In grades 5–6 and 7–8, the strongest evidence that the policy had an impact on behavior is for hyperactivity and anxiety (more than 50% of a std. dev.). The effects are rather strong, positive (more hyperactivity and anxiety) and robust to sample and method. In grades 9–10, the estimated effects are significant only for prosocial behavior, physical aggression and property offense.

Our results are in line with those reported by ERES, which found no effect on social adjustment, personal and emotional adjustment and intrinsic motivation. They also found that post-reform students felt less well-adapted to secondary school, male students were found to have lower self-esteem, and at risk students were less engaged in school work. We therefore conclude that the reform did not improve the behavior of students measured using the self-reported NLSCY behavioral indicators.

Needless to say – and as the authors themselves acknowledge – this study is not really “definitive” in any meaningful sense. This study was only able to measure short- and medium-term effects of this particular reform on some academic abilities and, to a lesser extent, some behavioral effects. And rigorous studies of large-scale curricular reforms are few and far between, so we don’t have a huge body of research to pull from and one study should never be relied on too heavily.

Nevertheless, this is considerable zeal in some quarters for widespread adoption of constructivist, progressive, or “student-centered” teaching approaches. Advocates of such approaches should find these results concerning.