Reformers and International Comparisons

Olympic_rings_with_white_rims_svgOn Tuesday at This Week in Ed I asked what the evidence is that education reform is the best way to fight poverty. The domestic evidence doesn’t strike me as obvious, but I was especially curious about the international evidence.

After writing that post I started nagging people on Twitter to tell me which countries have  – at least according to reformers – education systems we should be aspiring to.

My main reason for doing this was that if reformers really think education is the best way to fight poverty, then presumably countries with the best education systems should have substantially lower poverty rates as a result. My guess is that there is little evidence for this; if a developed country is successful at reducing poverty (and many are), that is usually because of government taxes and transfers, not education. Still, since reformers seem largely unwilling to identify the countries they believe have the most effective education systems, their claims about the benefits of reform (e.g., poverty reduction) are difficult to test.

Unsurprisingly, I didn’t have many takers on Twitter. A few reform-sympathizers chimed in that while we might want to learn from aspects of different countries’ education systems, reformers aren’t really committed to the idea that copying other countries would be useful in general. We have a different context, after all!

I agree whole-heartedly that you can’t judge the effectiveness of an education system just by looking at its outcomes. Context matters a great deal. But, of course, a fair amount of the education reform movement is premised on the idea that we can tell discern the mediocrity of our own education system by noting the mediocrity of our outcomes.  If the inferiority of our education system is demonstrated by our test scores, it’s hard to see why the superiority of other systems wouldn’t be. Raw international test scores either tell us about educational quality or they don’t.

Jason Becker stepped up with a more radical defense of reformers, arguing that they are not, in fact, making claims about educational effectiveness at all.

As far as I can tell, this is false as an empirical matter of fact. For example, here’s StudentsFirst with a series of ads explicitly claiming that “our education system” is a joke.

Hypothetically, though, could education reformers take the position that the our ranking on international test scores is interesting just because our outcomes as such have implications for our “competitiveness”?

I’m skeptical for two reasons.

First, the extent to which our relative educational outcomes matter for our international competitiveness is an empirical question that does not seem to me to have an obvious answer. America has not “lead the world” in educational outcomes for decades, if ever, and yet we seem to be reasonably successful and “competitive” (whatever that means). If reformers want to go this route, they’d have an awful lot of work to do in terms of demonstrating this relationship. (Note that they rarely bother to pursue this, which suggests that this is not really their main concern anyway.)

Second, and more fundamentally, would it even make sense for education reformers – as education reformers – to make a big fuss about our outcomes and competitiveness without making any claims (implicit or otherwise) about our educational effectiveness? Probably not: why bring up unequal education outcomes in a conversations about education reform unless you think the problem could be addressed by improving our educational effectiveness?

In other words, it wouldn’t make sense for education reformers just to point out that our outcomes are too low unless they are also – at least implicitly – trying to argue that those outcomes could be improved through changes to our education system.

Admittedly, reformers are often not explicit about what they think international comparisons demonstrate, probably because when you get right down to it they demonstrate very little. Nevertheless, those international comparisons either tell us something about education reform or they are non sequiturs in a conversation about education reform.

If we suppose reformers do not intend for international comparisons to be non sequiturs, there is still no plausible argumentative path forward for them.

1. Reformers may intend the international comparison to demonstrate that our education system is unusually or unnecessarily ineffective. This would be a bad use of raw test score data, which measure student achievement, not school effectiveness. Furthermore, as noted above, this would imply that reformers should be able to point to “more effective” education systems in other countries. They seem loathe to do this.

2. Reformers may intend the international comparison to demonstrate that, regardless of how effective our education system is now, it needs to be more effective in order to “catch up” our students. However, the plausibility of this argument rests in no small part on what we believe about our system’s current educational effectiveness. The more effective we believe our system already is, the less plausible we will find it that our international standings can be meaningfully altered by trying to make it even more effective. It will seem plausible that education reform can substantially close the gaps between the U.S. and the highest-achieving countries only if we think that our current education system is quite ineffective in relative terms. Which is to say, here reformers have subtly begged the question at hand, namely: whether our mediocre educational outcomes are best understood as resulting from educational ineffectiveness.

The most frustrating aspects of all of this are the obvious double standards.

In other contexts reformers have acknowledged, however grudgingly, that raw test scores don’t tell us much about effectiveness and what really matters is “value added”. When comparing scores internationally, however, suddenly all we need to know is that our test scores are lower than the scores of people in (some) other countries and this is supposed to make us very embarrassed about our education system.

Moreover, while these test scores tell us a great deal about how poor our own schools are, they are apparently of no use in evaluating other countries’ schools.

All in all, then, the reformers’ position often turns out to be simply that ranking countries by educational outcomes is useful for denigrating American schools…and nothing else. It is profoundly unlikely that that is the correct way to use international test scores.

This entry was posted in Education, Education Reform and tagged , . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.


  1. Jake
    Posted April 24, 2014 at 1:11 PM | Permalink

    I don’t think it is the reformers who point to Finland as the promised land, it is the Left. When people bemoan our lack of math skills compared to East Asians, I don’t think anyone is advocating for rote instruction and cram schools.

    • Posted April 24, 2014 at 5:38 PM | Permalink

      I agree that Finland is largely an obsession of progressive reform critics, but they’re much less likely to be making claims about relative educational effectiveness as opposed to claims about the effects of poverty and the like. But, certainly, many of Finland’s fans misinterpret its lessons, too.

  2. Jim
    Posted April 24, 2014 at 1:35 PM | Permalink

    If one compares the performance of different demographic groups in the US with similar demographic groups elsewhere the US comes out looking pretty good. For example US whites outperform all white dominated countries with the exception of Finland and I belive New Zealand. Asian-Americans (who are not all East Asian) outscore Japan and South Korea. US Hispanics outscore all Latin American countries.

    Of course China has the advantage of having a hell of a lot of East Asians.

    • Posted April 24, 2014 at 5:39 PM | Permalink

      In general I’m suspicious of even these sorts of comparisons since there’s a lot that’s non-comparable even within superficially-similar demographic/racial categories.

      • Jim
        Posted April 25, 2014 at 6:04 AM | Permalink

        Yeah the data is pretty dirty. Asian-American includes many groups such as Polynesians who have little similarity with East Asians. “Hispanic” is a close to meaningless term.

        There are two kinds of data – too small to be credible or too large and heterogeneous to be useful.

One Trackback

Leave a Reply