Explained Variation Is Not A Measure of Importance

246717376_aa7e238d67Back in early April the American Statistical Association put out a “Statement on Using Value-Added Models for Educational Assessment“.

Last month, Raj Chetty, John Friedman, and Jonah Rockoff issued a response, in part because so many commentators seemed to misunderstand the ASA statement and in part because the ASA seemed not to have incorporated some of Chetty et al.’s most recent research.

Diane Ravitch’s unimpressed follow-up involves a few all-too-common misconceptions:

What do Chetty, Friedman, and Rockoff say about the ASA statement? Do they modify their conclusions? No. Did it weaken their arguments in favor of VAM? Apparently not. They agree with all of the ASA cautions but remain stubbornly attached to their original conclusion that one “high-value added (top 5%) rather than an average teacher for a single grade raises a student’s lifetime earnings by more than $50,000.” How is that teacher identified? By the ability to raise test scores. So, again, we are offered the speculation that one tippy-top fourth-grade teacher boosts a student’s lifetime earnings, even though the ASA says that teachers account for “about 1% to 14% of the variability in test scores…”

The argument is that if teachers account for only a small fraction of the variation in student test scores, teacher quality is probably not a useful lever by which we can improve education outcomes.

This is wrong for at least three reasons.

First, to know whether 1%-14% is a lot of variation to account for we have to compare teachers to something else. It’s not entirely clear from her post, but Ravitch1 seems to want to compare teachers to all other factors put together, but that comparison tells us very little. The 86%-99% of the variation in student test scores not explained by teachers is not explained by a single other factor for us to focus all of our policy energy on; it’s an aggregate of a large number of factors, each likely accounting for a much smaller fraction of the variation.

Second, even if some factors explain more variation in test scores, that doesn’t mean we have to pick just one factor to care about. We may want to prioritize, say, poverty reduction over teacher quality improvements, but that doesn’t mean only the former matters.

Third, and most fundamentally, variation accounted for by a factor is not a measure of that factor’s importance. The ASA statement actually points this out somewhat obscurely:

Research on VAMs has been fairly consistent that aspects of educational effectiveness that are measurable and within teacher control represent a small part of the total variation in student test scores or growth; most estimates in the literature attribute between 1% and 14% of the total variability to teachers. This is not saying that teachers have little effect on students, but that variation among teachers accounts for a small part of the variation in scores.

The fact that this was included in the ASA statement has not prevented considerable confusion; VAM critics have latched on to the first sentence, but seem not to understand the significance of the second.

Let’s unpack that second sentence.

If a factor doesn’t explain much of the variation in student test scores, that could mean that the factor is relatively unimportant and that even large changes in that factor would not have significant effects on scores.

Another possibility, however, is that that factor doesn’t systematically vary between students.

Consider “access to breathable oxygen”. If you crunched the numbers, you would likely find that access to breathable oxygen accounts for very little – if any – of the variation in students’ tests scores.  This is because all students have roughly similar access to breathable oxygen. If all students have the same access to breathable oxygen, then access to breathable oxygen cannot “explain” or “account for” the differences in their test scores.

Does this mean that access to breathable oxygen is unimportant for test scores? Obviously not. On the contrary: access to breathable oxygen is very important for kids’ test scores, and this is true even though access to breathable oxygen explains ≈0% of their variation.

Now let’s return to the importance of teachers. If teachers account for only a small fraction of variation in student test scores, that may mean that teacher quality is largely unimportant. It may also mean that teacher quality does not vary systematically very much between students.

Another way to think of it is this: if every teacher was exactly as effective as every other teacher, teachers would account for exactly 0% of the variation in student test scores. This would be true regardless of whether these imaginary teachers spent the entire school day reading the newspaper or if they successfully taught advanced calculus to 3rd graders.

In other words, determining statistically how much variation is “explained” by teachers will not, by itself, tell you how important teacher quality is.

This is precisely where research like that of Chetty et al. comes in. It attempts to go beyond simple measures of “explained variation” to quantify teachers’ actual importance and impact.

We can still reasonably disagree about what that research tells us about “the importance of teachers”. What you can’t reasonably do is dismiss that research out of hand using measures of explained variation, as those are not direct measures of importance.

  1. Ravitch is by no means the only one who makes these mistakes, but she’s usefully illustrative here. []
Posted in Education Reform | Tagged | 2 Responses

My Twitter Reactions to the Vergara Ruling

We are already being inundated with analyses of what yesterday’s Vergara ruling “means”, so rather than write up yet another one I just compiled my thoughts from Twitter:

Posted in Education Reform | Tagged , | Leave a comment

There Is Probably No “Crisis” In American Education

Here is a chart of educational attainment in the United States since 1940:

Educational_Attainment_in_the_United_States_2009

When you look at that chart, do you see a crisis?

No? Me neither.

How about in these charts of reading and math achievement on the NAEP for 17-year-olds, broken down by race?reading17

math17

Still hard to see a crisis, at least to my eyes.

Certainly, it’s fair to say that American education has many problems related to effectiveness, efficiency, and equity.

And it’s probably reasonable to say that our country and the world face a number of bona fide crises – war, climate change, poverty, or criminal justice, for example – each  with some relationship to education, however complex or indirect.

But next time you hear someone claim – or are yourself tempted to claim – that American education is in a state of “crisis” or is being “destroyed” by education reform, remember these charts. Then ask yourself whether those terms are being used productively or whether they are being defined down in a way that obscures as much as illuminates.

Posted in Education Reform | Tagged , , , , | 16 Responses

Are the Common Core Standards Voluntary?

50323994Did states adopt the Common Core standards voluntarily, or were they forced to do so by the federal government?

I’m not sure why we would think the answer to that question matters very much. If adopting the CCSS would be good for a state, then their adoption by that state would be good. If adopting the new standards would be bad for a state, then the state shouldn’t adopt them. Information about how the standards are (or are not) adopted seems neither necessary nor sufficient to determine whether adoption should proceed.1

Still, suppose we’re really interested in whether or not the standards were “voluntary” for states. And let’s suppose, for the sake of argument, that it makes sense to talk about states doing things “voluntarily”, a substantial assumption given that “freedom” is a difficult concept to apply even to individual people.

So, were the standards voluntarily adopted by states? There is a prima facie case that they were, even if the Obama administration ended up supporting them. As Michael Petrilli says:

These standards started out as a state effort, with support from private entities like the Gates Foundation. It was the governors and state superintendents who came together, voluntarily, to draft higher common standards, because they acknowledged that their own state standards were set too low. There was already momentum behind the standards when the Obama administration intervened.

Along with the fact that not every state has adopted the CCSS, this is basically dispositive. If states mostly supported the standards prior to the Obama administration’s endorsement, it hardly makes sense to talk about the federal government “coercing” the states into adoption. If Arne Duncan hadn’t gotten involved it’s possible that some states may ultimately have decided not to adopt the standards they had helped to create, but “a few states were marginally more likely to adopt” hardly seems sufficiently coercive to justify the resulting libertarian outrage.2

Libertarians, however, have been outraged. Why? Here’s Neal McCluskey:

From the outset of the Obama administration, officials talked about a need for national standards, and under the mammoth 2009 “stimulus” they got a lever by which to push that: the $4.35 billion Race to the Top program. To fully compete for Race to the Top money states had to adopt standards common to multiple states, and only one set of standards fully met the definition: the Common Core.

This argument will baffle most people. How is being offered money to do something “coercive”? Presumably McCluskey is offered monetary compensation of some kind for his work with the Cato Institute; has he therefore been “coerced” into working there?

Most people, however, are not libertarians. McCluskey sneaks his premises in a little later in the argument:

Adopting the Common Core was, in principle, no more voluntary than having a mugger take your money — [state] taxpayer money — then let you “voluntarily” hand him the keys to your car to get the dough back.

Wait, what? McCluskey thinks paying your taxes to the federal government is morally equivalent to being mugged? You could be forgiven for thinking you misunderstood, but here he is replying to me last week on Twitter:

This is the classic anarcho-libertarian line on government: that its activity necessarily involves forcing people to do things they would prefer not to do and is ipso facto illegitimately coercive.3 Because this is such a fringe view many CCSS supporters have had a hard time understanding why anybody would consider, e.g., Race to the Top to be “coercive”. The answer is that some people think taxation is essentially theft.

Without getting too far into the weeds here, it’s worth mentioning why most people think being anti-coercion doesn’t mean that taxation is theft. First, “taxation is theft” strikes most people as a reductio ad absurdum of naive definitions of “coercion”. If you are using “coercion” in such a way that you have to conclude that taxation is the moral equivalent of armed robbery, something is probably wrong with your use of the word.

Second, most people believe that taxation is justified in one way or another. This may be because they view taxes as “the price we pay” for the various (typically larger) benefits of membership in society. Or it may be because they realize that your possessions and what they are worth are determined to a large extent by our social context, including our governmental institutions. If, e.g., the nominal and real value of your wages is a function of our social (and governmental) institutions, it becomes very difficult to discern how much of them (if any) is “yours” in a morally fundamental sense.4

So it is safe to say that the federal government did not coerce states into adopting the Common Core standards by offering  money to do so. But if carrots aren’t coercive, what about sticks? It is probably fair to say that sticks were in play:  

We’ve dispensed already with the “Fed $ from taxpayers under force of law” issue, but NCLB waivers appear to be somewhat different.

No Child Left Behind, you will recall, requires that states set learning standards for basic skills – reading and math, mostly – and get ever-increasing proportions of students over related proficiency thresholds. Over the years the requirements of NCLB have become more difficult for states to meet as proficiency targets have risen and Congress has failed to modify the legislation. As a result, the Obama administration offered states “waivers” to avoid many of NCLB’s accountability provisions in exchange for adopting other various reforms including, in many cases, the CCSS.

A somewhat more plausible case for federal coercion is this: States that did not adopt the CCSS would have a much harder time getting a waiver from NCLB, and without a waiver they would be subject to increasingly harsh penalties for failing to satisfy the law’s requirements.

Here, too, the libertarian arguments mostly don’t work.

First, it was possible to get a waiver without adopting the CCSS (as Virginia did) or just skip the standards altogether (as several states have).

Second, the provisions of No Child Left Behind are substantially voluntary. This has been largely forgotten, but states have several options for avoiding NCLB’s accountability penalties. NCLB allows states to set their own standards and proficiency levels, for example, and there are various ways of making sure your schools are not actually held to the highest proficiency standards. NCLB, moreover, is just another carrot: required only for states that wish to receive federal education funding.

In other words, when confronted with the possibility of adopting the Common Core standards, states actually had a substantial amount of flexibility.

There is arguably a naive sense in which states were “coerced” into adopting the CCSS in that some states may not have been completely satisfied with any of the options available to them. But by this standard virtually everything is coercive to one degree or another and the term is rendered meaningless for normative purposes. That conventional, lay-person use of “coercion” may be practically useful on a day-to-day basis, but it has no philosophical precision so it can’t do any philosophical work.

And this, I think, is why the debate over whether the CCSS are “voluntary” has been so intractable. The question is a philosophical (and possibly incoherent) one, but the parties involved – especially libertarian opponents – insist on using notions of coercion that are so imprecise (or loaded) that they can’t possibly settle it one way or the other.

  1. Of course, we should prefer to have a set of background institutions that effectively distributes power between different levels of government. Those concerns, however, are mostly neither here nor there when evaluating the merits of CCSS adoption since the standards have only minor – if any – implications for our background institutions. []
  2. Note also that the Obama administration’s belated involvement in the adoption process has been so polarizing that it’s not obvious the net effect was pro-CCSS in each individual state. There was a point at which CCSS supporters started asking them to back off the advocacy precisely because it seemed to be backfiring. []
  3. McCluskey claims not to endorse the anarcho-libertarian view, but it’s hard to know how else to interpret an analogy between taxation and armed robbery. []
  4. We have socially and economically useful private property institutions, but they depend heavily on coercion, especially government coercion, themselves. []
Posted in Education | Tagged , | Leave a comment

Everything is Untested Until You Test It

9486459224_e1517c1dab_nTeacher tenure is a hot topic in California these days thanks to the Vergara trial, so it was newsworthy when the San Jose chapter of the California Teachers Association asked the State Board of Education for a waiver from state law extending the probationary period for some teachers using a system of peer review. They didn’t get it:

The State Board of Education on Thursday denied the San Jose Unified School District and its teachers union their request for the authority to require some probationary teachers in the district to work an additional third year before becoming eligible for tenure.

Granting a one-year waiver from state law, Jennifer Thomas, president of the San Jose Teachers Association, had argued, would send a message that “a union can be an incubator of innovation in pursuit of educational excellence.” Board member Carl Cohn, a retired superintendent of Long Beach Unified, agreed, urging the board to encourage the cooperation between a teachers union and a district reflected in the request. “Sometimes it is easier to teach kids to read than to get this kind of trust,” Cohn said.

But with its 7-2 vote, the majority sided with the California Teachers Association and the state Department of Education’s position that the district and the union should ask the Legislature to change the tenure law or grant it an exception. Waivers, said board member Sue Burr, should be narrow and limited to circumstances involving individual teachers, such as a teacher who went on maternity leave while on probation. “For better or worse, we’re being asked to waive a fundamental personnel protection,” she said.

One of the big disappointments here is that San Jose Unified and its teachers union were cooperating(!) to try something new and different and it’s state law – not mutual mistrust – that’s preventing them from moving forward. The proposed experiment wasn’t a big one in absolute terms – it’s unlikely that many teachers would have their probationary periods extended in any case – but it involved grappling with hot-button issues of teacher evaluation and tenure that rarely receive even that level of experimentation.

One of the lines often leveled against education reforms is that they are “untested” and therefore shouldn’t be imposed on schools because we don’t have a good sense of what the effects are likely to be.

In many cases it’s true that a proposed reform is untested, but note that this can often be avoided by letting schools and districts test them.

Of course, it may not always make sense to give local education agencies flexibility on every issue. But the disappointment you see expressed over the San Jose case is due, in part, to the sense that it should be possible for the district, especially in collaboration with the union, to perform modest experiments with tenure and peer review.

Much has been made of the fact that the CTA lobbied against the local chapter’s request, but it’s not hard to understand why the Board would be reluctant to waive away the legislatively-established two-year probationary requirement. Arne Duncan’s NCLB waivers have been harshly criticized in some circles as bureaucratic overreach, and that’s a situation in which virtually everyone agrees that the law being circumvented is long overdue for revision. There is less consensus about California’s tenure laws, so it’s natural that the Board may not want to pick that fight with the legislature.

So while the outcome of this decision is frustrating, the underlying issue in this case – and likely many others – is that the laws themselves may need to be loosened to some degree to allow reasonable experimentation at the local level.

And if we can’t test new ideas at all, it’s not clear how we make progress.

Posted in Education Reform | Tagged , , , | Leave a comment

Why Education Reform is Probably Not The Best Way to Fight Poverty

12383520_78bc1448e2_nDoug Lemov is skeptical that I’m right about education being a (relatively) ineffective way of fighting poverty. His response is thoughtful and deserves a response of its own.

First, it’s worth being clear that the only claim I’m prepared to advance and defend is that education reform is not the best way to fight poverty in the United States today. This is entirely compatible with the ideas that (1) education might be modestly effective at fighting poverty, even if only indirectly, and (2) improving education is worthwhile for reasons entirely unrelated to poverty. In fact, I believe both of those things.

The problem I have is with the contention – made by many prominent reformers – that education is the best way to fight poverty. And yes, this is often what they say.

Consider, for example Michelle Rhee, who claims that when it comes to poverty, “teachers can be the answer for so many kids”; otherwise you are going to “write a generation of children off”. Arne Duncan apparently thinks that “the only way to end poverty is through education.” Mike Petrilli claims that “schools are the indispensable anti-poverty program”. He also, curiously, thinks that poverty is “(mostly) not about money”, which is both (mostly) false by definition and an implicit dismissal of money-based methods of fighting poverty.

It’s that denigration (implicit or otherwise) of other poverty-fighting programs that’s the problem. If education is the best way to fight poverty – or the only way – then that means we should worry relatively less about other methods. Here is Rhee again explicitly disparaging other ways of fighting poverty, saying “the best tool that we have to fight intergenerational poverty and break that cycle of poverty is education. It’s not a particular social program or this or that.”

So I object to the claim that education is the best way to fight poverty (in the United States) because (1) we should not in general claim things that are false and (2) we should not in any way discourage people from supporting other, more effective methods of fighting poverty.

With all of that being said, let’s look at Doug’s defense of education as a poverty-fighting measure and why I find it unpersuasive, at least on this narrow question of relative effectiveness.

My Original Post

First, Doug is absolutely right that my international comparison in my original post does not in any way prove that education is unrelated to poverty. All I did was show that countries with higher PISA scores today also tend to have higher pre-tax/transfer poverty rates. There would be lots of ways of running that analysis more persuasively, but my goal was really just to try to prod people into making the affirmative case for education as a poverty-fighting tool.

Part of the issue, here, is that I view the claim that education is the best way to fight poverty as radical, and therefore requiring substantial evidence. I would like to see reformers make the case that all of the other developed countries that have dramatically reduced their poverty rates have done so through education, but I don’t think I’ve ever even seen that attempted.

If we want to ignore my graph, that’s fine, but let’s also be clear that strong claims about education reform’s poverty-fighting power have not (yet) been supported with much international evidence.

On the other hand, I did also point out that domestically it is noteworthy that decades of improving educational outcomes have not done much to reduce poverty rates. I think that’s a more substantive point that I’d like to see addressed. There’s a lot more to say about it – for both sides – but it’s a good starting place (and one I’ll briefly return to below).

Doug’s Affirmative Case

Doug’s affirmative case for education as a poverty-fighting measure rests primarily on three arguments. First, that educational outcomes strongly predict economic growth; second, that the individual economic benefits of more or better education are substantial; and third, that we should worry less about relative measures of poverty and more about absolute measures of poverty.

Education and Economic Growth

Doug bases his case for the importance of education to economic growth on Eric Hanushek’s analysis finding that

If one country’s test-score performance was 0.5 standard deviations higher than another country during the 1960s—a little less than the current difference in the scores between such top-performing countries as Finland and Hong Kong and the United States—the first country’s growth rate was, on average, one full percentage point higher annually over the following 40-year period than the second country’s growth rate.

Doug continues:

That may not seem like much, but, as Hanushek notes, world economic growth averages about 2 to 3 percent of GDP annually, so a difference of a full percentage point in growth is massive.

I, for one, have little trouble believing that education is correlated with economic growth. While his analysis is in many respects interesting, however, there are lots of reasons to doubt that Hanushek has found useful causal phenomena of the magnitude he suggests.

Hanushek admits that “[o]ther economic research has identified two additional factors that affect a country’s economic growth rate: the security of its property rights and its openness to international trade.” This highlights the fact even though economic growth is likely a complicated process dependent on many factors, we understand very few (only two?) of those factors. This makes it unlikely that it is currently possible to adequately control for all of the relevant factors to identify the “true” explanatory power of education.

Additionally, explanatory power in a correlation is not the same as magnitude of causality. Correlation is not causation, etc. I suspect that the correlation Hanushek identifies reflects some causality in the direction he and Doug expect, but our certainties about how much causation runs in each direction should be quite low.

More importantly, there are at least two reasons that Hanushek’s analysis does not speak directly to the question Doug and I are discussing (viz., whether education is the best way to fight poverty in the U.S.).

First, when we talk about “the importance of education” in the United States, we are really talking about the importance of education reform. We already have an extensive education system which means we are already reaping a substantial fraction of the potential benefits of education. The question is about the extent to which we can additionally improve our education system, and we can’t just assume we’re going to be able to get anywhere near the sorts of gains that Hanushek uses to illustrate “the importance of education” for economic growth. Those are hypothetical numbers, not options for us to choose off of a public policy menu.

Second, and as I mentioned to Doug on Twitter, economic growth is not the same thing as poverty reduction. Economic growth means increases in productivity and wealth, but poverty reduction is about the extent to which those economic gains flow to, and improve the material well-being of, the poorest people. To see why it’s a mistake to conflate economic growth and poverty reduction, these graphs of inflation-adjusted median household incomes are a good place to start:

click to enlarge

click to enlarge

Median household data can be misleading – e.g., family sizes have shrunk, inflation measures are imperfect, and people can move between income quintiles  – but this is not a graph of an economy where the benefits of economic growth are flowing proportionally to the poorest households. During the time period examined in that graph, real GDP roughly tripled and per capita GDP roughly doubled, but our economic “floor” budged fairly little.

And lest you think that some sort of Simpson’s paradox is obscuring substantial gains by subgroups, consider how the household income numbers break down by race:

click to enlarge

click to enlarge

Again, remember that this covers a time period of well-documented academic gains for most subgroups of students and for Black and Hispanic students in particular, but their incomes have not obviously risen accordingly. There are ways to slice the data that make the economic gains look somewhat better, but you have to slice them pretty fine to make it look like educational improvements are contributing to substantially reduced rates of poverty as a result of economic growth per se.

The Individual Returns to Education

Here’s Doug:

In the US, I reminded myself, not finishing high school or finishing high school with skills that don’t allow you to go on to college is one of the fastest way to increase your likelihood of ending up in poverty. (Would anyone dispute that? The trend in fact has steepened recently as middle class industrial jobs erode and “knowledge work” is the overwhelming way to the middle class and above).

Make no mistake: education seems to have positive, causal effects on earnings. Frankly, I find that literature daunting and my understanding of it is limited.

For present purposes it will suffice to say that however substantial the individual returns to education may be, they do not appear to be putting a major dent in our poverty rates. I would guess various factors are at play. For one thing, the returns to education are heterogeneous. Consider, for example, that a substantial fraction of recent college graduates earn less than the average high school-only graduate.

So while college is lucrative for the average student – and very lucrative for some fraction of graduates – a great many college graduates appear to be reaping much smaller economic gains from their degrees. If this is particularly true of marginal college students – those who would not have been ready for college at all if not for reform and who are perhaps less likely to enter the most lucrative fields – the poverty-fighting power of additional education may be limited.

Second, large shifts in overall levels of education may have effects on individuals’ returns to education. If, for example, the supply of high school graduates increases, that may put downward pressure on wages for high school graduates or drop-outs. This sort of dynamic effect would (I think) further reduce the impact of education on poverty at the national level.

Finally, it is worth remembering, again, that we are considering the effects of education reform, not of education as such. The standard line among education reformers today is that the best route to improving educational outcomes is through improvements in teacher quality, but you apparently have to improve a teacher’s quality by a full standard deviation to get a student even a 1% income boost at age 28. That’s not nothing, but it’s also a purely theoretical hypothetical; it’s not the kind of return that gives us much reason to think education reform is going to be the best way to fight poverty.

Measures of Poverty

Here’s Doug again:

My understanding is that Paul may have used GINI as a proxy for poverty but according to the World Bank, GINI is a measure of income distribution rather than of poverty. And this could be a major confounding factor. For example, roughly speaking, being in the bottom quintile of incomes in the United States makes you “poor”…relatively speaking. I do not intend to mitigate the suffering or difficulties that come with that poverty. (In fact, I work primarily with high poverty schools because i care so much about it) but being in the bottom quintile in the US would not make you poor in India or El Salvador.  Your income and quality of life would be far above the mean.  So your opportunity and quality of life even at the bottom of the economic hierarchy in the US is far better than that of people far higher on a relative ranking of poverty within other countries. So if i improved my nation’s schools and GDP growth went up steeply, everyone would become more prosperous and better off and poverty would be mitigated but my country’s GINI scores might not change. In fact they’d be just as likely to become more pronounced–poverty and income distribution are different things.

To clarify, the measure of poverty I used in my original post was the “poverty rate before taxes and transfers” with the “poverty” line set at 50% of each country’s median income. This is not technically the same as using Gini though most of the differences are unimportant here. Doug is correct that the measure of poverty I used is defined in some sense relative to each country’s overall wealth and that in absolute terms the poorest Americans are often better off than the median residents of much poorer countries.

Still, I think my chosen measure is preferable.

The fact that my chosen poverty measure is in some sense relative is a feature, not a bug. As far as I am aware all conversations about education reform in the United States are about improving welfare in the United States. Whether the poorest Americans are richer than the average residents of poorer countries is therefore immaterial. In a hypothetical conversation in which I have to choose between improving the absolute well-being of the poorest people in the world and improving the absolute well-being of the poorest quintile in the United States, I will likely prioritize the former. In our actual conversation about fighting domestic poverty, however, it seems not just useful but essential to factor in our overall level of wealth when thinking about what is possible or acceptable.

Also, even if we prefer to ignore our considerable national wealth when evaluating how effectively we are ameliorating suffering for our poorest citizens we still come out poorly in international comparisons. Here, for example, is a comparison of international child poverty rates measured both in relative (median income) and absolute (PPP) terms:

click to enlarge

click to enlarge

Note that even using this more “comparable” measure the child poverty rate in the U.S. is still double or triple that in many other developed countries.

Compared to What?

The effectiveness of an intervention should always be understood in comparative terms. If my claim is that education is probably not the best way to fight poverty, that must be because I think there are better ways and I have been remiss in not stating them explicitly.

We know that the United States has already significantly reduced its poverty rate by collecting taxes and then transferring the money to less-fortunate families through tax credits and benefits:

dajoi-fig-1

We also know that it is possible to have substantially lower rates of poverty than we have in the United States because other countries have successfully reduced their poverty rates well below ours. (Remember, this is true regardless of whether we are measuring poverty in relative or absolute terms.)

How have these countries achieved these reductions in their poverty rates? For the most part, they do it the way we’ve done it: via government redistribution (i.e., taxes and transfers). Here is a comparison of OECD countries’ pre- and post-transfer poverty rates:

click to enlarge

click to enlarge

This is what education reform is competing against for the title of “best way to fight poverty”.

It will not do here to say that tax-and-transfer programs are less effective by hand-waving away their political viability. Their benefits are substantial in magnitude, we’ve implemented major programs in the past, and many programs today could still enjoy bipartisan support (especially if we avoid the ‘education-is-the-best-or-only-way’ narrative). We may not be close to implementing other, more effective poverty-fighting schemes, but we’re not obviously “close” to achieving any particular educational reform panacea, either.

Conclusion

Education is good and we should improve education when reasonably possible. Education may even have some utility when it comes to reducing poverty. Nevertheless, the claim that education is the best way to fight poverty seems both empirically false and politically destructive for other, more direct and effective poverty reduction measures.

Posted in Education Reform | Tagged | 11 Responses

Reformers and International Comparisons

Olympic_rings_with_white_rims_svgOn Tuesday at This Week in Ed I asked what the evidence is that education reform is the best way to fight poverty. The domestic evidence doesn’t strike me as obvious, but I was especially curious about the international evidence.

After writing that post I started nagging people on Twitter to tell me which countries have  – at least according to reformers – education systems we should be aspiring to.

My main reason for doing this was that if reformers really think education is the best way to fight poverty, then presumably countries with the best education systems should have substantially lower poverty rates as a result. My guess is that there is little evidence for this; if a developed country is successful at reducing poverty (and many are), that is usually because of government taxes and transfers, not education. Still, since reformers seem largely unwilling to identify the countries they believe have the most effective education systems, their claims about the benefits of reform (e.g., poverty reduction) are difficult to test.

Unsurprisingly, I didn’t have many takers on Twitter. A few reform-sympathizers chimed in that while we might want to learn from aspects of different countries’ education systems, reformers aren’t really committed to the idea that copying other countries would be useful in general. We have a different context, after all!

I agree whole-heartedly that you can’t judge the effectiveness of an education system just by looking at its outcomes. Context matters a great deal. But, of course, a fair amount of the education reform movement is premised on the idea that we can tell discern the mediocrity of our own education system by noting the mediocrity of our outcomes.  If the inferiority of our education system is demonstrated by our test scores, it’s hard to see why the superiority of other systems wouldn’t be. Raw international test scores either tell us about educational quality or they don’t.

Jason Becker stepped up with a more radical defense of reformers, arguing that they are not, in fact, making claims about educational effectiveness at all.

As far as I can tell, this is false as an empirical matter of fact. For example, here’s StudentsFirst with a series of ads explicitly claiming that “our education system” is a joke.

Hypothetically, though, could education reformers take the position that the our ranking on international test scores is interesting just because our outcomes as such have implications for our “competitiveness”?

I’m skeptical for two reasons.

First, the extent to which our relative educational outcomes matter for our international competitiveness is an empirical question that does not seem to me to have an obvious answer. America has not “lead the world” in educational outcomes for decades, if ever, and yet we seem to be reasonably successful and “competitive” (whatever that means). If reformers want to go this route, they’d have an awful lot of work to do in terms of demonstrating this relationship. (Note that they rarely bother to pursue this, which suggests that this is not really their main concern anyway.)

Second, and more fundamentally, would it even make sense for education reformers – as education reformers – to make a big fuss about our outcomes and competitiveness without making any claims (implicit or otherwise) about our educational effectiveness? Probably not: why bring up unequal education outcomes in a conversations about education reform unless you think the problem could be addressed by improving our educational effectiveness?

In other words, it wouldn’t make sense for education reformers just to point out that our outcomes are too low unless they are also – at least implicitly – trying to argue that those outcomes could be improved through changes to our education system.

Admittedly, reformers are often not explicit about what they think international comparisons demonstrate, probably because when you get right down to it they demonstrate very little. Nevertheless, those international comparisons either tell us something about education reform or they are non sequiturs in a conversation about education reform.

If we suppose reformers do not intend for international comparisons to be non sequiturs, there is still no plausible argumentative path forward for them.

1. Reformers may intend the international comparison to demonstrate that our education system is unusually or unnecessarily ineffective. This would be a bad use of raw test score data, which measure student achievement, not school effectiveness. Furthermore, as noted above, this would imply that reformers should be able to point to “more effective” education systems in other countries. They seem loathe to do this.

2. Reformers may intend the international comparison to demonstrate that, regardless of how effective our education system is now, it needs to be more effective in order to “catch up” our students. However, the plausibility of this argument rests in no small part on what we believe about our system’s current educational effectiveness. The more effective we believe our system already is, the less plausible we will find it that our international standings can be meaningfully altered by trying to make it even more effective. It will seem plausible that education reform can substantially close the gaps between the U.S. and the highest-achieving countries only if we think that our current education system is quite ineffective in relative terms. Which is to say, here reformers have subtly begged the question at hand, namely: whether our mediocre educational outcomes are best understood as resulting from educational ineffectiveness.

The most frustrating aspects of all of this are the obvious double standards.

In other contexts reformers have acknowledged, however grudgingly, that raw test scores don’t tell us much about effectiveness and what really matters is “value added”. When comparing scores internationally, however, suddenly all we need to know is that our test scores are lower than the scores of people in (some) other countries and this is supposed to make us very embarrassed about our education system.

Moreover, while these test scores tell us a great deal about how poor our own schools are, they are apparently of no use in evaluating other countries’ schools.

All in all, then, the reformers’ position often turns out to be simply that ranking countries by educational outcomes is useful for denigrating American schools…and nothing else. It is profoundly unlikely that that is the correct way to use international test scores.

Posted in Education, Education Reform | Tagged , | 6 Responses

Some Advice for Common Core Supporters

659315_5ba9794c89Yesterday at TWIE I gave CCSS supporters a hard time for seeming to give up the affirmative case for the new standards:

These days supporters seem to dedicate most of their time to assuring us that the CCSS are not to blame for “fuzzy” math curriculua or “whole language” or questionable history assignments. We are even told that it’s just as well if states opt out of the Common Core altogether because they’re unlikely to gain much from implementation anyway.

On Twitter Morgan Polikoff asked what I would prefer for CCSS supporters to do instead of making these half-hearted defenses. Fair enough, though I don’t really think of myself as part of the team so take all of this with a grain of salt. I’d recommend the following, which are arguably all variations on one or two themes:

1. CCSS supporters need to acknowledge that they overestimated the potential for standards per se to improve curriculum and instruction. Reformers are rarely comfortable wading into the minefields that are debates over C&I, but one of the lessons here should be that you can’t just lean on standards to do that work for you. Frankly, the science of teaching is just not sufficiently advanced and accepted that educators and families will all fall into line if you just give them the right goals to shoot for. Teachers already think their pedagogy is about right for whatever learning objectives you want to establish; if you want them to think differently you need to convince them directly. It is also increasingly apparent that you can’t avoid nasty battles over curriculum by saying “standards are not a curriculum”.  You may as well wage that war directly.

2. CCSS supporters should acknowledge that the new standards are not really as unambiguous as they had thought. By convincing themselves that opponents just don’t understand what the standards “really” say, supporters end up repeatedly missing the point. That tactic may work in a few cases, but whether, for instance, they encourage “fuzzy math” is just not something you’re going to be able to conclusively establish using nothing but the standards themselves. To the extent that teachers think the standards encourage fuzzy math, the standards do encourage fuzzy math. Supporters need to realize that while the CCSS are not “to blame” for peoples’ pre-existing edu-confusions, those confusions are really what we should be dealing with. As with #1, wage those wars directly.

3. CCSS supporters should focus more on Common Core-aligned assessments. What the CCSS “really” mean will be determined in large part by the tests used to hold teachers and schools accountable. So while it’s all well and good to assure us that, e.g., the CCSS “require” a “content-rich curriculum”, that won’t really be true unless the eventual assessments require a content-rich curriculum. I don’t think I’ve seen nearly enough attention paid to this and we’re already pretty far along in the assessment-design process.

4. CCSS supporters should spend more time highlighting “good” Common Core-aligned lessons. Opponents are already more than happy to talk about “bad” lessons;  supporters seem to have decided the best thing for them to do is join in. This is contributing to a growing sense – however unjustified – that the CCSS promote all sorts of educational silliness. That’s a recipe for further political losses and it means that educators aren’t getting the CCSS guidance they deserve.

Personally, I’m agnostic about the Common Core standards. People whose judgment I trust often say very good things about them. At the same time, the new standards are extremely expensive and disruptive and there does not appear to me to be nearly enough in the way of consensus among supporters about what the standards “really” say to make me think they’ll be good for curriculum and instruction on balance. For every supporter who tells me the CCSS “require a content-rich curriculum” there’s another who insists that the new standards “prioritize skills over knowledge”.

So I am not always encouraged. But there’s still plenty of time for “the good guys” to exert influence over CCSS implementation if they’re willing to start playing offense.

Posted in Education Reform | Tagged , | Leave a comment

VAM and Status Quo Bias

4011991883_8cc3a77d69_nYesterday at This Week in Ed I wrote about the American Statistical Association’s report on value-added modeling in education:

[I]f you were to hear about this report only from the staunchest, most ideological opponents of VAM, you would think it says something else entirely. Valerie Strauss, for instance, claims the report “slammed” the use of VAM to evaluate teachers and Diane Ravitch seems to think it is a “damning indictment” of such policies.

The report itself is not nearly so hyperbolic.

For a useful summary check out Stephen Sawchuk, but the report itself is a mere seven accessible pages so I encourage you read it yourself.

The bottom line for the ASA is that they are optimistic about the use of “statistical methodology” to improve and evaluate educational interventions, but current value-added models have many limitations that make them difficult to interpret and apply, especially when evaluating individual teachers.

There’s been considerable confusion about the report because some people seem to be having a hard time getting their heads around the possibility that VAMs have serious limitations but may nevertheless be appropriate for use in education generally or teacher evaluation specifically. 

So, for example, it’s true – as I was told over and over on Twitter – that the report states that VAMs “typically measure correlation, not causation” and “do not directly measure potential teacher contributions toward other student outcomes [besides standardized test scores]” and offer “scores and rankings [that] can change substantially when a different model or test is used”.

Crucially, however, none of this establishes that VAMs shouldn’t be used to evaluate teachers.

I mentioned this in the post, but it bears repeating: all methods of educational evaluation have limitations. That includes the methods we currently use! The trick is to identify and understand those limitations, and then select the best method (or combination of methods) for meeting our objectives.

It just so happens that the methods we use to evaluate teachers now consist mostly of classroom observations by administrators, usually employing a lengthy rubric with varying degrees of clarity and specificity. Those observations are by no means free of limitations.

In fact, it’s hard not to suspect that if, in an alternate universe, reformers were trying to move us away from widespread use of VAMs and toward an observation-based system, many critics would be up in arms about the fact that classroom observations are subject to many of the same limitations as VAMs (plus a few of their own).

Do classroom observations “measure causation” with respect to student outcomes? Do they “directly measure teacher contributions” toward student outcomes at all? Are teacher ratings based on observations stable even when evaluators and rubrics are changed?

As with VAMs, the answer to all of those questions about classroom observations is, of course, “no”. And yet we rely – heavily! – on these observations anyway.

Because classroom observations are a ubiquitous part of the status quo today, we take them for granted and their limitations don’t seem nearly so horrendous. VAMs, on the other hand, represent a departure from the status quo, which means that we’re much more likely to turn a critical eye toward them.

That’s a natural tendency, but it’s also basically irrational. The relevant question is not just, “do VAMs have limitations?”, but “compared to what?”

Posted in Education Reform | Tagged , | 6 Responses

#EduFictionMoviePitches

After watching the trailer for Scarlett Johansson’s new action flick last week, Marc Porter Magee and I agreed that the movie seemed perfectly acceptable until Morgan Freeman intoned that “most human beings only use 10% of their brain’s capacity”. I’m pretty forgiving when it comes to contrived movie gimmicks, but as an explanation for the main character’s superhuman abilities that’s just too goofy to swallow.1

The movie’s use of the “10%” myth got me thinking, however, of other education-related misconceptions that might make for good movie pitches, so I threw a bunch up on Twitter over the weekend. A few that I was especially pleased with:

There were also several good submissions from others:

You can check out all of them here. By all means, let me know if you think up new ones.

  1. I was particularly optimistic about this new movie because director Luc Besson has made some very good action/thrillers, including The ProfessionalThe Fifth Element, and La Femme Nikita. []
Posted in Education, Movies & TV, Pop Culture | Tagged | 1 Response