The E2e weatherization study: generating more heat than light

A recent academic working paper on the low-income weatherization assistance program in Michigan, by researchers associated with the E2e project, has created much controversy. My colleague Marty Kushler recently published a blog post criticizing the paper and its accompanying policy brief; many others have done the same (see links at the bottom to read more). As the critiques have noted, there are two primary concerns with this paper:

1. The paper looks at one program in one state and inappropriately seeks to apply the results to all residential energy efficiency programs.

2. The evaluation focuses only on energy savings and ignores the fact that low-income weatherization is not only designed to save energy, but also has other objectives, such as improving occupant safety, health and comfort, addressing structural problems in homes, and reducing utility bill arrearages.

In this blog post, I’d like to take a broader look at some important issues this paper raises, building on these two points and also raising a couple of additional issues.

Methodology

First, parts of this study use an experimental research design called a randomized controlled trial, where researchers define a pool of eligible customers and randomly assign some of the eligible customers to a treatment group and some to a control group. In this way the treatment and control groups are identical. This is a powerful research approach but also has an Achilles heel: you cannot generalize the findings to households and programs that are different from those that you studied, because different types of households and different programs can have characteristics that would change the findings. In this case, they looked at weatherization of low-income households in a few counties in Michigan, and the results cannot be generalized to other states, to other programs, or to other income groups.

To illustrate the point, a study by Oak Ridge National Laboratory is looking at 30,000 homes that participated in the low-income weatherization program across the country (and not just one state). Preliminary results from this study find that, relative to a control group, benefits are on average about 1.7 times the cost––a very different result from the E2e study. While there are some differences between E2e’s approach and the quasi-experimental approach used by Oak Ridge, the big difference is in geographic coverage.

Missed benefits

Second, many energy efficiency programs and investments serve multiple purposes. To compare all costs against only some benefits is to bias the analysis. In this case the paper in one place seeks to assign all the costs to energy savings and in another place puts all the costs on carbon reduction; nowhere do they look at other important benefits of the program. To fairly evaluate such programs and investments, either all of the benefits need to be quantified and compared to the full costs, or if only energy benefits are quantified, then only the incremental costs of the energy efficiency improvements should be counted. Given that the weatherization program being studied is deliberately intended to achieve multiple benefits beyond simple energy savings, a more comprehensive assessment of benefits should have been done.

Market-only agenda

Third, it appears that one of the aims of this study was to discuss the most efficient approaches to reducing greenhouse gas emissions (this is the title of item #1 in their policy brief). In the brief they recommend market-based approaches like cap-and-trade programs and carbon taxes. While ACEEE supports such market-based approaches, we recognize the need to use multiple approaches to substantially reduce greenhouse gas emissions, including energy efficiency programs. The country will also need to use multiple approaches in order to meet broader energy and social goals. A carbon tax is unlikely to do much to address the needs of low-income households, just as greenhouse gas reduction is not the main purpose of low-income weatherization. Rather than tear down one approach at the expense of another, it would be useful to assemble a set of complementary approaches, using evaluation to make each approach as good as it can be. Biasing research, even toward a noble end, can undercut efforts to build support and find “win-win” synergies.

Missing review

Fourth, this paper is labeled a “working paper.” It has not yet been peer reviewed. Rather than release an un-reviewed study to the press to great fanfare, it would have been much more appropriate to share the results with energy efficiency evaluation and low-income weatherization experts (and not just economists) to get feedback on the draft report, learn about any problems with the draft study, and revise the study to address the problems. The issues that are being raised on many blogs regarding the E2e paper would have been much better raised in a peer-review process, rather than in the press and other public forums. This is not the first time we have had to criticize unbalanced research or inappropriate spin coming out of E2e (for other examples see here and here). The authors are all university professors; we presume this is not the way they are teaching their students to publish research.

Michigan, Wisconsin, and the media

What does this mean going forward? While we strongly support research to determine how well energy efficiency programs are functioning in practice, we hope that the E2e project does a better job in the future to seek external review before they release their findings. They should also fairly balance both the energy and non-energy benefits and costs of the programs they examine.

They can start in Wisconsin. One of the authors of the Michigan study told the Associated Press that he is finding similar results in a study of middle-income homes in Wisconsin. We strongly suggest that this new study receive expert review before it is released, and that it consider non-energy costs and benefits. Expert-reviewed research by ACEEE and others has found that non-energy benefits are also prevalent in middle-income homes. It also bears noting that middle-class weatherization, like low-income weatherization, is relatively expensive. It would be doubly irresponsible to generalize the new results to other energy efficiency programs, after picking two of the most expensive examples. For a little perspective, Lawrence Berkley Laboratory found in a recent report that whole-home weatherization programs cost utilities an average of 9.4 cents per kWh saved, while the average residential program costs only 3.3 cents per kWh saved.

We hope that the press is much more careful when it examines future E2e papers. While there is some excellent research coming out of this project, there are also a distressing number of questionable papers. Good research is immensely valuable, but poor or mischaracterized research can be far more harmful in the short term, given the media’s penchant for counterintuitive stories. The result is to hinder the ability to make rational policy decisions.

The E2e weatherization study: generating more heat than light

Methodology

Missed benefits

Market-only agenda

Missing review

Michigan, Wisconsin, and the media

Click on the links below to read more discussions on the study.

This Article Was About

Authors

Other Interesting Reads