Net Savings For Non-Evaluators: Some Concepts Rather Than Statistics

Blog Post | January 15, 2014 - 10:04 am
By Martin Kushler, Senior Fellow

 How much energy savings resulted from a given energy efficiency program? This is the fundamental question that program evaluators have faced since the days of the very first energy efficiency programs in the 1970’s.

What makes this task more difficult than it sounds at first is that the results, or gross savings, observed among program participants must be compared to something that cannot actually be observed or measured: what would have happened if the program had never occurred. If gross savings alone are used, there can be misleading results due to the inability to consider the effects of “free-riders” (customers who participate in an efficiency program but would have taken energy-saving actions even if there were no program) and “spillover” (customers who do not directly participate in a program but are indirectly induced by a program to take action).

The term of art for this elusive variable is “net savings,” and evaluators have developed a plethora of techniques and methodologies for estimating it. While estimating net savings has always been a challenge, in recent years it has become more challenging for three key reasons. First, programs have become increasingly complex, including efforts to influence entire markets (and not just specific program participants). Second, there are many entities besides utilities in the marketplace providing energy efficiency information and incentives. Third, evaluators have developed increasingly complex methodologies to attempt to estimate net savings.

At the same time, with the spread of regulatory policies for things like utilities earning shareholder incentives for energy efficiency program performance, and “lost revenue recovery” in some states, the dollar “stakes” attached to the estimate of program savings have never been higher.

For all of those reasons, the subject of “net savings” has become a very hot topic in the industry. In order to help policymakers, regulators, utilities, and other interested parties better understand the factors and choices involved, ACEEE conducted a national review of state approaches to the net savings issue. ACEEE surveyed all 50 states and the District of Columbia, reviewed a large amount of recent industry literature, and conducted interviews with national energy efficiency program evaluation experts. The findings are summarized in our new report, Examining the Net Savings Issue: A National Survey of State Policies and Practices in the Evaluation of Ratepayer-Funded Energy Efficiency Programs .

Briefly stated, the results of this study demonstrate that there is a great deal of diversity in how states are approaching this issue. At one end, nearly a quarter of states simply report gross savings. Another large segment, likely a majority, nominally report “net” savings, but with a fairly simplistic approach (often just using “deemed” net-to-gross [NTG] ratios). Finally, a smaller number of states (many profiled in this report) are pursuing more sophisticated measurement of net savings, including spillover and in some cases, broader market effects.

Some of our key take-aways from the study include:

  • There is no single optimal method for estimating net savings, and “triangulation” of multiple methods is recommended, where feasible;
  • Common practice in the states regarding net savings estimation is improving. One noteworthy example is the increasing recognition that it is inappropriate to only apply downward adjustments to gross savings to account for “free-riders” without also considering the potential upward adjustments from “spillover”;
  • There are important trade-offs to be considered between the time and monetary resources that must be devoted to evaluation vs. the level of sophistication of net savings analysis that can be conducted;
  • Whatever method a state chooses for estimating net (or gross) savings, reports should be transparent in describing the methodologies and assumptions used, so reviewers can understand what was done and how the results might compare to results from other states;

Finally, one additional note about this subject should be emphasized. Although opponents sometimes point to discrepancies among states in their use of evaluation methodologies as a reason to discredit the concept of energy efficiency as a utility resource, such assertions are without merit. Nothing about the diversity of approaches used for estimating net savings in any way diminishes the over-arching conclusion that energy efficiency is the least-expensive utility resource option available. Energy efficiency has been shown to be robustly cost-effective across jurisdictions using very different methodological approaches to quantifying net savings (as documented in a 2009 ACEEE report and will be documented once again, in a soon-to-be-released ACEEE update that reviews results from 20 different states). Debate over particular net savings methodologies is to a large extent an “inside baseball” issue for utilities and regulators who must arrive at specific numbers for reporting, rather than something that should call into question the overall policy of supporting utility sector energy efficiency programs.

The net savings issue is one where methodologies and state approaches continue to evolve. We hope that the assessment of current status across the nation provided in our new report will contribute to the discussion.