The Challenging and Changing World of Industrial Efficiency Program Evaluation

Today ACEEE released a new report that digs deep into the complicated but important work of evaluating the impact of energy efficiency programs. This report targets the evaluation of industrial energy efficiency programs in particular, taking aim at some of the most contentious areas of evaluation.

Regulators or efficiency program administrators typically hire third-party evaluators to assess how a program is performing. One of these types of assessments is called an impact evaluation, which measures the actual impact of an energy efficiency program. This typically includes energy savings, non-energy benefits, and energy efficiency investments that the program encouraged elsewhere in the market. It also typically measures “free riders”—participants in an energy efficiency program who would have made their energy efficiency investment even if the program never existed.

The challenge in industrial efficiency program evaluation is the manner in which these impacts are defined and measured. Industrial facility managers make energy efficiency investments due to all sorts of market factors, business needs, and influences. These managers make decisions about energy efficiency investments over long periods of time, often waiting a year or more to make an investment for strategic business purposes. They often consider the non-energy benefits of energy efficiency investments to be equal or greater in value to the energy savings themselves. And they learn about new technologies and opportunities from their peers and trusted vendors as well as from their local energy efficiency program.

By design, evaluations of energy efficiency programs often overly simplify the nuances of decision-making in the industrial sector. The evaluations also are conducted on timelines that might not align well with the behavior of an individual industrial entity. A facility may make an investment decision several years after an efficiency program initially influenced their decision, but evaluators may determine the reasoning behind the facility’s decision well after the investment has been made—and well after the customer might be able to recall the original influence of the program.

Today’s report targets issues like the timing of evaluation activities as well as:

Regulators want measureable energy savings at the low cost that efficiency can offer. Efficiency programs want (and need) to get credit for their hard work, even though that work is sometimes very hard to quantify. Evaluators want to challenge and prove the assumptions held by the efficiency programs. And industrial energy managers want to make informed investment decisions without being overly bothered by those wanting to survey their motives and dig into their data.

So what’s the efficiency community to do? One answer is to determine which metrics collected by evaluators need to be derived with high degrees of precision—and which ones do not. Free ridership, for instance, is probably not something that can be perfectly measured (the current efforts of evaluators notwithstanding). Surveys and ex-ante reviews (reviews conducted at the beginning of a program participant’s interaction with a program) could likely yield “good enough” estimations of free ridership. A free ridership value with four significant figures is precise but probably not very accurate.

Another idea is to determine which evaluation metrics are being used to determine cost-recovery and shareholder incentive amounts. Those metrics are often the most fought over and the ones where small differences in numbers can make a major difference in a utility’s bottom line. Understanding the motivations of different stakeholders can help inform a dialog between all the parties with something to gain or lose from evaluation findings. 

As states increasingly look to their industrial sector to help meet future energy efficiency goals, determining what is influencing that sector is going to become more complicated. New air quality regulations will further incentivize energy efficiency investments, especially for facilities with aging boilers. Evaluators will be challenged with teasing out the influence of energy efficiency programs from that of new air regulations.

These and other changes, such as the growth of energy management programs, will challenge the status quo of industrial program evaluation. Success in industrial energy efficiency programs may be harder to attribute to a specific year or effort, and influences may be harder to pin down. The industrial efficiency community and its evaluators and regulators will need to determine, together, which metrics would be universally useful and how they could best be derived.