Utility energy efficiency programs invested $8 billion nationwide in 2018, so determining their impacts and effectiveness is critical. How much energy was saved? Were program dollars well spent? Did programs meet their goals? How did customers benefit? And how much carbon was kept out of the atmosphere? Asking and answering these questions is the realm of energy program evaluation—practices that are integral to today’s energy efficiency programs and that ensure efficiency’s multiple benefits accrue to customers, utilities, and society.
A new national survey of energy program evaluation policies and practices finds wide variation among states, as earlier studies have also shown. Despite the variation, the survey also found common underlying methodologies and evaluation techniques that have benefitted from industry efforts over the past several years to improve their consistency and quality.
The art and science of energy program evaluation
A fundamental challenge for evaluating energy efficiency programs is estimating the energy savings that result from a given program—that is, quantifying the amount of energy that was not used. There is no such thing as an energy savings meter. Estimating savings typically involves applying statistical, engineering, and modeling techniques to customer energy data and other information. Such techniques have evolved, becoming more advanced through the ability to collect and analyze highly granular customer data. Program evaluators also rely on widely accepted methods, data, and algorithms to estimate savings, increasingly documented and required in technical reference manuals (TRMs), which serve as state guidelines for program evaluation. These practices and advancements have built and strengthened the science of evaluation, but there remains an element of art in program evaluation. The evaluator must know which techniques to apply in different situations given limits, conditions, and available resources.
Common structures, varying practices
State policies and practices for evaluation are not uniform across the United States. This is not surprising, given the great diversity that exists among states in the structure of their utilities, the policies and regulations that govern them, and the nature of their energy resources. Prior ACEEE research (completed in 2012) found similar diversity in energy program evaluation policies and practices across the states.
Utilities and regulatory commissions rely on evaluation results to guide decisions on energy efficiency and other demand-side programs, such as which programs to offer and how much to spend on them. To make good decisions in the public interest, oversight and administration of program evaluation needs to be structured to ensure accurate and unbiased results.
In ACEEE’s new update to our earlier research, we found that program evaluation continues to be performed largely by independent contractors. Assigning this role to evaluation professionals helps ensure that results are based on sound methodological practices and that they can be relied upon for good decision making by program administrators and regulators.
Another key issue is evaluation oversight. We found that nearly all regulatory commissions use evaluation results for general oversight of programs and provide opportunities for public involvement. For example, in Massachusetts and Arkansas, the stakeholder collaboratives that advise and review program development, implementation, and evaluation include members of the public.
Shared core principles and new tools to meet new challenges
Our survey confirmed that program evaluation is solidly grounded in a set of core principles. Evaluators today follow well-established professional practices and can draw on a wealth of resources, such as TRMs and the Database of Screening Practices, many of which have been developed in recent years. However, changes in energy markets, technologies, and program goals are affecting program designs and associated evaluation practices. Evaluators face new challenges, such as accounting for carbon reductions or valuing distributed energy resources. To meet these new challenges and improve traditional practices, evaluators can use various advanced technologies, such as smart meters and data analytics. We found that the uptake of such advanced applications is slow. Slightly more than half of the states we surveyed have not yet used advanced metering technologies or data analytics for program evaluation.
Because most utility energy efficiency programs are funded through customer bills, ensuring that the programs are cost effective is a primary function of program evaluation. Several cost-effectiveness tests are used, each representing a different perspective and sets of costs and benefits. As in our 2012 research, we found that the application of cost-effectiveness tests continues to vary widely among states. Despite the variation, there is a growing trend toward embracing core principles that yield balanced and comprehensive assessments of costs and benefits—something some past practices did not do, though it is critical for making prudent choices about investments in utility infrastructure and avoiding wasteful or unnecessary spending.
Energy efficiency is increasingly being called on to deliver multiple benefits to consumers, utilities, and society. In addition to its more traditional focus on reducing costs for customers and utilities, energy efficiency is a fundamental tool for addressing climate change, public health, environmental justice, and energy equity. Evaluation is the way we measure and assess how well our energy programs are meeting these objectives. Our survey shows that evaluation policies and practices are generally well poised to meet current and emerging challenges for assessing the performance and impacts of energy efficiency programs. The critical role of evaluation, however, requires continued oversight and public involvement to ensure accurate and unbiased results that can be used to make wise choices about our energy resources.
This Article Was About
Evaluation, Measurement, and Verification (EM&V) Energy Efficiency as a Resource Distributed Energy ResourcesAuthors
