Methods: An Overview
The term return on investment (ROI) is used broadly and encompasses ROI, cost-benefit, and cost-effectiveness analyses. While each type of analysis is slightly different, the overall methods are similar.
1. Developing a framework
The initial, and sometimes most difficult, part of the ROI analysis process is developing a framework. In addition to identifying the focus of the analysis (i.e., the question the analysis will address), developing a framework involves: determining what type of analysis is most appropriate, from what perspective the analysis will be performed, the population to include, and the timeframe of the analysis. The framework drives what data will be needed for the analysis. One common error is to allow the available data to dictate the question the analysis will answer. The framework for the analysis must align with the overall objective of the analysis and the desired use of the results.
1.a. Analysis objective and primary purpose
These analyses are much more than a simple economic evaluation of an existing program, which can be done with basic accounting principles. One of the most powerful uses of an analysis is to answer “what if” questions. What if this program were expanded to more patients? More geographic areas? More practices? Comparing different models, scenarios, or options can be extremely useful for decision-making and strategic planning.
1.b. Understanding the different types of analyses and terminology
In general, ROI, cost-benefit, and cost-effectiveness analyses are similar. It is important to understand what differentiates the types of analysis and how different stakeholders may use the terminology, even if incorrectly. ROI is a common term used in business analytics. Program evaluations are more likely to include CBAs or CEAs due to the ability to include indirect and intangible costs and benefits.
One of the primary differences in the types of analysis is how results are presented. ROI analyses present the ratio of net benefits to total cost as a percentage. Results greater than 100 percent mean that returns (i.e., net benefits) are greater than the total cost. CBAs present the net benefit—total benefit minus the total costs—as a ratio of total benefits to total costs. Positive net benefits or ratios greater than one indicate an overall benefit or savings. For ROI analyses and CBAs, all costs and benefits must be assigned a monetary value, including indirect and intangible benefits and costs. CEAs are useful when comparing options with similar outcomes or when outcomes are difficult to value. CEA results are presented as the ratio of total costs to the measure of effectiveness.
1.c. Perspective, population, and timeframe of the analysis
The objective and primary purpose of the analysis will drive the details of the framework, including the perspective, population, and timeframe for the analysis. In most cases, the perspective of the analysis will match the primary purpose. For instance, if the purpose is to convince payers to invest in a program, the analysis should be from the payer perspective. Similarly, the population should be of interest to the primary audience. In the payer example, the population might be commercially insured enrollees in the geographic area the health plan covers. Finally, the timeframe should be determined by what is relevant not only to the primary audience but also to the topic. If the program being analyzed takes three years to implement and show results, then the timeframe for the analysis should be at least three years.
2. Identifying and measuring costs and benefits
Once the framework is established, the next steps include identifying and measuring the specific costs and benefits. Generally, costs are easier to identify and measure than benefits. Benefits can often be intangible or indirect. Additionally, measuring benefits can be challenging if the link between the intervention and the outcomes is not established. Finally, data are often extrapolated to fit the comparisons being made in the analysis. It is important to frame costs and benefits in the analysis so that both accrue to the same stakeholder group.
In the process of detailing the costs, Alliances began to understand the difference between an accounting of the actual costs and an economic analysis. For example, a program cost that was paid for through AF4Q funding for the initial program implementation may become a cost to payers in a scenario examining the cost-benefit of payer investment in a program. While the previous implementation experiencedetermines the magnitude of the cost, how the cost is handled in the analysis depends on the framework and comparison options being presented.
Identifying and measuring benefits proved to be more challenging for Alliances. Generally, the challenges to identifying and measuring benefits relate to the lack of a link between the intervention and the downstream benefits and to the data being currently unavailable.
3. Analyzing results and conducting sensitivity analyses
Once the detailed costs and benefits are identified and quantified, calculating the results is fairly straightforward. Conducting sensitivity analyses is an effective way of addressing questions and concerns with assumptions made in the analysis that could affect the results.
Greater Boston conducted a simulation analysis of a variety of scenarios for conducting its statewide patient experience survey. The analyst varied parameters of the model to determine the optimal combination of factors to achieve the greatest overall benefit to the Alliance.
4. Making recommendations
Once final results are obtained, understanding how to interpret and present the results to the primary audience is important. Many of the Alliances faced barriers and challenges to being able to complete their analyses in the way that they had initially envisioned. However, all of them had a clear understanding of their framework and limitations of the results. Despite limitations, the value of the results is to open the discussion with key stakeholders about how the results could be improved with better data.