Interview with Jack Phillips on ROI for eLearning

Jack Phillips is founder and CEO of the ROI Institute, Inc., a research, benchmarking and consulting organization. Phillips developed the ROI Methodology™, a critical tool he has used for measuring and evaluating programs such as training, human resources, technology and quality programs and initiatives. Follow Jack on LinkedIn, Facebook, Twitter, Google+, or YouTube.

Why do companies hire you?

Companies need us because we help show the value of what they do. The pressure in the business world today is to deliver value and financial return on investment. So we teach people how to do that and we have 100 consultants to help. We also provide certifications so that people can do the ROI analysis on their own. The challenge is that people who own programs may not want to measure them at the impact and ROI levels. They are afraid of the results, as if they are negative they may reflect poorly on their own performance. Yet the executives who are funding these programs want to measure them. Another challenge is that it takes extra effort in the planning stage to do ROI analysis. The good news is that this is the best way to keep the funding stream going, by showing the value of what you are doing. We look at this as a process improvement exercise. If the program is not delivering value, you can determine why and improve from there.

What is difficult about measuring and achieving ROI in eLearning?

First, I’ll describe the different phases. The first level, which is actually Level 0, is just capturing basic data such as how many people are involved in the course and the cost per person. That is measuring Input. The next level, or level 1, is Reaction, which measures if the content is relevant to your job and your success and also, is the experience user friendly? Reaction is critical, because if it is not positive, people will not complete a course. Level 2 is measuring Learning. Do users understand the concepts? ELearning shines here because the technology allows us to easily use tests, quizzes, simulations, games, and demos along the way to reinforce the learning. Yet, learning is not a guarantee that someone will actually use the information, so Level 3, Application of Knowledge, is needed. This is where you begin to attract the attention of the executives. They want to know that their people are actually using the skills and knowledge they have learned. Level 4 is Impact, which details how the applications affect the business in terms of sales, new accounts, shorter cycle times or to prevent a compliance discrepancy, for example. The final and fifth level is ROI, which takes the impact measure and converts it to dollars and compares that to the cost of the eLearning program. That is your financial ROI.

How often do companies get to those last two levels?

We rarely find Level 3 and Level 4 measurement in eLearning, and ROI measurement is extremely rare. Yet, by design, most programs do not warrant the five levels of ROI. Even in facilitator-led training, only 5-10 percent of programs should be measured at Level 5. This is something that should be reserved for strategic programs every year. Still, I recently reviewed the Brandon Hall award winners for eLearning. Only a couple of them measured impact and really none of them attempted to measure ROI. although they all talked about it.

This is an issue. We need to have developers tuned in to measurement and design courses with that in mind, considering not just the ease of learning but ease of application and use at work. In our studies on eLearning, the process tends to break down at levels three and four.

Why is that?

First, designers do not always work with objectives for application and impact. Second, in classroom learning there is a bond with the facilitator that helps to drive meaningful impact and helps students make a commitment to use the knowledge. In facilitator-led learning, there are also more tools built into the process such as action plans. The biggest difference in eLearning, however, is that management is left out of the process. Research has shown that manager involvement is key to reaching application and impact goals. If the manager supports the process, he or she is usually involved in classroom training, whereas in eLearning the manager may not even know if the employee is taking a course. ELearning can also be seen as informal learning, which means people may not take the process as seriously as they should.

So what can trainers do first to work on ROI?

Getting managers involved is critical. Sales managers, for example, can mandate that all the sales associates take a course and then do follow-ups with each associate, be a coach to them, and encourage them to use the skills. The announcement of a new program should also come from an executive who should communicate the goals and track progress.

Share this on