Why Nobody Believes the Numbers: Al Lewis takes on Milliman and Mercer (transcript)

This is the transcript of my recent podcast interview with Al Lewis. If you’re looking for a productive post you’ve come to the right place.

David E. Williams: This is David Williams, cofounder of MedPharma Partners and author of the Health Business Blog. I’m speaking today with Al Lewis. He’s president of the Disease Management Purchasing Consortium and he’s also the author of a new book: Why Nobody Believes the Numbers: Distinguishing Fact from Fiction in Population Health Management.

Al, thanks for joining me today.

Al Lewis: Thanks for having me, David.

Williams: It’s a provocative title for the book. Tell me a little bit more about the thesis of this book is and share a few key takeaways.

Lewis: The thesis of the book is that the entire valued-added services industry in the managed health care sector, including for self-insured employers, is just rife with incompetent measurement of outcomes. As a result of this incompetent measurement of outcomes, massive amounts of money get misallocated to programs of extremely dubious value.

However, the good news is that if in fact you can measure the outcomes correctly, you can probably reduce the amount of money you spend on these programs while getting a modest but real savings and improvement in health for your employees.

The three biggest takeaways of the book are; number one, that most of this stuff is done incorrectly; number two, you can do this correctly, you can measure your outcomes correctly using what I call ‘ingredients you already have in your kitchen’. You don’t have to do anything fancy involving statistics or actuaries or anything, simply the fifth grade math that you learn to get an answer, which is reasonably valid. And number three, when you get that valid answer, you may find that you saved small amounts of money and that you’ve made some noticeable improvements in the health of your population, but it will not solve your problem of increasing cost year over year.

Williams: There is a group out there called the Care Consortium Alliance. They have guidelines for measuring outcomes, but I take it that you don’t think too highly of their approach even though it’s fairly well accepted in the industry.

Lewis: Math is not a matter of opinion, David. And that’s the beauty of it. It’s not ‘he said, she said’. It’s proof versus disproof. So it’s not a question of whether I think highly of their guidelines. The question is whether their guidelines are valid. And I actually found a major mistake in their premise that pre-post measurement is valid. I proved the mistake in the book.

But people should not be scared away by the word “proof” in the book; this is all fifth grade arithmetic. In fact, I actually dedicated the book to my fifth grade math teacher. I said, “To my fifth grade math teacher for apparently doing a better job than the other kids’ fifth grade math teachers.”

The mistake causes savings to be overstated any time a pre-post measurement is used. Rather than get it in to a shouting contest in math, which is not appropriate in math, I simply said, “This is the answer and if anybody can find a real or hypothetical situation with numbers,  where the guidelines in the book will give a wrong answer and the Care Continuum guidelines will get the correct answer, I will pay that person $10,000.” Well, guess what, the $10,000 is still sitting happily in my bank account.

Williams: You mentioned actuaries. And of course there are big groups like Milliman and Mercer that are active in the measurement business. They have actuaries, they have a lot of people that made it even further than the fifth grade. But you do take issue with a lot of their work. You say that many of their findings are actually impossible.

Lewis: Yes.

Williams: They’re really that out of line?

Lewis: Yes. “Impossible” is the correct  word. Once again, that’s the beauty of math. The numbers either add up or they don’t. It’s not arguable. In the case of Milliman, they found savings for North Carolina Medicaid for their children’s admissions that were more than twice as high as the amount of money that the State of North Carolina spent on children’s admissions.

And by the way, that number did not fall over the period in question, so Milliman was actually impossible in two different ways.

Then Mercer, well, they had several examples of impossibility in the book including in the State of Georgia where they found mathematically impossible savings. It turned out that the vendor, APS Healthcare in this case, later agreed to return Georgia’s fees because they acknowledged not making the outbound phone calls.

So essentially, Mercer was able to find mathematically impossible savings where nothing was actually done to achieve those savings. So am I very popular with Milliman and Mercer? No. In the case of Milliman, I offered them a $50,000 bet to take their report and my report and submit it to Harvard University and have Harvard University say which one is valid. I have not heard back from them.

Williams: What do you suppose leads to these sorts of very large errors and impossible results? I mean, you’re picking on some programs like North Carolina that are fairly well known and often touted as successful. Is it just a matter of not hiring the right people, or vendors are saying what people want to hear. What’s going on?

Lewis: That’s an excellent question and if you had asked me a few weeks ago, I would have said they’re telling the consultants what to tell them back. Having talked to Milliman, having talked to North Carolina, first, the North Carolina people have been complete adults about this whole thing, which frankly, is more than I was at first.

And I genuinely think that they do want to find the right answer. It’s hard for them to accept that perhaps the program has not saved money because they’re very passionate about it. And frankly, if I were in Medicaid and I could move to any of the 50 states, I’d probably move to North Carolina. But that’s a lot different from saying that a program saved money.

Williams: Why is it that you could have this level of error?

Lewis: Well, in disease management, you get it because the pre-post guidelines are mathematically invalid and you can literally do nothing except measure year over year using  the recommended algorithm and you’ll save money.

I actually recommend doing that year over year in a program where you do not have disease management to see how much costs go down without doing anything. And I give a couple of examples of where that happens in real life; Illinois Medicaid is an example in my book.

In disease management, all you need to do is follow the guidelines and you’ll make mistakes. In wellness, the mistakes are just massive. I say in the book that wellness is five years behind disease management in measurement which is like being five years behind Iraq in democracy.

Number one, you can’t compare participants to non-participants and have that be a valid answer. If I split the population, if I say to people I’ll pay you $100 and I’ll teach you how to compete in a triathlon, the people who sign up for that, even if you do nothing, will have lower cost than people that don’t participate.

That’s obvious in the case of the triathlon, but it’s true everywhere. It’s just not recognized. So number one, they compare participants to non-participants. Number two, all the time they’d say we took the high-risk people and we did all the stuff with them and the risks went way down.

Well, Dee Edington who  spent more time studying the measurement of risk than probably everybody else on Earth combined, has noted that if you do nothing to the high-risk people that their risk factors would fall –I may be misquoting a little– by 30 percent over the next two years or something like that. That doesn’t mean the people are getting healthier, that simply means that the lower-risk people are becoming higher risk. But they’re not counted, so it doesn’t matter.

So you have participant versus non-participant bias, you have a regression to the mean in risk factors, and the third thing you have is just the simple unwillingness or inability to step back and say, “Are these findings even possible? Are they mathematically within the range of possibility?” And the answer most of the time is no.

I mean, they say, well, cost went down by 30%. Well, for cost to go down by 30% through a wellness program, you’d have to eliminate essentially every single health-sensitive hospitalization, and health-sensitive ER visit in the entire population in your company.

So you think somebody would say, “I wonder if we’ve reduced the number of heart attacks or the number of diabetic events in our population in the last couple of years.” No one ever asks that very basic question.

You’re trying to improve the health of people’s hearts. Somebody should find out if in fact the number of unhealthy heart events went down.

Williams: Let’s jump coasts and go from the public to the private sector for a minute. I want to ask you about Safeway, which is a well known, highly touted company in terms of the approaches they’ve taken on trying to get their health care cost under control and improve the health of their employees. Is that one that’s robust?

Lewis: Well, I had never challenged Safeway myself because it was so well publicized and everybody cited it so much that I didn’t think it could possibly have been wrong. And then I just read in the New England Journal of Medicine, a study that in fact showed that Safeway’s zero trend, which was due to many things that they did in benefit design, predated their wellness program by three whole years.

So despite the fact that Congress appears to have relied on Safeway numbers when they put wellness incentives into the Accountable Care Act, Safeway’s savings were emphatically not due to wellness.

So you have two situations here; one in the public sector, North Carolina Medicaid, and one in the private sector, Safeway, where entire government policies have been largely based on two sets of outcomes that no one bothered to challenge. And then when you look at them, they turn out to be incorrect.

Williams: On the one hand, I can understand how purchasers of wellness and disease management services and policy makers would be pleased with your work. On the other hand, there is a huge need to save money and improve population health. So I wonder, does anything actually work?. I mean, are we all pointing in the wrong direction here?

Lewis: Well, there is a line in the book that says that everything in life has an 80-20 rule. Meaning, 20 percent of your effort gets you 80 percent of your result. In health care, the 80-20 rule is that 80 percent of the time there is no 80-20 rule.

So you can’t push a button in health care, or two buttons, or three buttons and have costs go down. All the initiatives that are being discussed at the public and a private level, except for very simple things like pushing the money back on to the consumer, are only going to make marginal reductions in savings.

And let’s look at wellness now, in particular. Let’s take the case of Greece and Germany. Germany is very productive and Greece isn’t very productive. So if I said to you all we have to do is have the Greeks be as productive as the Germans and their economy will be saved, and not only that but we were able to pay 20 Greeks who were willing to go to Germany and work there and have their productivity go way, way up. Therefore, Greece can be like Germany.

You would say that’s ridiculous. But in fact, that’s the way wellness works. They start with a premise that healthy people are more productive than unhealthy people, which is true. Nobody would argue with that. Then they find a few unhealthy people who became healthy and they say anybody can become healthy. But in fact, you can’t make Greece into Germany. Only a few people are going to migrate from unhealthiness to healthiness.

You can pay everybody in Greece all you want to try to be more productive, but they’re not going to turn into Germans. Likewise, in wellness, you can make all the incentives you want for unhealthy people and very few of them are going to be essentially paid off to become healthy.

Williams: Final question, Al, I want to know whether you consider yourself to be an optimist or a pessimist?

Lewis: I am neither. I suspect that over time, health care will increase at the rate of GDP plus a little bit beyond that to account for aging and better treatments. And this is not such a bad thing if in fact you have better treatments. I myself have cancer and it’s responded extremely well to the type of better treatments that were not available 20 years ago, when I therefore wouldn’t have spent any money on them.

So that’s not a bad thing. By and large, I’m not pessimistic about the future for our health care. A lot of the extra spending is going to be for treatments that simply were not available awhile back that keep people like me healthy and productive.

Williams: I’ve been speaking today with Al Lewis from the Disease Management Purchasing Consortium. We’ve been talking about some of the interesting findings from his new book which is called Why Nobody Believes the Numbers: Distinguishing Fact from Fiction in Population Health Management.

Al, thanks for joining me today.

Lewis: Thank you for having me, David.

Share