Today, we're going to continue our discussion of the different types of evaluation.

Last time, we differentiated between evaluation and monitoring.

Remember, evaluations are periodic,

objective assessments of a planned, ongoing,

or completed project, program, or policy.

Monitoring, on the other hand is a continuous process that tracks

the program mid-stream and uses data to inform

program implementation and day-to-day management.

We also discussed the three types of evaluation questions,

descriptive, normative, and cause-and-effect.

You will remember that we're primarily after the cause and

effect type of question and program and policy evaluations.

As Paul Gertler and co-authors described in their textbook,

Impact Evaluation and Practice,

"The basic impact evaluation question can be formulated as,

what is the impact or causal effect of a program on an outcome of interest?

In other words, it will answer the question,

"Is a given program effective compared to the absence of the program?"

As we will discuss later,

as we dive into methods,

this type of evaluation often relies on comparing

a treatment group that received a project, program,

or policy to a comparison group that did

not in order to estimate the effectiveness of the program,

more on that later.

But I do want to highlight one important term and the distinction here.

You will note that we use the word effective or effectiveness here, as in,

is given program effective compared to the absence of

the program or what is the efficacy of a program in achieving its goals.

What's the difference? Well, effectiveness is real world,

providing evidence from interventions that take place

in normal circumstances or real life.

Efficacy, on the other hand,

provides evidence carried out under specific circumstances.

Oftentimes, we think of this as under ideal conditions.

The canonical methodology for efficacy is the randomized controlled,

and I emphasize controlled trial.

Can you guys think of some tradeoffs between efficacy and effectiveness?

Let's give some examples.

We will use two policy evaluations to illustrate.

First, let's start with an efficacy example.

Let's imagine we're studying how

a financial incentive program can improve smoking cessation rates.

In other words, we'll pay people to quit smoking and stay abstinent,

and see if that improves long-term quit rates.

This was a real study done by my colleague,

Kevin Volpe, in the form of a randomized controlled trial.

There were inclusion and exclusion criteria,

like a patient had to smoke at least five cigarettes a

day and not use any other tobacco products.

This homogenized the patient sample,

and patients were then randomized to financial incentive or not.

You can see that this was a highly controlled situation.

Now, let's consider an effectiveness example.

An illustrative evaluation would be assessing the impact of using

bundle payments to pay for joint replacement surgery on cost and quality.

In this Medicare program,

hospitals volunteered to be paid a fixed price for the acute hospital care,

plus any services a patient used over the 90 days that followed.

This was implemented at hundreds of hospitals across the country.

We'll call this the bundled payment example.

Okay. Back to efficacy versus effectiveness with

the smoking cessation trial and bundled

payment program examples in the back of our minds.

In general, we think of efficacy studies

as providing a truer effect of the intervention itself.

You may think of this as a proof-of-concept,

or testing whether a program or policy has the potential to be impactful,

or how big the size of that impact could be.

These studies have high internal validity.

Internal validity is the extent to which a causal conclusion,

based on the study, is warranted.

Which is determined by the degree to which a study minimizes

systematic error or bias or confounding.

Another way of saying this is,

the key question in internal validity is,

whether observed changes can be attributed to your program or intervention,

that is, the cause,

and not to other possible causes,

sometimes described as alternative explanations for the outcome.

So, efficacy studies have high internal validity,

but we aren't sure when we replicate them in practice,

whether we'll get the same result.

This means they may have lower generalizability or external validity.

In general, we are experiencing a crisis of replicability in science,

and that we find it difficult to replicate some major influential study findings,

much to the chagrin of researchers and the scientific community.

This is not at all restricted to policy evaluation itself.

In part, it's related to methodology and factors like efficacy versus effectiveness.

But there are many other factors at play too like publication bias,

mining data for p-values, et cetera.

That being said, let's set that aside for now.

Let's use one of our examples.

In the smoking cessation study,

the investigators went to great lengths to

make sure that the financial incentives were actually

the cause of the 10% point higher quit rates.

Randomization, sample selection and the like

were all used to increase the internal validity.

However, we may worry that

the next workplace any employer tries to implement this program,

they may not get the same results.

That is, they may not get the same effect if they're younger employees

or those that have been smoking for

shorter or longer than those individuals in the trial,

or other environmental forces like more chewing tobacco or bars around,

alcohol and smoking seem to go together.

So, this randomized controlled trial had high internal validity,

but we may worry it won't apply to other situations.

Effectiveness studies, on the other hand,

provide estimates of the program impact in real life.

That is, estimates that might be highly

relevant to replicating this program in other places.

However, we might have other factors at play that could explain the effect.

That is, the effect could be confounding.

So, effectiveness studies have high external validity.

External validity is the validity of

generalized causal inferences from our policy evaluation.

It is the extent to which the results of a study can be

generalized to other situations and to other people.

Saying this another way, external validity

is the degree to which conclusions in your study would hold for other persons,

in other places, at other times.

Our bundled payment program was implemented in the real world.

The participants had to contend with

what post-acute care providers were available to them,

what the capacity of those post-acute care providers was,

whether there were other bundled players in the market, et cetera.

Seems pretty real worry?

That being said, we may worry there's something totally

separate is driving the effect we are

seeing of bundle payments improving cost and quality.

Like, what if all those bundle payment providers

were also accountable care organizations,

and that was really why they were doing so well?

Also, remember that those bundle participants volunteered to be in the program.

So, we have some potentially very important selection bias

that could affect our causal estimate of the impact.

Here, we might say that the bundled payment example has

good external validity and question the internal validity.

Know that a caveat in

external validity is that hospitals volunteer to participate in bundles,

but we'll get to that in just a moment.

How would we have to change the design of each study to

shift the relative strengths of those examples?

In the smoking cessation study,

what if we studied implementation of a program across 15 different work sites and

three different employers consider all smokers and watch what happened over many years?

This is starting to sound more like an effectiveness study.

No. Or in the bundled example,

what if we mandated hospital in random metropolitan areas to be in

bundled payment programs and put strict criteria on which patients count and which don't?

This is real by the way. Medicare announced intent

around a future program of mandatory bundles in December of 2016.

This is making the bundled payment study sound more like an efficacy study, right?

Yet, it still retains a core real world aspect to it.

So, there're ways to try to bridge this gap methodologically,

but it will almost always be impossible to do so completely.

We covered a lot of ground today.

We discussed efficacy versus effectiveness and dove into examples.

See you next time for even more on evaluation types.

Efficacy and Effectiveness

Health Care Innovation

Rencontrer les enseignants