One of the things I think that's fascinating about this study is, given your short time periods. So you did really good job of measuring implementation, strength, and quality care. You talked about the indicators there. How did you or why did you decide to actually go out and try to measure mortality? As opposed to say, "Well, we'll look at quality of care and implementation strength, and that'll be good enough. Well, I guess the credit goes to the Canadian Catalytical Initiative. They thought about investing in a mortality measurement when they invested $100 million to the program. Many of the donors perhaps would not invest such amount of money for a solid evaluation. I think that's a good practice. It's a rare practice. Yeah, it was also The Global Initiative and the need to evaluate program implemented in the real-time setting, not in a research setting. I think that was also the impetus to have this impact evaluation and also building a local capacity to do evaluation and make impact level evaluation an integral part of program implementation. I think all this, it just created the right opportunity for this program to have evaluating mortality. The government wants to see the evidence of mortality reduction. They were quite demanding that we want to see the investment generate concrete results. They want convinced about it. And measure mortality because we didn't want to stop midway and say, "Okay. Well, the program is strong and then stop and let's go home." We want to make sure that we measure it and show results. Remember through UNICEF leverage, we're able to mobilize fund from the Gates Foundation to support the Inland large inline mortality survey as well, which was quite a unique opportunity. Yeah, I was looking that huge household survey. I mean, it's like the equivalent of a DHS or something. Was that because you want to measure baseline and inline mortality? You need to increase the sample size because you have limited time period. Yeah, exactly. The limited time period meant that we needed a large sample size if we are to detect anything, and also be able to measure mortality inline and at baseline as well. We did this full birth history strategy that allow us to measure mortality on the 18-month period, but then step a bit back again to during the pre-implementation at baseline to measure mortality as well. Was quite a good experience. Let me ask you, too. Sometimes when you do these evaluations, the actual kind of your plan for how you're going to do the analysis and how the data is going to get out there and who's going to use it? Again with these multiple partners, how did you guys agree on all of this stuff about setting, "OK, we're going to measure mortality, how we're going to do the analysis"?The whole analytic plan, how did that work? Was UNICEF the driving force or MOH or was it all of you working together? I think UNICEF was on the side of the implementation. J2 and implementing the research partners in-country were on the side of the evaluation. They did independent evaluation. Although we were working together because we couldn't move without the implementing partners as well. So we had some sort of a standard measurement plan in place where we had planned to do a baseline, for coverage measurement. Those interim data collection on implementation strands, quality of care, the quality TIF surveys. We also assess the monitoring of the program, and [inaudible] can tell you about what was being set up at a time to monitor the program. Then the endline, that will just ultimately measure the mortality. That was the standard measurement plan that we had in place to deliver result on this. We wanted also to keep the integrity of the evaluation by keeping these two roles, so UNICEF with the implementing part and us who are responsible for implementation along with the stronger monitoring, whereas JHU led the evaluation, the baseline, and the endline, and also the midterm assessment in the quality of the implementation strength assessment. For the monitoring part, because we use that data to adjust our implementation, we have identified some key markers like whether supervision has happened or not, training is completed or not. Also what does the standard and the quality of the training also looks like, as it as per the guidelines, as per agreed upon national standard, whether this partner trained this way or the other partner trained. We wanted to have a standard way. Also as I mentioned earlier, whether the service provided was consistent of quality. Again, whether it is in line with the National Guideline. We had this stronger monitoring system. But putting in place a database for all these data points, for all three or four implementing partners, it was just a big challenge. But at least, this periodic quarterly review meetings really provided the information we need to identify where is the critical gap that we need to make proper adjustment and timely adjustment so that in the end, we'll be able so that our independent evaluation team could properly identify or answer the proper question, the question we started to begin with. Implementation and the evaluation track go alongside of each other, coordinated. There was a day that I could never forget when the JHU team brought the results from the field that showed the program implementation input was great in terms of training and supplies, but utilization rate is really low. There are not many families come to the house, extension workers to seek care. The way we're shocked and they said, " [inaudible] , with all the input, what are we going to do?" That's such a wake-up call. That's an example where evaluation finding informs the timely correction of program implementation, and take correct measures as we go. If I recall correctly, [inaudible] , the whole evaluation had three parts: the mortality track, the implementation strands, part done by Nathan Miller, and then the community perceptions done by Brian. Right. I was very impressed with the monitoring of the implementation strands alongside mortality study. There was one day, I remember Nathan came to me to sit out to [inaudible]. It seems I cannot recruit enough patients for my study to observe the quality, to have enough patients to answer my study questions. I said, "That's not possible." These really showed us that the utilization in the initial phase of the iCCM program implementation was quite limited and also is something we overlooked. We paid a lot of tension to inputs, to supplies, to training, to supervision. We didn't pay enough attention to community demand generation to make sure there's adequate utilization. That's just a good example how a good designed implementation research as part of the evaluation could inform program corrections and the lessons learned and to improve the implementation. That's great. Let's go ahead. This is interesting that you mentioned that, guess in terms of the design of an implementation of the quality of care and implementation strategic study, we needed to observe the health workers, the extension workers treating children that are sick. But you'll send teams in the morning, very early there to the health posts and nobody comes. What we had to do is to go around in the community and look for sick children. Not far away from the health post. Exactly. It was a bit skeptical. Who goes around looking for sick kid. But to our surprise, we we're able to find enough children in the immediate surrounding of the health post. Some of them actually had a severe classification needed referral. Exactly, so it was quite interesting. When we took the finding to the original health bureau, Oromia Regional Health Bureau under the Federal Ministry of house, as well as implementation partners. We said, look, we've been working hard for input, but we may not be able to show results in mortality reduction unless we increase the utilization rates and also make the investment worthwhile. That really started a good debate with the government how to improve the program implementation in terms of utilization. Well, let me ask you. We had this result sharing, and it was for everybody shocking because all of us have this assumption, children are dying from preventable illnesses because care is not available closer to home. Nothing, no other factor. If we make that available, of course, you know the people will use it and we'll see mortality reduction. But after we have spent so much, and we didn't see enough children using it. We have to ask the question in, at least earlier, early data showed that most of the time the health post is closed because this health extension workers have 16 other packages that they are responsible. IC SM is just the small part of one of the packages and 75 percent of their time they need to be doing house to house visit. So for that reason, the health post is closed, it may be open whenever they are in the health post. That creates this uncertainty, what time is she coming? What time is the health post open? The regional health bureaus then and with the Federal Ministry, they issued at least one of the health extension workers need to be in the health post and the health post needs to open all the time. At least you know this earlier data, provided the basis, the evidence to change this policy. That become then the standard operating procedure for the health post to be open at least five days a week, if not it wasn't possible to have 24 hours, seven days a week service. That's okay. Let me tell you, that's the perfect example of somebody using results to alter. So was there any pressure when you got negative results to luck? Well, then let's don't talk about this, let's just fix it and act like we did it right the first time, in terms of just. One of the things I've always found interesting about this evaluation is the dissemination was quite broad. The government seemed to get into share information, publish papers. How did that happen? I mean, for us the implementers, especially who are working really hard to improve in all of these supply-side. In UNICEF, we were all the time on the road, our team, including Dr. Luwei, working 24 hours to make sure this is done. For us, of course, it was really a hard pill to swallow. After we have done so much, how come this and I was hugely disappointed at first and I said, Nate, are you sure this is really correct data? But then once you pass that, then immediately you have to ask, okay, what do I need to do different and what do I need to make the correction? In the same manner, Federal Ministry, all other partners also. Initially, it was disappointing. But yeah, this is the real-time, this is [inaudible] This is how data should be used. Data is not getting what you want to see, data is getting the truth and the truth is this and use the truth to make change. It is also an indication of not just low utilization of pneumonia treatment. It is also related to utilization of other primary health care package, but the basic services. There's lot of lessons to be learned from this implementation strength study that leads us to think more how to provide family-centered, patient-centered, child-friendly services. How do you make sure that at the convenient time, not to the health care providers, but to the household, to the mothers, services should be available? How do you balance the services at the health post and household visit. That led to a lot of program discussions that was really interesting. Yeah. Following that. I mean, you may want to talk about the health development armies, the spark following these evaluations and this movement where now from the top there has been some decision to reorganize the community in order to be self-sustain in seeking care. For mobilization. For mobilization. Yeah. This was also in part to relieve some part of health extension [inaudible] so that they focus on some of the key services like family planning, or treating children for common illnesses while volunteers can do the promotion, teaching mothers to breastfeed and also to seek care. If a child is sick, there is a care in the health post. There is drug in the health post to take your child. These things can be done by these volunteers. This is another major policy change. Now in every community, about one to five network, about 30 networks of women group is created. They have received training, they have a counseling carved that they can use to counsel mothers. Radio. Yeah. Radio programs, religious leaders. There has been a huge demand creation projects actually after iCCM concluded, the funding ended. But UNICEF and other partners also mobilized resources that government also are located. Then a whole strategy for demand creation was developed actually. Well, I really do think that that's one of the surprising things to me. You rarely hear of evaluations of programs that didn't meet their overall goals and mortality reduction. But in fact, it seemed to have had quite an impact that led to a better result than perhaps if you had found a mortality reduction. All of you have been involved in evaluations and program implementation. Is this, what happened here standard or is it better than most and what made it work or not work from your perspective? I want to ask each of you if you had any concluding remarks on this particular evaluation. Who wants to go first? I think so far, just like the IMCI, the theory of change for ICCM is quite strong. Like [inaudible] was saying. If the child is sick, give the child drug. The drug has been proven to be efficacious. The theory, the impact model is strong. Now, can we deliver that in a real world? That's what the evaluation was about. Make sure that we identify the bottlenecks, what is not going well in order to improve the program, not to destroy the program. That's the important piece because we're not up to the goal of saying, well, this ICCM doesn't work. Remove ICCM and close and think about something else, but how to strengthen the program. We're glad that the country was able to take up those results and see what are some of the hole in the rollout of the program that they can close in order to strengthen the program. I think that was quite an interesting experience. I must say the collaboration with UNICEF. I have to say UNICEF has been the driver of the evaluation in a sense, in agreeing to actually pay for it. We went to the evaluation without much money to pay for the baseline, which came out quite a big tab and unexpected, but we kept charging forward and with leveraging funds here and there we are able to complete the evaluation, and I was quite pleased and fascinated by the evaluation. I think for me, you know, it is a lifetime experience or lesson. For me, it has been a public health school for me actually. What is very peculiar for this is that we are doing it in the real situation where it is messy, where there are so many moving parts, and you have to do implementation and at the same time also the evolution goes in parallel, and you have to make that connection and communication. That was really unique. In most cases, evaluations are viewed as, if you don't meet project goal, you are viewed as failed project and there will be some consequences. Your donor is disappointed, the government is disappointed, as an organization you are in some box, but this is not there. This was truly learning, truly evaluation is to improve the program. Because of that, I think we were free to use that data to improve the program in the governmental, so in the same manner. To be frank also, when we presented also the mortality reduction, they say, "Okay", This may not be significant statistically speaking, when we measure population level. But we have saved some lives. Even if it is one child, two child, it matters, that child matters, and it is not a loss. This doesn't mean we have failed. This has been how that was viewed and used. I guess what is important for the student to learn from our lesson is that good evaluation should be designed around a solid program, not the other way round, and we encourage each of you to look for good programs to put your effort to design good evaluations.
