So I'm delighted to have Clarissa Brockelhurst here as our guest today. Clarissa is one of the foremost leaders in the WASH sector today around the world and she's worked for many organizations in the WASH sector. She's worked for the World Bank's Water and Sanitation Program, Water Aide, many other clients. From 2007 to 2011 she was chief of UNICEF's Water, Sanitation, Hygiene section. Today she's an independent WASH consultant operating out of Ottawa, Canada and she's also an adjunct professor here at UNC. So Clarissa, I'm so happy to have you here. Thank you so much for coming. >> Thank you, it's a pleasure to be here. >> Great. So let's talk about the joint monitoring program. When I started in this sector years ago, we really didn't have a very good picture of what was going on on the ground and the water in sanitation field, and that's changed now with the joint monitoring program. So, tell us how it happened. >> So, the joint monitoring program has a long history. It's a program that is implemented by as a partnership between WHO and UNICEF. And it's probably one of the more successful intra-agency partnerships. To call it a program is a bit of a misnomer because a program makes it sound like its got hundreds of staff and maybe a uniform or something like that. And in actual fact the two agencies run it using the resources of existing staff members. And the two agencies take turns every two years to produce a comprehensive report on progress on water and sanitation. And part of the history is that originally the information that was available, particularly to the WHO that first did this monitoring was information that was provided by governments. And there was a general feeling that this was of mixed quality. Some governments were very comprehensive in what they reported and some governments less so. But this is the information that was produced. And then, around the time of the World Summit for Children, which was in 1995 I believe, water and sanitation became a big priority for UNICEF. And so UNICEF sort of joined forces with WHO and the two agencies started reporting on water and sanitation together. >> This was before your time. >> This was before my time. And the next big sort of watershed was that because UNICEF Post World Summit for Children had a need for much more data on what was happening in children's lives around the world, UNICEF introduced a household survey called the Mutli-Indicator Cluster Survey. And this joined another comprehensive household survey done by the United States done by USAID called Demographic and House Survey. And these two surveys, done spacing about three to five years in most developing countries, contributed to some data revolutions. So all of a sudden, after a few years, there was far more household survey data than there had been before. >> How big were the sample sizes on these new surveys? >> The sample sizes vary from country to country. They tend to be large enough so that you can say with reliable certainty what is happening in the country. But in very few countries are they large enough that you can desegregate below national level. The only exception is India. India has a large enough sample size that you can actually do analysis at state level, which as you know for India is extremely important because the states are so different. I believe that the DHS surveys have a larger sample size than the Mutli-Indicator Cluster Surveys. But the two surveys coordinate very closely with each other. And one thing that the two surveys have in common is they both have a module of questions specifically on water and sanitation. So after a few years it became clear that now there was a body of data from household surveys on what users said they were doing in terms of water and sanitation. What was their primary source of water? Where did they defecate? How far did they walk for water? And the people working on the JMP at the time said, okay, it's time to switch from data provided by governments to data that comes from these household surveys. So this was a major change. And I think it has contributed enormously to the reliability of the information published by the JMP. It has also changed the way that we can do analysis with the JMP data. Because it comes now out of household survey data we can now do correlations and analysis that relies on all the other information that comes out of those household surveys. So for instance we can do wealth quintile analysis. Because the household surveys also collect information on assets at that household level from which you can come up with wealth quintiles. >> So what were the biggest obstacles you faced when you made the switch from government supplied data to this other way? >> I think one of the major problems is that there are certain things you cannot ask a household. You cannot ask a household, is your water safe? They don't know. They don't know the bacteriological quality of their water. So for instance, in terms of trying to track the NDG target, which is, once the NDG's came into place the JMP was requested to track progress against the water and sanitation NDG's. Safe water is what appears in the MDG target for water and sanitation. But we can't actually track safe water by going out and doing water quality testing at a national level in every country. So we have to use a proxy. And the proxy is that households are asked what type of technology do they use for water supply? So they can say they use a hand pump. They can say that they have a piped water source in their house. They can say they use an open dug well. Or they can say they use a river or a stream for instance. And then those technologies are divided into two classifications. One being improved and the other being unimproved. And only the improved ones count as providing coverage, providing access. >> Was it hard to get the same questions consistently asked in different- >> Yes. Yes, and certainly this was still a problem when I first came into UNICEF in 2007. We had some big household surveys in which the response categories differed from the main survey. >> You had to match them up. >> I can remember one of my first experience was being asked if I would say whether a latrine with a shelf was an acceptable sanitation technology. I said I don't even know what a latrine with a shelf is. And this is what the Chinese survey had returned, and it was obviously a translation problem. But we never did manage to figure out what a latrine with a shelf was. So one of the things that the JMP did is a produce a guidance on harmonized questions. So there's actually a document that anybody designing a household survey, not just MICS or DHS. But if a country doing a survey or another agency is doing a national household survey, they can take a module of questions straight out from that. All the response categories are standardized and they can just plug that into what they are doing. So that is hugely improved the comprability of data. >> How would you characterize the quality of the data coming out of the JMP today? >> As the time series of data gets longer and longer, of course it's more robust. And one of the things I think it's important to understand is how the data are analyzed. So the results, for instance, how many people are using report that they are using an improved water supply. That gets plotted on a grid and then a linear regression line is drawn. Now, when you only one or two surveys, and at the beginning there were many countries that only had one or two surveys, you got a very simple regression. But as the numbers of surveys is added and now we have countries that you know, ten surveys for instance. Then of course the accuracy of that regression line and the accuracy of the projection to the estimate gets much better. >> So, what do you think the future of the JMP is after end of the NGD goals? >> So one of the things we have struggled with is that there are flaws in the JMP. For instance this proxy for safe water is one of them. And there were also flaws in the MDGs themselves. For instance hygiene was not part of the MDGs. And in fact as a result, we don't have good ways to track progress on hygiene behaviors. The other problem is that the MDGs didn't include anything beyond the household. So there were no targets set for school water supply or health facility water supply. So, one of things that the JMP has facilitated is a global consultation to try to make proposals for what future WASH targets could look like post 2015. And that has led to a lot of interesting debate about how we can sort of correct the errors of the past. And the proposals that have come out suggest that we should have targets for households, for health centers, for schools. >> That we should put in our definitions of improved water supply, a measure of water quality, not just a proxy. That distance to source should be included, which is something that was also not part of our current monitoring. >> That'd be hard to do, wouldn't it? If you were asking households how far it'd be, and how far they're walking. Yeah. >> So that opens up the whole thing about now that we're going to do this, and another thing that we want to do is we want to look at full management of fecal material. So the current system just looks at if you have a pit latrine or if you have a flush toilet, you count as being served. We don't know if your pit latrine is ever emptied, or, if it is emptied, if the contents are safely disposed of. >> You don't have that question in the- >> No, no. Because households often can't answer that. And you don't know if when they flush the toilet you don't know where the waste goes, because in many cases households have no idea, that's one of the problems. But now we are actually going to be challenged, we are challenging ourselves, to have measure of this full management of excreta. That is going to force the JMP to find whole new sources of data. Household surveys just are not going to be enough anymore. So it's a huge challenge now to try to look for where those data might come from. How we could enhance the quality of the information that we're getting out of the household surveys. Bearing in mind that it's very difficult to make changes to these big, well established surveys. Someone like the DHS, for instance, I don't know if you know how many questions there are on a DHS survey? >> I don't actually know. >> 850. >> Yeah it's a long interview. >> It's a long interview. So you can imagine that the DHS is very reluctant to add anything. Because the DHS collects information on whether the families sleep underneath a bed net. And whether they have a radio. And if they've ever lost a child in infancy, that type of thing. And there's lots of skip sequences you don't ask all 850 questions of every single interviewee. >> Have you thought of using community surveys? I mean, the LSMS used community surveys to collect some of this kind of data. >> Certainly, one of the things I think we need to start looking at is what happens at a community level? For instance, open defecation. In many cases, what you're looking for is not whether a individual household practices open defecation, but whether an entire community is open-defecation free. And of course, household surveys can't tell you that. So if that's going to become one of our metrics in the future, we're going to have to come up with other sources of of data. >> So what do you think the WASH community needs to do moving forward, in terms of collecting better data for evaluation and planning? >> So I think what we have to do is we have to look at the history of the JMP, and remember that it wasn't so long ago that we started on a whole new data source. That that step that the people working on the JMP made when they gave up one data source, which was government reports, and moved and made this great leap of faith to move to household surveys. And they were starting at a very low base. In many countries there were very few household surveys. And, they had to be brave. They had to be resolute, and it worked. And now I feel that we're setting the reset button again. We're starting again and we're saying we're going to have to start again with some parameters are going to be measured using very unevolved sources of data. We'd like to use regulatory agency data, for instance. But sometimes it's not particularly well comparable between different agencies. So, we're just going to have to be brave all over again. We're going to have to start, and we're going to have to say to the world, look for the next few this years, this might be kind of ropy data. But you've got to bear with us because we've got a proof of concept. We can make it work and we can come up with these time series which will then give people confidence that they can rely on these data. >> So what does it cost to collect and manage the JMP data? And how much more money do you think you need for this next phase? >> The JMP gets the data for free. The current model is that the current JMP harvests the DHS and the mixed data. And, of course, bilateral donors pay for those surveys to be carried out by USIAD and by UNICEF, and then make the data sets available to anybody who wants. >> But is there a group of analysts working on this data? >> That's right. >> How many people are involved in sort of working on this data? >> There's a team of about six. Maybe three from UNICEF and three from WHO. But none of them, very few of them are full time. So for instance, when I was chief of UNICEF I might have been spending maybe 5 or 10% of my time on the JMP. I had a couple of staff members who probably each spend maybe 30% of their time. And then maybe we might have one or two people who are specifically working on the JMP. >> Can anybody get access to the raw data? >> Yes. It is all- >> It is all up on the web. >> It is all up on the web. Yeah yeah. >> So what about this next stage, what do you think it's going to cost to do this, this phase three? >> Phase three. I think it's going to be huge because we are going to be challenging ourselves to monitor far more difficult things and to monitor them better than we ever have. So, let's just take water quality, for instance. The DHS and MICS are exploring options for including water quality testing in among their enumeration activities. >> For a sample of households, or everybody in the survey? >> No, for a sample of households. You can imagine, this is going to be quite big. >> It's going to add a level of complexity to To the implementation of the surveys and then there's going to be complicated data which will have to be analyzed in terms of what the water quality tests tell us. So if we start doing that in every single country in the world, at a national level, that's huge in terms of the data analysis burden. >> Have you thought about adding spatial information so you know exactly where the households are? >> Yeah. >> Yeah >> So and there are things that are now available to us which of course 10, 15 years ago we didn't really dream of. There may be ways of collecting information using mobile phones or whatever that we haven't really thought of. And yet at the same time we have to continuously apply the same sort of standard of rigor that we've always tried to keep on the JMP. A lot of data never makes it into the JMP. If we find out about a household survey that has been implemented in the country, we will go and look at it. Is it truly nationally represented? Is it a good quality? And if it doesn't meet either of those criteria, it goes out and people are constantly frustrated that they think that certain data should be in the JMP. But it doesn't meet our standard for rigor. So we don't put it in. And I think the same is going to happen with my new data sources. Is it's going to be very difficult to balance out our desperate need for data against our desperate need for quality. >> So just looking back over this sort of second phase of the JMP and this new data, was there anything that surprised you that came out of it that you really didn't know before, think something that kind of jumped out. >> That came out of the consultations? >> Well, came out of the data from the surveys. >> From the current JMP. >> Yeah. >> Yeah, I think one of the things that has been the most interesting and the most revealing is the income, the wealth quintile analysis, the equality analysis. And JMP has starting doing this more and more. >> We did it first experimentally a few years ago and then discovered that this was something that really, as things like the human right to water and sanitation became more and more debated, these data were far more relevant. And what happened when we did wealth quintile analysis particularly for sanitation, is it showed glaringly what many of us had already suspected, which is progress on something like sanitation is highly unequal. So, for instance, when we did the wealth quintile analysis for India, we discovered that the entire bottom quintile, one-fifth of the population of India, practices open defecation. It's not that some of them do, not that most of them do, all of them do. This was really quite a surprise, and actually resulted in some difficult conversations with the government of India, because, of course, the JMP wanted to publish that information. And they were very good. They wanted to get that information out there too. They saw it as a way to get some political will around sanitation. >> Anything on the water side that surprised you? >> Pipe water. Pipe water. Even though water, in general, is even more equally distributed in terms of the progress, one thing that became clear is that poor people do not get piped water supplies. Piped water supplies are almost exclusively with the upper wealth quintiles in urban areas. And again, this is something that perhaps intuitively we knew. But then the data is showing that and show how unfairly progress on piped water supply has taken place. It's very interesting. But also the variation between countries and regions. So China has made far more progress on getting piped water supply to the poor and rural dwellers than most other countries. >> Great, well, anything else you'd like to tell us about the JMP, it's so nice of you. [LAUGH] >> [LAUGH] Nothing, except yeah, I think we're entering a whole new era of data challenges. But I do think it's very interesting that the JMP has continuously challenged the WASH sector to look at itself more carefully. What it's actually achieving and where the progress is? And I hope that, that's going to go on happening. >> Great, well, thank you so much for taking time to talk with us today. >> Okay. Thank you. >> Yeah.
