Hello and welcome back. Today, we'll be talking about observations, the art and science of observing your users. Why would you want to observe your users? There's really three reasons probably more but the three major one's are that some actions are really difficult to articulate. So, let's say that I ask you, how do you ride a bike, you just get on and ride, right. It's only through observing you that I can actually tell the minor ways that you balance or the minor ways that you actually approach the activity. Or if I ask you, how do you answer your email? You may have somewhat of an answer. Maybe you say, well, I look at it once a day and I answer it all in one single pile but by observing you. I may actually find that you do it somewhat differently then you actually return to it throughout the day. Or certain emails when they come in if they ping your phone are more problematic than others. So sometimes these descriptions of behavior that we give, so they may say, I check my email once a day, actually conflict with the behavior that we take when we go through our daily lives. And This is not because a participant is lying necessarily, but just because we structure things in our head the way we plan our day, so different from the way it might actually go. THe other thing is that participants may not know which details of their activities are actually most relevant to you. So maybe the reason why I didn't say that I actually check my email throughout the day is, because I wasn't sure if that particular aspect of my interaction was actually something you were asking or actually was relevant to you. Maybe I didn't think that checking my email on my phone, for example, counted as checking my email. Or maybe I didn't think that if someone approaches me in the hallway and say, by the way, I emailed you that thing. Do you mind taking a look at it? And I look at it that that still counts as checking email outside of my one hour of checking email per day. In fact, by observing somebody as they go about they day, or as they go about a particular activity, can yield insight that's different from just asking them about that activity. So let's take about a quick example of a participant observations study. So as I've talked about in previous reviews I'm really interested in understanding the potential role of technology, and helping people recover from addiction in alcoholism. And so as part of that process I really wanted to understand how technology was currently used, and also how people used social support in their recovery. So part of that process was attending six months of participatory observation. I went to a hundred 32 open meetings through all step meetings and open meetings actually open to anybody not just people who are currently in recovery. And so I was able to come in and kind of observe and make friends with people and ask them questions about how they currently practice their recovery. But also really just kind of sit in the room and see what happens. I also attended 18 organizational service meetings. So all these meetings that happen every day all around the world they need to be organized somehow. So frequently it's kind of this volunteer system, grassroots system of service level meetings where common issues are considered for a particular area, region or state and decisions are made. And also documented artifacts and current technology used. So when I say documented artifact, I mean doing things like taking photographs of a particular technology that maybe used by participants. Whether that's an app ro something like a white that you see on the photograph, on the heavily photograph at the bottom. So through that process there are a few things that I learned. So I learned how meetings and event logistics are organized at the group, area, and regional levels. And by actually watching how the sausage is made so to speak. Not just asking people well how is a new event organized where I might get kind of a sanitized or a whitewashed way of describing it. I looked at how disruptions and norm violations are handled at the group level. I was quite fascinated. So sometimes you have somebody come in who is early in recovery or, perhaps, in fact, still high or still drunk. They will be speaking at a meeting and really being disruptive rather than actually contributing to the discussion. And what I saw was kind of the old timers of the meeting, people who come regularly, and have a lot of time clean will exchange looks, one of them will nod. Rhey'll take that person out of the room so they're not disrupting that meeting, but they'll talk to them for as long as they want to talk. And that was really powerful to see that way of approaching norms and violations, which wasn't punitive. But rather try to be constructive for all the people involved. And lastly, I got to observe how the organization as a whole creates and approves content. At the time that I was doing this observation, the narcotics anonymous program. The NA program, was in the process of approving a new book, and this is actually kind of approved, and written at the grass roots level. So drafts were passed down from the world level to regions, areas, and even groups. People would read and comment and pass those comments back up to the world level. So it was really powerful to kind of see that process as it was happening rather than to just ask about this process in retrospect. So let's talk about how to observe participants. So four steps here and there's a few of these videos have a very similar structure so some of this may sound familiar. So, step one is to prepare your note structure. Step two is to schedule and arrange access. Step three is actually conducting the observation, and then four is analysis. So the key to preparing your own structures understanding, what it is that you hope to capture in your notes? Are there any other types of evidence that you want to capture? So maybe you want to take a video of something that would clearly be inappropriate in any meeting or a meeting. But photographing something like a white board or other artifacts are getting screen caps of people's apps that they use for recovery, was reasonable forms of other forms of evidence that I could gather. What granularity is actually relevant to? Do you want to know each time somebody speaks up at a meeting? So that you know how many people actually participate, or if you just want to sit and kind of sit back and look at the larger themes that happened, we kind of have more structure way of taking notes. And lastly, will you actually try to quantify these notes in any way? So for example if I did know every time somebody at a meeting spoke, I could actually have an idea of how many people on average speak at a meeting from the 132 meetings I attended. These images actually show the three extremes, or maybe not extremes, but a spectrum of taking notes. The first might just be a notebook and a pen so completely unstructured. You're coming in, you don't know what you're going to find yet, but you're looking for any interesting insight that might come from that process. And that's really how I did the AA and the observations. I really just had a notebook that I was taking notes in after the meeting as field notes. Because I didn't quite feel that it was appropriate for me to put the whole meeting under a microscope, and to count how many times different people spoke. The one in the middle shows kind of a medium structure. So perhaps it might be something that's more digital and you might be actually recording specific time stamps and something you state happening. And then noting things like particular trouble that our participant had with the system or what they did in a particular moment in time. So it's kind of a lot more structured because you get time stamps and you're doing it in the process in a moment as you are observing. And then the last one here is also from the study that I run observing how kids use the particular technology. And one of the things we were interested in this study is, how often do the kids look up at the screen is providing them content, versus play with the physical toys that kind of, the things that activated the content on the screen. And the argument we were making is if they don't attend to the content then they can't possibly learn. So if they're looking away or are they looking down they're not actually attending to the content we're trying to teach them. And so what we had is a very structured sheet, where for every 30 second segment we kind of specified what was happening mostly in that segment. So we use the downwards arrow to say the kid was looking down and upwards arrow to say the kid was looking at the screen and an O to say that the kids attention was somewhere else, and not even on our system at all. And so later on we could actually quantify that. We could for each participate say okay, how many downwards arrows did they have, how many upwards arrows did they have? That would actually allow us to for example compare systems. The spectrum goes from very little structure for how you take notes to quite a lot of structure where every 30 second segment of your time is structured. And all these are valid. It's just a matter of what is valuable to you in the study. And what do you kind of anticipate seeing from the observation process? The second step is scheduling and arranging access. So we've talked about this before and all the same issues as interviews apply. Consider the setting, consider the presentation. How you present yourself. Consider how you'll be viewed and whether your presence will bias a participant's responses or how they act. Follow up to confirm the schedule and of course be safe. So do the alteration somewhere either with somebody els, or somewhere where you're in a public setting and you can be safe. The third step is actually conducting the observations. So I find it as really important to provide clear Instructions for the participant. If necessary, there are cases where you may be observing people in a public setting where they actually don't know they are being observed. So maybe your doing something like counting how people walking in and out of a supermarket. You're not going to ask consent from every person who's walking in and out but maybe you're doing some basic counts. But most the time when your observing participants, they know about it. They give you their permission to observe them. And sometimes you even provide them specific tasks, like you may asked, okay, I just want to see how you would debug a particular piece of code. Can I sit back and watch you while you do this? And they'll go ahead and do it for you. But I think those instructions are very important. Particularly, also, tell the participant if you want them to act as if you're not there, or if you want them to kind of be commenting on the activity as they're doing it. You're going to get a very different result, depending on the two. You want to go ahead and set up any necessary equipment, so perhaps you're audio or video recording the session. If you have notes, you want to have all your notes sheets. Printed so that you can take notes either in a structured way or in a digital way. Then as the study progresses you take notes using your pre-prepared structure, or lack of pre-prepared structure if you're just taking general notes. After the observation, this is really important, review your notes immediately afterwards. So I like to schedule at least half an hour in my schedule immediately after user observations. Just to sit down, write field notes, describe what happened, anything that may have popped up that I didn't anticipate or that I didn't note in my other notes, I would write down there. Because all this stuff is very easily forgotten. If you wait until you do three observations in a row, you're not going to remember what happened in that second observation, or some piece of insight that you might have had as you were observing. So, that process of field noting is very important. Now, analysis is covered in a later lecture, and because here we did talk about observations that could be very kind of unstructured, and qualitative, and observations that could be quantified that are kind of very specific. The two types of analysis will be covered in the qualitative analysis and quantitative analysis lecture. And just to give you an example, so the on the left it's analysis that kind of a more open interview type study, where you just observing and your noting general things that might be of interest. And the one on the right is a very structured analysis where we actually looked at how specific children played, and whether they were social with each other using one of our four prototypes that we developed. And so this actually generated very specific numbers. Level of social play that we were then able to compare across children and across conditions. So lots of different ways of analyzing observations. You know the other thing I want to talk about is this idea of digital observations. So it used to be that you have to be in a particular place in order to observe people. But now there's lots of activity that happens online and you can actually conduct an observation study remotely from participants. So the image here shows AA or NA, I'm not sure exactly, video meeting that is actually happening on an online webpage. So this is, of course, very heavily anonymized here as we're showing in this image. But we were able to attend those and observe those, and we didn't actually have to leave our office in order to do that. So we could follow the same process of taking notes and seeing what was happening. But it was something that was actually happening in a digital environment. So you can also do something similar by having the participant screen share. So maybe you do want to observe how somebody goes about their day at work. But you can't be there to shadow them if a lot of work, and their work is digital and they agree, maybe they can just share their screen throughout their day. And other kind of context where I think this is really powerful is actually multiplayer video games. So going to something like World of Warcraft, and observing how people behave can be as interesting, and as powerful as going to a playground, and observing how people behave there. So what are the strengths of the observation method? One of the strength as interviews, it can work with no knowledge, or prior hypothesis before starting, especially if you're taking kind of unstructured notes. Again, it also provides you this rich, nuanced data about the participant's actual process. Not just what they say it is. It reveals the specific, concrete, and unedited actions. The participant doesn't really have time or the opportunity to necessarily kind of abstract, or represent or focus about how they're being viewed, or whitewash their activity. They just do it and you observe it. And so in a lot of ways, it can be kind of provide you more honest perspective on the actual steps taken in the workplace or at a particular leisure activity. Now of course it has weaknesses as well. So usually you do need an access to and the cooperation of the participants in this specific context. Then maybe more difficult to get. I think sometimes people are more comfortable with being interviewed, because they do have that ability to kind of edit a little bit of what they're saying. Or how they're presenting themselves. It's a pretty vulnerable experience to let somebody just come in and observe you. It does, also, rely on the situation of interest being frequent enough to observe. So, for example, I was interested in the AAA and NA meetings how particular moments of disruption were handled. But in those 132 meetings, those moments of disruption probably only happened about 10 or maybe 12 times, out of those entire set of meetings. So if I had just gone on what to one or two meetings, I would not actually have been able to observed that and that interaction. So it has to be in frequent enough to observe given the number of durations you're taking taking and lastly, just by looking what somebody is doing. It may be really hard to understand not only what they're doing, what exactly is the purpose of their activity, but also why it is happening. And so this is kind of where interviews may be able to step in, and help out but if you're just looking at what the person is doing it may be really difficult to understand why. So if you're interested in getting more information about observation one of my favorite books. And it doesn't just talk about observation it talks about kind of Analyzing Social Settings in general, but it has some very solid chapters and observation. My favorite book is this book by Lofland and others on analysing social settings. So take a look at it if you get the chance. That's it and hope to see you next time.
