[SOUND]. So in the previous video we just saw what happens when a computer savvy user whose not a security specialist, encounters an SSL warning in their browser. Now we're going to to focus more on that as an academic intellectual case study, and to do that we are going to read this paper, Crying Wolf: an Empirical Study of SSL Warning Effectiveness. You'll find a link to the PDF of this paper in the materials for this week on Coursera. And you should read the paper before going through the lecture, because then you'll have a better background for it. Now, just as a reminder, these SSL warnings are messages that come up in your browser when it encounters a problem with the security certificate for a secure website. So that could be that the certificate is expired, that it's signed by the wrong domain, or that the browser doesn't recognize the signing authority for the certificate. When a user encounters that, it's possible that there's just a meaningless error, like if the clock is set incorrectly on their computer, making security certificates look expired when they're not. But they also can be indicators of man in the middle attacks, or other security problems that will prevent a user from securely accessing a website. The browser's goal is to discourage users from going forward, especially to secure sites, when it can't validate the security certificate. This paper looked at how effective those warnings are, and how we might design more effective warnings. After this lecture, the next one you will see is an interview with Lorrie Cranor, the last author listed on this paper. And she's going to talk a little more about the process they went through in designing the alternative SSL Warnings that we're going to see discussed in this paper. So let's start by taking a look at the three SSL warnings that were studied in the paper. The first one is this, which is the security warning that comes up when an invalid certificate is shown in Firefox 2. Again, if you print out the PDF of the paper or download it, you can read these messages more in depth, and you should take a look at them before you go through this lecture. The second is very similar to the message that we saw in the previous video showing that a secure connection failed. This is the message from Firefox 3, and finally they looked at this SSL warning that comes up in Internet Explorer. So these were the three major browsers in use at the time that the paper was written. And the author started out just by trying to see how people thought about these errors and how they responded to them. Their survey provided some interesting results. First, they looked at three different types of errors that could cause an invalid security certificate. And if we look on the horizontal, the X axis here, we can see that those are having an expired certificate, having an unknown certificate authority, or having a domain mismatch. Where the certificate is for a different domain than the site the user's trying to access. They showed these errors to people along with how they would appear in each of the three different browsers that we just saw. And they asked them whether or not they would continue on to the site they were accessing. This table shows the percentage of respondents who said they would continue, that they would not continue or maybe they would. You can see for an expired certificate many more people said they would continue than in the other two cases. And you can see in all of these cases the people seeing the Firefox 2 error proceeded more, though in the last case of having a domain mismatch, that's a really small percentage of people. Still, we see 30, 40% of people say they would continue even if they see these SSL errors. Now depending on the site they're going to, that might be okay. If you're going to a site where you're just browsing, there's no secure or interesting personal information there, it may not matter that you don't have a secure connection. That's something that's addressed later in this study that we'll talk about. But another interesting result that came out of this survey were comments from people about why they would continue, that includes things like I use a Mac, so nothing bad would happen. This user trusts their system, so they don't think they have anything to worry about with the SSL certificate expired. While it is the case that Macs have fewer viruses that they run into than a Windows machine does, viruses aren't really the risk with an invalid security certificate. So, this person's confidence in their operating system, actually doesn't help them in the case of the security error that they're running into. Another user said, since I use FreeBSD rather than Windows, there's not much risk. Again there is a similar understanding of the general security risks that come with a certain operating system, that don't actually apply to the security case in question. And finally one use says, On my Linux box, nothing significantly bad would happen. So these users are all focused on their computers, and the risks their computers face. But, they don't quite understand that they're actually facing a risk of transmitting personal information that could be intercepted, and used in ways that they don't like. So the study, after doing that survey, wanted to see if there were possibly ways they could create a better SSL warning. So they started off with this experimental warning page. And they say, the secure connection failed, and then they ask people what type of website were you trying to reach? A bank or another financial institution, an online store or an e-commerce website, other, and I don't know. Those first two are places where you might be transmitting sensitive information like your banking information, log in information, or things like credit card or other personal information if you're shopping. If you're not going to a site where you'd send some of that secure financial information, it may be the case that you don't actually need to worry about the certificate being expired. So, they had a second page that came up. If users were not going to one of those sensitive websites, they would go ahead and pass them on to the website. So, if I said I was going to another website and clicked continue, it would say, okay, and send me along to my destination. If I had indicated I was going to one of those first two more sensitive websites, then I would get this very scary warning that comes up all in red. That says there's a high risk of security compromise, it says that we're having our connection intercepted by another party, or someone is impersonating the site that we're trying to go to. The attacker's trying to steal information, and we have this option to figure out, do we just get out of there, do we have an option to see why the site was blocked? And, down in the corner you see there's an option to ignore the warning, which is small, and so it's not something that someone's instinctively going to click on. They're more likely to look at the buttons. Users participated in an in-lab experiment, where they were shown a series of different errors and asked whether or not they would continue on to the website they were trying to access. Those errors included the three that we saw for Firefox 2, 3, and for Internet Explorer, along with two experimental conditions. One of them is a multi-page system that first shows this error, and then if the person's trying to access a website with sensitive information, it takes the user to this scary high warning page. And the second experimental condition only shows this high risk security page. So we have two experimental conditions, and three existing SSL certificate warnings. And users were sent to either The Carnegie Mellon Library website, or to a banking website. These are the results that the researchers found. On the horizontal x axis, we can see all five experimental conditions. The first three are the SS Cell Certificate warnings from Firefox 2, Firefox 3 and Internet Explorer. Then we see the Single-page warning which is that red box that looks very scary. And finally the Multi-page warning that first asks users which kind of site they're trying to go to and then, if they're going to a more secure or sensitive website, takes them to the red warning page. On the vertical axis we see the percentage of users who said they would ignore the warning and go on to the website. Finally, there's two bars shown for each of these different browser warnings. The blue one shows what percentage of users would go on to the website if they were accessing a banking website, and the green bar shows the percentage of users who would continue on to a library website, which is less sensitive. What we can see is that the Single-page warning, which is just that large red error box, had significantly fewer people in the banking website who said they would actually go onto the site. So that's good, you want people who are potentially going to transmit sensitive information to be less likely to do that when there's an SSL error. However, for the library website, the Firefox 3 error kept the most people off the site. The Single-page allowed, the second largest number of people to go through. But you can see for the library website, overall, a lot of people were willing to ignore the SSL warning, and go on to the website. It tended to be lower for banking, though looking at the Firefox 2 and Internet Explorer warnings, still 90% of people were willing to ignore that warning, and go on to the banking website. The different SSL warning interfaces also had an impact on users who logged in, bases on whether or not they read or didn't read the error message, and based on whether or not they understood it. Now, an important thing to note here, is that we're looking at the percentage of users who logged in, and so, none of these numbers, you should expect to add up to a 100%. So in the Firefox 2 condition the first row, we can see that among people who read the error message, 20% of them logged in. Among people who didn't read the error message, 70% of them logged in. Moving on, when people have read and understood the error message, 35% of them logged in. But when people read but did not understand the error message, 55% of them logged in. And we actually see that there's a big variation in all of these numbers across the different interfaces. So, for example, when we look at the single message, that was the experimental condition that just shows the red error box, 20% of people who read it logged in, 25% of people who didn't read it logged in. Among people who understood it, 20% logged in, where 25% of people who didn't understand it, logged in. The differences between these numbers are smaller, than the differences between the numbers for Firefox 2. Which suggests that it could be the interface that's warning people off, regardless of whether they read it, didn't read it, understand it or don't understand it. Whereas in the Firefox 2 example, reading it makes a significant difference as does understanding it. What this table is illustrating for us, is that the interfaces can make a difference as to whether or not people log in, regardless of whether they read or don't read. And for some of the interfaces it's very important that they read, where in other interfaces it doesn't seem to make much of a difference. The same goes for understanding, when people understand, they may log in much less than when they don't understand, but in some interfaces that's not the case. Thus the interface makes a big difference in users' behavior. So what lessons can we take from this study? Well it's not necessarily that any one of these SSL warnings is superior to the others. Rather, this study gives us the insight that different interfaces can have major impacts on the security behavior of users. We saw in their results that, depending on which SSL warning they saw, some people were more or less likely to continue on to the site that they were trying to access. We also get some lessons that depending on what they're shown, users may be more or less likely to read the message in the first place, and depending on how its presented, they may or may not understand it. We also saw very big differences in the rate of logging in among people who read and didn't read error messages, and people who understood and did not understand them. And that was related to the interface they saw, it also gives us some lessons about how to design these things. We want to think about what do we want users to do? In this particular case study, when there is an Invalid Security Certificate, if a user's going to a site that requires sensitive information, like a banking site, we want them not to continue. That's the best security decision. So, that's what we want users to do, and that leads us to the question, what do they, the users, need to understand, in order to do that. So if we want the users to stop, if they're going to a site that requests sensitive information, when we can't validate the authenticity of that site, what do we need to tell users so they'll make that decision? These different interfaces don't necessarily convey information in a way that users will understand it. Again, we saw varying levels of understanding, and in our previous video where we saw our computer savvy non security person accessing one of these errors, we could tell that he wasn't exactly sure what was going on. That was echoed in this paper in the survey questions where a lot of users were giving answers related to their system, rather than to the communication of their information, which is actually what's at risk. So we need to think about, how do we convey the risks and the situation to users, because that will help them make the right security decision. And then finally, we think about, how can we make it more natural for users to do the right thing? We saw that in this study with the scary red page error. They had moved the option to ignore the warning and continue to the page to a less-obvious place. So users wouldn't just click on a button to get past the error, they'd really have to read and find that link in order to proceed. These are the lessons that guide a lot of design decisions and especially security decisions. We think about what the users should do, how do we make it clear and natural for them to make the right decision. In the next video we're going to talk to Lorrie Cranor, who's one of the authors of this paper, and get her take on some of the lessons and implications of this work.
