Alright, now let's get started with question number six here we will actually start to predict each one of our different labels, as well as start storing the probabilities. And that's because as we spoke about during lecture, one, we want to do certain scoring such as the ROC curve. We're going to need the probabilities. Whereas when we're trying to do the accuracy or the precision or the recall only the actual predicted labels. So we're going to start off with creating an empty list, which is just going to be our different predictions for each one of our different models. And then our list of probabilities, which is just going to be the probabilities for each one of our different classes for each one of our different models. So for each model, there'll be six different probabilities. And we'll see how that looks in just a second. But then going to create this list of coefficient labels and coefficient models. Which is going to be similar to what we've seen before so that when we zip these two together, we'll be able to access both a labeling as well as the actual model. And we're going to access those as lad and mod. So the first thing that we're going to do is take our empty list to wide pread and we're going to append a panda series. That's just going to be the prediction for each value in our x test. So, x test is our holdout set. We can call model dot predict like we've done before with other, with our linear regression model. It works the same, this time it will output a label either 0, 1, 2, 3, 4 or 5. And then we're going to name that series. Every series is similar to a column within a panda's data frame, we will name that according to the label, whether that's lr, l1 or l2. So that we'll have a prediction for each one of these different models that we created. The next thing that we're going to do is create a multi level index again. Using the levels are going to be at the top level is going to be which model we're using. So here we're using LR, L1 or L2. And then for each one of those, we're going to have an output of probabilities. So we need a different column for each one of our different labels. So it'll be the probability for labels 0 for label 1 for label 2, etc. We're then going to call model dot predict problem here. And that will get us that probabilities. And to show you what that looks like. I'm going to pull out one of our models which is LR and call that predicts proper and we're going to do that on x test as we have here. And if we look at the shape, we'll see that we have 3090 rows and 6 columns. So x test was going to be 3090 in length and then for each one of those different rows, we predict the probability of each one of the closets. And this looks a little bit cleaner. And if we look at a data frame rather array, so let's take that original array and look at what this actually looks like. We have label 0 is very unlikely, whereas label 3 here is very likely because it's around 0.99. And we'll see how that factors into actually coming up with the prediction as well in just a second. So we have that, again, it's outputting a data frame for each one of our different models. So we're going to have that again, multi level index where we have both the model as well as what we see here. 0, 1, 2, 3, 4, 5, the predicted probability for each one of the different classes. So we're going to pass that coeff label that we created as our columns for our data frame, and then we're just going to concatenate. Here it's going to be the prediction for each one of our models, and here it's going to be all the different probabilities for each one of our models. So now, let's look at what y pred.head now looks like. It predicted 3, 5, 3, 1, 0 for lr. It actually predicted the same values for lr, l1 and l2. We can see which ones were different by doing something like y pred such that y pred.l R is not equal to y.pred.l1. And here we'll see some examples where it came up with different predictions, but there's not that many here. We can then also look at our probability data frame that we created. And we see that it has for each one of these different classes. Lr, l 1, sorry L 1 and L 2. We have the different probabilities for each one of the classes. Here we see that by far, class 3 is the highest for lr as well as for l1 as well as for l2. And that makes sense given that if we look back here, that it predicted class 3 for each one of those different models. So the way that it predicts is just which one has the highest probability. So now we have our probabilities. And we have our predicted labels. So we can start coming up with the scores that we'd want in order to actually see how well we performed. So we have here the, we're going to import new metrics, the precision, recall, fscore, support is going to output each one of those separate scores in one function. It'll output the precision The recall the F score and then the support is just going to be essentially the same as the value counts for our labels. So we'll see this in just a second and we're importing that as score, but then also going to import the confusion matrix so we can look at our different confusion matrices. Our accuracy score are our OC AUC score. Which is that area under the curve for the ROC curve, and again for that we're going to need the probabilities. Which is why we outputted all the different probabilities. And then we'll see how this label binaries comes into play. It's just the way that the ROC accepts certain values. So the first thing we're going to do, I'm going to import all those libraries by running that cell. And then I'm going to show you what the score comes out as. So let's say we're working with y tests, that's our actual values and we want to see that score. Again, it's going to have to use the actual predictive labels not the probabilities. Since we're using precisions or trying to find precision we call in f score here. So I'm going to say y test and then I'm going to say y pred and we want to specifically here for LR. And we see all the different scores, we have an array we can look at the shape. It's a tuple excuse me. We can see here that we have 1, 2, 3, 4 arrays. The first array is going to be our precision for each one of our different labels. So remember it's one versus the rest. So, for those that predicted the first class, it was able to get all those were correct. That's our precision. We can and then the same for class 2 and class 3 each one separately. One is going to be the recall for our scores. 2 is going to be the fscore. And then the last one, which is support, which we're not going to use. It's just going to give us the breakdown, again, similar to value counts of each one of our different labels. Now, when we say average equals weighted, which is what we do here. We see that instead of giving a list of all the scores, it's actually going to average out those scores according to that breakdown. So we no longer need support. It's going to weight the scores. So it's going to come up with a weighted average. For each one of the scores so we have a single score for precision, recall, f score and then support no longer is necessary. So that's going to be what we use here with score. Our accuracy score, we just need to pass in y test, as well as our prediction for each one of the labels being the different models. And then for the ROC AUC score, we see here that we pass a label binarize. That's because when you want to look at each one of the different scores and each of the probabilities, the input just has to be similar to what we get with a one hot encoding. So, if I call label binarize. We see here that instead of saying what was our original values, which let's plot y test.head, we see that it's 3, 5, 3. We see this is 0, 1, 2, 3. And we have that one at the third label. So that's what it's replacing. Then here we see it's a 5. And that's because it's 0, 1, 2, 3, 4, 5, and that's the last label. So it's just one hot encoding what we have here, that's what the label binarize will do for us. And then we're going to pass in the probabilities, which is the same shape. When I call y prob, recall that is data frame, I believe with 18 different columns. Write six for each of the models of our three models. When I call a certain one of those models, we get it down to just the six columns so that these will match up with the shape of our label binaries. And then we are able to calculate for different thresholds at ROC, AUC score and see that area under the curve. We're then also going to save our confusion matrix. Here, we're creating a dictionary. So we just get for each one of our labels what the confusion matrix is, and we'll see this in the next cell. And then for our metrics our list, we're just going to create a series that has our precision or recall, our fscore, our accuracy, our area under the curve. For each one of our different labels, we can coordinate that together and we can see our metrics for each one. Which are fairly high for each one of these different metrics, save our precision recall F score and accuracies and the area under the curve for each one of these different values. The final thing that we're going to do here is we're going to plot out our confusion matrix for each one of our different labels. This is going to be similar to before, we have four different subplots that we want here. So I'm going to run this and let it run for a second. We create four subplots, but really there's only three models. So we're saying 2 by 2 we flatten that out as we had before. So our x list is just each one of those bounding boxes. We call flatten so we can run a clean for a loop through it. We're setting the size of that full image to 12 by 10. We're shutting off the access for that last one because we don't want to actually plot that last one, x plus negative 1, that's going to be the last one. And then we're saying for x and lad in each one of our different bounding boxes up until the last one and our coefficient labels. We want to pull out our confusion matrix that we saved before in a dictionary. We want to pull out for each one of the specific models. And we're going to create a heat map. And that heat map will give us the densest values where each one of our different predictions came from. So hopefully for 0 and 01, and 1, all those darkest values are across that diagonal because that means we predicted correctly. And we're also annotating so we see the actual values, and we're using the same app colours that we brought in earlier. So if we look here, what we actually have. Across the rows are going to be the actual classes, whereas across the different columns are going to be what was predicted. So we see that one is going to be the actual class, and it predicted two. And those seem to be confused with one another when we use lr same with l1 and same with l2 and those seem to be the most confused classes. If you look back to I believe it was. Let's scroll all the way to the top so we can see what that reference is to. We passed in, we use our labelling quote or here. I'm going to reinitiate this We have to reinitiate each one of these steps. And coming back to the bottom we see that we're using le, we can look at our le dot classes. And we can see which ones are getting confused. Lang is 0, sitting as 1, standing is 2. So it seems that sitting and standing are getting confused with one another. Which may be is why any of you have Apple watches that remind you to stand every once in a while and you're like wait, I just stood. This is similar to the algorithm that's being used in order to predict sit or stand. That closes out our lab here on logistic regression, as well as error metrics. And with that we look forward to seeing you back in lecture. Well, where we will get started with K nearest neighbors. All right, I'll see you there.