My name's Adriaan. I'm a PhD student at Imperial College in London. This week, I'll be guiding you through some coding tutorials. They'll be on this week's topic which is saving and loading models. Let's get straight into it. In the last video, Kevin showed you some of the ways in which you can save a TensorFlow model during training and reload it back for later use. In this tutorial, you'll be doing this yourself in your own notebook. Let's fire up this notebook and get straight into it. Once we fired up this notebook, let's go and import TensorFlow. It should give us Version 2.0.0, which it does. In this session today, you'll be using the CIFAR-10 dataset. It's quite a standard machine learning image classification dataset. It's got 60,000 color images in total with one of 10 labels each. So you can see the 10 labels here as well as a link with some more information. Since it's so standard, you can actually import it directly from Keras. So I've provided the code for you to do that right now. I've actually given you a slightly smaller subset, so just the first 10,000 training images and the first 1,000 test images. This is just a speed training and evaluation up a little bit. So in your own time, you could do it with a full dataset if you like. The idea will be the same, but this just speed things up a little bit. So let's go ahead and run this cell to actually import that dataset. Now the first time you do this, you might get a print statement that says it's downloading, but it's already on my machine, so you won't get it the second time. What I've also done is I've provided some code for you which plots the first 10 CIFAR-10 images. So this is just to get a feel for that datasets. Let's actually run this. You might have run the cell twice to encourage the notebook to actually give you the output, but now you see what we're working with. They're fairly small color images in one of 10 categories; a truck, a frog, a horse, a ship. You get the idea and we're trying to build a neural network today which classifies these 10 images. What I've also provided for you are two useful functions. So the first is called get_test_accuracy. What this does is it takes a model and some testing data and it just spits out the classification accuracy, and also I've got a function called get_new_model and you'll see why this is useful in a second, but what it does is it creates a new instance of a simple convolutional neural network. You can see the layers here, and I've also imported. You see it's got dense, flatten, some convolution and some max-pooling layers. It also compiles it, but it doesn't train it. So it just creates a new initialized instance of a sequential convolutional neural networks. Let's go ahead and define that. Let's use this function right away to actually create a new model. Let's write "models=get_new_model" like this. Let's actually print model.summary. So let's see what model we're working with here. You can see the type of net this is. So it's a convolutional neural network. It's got two convolutional layers; a MaxPooling layer, and then flattens everything, and then two dense layers where the last layer has got 10 outputs, and it's a softmax. This is how you are actually classifying these things. Let's also get the test accuracy of these models. So if I write get_test_accuracy of model x_test and y_test like this. Well it's untrained. We've got 10 possible classification. You'd expect it to get around 10 percent accuracy as good as random guessing. You see it's actually the case. At 10., you're just a little bit higher, but yeah, roughly 10 percent accuracy. By the way, the number you get for the test of accuracy will be slightly different. So I get 10.2 percent, but you might get a slightly different value due to a different initialization of this network but it should be around 10 percent. Now let's go ahead and train this model. Now as Kevin mentioned, one of the ways in which you can save TensorFlow models is via model_checkpoints, which essentially it's snapshots of the model taken during training. Let's go ahead and create one of these things. First thing to import is this model_checkpoints or from tensorflow.keras.callbacks import ModelCheckpoint, let's go ahead and do that, and let's go ahead and create one of these checkpoints. So I'll write first checkpoint_ path. So this is going to be model_ checkpoints and then checkpoints. So what this does is it creates a directory called model_checkpoints and then a file called checkpoint inside it. Let's go ahead and create the actual checkpoint now. Checkpoint is model. Checkpoint is the thing I just imported, I'll get file_path, it's going to be checkpoint_path. That should be a comma, checkpoint_ path frequencies. I want to save every epoch. So at the end of every epoch during training, I'm going to save this thing. Let's save just the weights. I'll write save_weights_only is true. So I don't save the model architecture. Let's put a verbose equals one so we get some print statements. What I also like to mention is that, well this model_checkpoints/checkpoint is the same fall every time. So at the end of every epoch, it's going to overwrite the same filename. Let's go ahead and actually use this thing. The way we use it is we call this checkpoint as a callback in our model.fit. So model.fit, I'm sure you're familiar with now, x is going to be x=x_train like this, y is y=y_train. Let's say epochs is three. So we've got to train for three epochs, and then let's input this checkpoint as one of the callbacks. All right, checkpoint here. Let's go ahead and see what happens. So we're going to run this model.fit. I haven't forgot to actually run the cell and let's go and run this thing now. So you see it's training. Well, it's taking a little while, so what I might do is I want fast-forward and meet you guys at the opposite end of this training session. Okay, my model's now finished training, so let's see what the print statements were. So we've gotten these normal epoch statements, the training statements that we're used to seeing. I also have got this new statement which is saving model to model_checkpoints/checkpoints. So what you see at the end of every epoch, it saved the model in the directory model_checkpoints to a file called checkpoint and does this to every epoch. So at the end of every epoch, we overwrite the same checkpoint file. Let's have a look at what files were actually created here. So what I can do is if I use an exclamation mark, I can run UNIX commands from a notebook. So if I do ls -lh, what this does is it lists the directory and this -lh means that it gives a little more information about the files and also in human-readable format. So I'll do it, and I'll do that with model checkpoints. What this does is it lists the contents of the directory model_checkpoints. Let's go ahead and run this and see what's being created. So you see three files here. We have checkpoint, checkpoint.data-00000-of-00001, and checkpoint.index, and you'll learn later what these files are actually are, but for the moment, all you really need to know is that this checkpoint.data which is by far the biggest file, 174 kilobytes for me, this is where the actual weights are stored. The other things are a little bit of metadata. Now let's go ahead and also get the test accuracy for these train models. I'll do get_test_accuracy of model again, x_test and y_tests. Let's see what it gives me. So it gives me 47 percent. So just less than half. We only train it for a very small time. By the way this value, just like the value you had before, it'll probably differ for you. So it'll probably be close but obviously since you had some slightly different initializations, the value's going to differ a little bit for you. I mentioned that this checkpoints saves the weights, so if we create a new model, let's see if we can load these weights back onto it. So what I'll do is I'll create a new model like this, so I'll do models get_new_model, and then let's get the test accuracy for these things also. Let's get_test_accuracy of model x_test and y_test. Now this is going to be newly initialized, so you expect this to have roughly 10 percent essentially guessing accuracy at the moment. So yeah, a little bit less than 10 percent, but around 10 percent. Again this value might differ for you, but it should be around 10 percent. How do we load these weights back on. Well we can use the inbuilt model.load_weights, this thing here. So I'll do model.load_weights. All right, checkpoint_path, this is the thing I defined earlier. Let's get the test accuracy. Let me scroll down a bit. Let me just get the test accuracy for this new model. This should be the same value we had before. So for me it should be 47.7 percent. For you, it'll differ a little bit but it should be the same number as the one up here. Of course, the fact that this model has the same testing accuracy is the old one isn't really approved that has the same weights, but if you want to verify this for yourself, you can go ahead and check that. They really do have the same weights. You really are loading on the same weights. I've given you just one more piece of code which is another Unix command. What this does is it just clears the model_checkpoints directory. The reason you might want to do this is that if you want to continue this and mess around this code yourself, it might be good to delete the directory with the checkpoints that we just created together, so that if you create your own checkpoints you don't get mixed up with these ones essentially. So if you want to run that cell, all it does, it cleans up your workspace a little bit so you can tinker around these stuff yourself. So you've just seen how to use model_checkpoint which is just one of the ways TensorFlow allows you to save a model. You can imagine this is especially useful during training. You can save models and continue training later or just ensure you haven't lost work if things cut out. It's your turn to explore this functionality a little bit more. Try for example, to design a script that loads a checkpoint and continues training, or can you design and save different model architectures than the one I gave you to try and beat my classifier. In the next few coding tutorials, you'll learn about ways to customize checkpoints and other ways to save models. See you then.
