In this last session of our course, we'll go over the interpretation of simulation results and the subjects of running long versus short simulations and comparing different decision options using simulation toolkit. In the previous session of week 4, we looked at the continuous probability distributions. And the example, a company designing a new apartment building, we will need to use a simulation to evaluate different options. We have set up and run the simulation to evaluate a particular decision, and now it is time to analyze the results. So in the previous sessions, we have simulated ten pairs of demand values for regular and luxury apartments and calculated ten corresponding values of the profit for the Stargrove company. The simulation and its results are stored in the Excel file Stargrove. Let's go to this file and collect some summary statistics information that will help us to analyze the results of the simulation. So, here's Excel file Stargrove. In cells B19 through B28, we have ten random values that we use to generate the values of the demand for regular apartments. These ten random values are instances of the normal random variable with mean of 90 and standard division of 25. In the cells C19 through C28, we have placed ten random values used to calculate the demand for luxury apartments. These random values are instances of the normal random variable with mean of 10 and a standard deviation of 3. Cells H19 through H28 contain ten corresponding profit values. Let's generate the so-called summary statistics for each of those three sets of values. First, let's look at the summary statistics for the values in the cells B19 through C28. Let's go to DATA tab, select Data Analysis, check Descriptive Statistics, and click OK. In the appearing dialogue, we select, B18 through C28 as input range. Check labels in the first row. Select B30 as the output range. Request summary statistics and confidence level for mean at 95%. And click OK. The requested summary statistics for the random values in cells B19 through C28 appears in the new table. Let's do the same for the values in cells H19 through H28. So we go back to DATA > Data Analysis > Descriptive Statistics. We'll put the new input range age 18 through age 28 Check that there are labels in the first row and select G30 as the output range and click OK. Now let's go back to our slides and analyze these descriptive statistics values. Okay, here's the descriptive statistics for the ten random values taken from a normal distribution with mean 90 and standard deviation 25. As you could see, the sample mean, that is the average of these ten numbers, is around 100. And the standard deviation of these numbers is around 39. That is very different from 90 and 25. And that is to be expected since we only generated a small sample of size ten of that normal distribution. Suppose for a second that we do not really know the true expected value of the distribution from which the sample of ten numbers comes from. The good news is that Excel has a statistics based way of estimating, with a good level of confidence, where the true expected value might be. In particular, the confidence level that descriptive statistics provides can be used to construct a 95% confidence interval for the true expected value of the distribution from which we observe only a sample. Here's how this interval is constructed. You take the sample mean, and you add and subtract the 95% confidence level value. What statistics tells us is that there is a 95% chance that the true expected value of the distribution from which our sample came from is located inside this interval. We actually know that the true expected value here is 90. And it falls within this interval as it will in 95% of simulations like that. But why are we interested in a confidence interval for the true expected value of the distribution for the distribution of which we actually know the true expected value 90. Of course, we're not really interested in that, but we do have a distribution for which we do not know and want to know the true expected value. That is the distribution of company's profit. Remember, the expected value of the profit we use as a measure of reward, but all we have is the average of 10 profit values. In other words, we have a sample mean of a small, size 10, sample of profit values. And in this simulation, the sample mean is around $50.3 million. But this is just an estimate of the true expected value of profit, and it can be quite far away from the true expected value. How far away? The 95% confidence levels help us to understand this. We can use them to calculate the 95% confidence interval for the true expected value. Now in this case, it's quite wide between roughly 44 and $57 million. So based on this calculation, we're 95% confident that the true value of the reward associated with our decision of building 96 regular and 12 luxury apartments is between roughly 44 and $57 million. Now, that is pretty uncertain. The good news is that if we increase the number of simulation runs, our confidence of whether to reward is will also increase. Here's what happens if we increase the number of simulation runs from 10 to 1,000. We have created a file Stargrove_1000 and put the results of this new simulation as well as descriptive statistics of the output into that file. In generating the random variables in columns B and C of that file, we've used the same seed values as we did for the simulation with 10 runs, seed of 123 for column B and the seed of 1234 for the column C. We just change the number random numbers parameter from 10 to 1000. Repeating the calculation of the 95% confidence interval for the true value of the company's profit, we get a much tighter interval between 51.3 and $52.1 million. In a similar way, we can calculate the 95% confidence interval for the risk measure, the probability of the profit to fall below $45 million. We can use our simulation results to compare performance of different building plans. For example, suppose that in addition to the decision of building 96 regular and 12 luxury apartments, Stargrove is also considering a decision of building 88 regular and 16 luxury apartments. We can use a simulation with 1,000 runs to make such side-to-side comparisons. You can find the details of the Excel setup in the file Stargrove_1000_TwoDecisions. Here we're showing you the results, descriptive statistics for the simulated profit values as well as likelihood for the profit to fall below the threshold. Let's compare the two decisions based on the simulation results. As you can see, the 95% confidence interval for the true expected profit value under the 96, 12 decision is located, quote unquote, to the right of the 95% confidence interval for the true expected profit value, under the 88, 16 decision. In other words, based on the results of the simulation, Stargrove should be 95% confident that it will make higher expected profit, if it decides to build 96 regular and 12 luxury apartments. If we look at the 95% confidence interval for the risk measures, we can see that they quote overlap unquote. So, we cannot really say with 95% confidence which policy generates lower risk. We can use this approach to compare more than two decisions. If we cannot tell different decisions apart, we should try and run longer simulation to achieve higher degree of confidence in our choice. So, the overall approach to using simulation to find the best option is to choose options that satisfy the selected level of risk tolerance and among those, choose the one with the highest reward. In this final session of week four, we have looked at how the simulation toolkit can be used to compare alternatives under uncertainty.