In this lesson, we will discuss data quality. This Harvard reports suggest that only three percent of companies data meets basic quality standards. This is shocking when we realize that we are in the age of artificial intelligence, when most of the top companies of the world have AI or data science as the backbone of their business strategy. In fact, data quality and accuracy are such big concerns today that Experian in its latest research report on data management, coined the term data debt. As a phenomenon, data debt can be described in the following way. When you have a data asset that isn't necessarily fit for the purpose or have high degree of inaccuracy and if you don't invest upfront in fixing these issues, all your future operations on this data will be sub-optimal, which means it's a recurring cost, like an interest payment on debt. In the same research that covers various countries and industries, Experian further reports that 78 percent of the organizations suffer from data debt, which is hurting them in more than one ways, such as lack of trust on insights coming from this data, poor return on investment on their tech and data investments and data debt is acting as a barrier to being data-driven. A real life example of data that could be Amazon's internal AI recruiting tool. In 2014, Amazon started working on an AI driven tool that can select the best candidates based on their resume. The algorithm was give one past 10 years of data to learn from. If successful, that tool would have generated tremendous value for the company internally, it might divert also the potential to become a product offering in itself. However, in 2018, Amazon had to scrap the project because the tools results were biased against women. While this may seem like a problem with algorithm per say, that data can also be a culprit here. Essentially, the algorithm was responding to the patterns in the data. If in the past, the tech company, just like the entire industry, has recruited more males than females, this bias was present in the data. The limitation of the algorithm was that it was not able to reduce the bias. One of the ways you may avoid suffering from past biases in the data is to invest in generating data through experimentation. I'd like to share an example of Capital One. This is a company that ran 28,000 experiments in the year 1998. These experiments involve giving loans to people disregarding their credit history, the rates were also randomly offered. This was a companies way to generate data which was free from past biases. In 1999, the company's strategy was called Marketing Revolution and the company has sustained it's innovative DNA so much that at one in time it was running 80,000 experiments every year. Today, the company Capital One is one of the leaders in credit card industry. So through this discussion, I hope I've been able to convince you of the importance of data quality and why should firms consider making upfront investment in creating quality data rather than suffer from data debt
