What Do We Really Need to Trust?

Machine Learning (ML), Trust and Reliability

December 28, 2021

Are predictions from AI systems misleading or can we rely on them? As a real-world example, in this post we look at sales around Christmas using time series prediction models.

Author: Manu Somasagar Kamalakar

Explainability of ML Models

New Year, new happiness and hew predictions with Artificial Intelligence (AI)! In this blog today, we address the question of how predictions from AI systems can be misleading or misunderstandable – especially when the model itself does not provide explanations [1].

To this end, we have developed a time series prediction model to serve as an illustrative example (Figure 1). Suppose we trained a time series prediction model that predicts the profit of a supermarket before Christmas.

© Fraunhofer ITWM
Figure 1: The blue curve represents known data. The predicted data are shown in orange (lighter orange curves indicate the bounds of the confidence intervals) [2].

Reliability of AI Systems

Ideally, supermarket profits increase during the Christmas time because the number of customers in supermarkets increases as does the number of products sold. When we train the predictive model based on the historical data, the model expects profit to increase during the Christmas season. However, this could be misleading or even wrong in the real world because it is subject to various external factors in 2021 such as the number of COVID cases, government regulations, etc. So in this example, the question of how much we can rely on an AI system remains rather unanswered.

Already in 2020, the models did not provide good forecasts [3], because it was the first year of the pandemic. Everything that happened to sales that year was different from anything previously observed by ML models. Nevertheless, this was not unexpected for Christmas 2021. In the third year of the pandemic, most models already »know« the situation and are prepared. They have initial historical data to recognize and learn from a »Covid19 pattern.« However, there will always be situations, such as unexpected lockdowns, shutdowns, infection rates, etc., that make forecasting difficult.

Crisis Winners During Corona

Here, too, there are special cases: Sales have fallen due to the pandemic [4], but sales have not fallen equally for all products. Some companies have managed to score through online retailing. While COVID has been a financial disaster for many companies (e.g., the automotive industry) [5], others have still managed to profit from the situation, such as the entertainment industry, especially streaming platforms [6].

Figure 2: Worldwide Google searches and COVID 19. Taken from [7] data from the Johns Hopkins Coronavirus Center.

As can be seen in Figure 2, online searches are directly related to the increase in COVID cases. Now we ask ourselves the following questions:

What can we expect from a transparent ML model that provides explanations for its own predictions?
What would be different if, instead of a model making predictions that are far from reality, more accurate predictions had been made (better adaptability to change)?
There are certainly such models that make more accurate predictions. For example, some machine learning algorithms exist that learn not only from what happened long ago, but also from more recent data and adjust their predictions on their own. Nevertheless:
Would such an adjustment have happened quickly enough?
Does the model even have all the relevant information to adjust?
Do we need a more accurate forecast? Of course, a more accurate model helps. We could have known that sales were falling and therefore invested less. But as we said, in most situations a key figure or parameter does not describe the real world. Because the real world is complex, even if it's just about sales before Christmas. Such a model does not help us. But:
What if we were able to give the model all the relevant information to calculate?
What if the model had provided »explanations« itself instead? Maybe we could have invested in products that sell better.

Let's develop reliable models that really help by learning from feedback that is not only right or wrong, but also contains information about why it needs to be evaluated as wrong in the first place. That certainly won't be easy.

References

[1] Importance of Interpretability | Interpretable Machine Learning (christophm.github.io).
Doshi-Velez, Finale, and Been Kim. »Towards a rigorous science of interpretable machine learning,« no. Ml: 1–13. (2017).

[2] Magoosh Statistics Blog

[3] Article »The failure of COVID-19 prediction and what we learned from it.«

[4] Impact of the coronavirus pandemic (COVID-19) on retail sales in 2020 from the Office of National Statistics

[5] Covid and the impact on the automotive sector

[6] Boom in streaming platforms

[7] Global Google search patterns

Blog Overview

Blog Entry