How AI and Algorithms Can Fail to Detect Anomalies or Fraud

Underrepresentation – A Challenge in Recognizing Anomalies or the Christmas Tree Case

Blog Entry EP-KI-Blog / December 21, 2023

Who hasn't experienced this: AI-supported systems suggest a wide variety of products, even though we have only looked at them once. This may be annoying or even irritating, but it becomes much more important when systems are supposed to detect potential fraud or errors. Then we need to be able to rely on them to behave as we intend. Unfortunately, the world of data is rarely uniform, but very diverse. In our new blog post, we share with you what we need to watch out for when developing algorithms that are underrepresented.

Our junior research group »EP-KI: Decision Support for Business Processes Using New AI Methods« is constantly developing new methods for detecting anomalies. This is because several projects in our department are concerned with detecting fraud and audits. Normally, the first step is to better understand the data. This is followed by identifying the type of anomalies that the customer wants to find. Based on this, we develop a suitable machine learning model. In recent years, we have noticed time and again that our clients often have a very different idea of our workflow. The request is something like this: We give you the data and then you put it in a magic box that subsequently spits out the anomalies. However, developing a suitable model for detecting anomalies involves a lot of work and can contain many pitfalls.

In this blog post, we write about how AI algorithms can fail in this environment. Before we go into the problem, we briefly explain what is meant by anomaly detection, why underrepresented data is a problem for anomaly detection, and finally give a real-world example.

Let's go! Christmas without Fraud

We use a hypothetical data set to illustrate a real problem. We encountered this with data from a project partner. Let's imagine we are looking at the data of a chain of stores that sells Christmas trees. The company's clientele comes from Germany and France, but there is also a small group of customers in Hungary. As expected, the company experiences a significant increase in sales during the Christmas period, while sales for the rest of the year are practically zero. As sales increase, so do fraud attempts. The company has therefore decided to implement an alert system based on anomaly detection. Our task is to detect fraudulent transactions.

We Have Information About

the state in which the sale took place
the stores where the Christmas trees are sold
the amount paid for the tree, i.e. revenue for our company

To assess the impact of less frequent observations for a characteristic, we create four different data sets, which are shown in the table below. The percentages refer to the share of the respective country in the data.

	Raw	Currency transformed	Balanced	Balanced and Currency transformed
State (share of total sales in percent)	Hungary (8) Germany (50) France (42)	Hungary (8) Germany (50) France (42)	Hungary (28) Germany (40) France (32)	Hungary (28) Germany (40) France (32)
Stores (number of stores in the state)	Hungary (2) Germany (6) France (5)	Hungary (2 Germany (6) France (5)	Hungary (2) Germany (6) France (5)	Hungary (2) Germany (6) France (5)
Amount	EURO, HUF	EURO	EURO, HUF	EURO

What Do You Think the Algorithm Recognizes as Anomalies in Our Data?

We have already explained how a decision tree works. A well-known method for recognizing anomalies is the »Isolation Forest«, which randomly creates multiple decision trees. If a point can be easily separated by splitting on a small number of features, it is assumed that this point is anomalous. When using this algorithm, the following number of anomalies is detected for each data set:

State	#Anomalies		Total	#Anomalies		Total
State	Raw	Currency transformed	Total	Balanced	Balanced and Currency transformed	Total
Germany	4	3	50	6	4	40
France	1	3	42	1	6	32
Hungary	8	8	8	27	0	28

But why does the algorithm prefer to recognize sales in Hungary as anomalous? To answer this question, we should first take a closer look at how anomaly algorithms work.

How Do Anomaly Detection Algorithms Work?

To explain this in more detail, let's look at a data set where the majority of the data is structured similarly, while the composition of a small subset differs from the majority. This small subset is defined as »anomalies«. Certain methods allow us to detect these anomalies. For more details on anomaly detection, we refer to the section »The Detectors« in our second blog post, where we also explain how to distinguish algorithms from other classes of AI algorithms.

The detection of anomalies can be supervised or unsupervised, depending on whether previous cases of anomalies are known, i.e. whether labeled data is available. However, as we mention in our blog post, it is common in industrial applications that there is no labeled data.

© Fraunhofer ITWM
Graphic: Global Anomalies and Underrepresented Classes

Global and Local Anomalies

Regardless of whether labels are available or not, the use of unsupervised learning algorithms is a good idea in most cases, as they enable the detection of new types of anomalies that are not present in the existing data.

Very different types of anomaly detection algorithms can be used to detect different types of anomalies (Goldberg, Uchida, 2016)

One version of these unusual data are the »global anomalies«. These are structured quite differently compared to other observations in our data. The figure above shows a visualization of these anomalies. We can see that the points labeled X 1 and X 2 are far away from the other points.

Another type of anomalous data are the »local anomalies«. These are located near clusters of normal data, i.e. collections of non-anomalous data, but do not fall into these clusters. Clusters of anomalies are a different type of anomaly. In the figure above, C 2 could be such a cluster of anomalies. However, it depends on the use case whether it is really a cluster of anomalies or a cluster of normal data with lower frequency.

There are many applications where anomaly detection is used, for example in fraud detection. Here the anomalies correspond to cases of fraud. Use cases include detecting false information in records from care providers or emails from criminals, detecting abnormal medical conditions or patient behavior, detecting potential errors or machine failures, and much more.

Why Are Low Occurrence Features a Challenge for Anomaly Detection Algorithms?

What is the difference between small occurrences, which may also be insignificant, and anomalies? For the algorithm, a significantly less frequently occurring value of a characteristic is actually an anomaly.

In our example, simply collecting the data and applying an algorithm to detect anomalies presents at least two challenges:

Our data may contain a numeric column derived from different scales. In our example of the chain of stores selling Christmas trees to Germany, France and Hungary, we have currencies in euros or forints. In the raw data, 50 percent were sold to Germany, 42 percent to France and only eight percent to Hungary. Due to the exchange rate of the euro and forint, the amount for individual sales to Hungary is higher than to Germany or France. This is true even if the data is almost balanced. Now one might assume that converting the amount for Hungary solves this problem. But still we have the same anomalies, why?
After the currency has been converted to Euro, the algorithm will no longer detect anomalies due to different scales. However, the amount of data in Hungary is still much smaller compared to Germany or France, which is why the system will recognize this data as anomalous.

In both cases, the observations in Hungary are a relatively small number of observations. Therefore, the algorithms that look for global outliers, i.e. a small number of observations that are far away from the others, will recognize them as anomalies. This explains why Hungary is no longer recognized as anomalous in balanced data with uniform currency.

We hope that anomaly algorithms are well implemented and you can get your Christmas tree in the store of your choice at fair conditions.