Data classification (business intelligence)

In business intelligence, data classification has close ties to data clustering, but where data clustering is descriptive, data classification is predictive.^[1][2] In essence data classification consists of using variables with known values to predict the unknown or future values of other variables. It can be used in e.g. direct marketing, insurance fraud detection or medical diagnosis.^[2]

The first step in doing a data classification is to cluster the data set used for category training, to create the wanted number of categories. An algorithm, called the classifier, is then used on the categories, creating a descriptive model for each. These models can then be used to categorize new items in the created classification system.^[1]

According to Golfarelli and Rizzi, these are the measures of effectiveness of the classifier:^[1]

• Predictive accuracy: How well does it predict the categories for new observations?
• Speed: What is the computational cost of using the classifier?
• Robustness: How well do the models created perform if data quality is low?
• Scalability: Does the classifier function efficiently with large amounts of data?
• Interpretability: Are the results understandable to users?

Typical examples of input for data classification could be variables such as demographics, lifestyle information, or economical behaviour.

Challenges for data classification

There are several challenges in working with data classification. One in particular is that it is necessary for all using categories on e.g. customers or clients, to do the modeling in an iterative process. This is to make sure that change in the characteristics of customer groups does not go unnoticed, making the existing categories outdated and obsolete, without anyone noticing.

This could be of special importance to insurance or banking companies, where fraud detection is extremely relevant. New fraud patterns may come unnoticed, if the methods to surveil these changes and alert when categories are changing, disappearing or new ones emerge, are not developed and implemented.

References

1. ^ ^{a b c} Golfarelli, M. & Rizzi, S. (2009). Data Warehouse Design : Modern Principles and Methodologies. McGraw-Hill Osburn. ISBN 0071610391
2. ^ ^{a b} Kimball, R. et al. (2008). The Data Warehouse Lifecycle Toolkit. (2. Ed.). Wiley. ISBN 0471255475