Data mining in finance: Using counterfactuals to generate knowledge from organizational information systems

摘要

A common view about data mining is that it is an exercise of “clustering” customers, markets, products, and other objects of interest in useful ways from large amounts of data. In this paper, I demonstrate that the real value of data mining, particularly in the financial arena, lies more in revealing actions that lead to interesting distributions of outcomes, distributions that are not directly observable in the data. Simulating actions or events that did not occur is often more useful than trying to cluster the data, which represents only those states of nature that occurred and were recorded. I show how the use of counterfactuals, which are hypothetical events, coupled with certain types of machine learning methods produce models that promote human dialog and exploration that does not otherwise occur in routine organizational activity. I demonstrate with real-world examples, how the combination of database systems, counterfactuals and machine learning methods combine to provide a powerful bottom-up theory building mechanism that is useful in enabling organizations to use databases for learning about things that are useful to them.