Big Data and Bayesian Nonparametrics

Опубликовано 22 июня 2016, 1:15

Big Data is often characterized by large sample sizes, high dimensions, and strange variable distributions. For example, an e-commerce website has 10-100s million observations weekly on a huge number of variables with density spikes at zero and elsewhere and very fat tails. These properties – big and strange – beg for nonparametric analysis. We revisit a flavor of distribution-free Bayesian nonparametrics that approximates the data generating process (DGP) with a multinomial sampling model. This model then serves as the basis for analysis of statistics – functionals of the DGP – that are useful for decision making regardless of the true DGP. The ideas will be illustrated in the indexing of treatment effect heterogeneity onto user characteristics in digital experiments, and in analysis of decision trees employed in fraud prediction. The result is a framework for scalable nonparametric Bayesian decision making on massive data.