Classic ML Algorithms
Naive Bayes is a fast & simple classifier that works well even with thousands of features. It has few tunable hyperparameters, so it’s a popular baseline.
1. Bayes’ theorem (applied to language model)
Posterior probability based on prior knowledge:
Here, L = language model and x=(x1, …, xd) are features. P(features x | L) = generative model specifying the hypothetical random process that generated the data.
2. The “naive” assumption
Calculating this model is hard; to find a rough approximation of the generative model for each class, we assume a naive assumption – features are conditionally independent given the class:
3. Variants (different naive assumptions about the data
Multinomial NB: describes the probability of observing counts among a number of categories, i.e. assumes features are integer counts drawn from a multinomial (or categorical) distribution—great for text (word counts).
M.d. - generalization of the binomial distribution. Example: probability of counts for each side of a k-sided die rolled n times. If k=2, n=1 => Bernoulli distribution. If k=2, n>1, => binomial distribution. If k > 2, n=1 => categorical distribution.Gaussian NB
Assumes each xj is real‐valued and follows a normal distribution with class‐specific mean and variance. Naïve assumption - data for each label is drawn from a simple Gaussian distribution (Gaussian or normal d. - continuous probability distribution for a real-valued random variable)
Red curve is Gaussian
When to Use Naive Bayes Classifiers
Stringent assumptions - will not perform as well as a more complicated model.
BUT: extremely fast to train, straightforward probabilistic prediction, very easily interpretable, very few tunable parameters
A good choice as a baseline model. Otherwise: perform well when match the data, well-separated categories, very high-dimensional data (model complexity is less important)