📝 Naive Bayes

Description

< What is it? >

• Meaning: Naive Bayes is a family of supervised classifiers that use Bayes' theorem to predict a class from observed features. Training estimates class frequencies and feature distributions; prediction combines that information to choose the most probable class.
• Why “naive”: the model assumes features are conditionally independent given the class. For a spam classifier, it treats the presence of free and win as independent once the spam/non-spam class is specified. This simplifies computation even though real words can be related.
• Example: given an email containing free and win, compare the scores for spam and not_spam, then choose the larger one. The worked example below predicts spam.

Key points

< What is prior probability? >

• Meaning: a prior probability is the probability assigned to an event or class before considering the new evidence. In Naive Bayes classification, the class prior $P(c)$ is the starting probability before inspecting the current input's features. “Before” refers to this new input, not to all historical data: a prior can be estimated from previous observations.

• Estimate from training labels: a common class-prior estimate is the fraction of training examples belonging to that class:

  $$P(c)=\frac{N_c}{N}$$

  Here, $N_c$ is the number of training examples in class $c$, and $N$ is the total number of training examples. See Stanford's Naive Bayes explanation.

• Example: the six-email dataset below contains 3 spam and 3 non-spam emails. Before examining a new email's words, the model starts with $P(\text{spam})=3/6=0.5$ and $P(\text{not\_spam})=0.5$. After observing free and win, it combines those priors with the feature likelihoods. The worked calculation gives a posterior spam probability of approximately 0.857.

  Prior: spam probability before inspecting this email = 0.50
      + Evidence: this email contains "free" and "win"
      → Posterior: spam probability after using that evidence ≈ 0.857

  The prior describes the starting class frequency; the posterior accounts for the particular email. The arrow represents a Bayesian update, not literal addition of probabilities.

• Unequal class frequencies: if 20 out of 100 training emails are spam, the empirical spam prior is $0.20$, not $0.50$ simply because there are two classes. Priors can also be specified using known population frequencies. A deliberately balanced training sample may not reflect the class proportions seen in production.

< How the classifier makes a prediction >

• Bayes' theorem: for a class $c$ and feature vector $\mathbf{x}$:

  $$P(c\mid\mathbf{x})=\frac{P(c)P(\mathbf{x}\mid c)}{P(\mathbf{x})}$$

  Term	Meaning
  $P(c)$	Prior: how common the class is before observing this input
  $P(\mathbf{x}\mid c)$	Likelihood: how probable the observed features are under that class
  $P(c\mid\mathbf{x})$	Posterior: the class probability after observing the features
  $P(\mathbf{x})$	Evidence: the normalizing term shared by all classes for this input

• Example — spam probability after seeing “free”: suppose we examine 100 emails with the following counts. This illustrative dataset uses one feature to explain Bayes' theorem; the Implementation section uses a separate six-email dataset with two features.

  Email type	Contains “free”	Does not contain “free”	Total
  Spam	16	4	20
  Not spam	8	72	80
  Total	24	76	100

  Let $c$ mean “spam” and $x$ mean “contains free.” Each probability comes directly from the counts:

  Term	Meaning in this example	Value
  $P(c)$	Probability an email is spam before checking its words — the prior	$20/100 = 0.20$
  $P(x\mid c)$	Probability it contains “free,” given that it is spam — the likelihood	$16/20 = 0.80$
  $P(x)$	Probability any email contains “free” — the evidence probability	$24/100 = 0.24$
  $P(c\mid x)$	Probability it is spam, given that it contains “free” — the posterior	What we calculate

  Substitute these values:

  $$P(\text{spam}\mid\text{“free”}) =\frac{0.20\times0.80}{0.24} =\frac{2}{3} \approx66.7\%$$

  Before reading the email, the spam probability was 20%. After seeing “free,” it becomes about 66.7%. Verify this directly: 24 emails contain “free,” and 16 of those are spam, so the probability is $16/24$.

  The direction matters: $P(\text{“free”}\mid\text{spam})=80\%$ and $P(\text{spam}\mid\text{“free”})\approx66.7\%$ answer different questions.

• Independence assumption: replace the joint likelihood with a product of feature likelihoods. The shared denominator can be omitted when selecting the largest score:

  $$\hat{c}=\arg\max_c P(c)\prod_{j=1}^{d}P(x_j\mid c)$$

  See the scikit-learn Naive Bayes guide.

< Training, smoothing, and numerical stability >

• Training labels: provide examples with known classes, such as emails labeled spam or not_spam. Standard Naive Bayes training estimates counts or distribution parameters; it does not require a neural network or backpropagation.

• Smoothing: add pseudocounts so an unseen feature value does not force a class likelihood to zero. For binary features, add-one smoothing estimates:

  $$P(x_j=1\mid c)=\frac{N_{c,j}+1}{N_c+2}$$

  Here, $N_{c,j}$ counts class-$c$ examples where feature $j$ is present, and $N_c$ counts all class-$c$ examples. The denominator adds 2 because a binary feature has two outcomes. Absence contributes $1-P(x_j=1\mid c)$.

• Log scores: implementations sum log probabilities rather than multiplying many tiny values, avoiding numerical underflow. See Stanford's Naive Bayes chapter.

Comparison

< Naive Bayes vs. Decision Tree >

• Features and prediction: both models classify inputs represented as features. Decision trees classify by splitting on features, while Naive Bayes combines class priors with feature likelihoods. Bag of Words (BoW) turns text into features such as word counts; either model can use this representation. Naive Bayes can also use other feature types, such as continuous measurements with GaussianNB.

• Practical starting point: try Naive Bayes for a fast text-classification baseline using word counts or word-presence features. Try a small decision tree for tabular classification when feature thresholds, combinations of conditions, and readable decision paths matter. These are starting points to evaluate, not guarantees that one model will win.

  Consideration	Naive Bayes	Decision Tree
  How it predicts	Combines class priors and feature likelihoods	Follows learned feature splits to a leaf
  Useful starting tasks	Spam filtering, document categories, support-ticket routing	Tabular classification with threshold-based decisions
  Feature interactions	Assumes conditional independence given the class; does not directly learn interactions	Can learn interactions through successive splits
  Many sparse text features	Multinomial and Bernoulli variants are natural baselines	Can use them, but a single tree may need many splits
  Interpretability	Inspect priors and feature evidence for each class	A shallow tree provides an explicit if/else path
  Training and tuning	Usually fast; choose the feature distribution and smoothing	Control depth, minimum leaf size, or pruning to limit overfitting
  Main limitation	Related features can count similar evidence repeatedly	Deep trees can overfit; small data changes can change the tree

  See the scikit-learn guides for Naive Bayes and decision trees.

• Example — spam classification: a Bag of Words representation could encode an email as [count of "free", count of "win"]. Both models can use this vector: a decision tree might split on count of "free" > 0, while Naive Bayes combines the likelihood of those counts under each class with its prior. For thousands of word-count features, MultinomialNB is a useful starting point; for binary word-presence features, try BernoulliNB.

• Example — lipstick recommendations: in the Decision Tree example, gender, age, income, and nationality are tabular features. A tree can first split on gender, then use a different age or income threshold within a child. This makes conditional patterns visible. Naive Bayes can also classify tabular data, but its likelihood model must suit the features; mixing continuous and categorical fields requires appropriate handling.

• How to decide: compare both on the same validation split using metrics appropriate to the task, such as precision, recall, and F1. Fit preprocessing only on training data. Inspect errors and evaluate probability calibration if probabilities drive decisions; neither model guarantees reliable probabilities. Choose the simpler model that meets the measured quality and runtime requirements.

< Common Naive Bayes variants >

• Choose the likelihood for the feature type: these variants share the classification principle but model inputs differently.

  Variant	Typical input	Example
  Bernoulli	Binary presence/absence	Whether each vocabulary word occurs in an email
  Multinomial	Nonnegative counts; often also used with TF-IDF	Word occurrence counts
  Gaussian	Continuous values, modeled as Gaussian within each class	Numeric measurements
  Categorical	Discrete category values	Device type or subscription plan

  scikit-learn provides BernoulliNB, MultinomialNB, GaussianNB, and CategoricalNB in its Naive Bayes module.

Implementation

< Classify an email with Bernoulli Naive Bayes >

• Training input: each email becomes two binary features: [contains free, contains win]. This deliberately small dataset makes the calculations visible; repeated feature vectors can represent different emails.

  Email	Contains free	Contains win	Label
  1	1	1	spam
  2	1	0	spam
  3	1	1	spam
  4	0	0	not_spam
  5	0	1	not_spam
  6	0	0	not_spam

• Worked prediction: both classes have prior $3/6=0.5$. After add-one smoothing:

  Class	$P(\text{free}=1\mid c)$	$P(\text{win}=1\mid c)$	Score for [1, 1]
  spam	$(3+1)/(3+2)=0.8$	$(2+1)/(3+2)=0.6$	$0.5\times0.8\times0.6=0.24$
  not_spam	$(0+1)/(3+2)=0.2$	$(1+1)/(3+2)=0.4$	$0.5\times0.2\times0.4=0.04$

  Normalize the scores to obtain the model's posterior:

  $$P(\text{spam}\mid[1,1])=\frac{0.24}{0.24+0.04}\approx0.857$$

  The predicted label is spam. This toy probability is not a measured real-world accuracy.

• Runnable Python: this standard-library implementation trains on the table and predicts [1, 1]. It also handles absent features by using their complement probabilities.

  import math
  
  # Features: [contains "free", contains "win"].
  X = [[1, 1], [1, 0], [1, 1], [0, 0], [0, 1], [0, 0]]
  y = ["spam", "spam", "spam", "not_spam", "not_spam", "not_spam"]
  classes = ["spam", "not_spam"]
  model = {}
  
  # Training: estimate priors and smoothed feature probabilities.
  for label in classes:
      rows = [row for row, target in zip(X, y) if target == label]
      prior = len(rows) / len(X)
      probabilities = [
          (sum(row[j] for row in rows) + 1) / (len(rows) + 2)
          for j in range(len(X[0]))
      ]
      model[label] = (prior, probabilities)
  
  # Inference: an email containing both words.
  query = [1, 1]
  log_scores = {}
  for label, (prior, probabilities) in model.items():
      log_scores[label] = math.log(prior) + sum(
          math.log(p if value == 1 else 1 - p)
          for value, p in zip(query, probabilities)
      )
  
  # Normalize safely to posterior probabilities under the model.
  maximum = max(log_scores.values())
  weights = {label: math.exp(score - maximum)
             for label, score in log_scores.items()}
  total = sum(weights.values())
  for label in classes:
      print(f"{label}: {weights[label] / total:.3f}")
  print("prediction:", max(log_scores, key=log_scores.get))

• Verified output:

  spam: 0.857
  not_spam: 0.143
  prediction: spam

  For a library implementation, use scikit-learn's BernoulliNB with alpha=1.0 on the same binary features.

Troubleshoot

< What can make predictions misleading? >

• Correlated features: several related features can count similar evidence repeatedly, producing overconfident probabilities. Assess calibration if probability quality matters.
• Wrong feature model: binary flags, counts, and continuous measurements require appropriate likelihoods. Do not assume every variant accepts the same representation.
• Evaluation: measure performance on held-out examples. A confusion matrix helps distinguish missed spam from legitimate emails incorrectly blocked. The six-row demonstration verifies arithmetic, not generalization.

Video Tutorial

Naive Bayes, Clearly Explained !!!

Naive Bayes Explained Visually

Naive Bayes FROM SCRATCH in Python

Related ideas

• Classification predicts category labels.
• Search & Retrieval explains tokenization and text preprocessing.
• Term Frequency–Inverse Document Frequency (TF-IDF) represents text with weighted terms.
• Decision Tree classifies inputs through learned feature splits.

Reference

• Naive Bayes (scikit-learn)
• BernoulliNB (scikit-learn)
• Naive Bayes text classification (Introduction to Information Retrieval)