📝 Logistic Regression

Description

< What is it? >

• Meaning: logistic regression is a supervised model that estimates the probability of a class from input features. In binary classification, it combines the features into a linear score and applies the sigmoid function to produce $P(y=1\mid\mathbf{x})$. See Google's introduction to logistic regression.

  With three input features, the vector notation expands to:

  $$z=\mathbf{w}^{\top}\mathbf{x}+b=w_1x_1+w_2x_2+w_3x_3+b$$ $$p=\sigma(z)=\frac{1}{1+e^{-(w_1x_1+w_2x_2+w_3x_3+b)}}$$

  Each $x_i$ is an input feature, each $w_i$ is its learned weight, and $b$ is the learned bias. For $d$ features, the weighted sum is $z=\sum_{i=1}^{d}w_ix_i+b$; the model still produces one probability for each example.

• Example — predicting a recommendation click: consider three movie-recommendation features with illustrative weights and bias $b=-1.5$:

  Feature	Meaning	Input $x_i$	Weight $w_i$
  $x_1$	Genre matches the user's preference	1	1.2
  $x_2$	User recently watched a similar movie	1	0.8
  $x_3$	Movie popularity, scaled to 0–1	0.5	0.6

  $$z=1.2(1)+0.8(1)+0.6(0.5)-1.5=0.8$$ $$p=\frac{1}{1+e^{-0.8}}\approx0.6900=69.00\%$$

  All three features contribute to one estimated click probability of 69.00%. These weights illustrate inference; the implementation below trains a smaller two-feature model. Its training label is the observed outcome: 1 for clicked and 0 for not clicked. An application can apply a threshold for classification or sort candidates by probability for ranking.

  User–item features → weighted sum + bias → sigmoid → estimated click probability
                                                               ↓
                                                   optional threshold → 0 or 1

Key points

< Features, logits, and probabilities >

• Terms in the model: each example has $d$ numerical features. Categorical values must first be encoded, for example with one-hot encoding; text can be represented using TF-IDF or embeddings.

  Symbol	Meaning	Movie-click example
  $\mathbf{x}$	Input feature vector	[1, 0]: matching genre, no recent similar movie
  $\mathbf{w}$	Learned feature weights	One coefficient for each of the two features
  $b$	Learned intercept or bias	Baseline score when both feature values are zero
  $z$	Linear score, also called a logit	Can be any real number
  $p$	Estimated positive-class probability	Probability of a click, between 0 and 1
  $y$	Observed binary training label	1 for click, 0 for no click

• Sigmoid transformation: negative logits give probabilities below 0.5, a zero logit gives 0.5, and positive logits give probabilities above 0.5. For finite logits, the mathematical sigmoid lies strictly between 0 and 1. See Google's sigmoid explanation.

  Input logit $z$	Sigmoid output $\sigma(z)$	As a percentage
  −5	≈ 0.0067	≈ 0.67%
  0	0.5	50%
  5	≈ 0.9933	≈ 99.33%

  The output approaches 0 as the input becomes increasingly negative and approaches 1 as it becomes increasingly positive. In logistic regression, it represents the estimated probability of class 1. Floating-point computation may round extreme outputs to exactly 0 or 1.

• Example — follow one prediction: the implementation below learns approximately $\mathbf{w}=[1.1775,1.1775]$ and $b=-1.1775$. For candidate A with features [1, 1]:

  $$z=1.1775\times1+1.1775\times1-1.1775=1.1775$$ $$p=\frac{1}{1+e^{-1.1775}}\approx0.7645$$

  The model estimates a 76.45% click probability for this toy example. This is a model estimate, not a guarantee that the user will click.

< Log-odds and the decision boundary >

• Linear log-odds: logistic regression models the logarithm of the odds as a linear function of the features:

  $$\log\frac{p}{1-p}=\mathbf{w}^{\top}\mathbf{x}+b$$

  Holding other features fixed, increasing feature $x_j$ by one unit multiplies the modeled odds by $e^{w_j}$. It does not add a fixed amount to the probability. Coefficients describe model associations, not necessarily causal effects.

• Classification threshold: choose a threshold $t$ and predict class 1 when $p\geq t$. At $t=0.5$, the boundary is $\mathbf{w}^{\top}\mathbf{x}+b=0$: a line in two dimensions or a hyperplane in higher dimensions. The probability curve is nonlinear, but the boundary is linear in the supplied features. Adding interaction or polynomial features can create nonlinear boundaries in the original input space.

• Example — changing a decision: candidate A's probability is about 0.7645. A threshold of 0.6 assigns class 1; a threshold of 0.8 assigns class 0. Changing the threshold changes the decision, not the learned weights or probability. Select the threshold using validation data and the costs of false positives and false negatives. See Google's classification threshold guide.

< How training learns the weights >

• Training objective: binary cross-entropy, also called log loss, is the negative log-likelihood of the observed binary labels. For $N$ training examples:

  $$L=-\frac{1}{N}\sum_{i=1}^{N}\left[y_i\log p_i+(1-y_i)\log(1-p_i)\right]$$

  Confident wrong predictions incur a large penalty. For a clicked item, $y=1$: predicting $p=0.9$ gives loss $-\log(0.9)\approx0.105$, while predicting $p=0.1$ gives about 2.303. See Google's loss and regularization guide.

• Optimization: initialize weights, compute probabilities, compare them with labels, and update parameters using gradients. For the unregularized mean loss:

  $$\frac{\partial L}{\partial w_j}=\frac{1}{N}\sum_i(p_i-y_i)x_{ij} \qquad \frac{\partial L}{\partial b}=\frac{1}{N}\sum_i(p_i-y_i)$$

  The implementation uses gradient descent. Libraries also offer other optimizers, such as L-BFGS; logistic regression does not require a multilayer neural network.

• Regularization: an L2 penalty discourages large weights; an L1 penalty can set some weights to zero. On perfectly separable data, unregularized fitting can drive coefficients toward arbitrarily large magnitudes. Regularization controls this behavior. See Regularization for the objectives and penalties.

< Evaluation and practical uses >

• Evaluate the required output: use log loss to evaluate probabilities, precision and recall for thresholded decisions, and ranking metrics for ordered candidates. A confusion matrix shows which classes are confused. Check probability calibration when decisions depend on the numerical probability itself.
• Keep data separate: fit preprocessing and the model on training data, choose hyperparameters and thresholds on validation data, and report final performance on held-out test data. Use time-based or grouped splits when repeated users or future interactions could cause leakage.
• Ranking: score each candidate using the same trained model and sort by descending probability. Sigmoid is strictly increasing, so sorting the logits gives the same mathematical order. Click-log training is a pointwise ranking approach; exposure and position bias can affect what the model learns. See Recommendation System.
• More than two classes: multinomial logistic regression uses one linear score per class and a softmax to obtain probabilities that sum to 1. Multilabel classification instead often uses separate binary models because several labels can apply simultaneously.

Comparison

< Logistic regression vs. linear regression and decision trees >

• Model choice: compare models using the same held-out data and task metrics.

  Aspect	Logistic regression	Linear regression	Decision tree classifier
  Main target	Class probability	Continuous numerical value	Class label or probability
  Example	Probability a user clicks	Expected viewing time	Click or no click from feature splits
  Model form	Linear log-odds, transformed by sigmoid	Linear prediction	Rules formed by recursive splits
  Typical training objective	Log loss, often with regularization	Squared error, optionally regularized	Greedy impurity reduction, such as Gini
  Feature interactions	Must be supplied or learned by upstream features	Must be supplied or learned by upstream features	Can be captured through successive splits
  Main strength	Compact, efficient, interpretable baseline	Simple model for numeric targets	Flexible nonlinear rules
  Main limitation	Limited by the supplied feature representation	Linear form may miss complex relationships	Deep trees can overfit and be unstable

• When logistic regression fits: use it as a baseline for binary outcomes, sparse text classification, or problems where coefficient interpretation matters. Consider trees or an MLP when nonlinear feature interactions remain important after feature engineering. A more complex model is useful only if evaluation supports the added complexity.

Implementation

< Train a movie-click predictor in Python >

• Input data: this small synthetic dataset uses two binary features. Repeated rows represent separate impressions with identical encoded features; their observed click labels may differ.

  Features: [genre_match, recently_watched_similar_movie]	Labels for two impressions
  [0, 0]	0, 0
  [0, 1]	0, 1
  [1, 0]	0, 1
  [1, 1]	1, 1

• Runnable code: Python's standard library is sufficient for this teaching example. It learns both weights and the bias with full-batch gradient descent and an L2 penalty. The threshold of 0.6 is an illustrative application choice.

  from math import exp
  
  # Features: [genre_match, recently_watched_similar_movie].
  # Each row is one displayed recommendation; y=1 means it was clicked.
  X = [[0, 0], [0, 0], [0, 1], [0, 1],
       [1, 0], [1, 0], [1, 1], [1, 1]]
  y = [0, 0, 0, 1, 0, 1, 1, 1]
  
  
  def sigmoid(z):
      # This form avoids overflow for large positive or negative logits.
      if z >= 0:
          return 1.0 / (1.0 + exp(-z))
      ez = exp(z)
      return ez / (1.0 + ez)
  
  
  weights, bias = [0.0, 0.0], 0.0
  learning_rate, strength = 0.5, 0.05
  n = len(X)
  
  # Minimize mean binary cross-entropy + strength/2 * sum(w*w).
  # The bias is not regularized.
  for _ in range(2000):
      errors = [sigmoid(sum(w * v for w, v in zip(weights, row)) + bias) - label
                for row, label in zip(X, y)]
      gradient = [sum(error * row[j] for error, row in zip(errors, X)) / n
                  + strength * weights[j] for j in range(2)]
      weights = [w - learning_rate * g for w, g in zip(weights, gradient)]
      bias -= learning_rate * sum(errors) / n
  
  print('Weights:', [round(w, 4) for w in weights])
  print(f'Bias: {bias:.4f}')
  
  # Inference needs features only; no click label is supplied.
  threshold = 0.6
  candidates = [('A', [1, 1]), ('B', [1, 0]), ('C', [0, 0])]
  for name, row in candidates:
      z = sum(w * v for w, v in zip(weights, row)) + bias
      probability = sigmoid(z)
      predicted_label = int(probability >= threshold)
      print(f'{name}: features={row}, logit={z:.4f}, '
            f'p(click)={probability:.4f}, percentage={probability:.2%}, '
            f'class={predicted_label}')

• Verified output: running the code produces:

  Weights: [1.1775, 1.1775]
  Bias: -1.1775
  A: features=[1, 1], logit=1.1775, p(click)=0.7645, percentage=76.45%, class=1
  B: features=[1, 0], logit=0.0000, p(click)=0.5000, percentage=50.00%, class=0
  C: features=[0, 0], logit=-1.1775, p(click)=0.2355, percentage=23.55%, class=0

  Ranking by predicted probability gives A → B → C. The classification threshold marks only A as positive. These outputs demonstrate training and inference; eight synthetic impressions do not establish real-world accuracy or calibration.

• Display a probability as a percentage: the model and its training remain unchanged. probability = sigmoid(z) already gives the estimated probability; int(probability >= threshold) separately converts it into a class label. To display only the percentage, replace the classification and print lines with print(f"{probability:.2%}"). Python's .2% format multiplies the value by 100 for display and appends %: candidate A prints 76.45%. Keep the underlying probability on the 0–1 scale for calculations. This percentage is the estimated chance of a click, not the model's accuracy.

• Library choice: for application code, use scikit-learn's LogisticRegression, usually with preprocessing in a pipeline. fit(X, y) trains the model, predict_proba(X) returns class probabilities in classes_ order, and decision_function(X) returns binary logits. Scale continuous features using training-data statistics and retain the same transformations at inference.

Troubleshoot

< Common problems >

• Diagnosis: match the remedy to the observed behavior.

  Symptom	Possible cause	Practical response
  Training fails to converge	Poor feature scaling or insufficient iterations	Scale continuous features and inspect optimizer settings
  Very large coefficients	Separation, correlated features, or weak regularization	Strengthen regularization and review the features
  High accuracy but few positives detected	Class imbalance or an unsuitable threshold	Inspect precision, recall, and the confusion matrix; tune on validation data
  Good training results, poor test results	Overfitting or data leakage	Audit splits and preprocessing; regularize and evaluate on independent data
  Curved or interacting patterns are missed	A linear boundary is insufficient	Add justified interaction features or compare a nonlinear model
  Probabilities differ from observed frequencies	Miscalibration, sampling effects, or distribution shift	Evaluate calibration on representative data before using probabilities as rates

Q & A

< Why is it called regression when it is used for classification? >

• Answer: the model regresses the log-odds of an event on the input features. The resulting probability supports classification when a decision threshold is applied.

< Is logistic regression a neural network? >

• Answer: binary logistic regression can be represented as a single output neuron with an affine transformation followed by sigmoid, without hidden layers. An MLP adds hidden layers and nonlinear transformations that can learn more complex feature interactions. In PyTorch, a linear layer with one output and BCEWithLogitsLoss implements the binary logistic objective; the loss accepts the raw logit and handles the sigmoid calculation internally.

Video Tutorial

Logistic Regression (4 min)

Logistic Regression (3 min)

Related ideas

• Linear Regression predicts continuous targets with a linear model.
• Multilayer Perceptron (MLP) extends learned linear transformations with hidden layers.
• Decision Tree learns classification rules through feature splits.
• Gradient Descent explains parameter updates using gradients.
• Regularization covers L1 and L2 penalties.
• Confusion Matrix evaluates thresholded classification decisions.
• Recommendation System places click prediction within retrieval and ranking.

Reference

• Google Machine Learning Crash Course: Logistic Regression
• Google Machine Learning Crash Course: Calculating a probability with sigmoid
• Google Machine Learning Crash Course: Loss and regularization
• Google Machine Learning Crash Course: Thresholds and the confusion matrix
• scikit-learn: LogisticRegression