📝 Linear Regression

Description

< What is it? >

• [Definition]

  • Linear regression is a statistical method used to model the relationship between a dependent variable and one or more independent variables. It assumes a linear relationship between the variables, meaning that changes in the independent variables are associated with proportional changes in the dependent variable. The goal of linear regression is to find the best-fitting line (or hyperplane in higher dimensions) that minimizes the difference between the predicted values and the actual observed values.

• [Model]

  For a single prediction: ŷ = θ₀ + θ₁x₁ + θ₂x₂ + ... + θₙxₙ

  • Matrix form, ŷ = Xθ, where:

    • ŷ (y-hat) is the vector of predicted values.

    • X (Design Matrix) - Shape: m x (n + 1)

      X is the design matrix of input features (including a column of 1s for the intercept/bias):

      • m = number of samples (rows)
      • n = number of features
      • First column is all 1s (for the intercept θ₀)

    • θ (Parameter Vector) - Shape: (n + 1) x 1

      θ (theta) is the vector of parameters (weights/coefficients), including the intercept:

      • θ₀ = intercept
      • θ₁ = coefficient for feature 1
      • θ₂ = coefficient for feature 2
      • ...

Key points

< Main advantages >

• The main advantages of linear regression include its simplicity, interpretability, and efficiency in modeling relationships between variables. It is widely used for predictive modeling and can provide insights into the strength and direction of relationships between independent and dependent variables.

< Functions >

• We find the optimal θ by minimizing the cost function (Mean Squared Error): J(θ) = (1/2m) * Σ(ŷᵢ - yᵢ)²

< Gradient Descent >

• Gradient Descent is an iterative optimization algorithm used to minimize the cost function by updating the parameters
• The update rule for gradient descent is:

    θ := θ - α * ∇J(θ) = θ − α * 1 / m * Xᵀ(Xθ − y)

• where α is the learning rate and ∇J(θ) is the gradient of the cost function with respect to θ.
• The same rule drives every model trained by optimization. For mini-batching, learning-rate schedules, momentum and Adam, see Gradient Descent.

Implementation

< Linear Regression Using Normal Equation >

Question: Write a Python function that performs linear regression using the normal equation. The function should take a matrix X (features) and a vector y (target) as input, and return the coefficients of the linear regression model. Round your answer to four decimal places, -0.0 is a valid result for rounding a very small number.
Input: X = [[1, 1], [1, 2], [1, 3]], y = [1, 2, 3]
Output: [0.0, 1.0]

• ŷ = Xθ
• θ = (XᵀX)⁻¹ Xᵀy
• Here matrix X is not rectangular but m x (n + 1), so inverse matrix X⁻¹ does not exist. In this case, we can use the pseudo-inverse as below, it works like the inverse matrix: (XᵀX)⁻¹ Xᵀ
  

Solution:

import numpy as np
def linear_regression_normal_equation(X: list[list[float]], y: list[float]) -> list[float]:
    X = np.array(X)
    y = np.array(y).reshape(-1, 1)
    X_transpose = X.T
    theta = np.linalg.inv(X_transpose.dot(X)).dot(X_transpose).dot(y)
    theta = np.round(theta, 4).flatten().tolist()
    return theta

Crash course

Reference