📝 Linear Algebra

Description

< What is it? >

Linear algebra represents data and model parameters as vectors, matrices, and tensors. It provides operations for transforming these representations and comparing them, including the vector similarity used in search.

Key points

< Tensors, axes, and shape >

• Definition: in machine-learning libraries, a tensor is a numerical array organized along zero or more axes. Scalars, vectors, and matrices are special cases. Tensors store model inputs, parameters, intermediate representations, and outputs. See the PyTorch tensor introduction.

• Shape: a tensor's shape lists the size of each axis. The number of axes is its number of dimensions, available as ndim in PyTorch. Axis meanings depend on the application and its data layout.

  Number of axes	Representation	Example	Shape
  0	Scalar	One score, such as 0.9	()
  1	Vector	Three features: [0.2, 0.5, 0.8]	(3,)
  2	Matrix	Two examples with three features each	(2, 3)
  3	Three-dimensional tensor	One RGB image, with channels first	(3, height, width)
  4	Four-dimensional tensor	A batch of RGB images, with channels first	(batch_size, 3, height, width)

• Embedding dimension: a 512-dimensional embedding contains 512 components. Stored alone, it is a one-dimensional tensor with shape (512,); a batch of $B$ such embeddings has shape (B, 512) and two axes. Embedding dimension counts coordinates per vector, while tensor dimensionality counts axes. See Google's embedding-space introduction.

• Matrix rank: the number of tensor axes is separate from matrix rank, which counts linearly independent rows or columns. A matrix is a two-dimensional tensor regardless of its matrix rank.

• CLIP example: with the openai/clip-vit-base-patch32 checkpoint, one processed image has shape (1, 3, 224, 224), its image embedding has shape (1, 512), and its scores against ten descriptions have shape (1, 10). The leading 1 is the image batch size. These shapes describe different stages of the CLIP classification example.

< Matrix rank >

• Definition: the rank of a matrix is the maximum number of linearly independent columns, which also equals the maximum number of linearly independent rows. Equivalently, it is the dimension of the matrix's column space: all vectors that can be formed by scaling and adding its columns. See Interactive Linear Algebra: The Rank Theorem.

• Linear independence: a collection of vectors is linearly independent when none can be written as a linear combination of the others. A linear combination means multiplying vectors by numbers and adding them. For example, $[2,4]$ is twice $[1,2]$, so these two vectors provide only one independent direction. See Linear Independence.

• Examples: matrix size alone does not determine rank.

  Matrix	Rank	Explanation
  $\begin{bmatrix}0&0\\0&0\end{bmatrix}$	0	Both columns are zero; neither contributes an independent direction
  $\begin{bmatrix}1&2\\2&4\end{bmatrix}$	1	Column 2 is twice column 1
  $\begin{bmatrix}1&0\\0&1\end{bmatrix}$	2	Neither column can be formed from the other
  $\begin{bmatrix}1&0&1\\0&1&1\\0&0&0\end{bmatrix}$	2	Columns 1 and 2 are independent; column 3 is their sum

  The last example shows that dependence can involve a sum of columns, even when no two columns are scalar multiples of each other.

• Geometric meaning: in $y=Mx$, the output is a linear combination of the columns of $M$. Rank counts the independent directions those outputs can occupy. A rank-1 matrix maps inputs onto a line through the origin; a rank-2 matrix maps them onto a plane through the origin.

• Full rank and low rank: for a matrix with $m$ rows and $n$ columns:

  $$0\leq\operatorname{rank}(M)\leq\min(m,n)$$

  A matrix is full rank when its rank equals $\min(m,n)$, and rank deficient when it is smaller. Low rank generally means the rank is much smaller than the matrix dimensions. A $4096\times4096$ matrix may have rank anywhere from 0 to 4096, regardless of its 16,777,216 entries. Rank is also separate from sparsity: a dense matrix can have rank 1, while a sparse identity matrix is full rank.

• Calculating rank by hand: row-reduce the matrix and count the pivots, or equivalently the nonzero rows in row-echelon form. For the rank-1 example, subtract twice row 1 from row 2:

  $$\begin{bmatrix}1&2\\2&4\end{bmatrix} \xrightarrow{R_2\leftarrow R_2-2R_1} \begin{bmatrix}1&2\\0&0\end{bmatrix} \quad\Longrightarrow\quad\operatorname{rank}(M)=1$$

• Connection to LoRA: if $B$ has shape $d_{\mathrm{out}}\times r$ and $A$ has shape $r\times d_{\mathrm{in}}$, every column of $BA$ is a combination of the $r$ columns of $B$. Therefore:

  $$\operatorname{rank}(BA)\leq r$$

  With $r=8$, a $4096\times4096$ product can have at most eight independent column directions. Low-Rank Adaptation (LoRA) applies this constraint to the learned weight update. The sum of the frozen base matrix and that update can still have full rank.

< Sparse and dense vectors >

A vector is an ordered sequence of numbers. A sparse vector has mostly zero components; a dense vector has mostly nonzero components. These terms describe the distribution of values, not the vector's dimension. A dense vector can contain some zeros.

• Example: both vectors below have eight dimensions:

  Sparse: [0, 0, 3, 0, 0, 0, 2, 0]
  Dense:  [0.2, -0.5, 0.8, 0.1, -0.3, 0.6, 0.4, 0.9]

  The sparse vector has two nonzero components and six zeros, so it is 75% sparse. The dense vector has a value other than zero in every position.

In TF-IDF, a vocabulary of 10,000 terms gives every document 10,000 vector components. A document containing only 100 distinct vocabulary terms has at most 100 nonzero weights. A one-hot vector, with one component equal to 1 and all others zero, is another sparse representation.

Neural embeddings commonly use dense vectors whose components are learned numerical features. For example, a 768-dimensional text embedding can encode information across its components rather than assigning one component to every vocabulary word. Google's Machine Learning Crash Course introduces dense embeddings as representations of sparse inputs. Density alone does not imply that a vector is learned or captures meaning.

< Sparse values and storage >

Dense storage records every component, including zeros. Sparse storage records nonzero values and their positions, together with the overall shape, leaving other entries implicitly zero. This can save memory and computation when relatively few components are nonzero; sparse formats also require indexing overhead. See SciPy's sparse-array guide.

• Example: the eight-dimensional sparse vector above can be represented using zero-based positions:

  Dimension: 8
  Nonzero values: {2: 3, 6: 2}

  It still has eight dimensions, even though only two values are recorded. Writing out all eight entries changes its storage representation, not its mathematical values. Converting sparse storage to dense storage does not create a learned embedding.

< Dot product >

The dot product takes two real vectors of the same dimension, multiplies corresponding components, and adds the results. Its output is one number, called a scalar:

$$\mathbf{a}\cdot\mathbf{b}=\mathbf{a}^{\mathsf T}\mathbf{b}=\sum_{i=1}^{n}a_i b_i$$

The superscript $\mathsf T$ means transpose: a column vector becomes a row vector for matrix multiplication.

• Example: for $\mathbf{a}=[1,2]$ and $\mathbf{b}=[3,4]$:

  $$\mathbf{a}\cdot\mathbf{b}=1\times3+2\times4=11$$

For nonzero vectors, the geometric interpretation is:

$$\mathbf{a}\cdot\mathbf{b}=\|\mathbf{a}\|_2\|\mathbf{b}\|_2\cos(\theta)$$

The dot product therefore depends on both magnitude and direction. It is positive for an acute angle, zero for perpendicular vectors, and negative for an obtuse angle. It is not restricted to the range $[-1,1]$. See Wolfram MathWorld: Dot Product.

< Inner product >

An inner product is the broader mathematical concept used to define lengths, angles, and orthogonality in a vector space. It is written $\langle\mathbf{a},\mathbf{b}\rangle$. The ordinary dot product is the standard inner product for real coordinate vectors, so the terms often refer to the same operation in machine learning.

For real vectors, an inner product must be linear in each argument, symmetric, and positive definite: $\langle\mathbf{a},\mathbf{a}\rangle>0$ for every nonzero vector, and zero for the zero vector. Its induced norm is $\|\mathbf{a}\|=\sqrt{\langle\mathbf{a},\mathbf{a}\rangle}$. Inner products also extend to spaces of functions and complex vectors. See Wolfram MathWorld: Inner Product.

• Example: a weighted inner product can give coordinates different importance. With fixed positive weights $w_i$:

  $$\langle\mathbf{a},\mathbf{b}\rangle_w=\sum_i w_i a_i b_i$$

  For $\mathbf{a}=[1,2]$, $\mathbf{b}=[3,4]$, and weights $[2,1]$, the result is $2\times1\times3+1\times2\times4=14$. Setting all weights to 1 gives the ordinary dot product.

< Cosine similarity >

Cosine similarity measures how closely two nonzero vectors point in the same direction. It compares their direction rather than their magnitude:

$$\operatorname{cosine\ similarity}(\mathbf{a},\mathbf{b}) =\frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|_2\|\mathbf{b}\|_2} =\cos(\theta)$$

Here, $\theta$ is the angle between the vectors. Cosine similarity normalizes the dot product by the two vector lengths. Each L2 norm, such as $\|\mathbf{a}\|_2=\sqrt{\sum_i a_i^2}$, measures a vector's length. See Google's explanation of similarity measures.

Score	Meaning
1	Same direction
0	Perpendicular; no directional alignment
−1	Opposite directions

• Example: compare these two-dimensional vectors:

  Vector A	Vector B	Similarity	Why
  [1, 1]	[2, 2]	1	Same direction, despite different magnitudes
  [1, 0]	[0, 1]	0	Perpendicular
  [1, 0]	[-1, 0]	−1	Opposite directions

Dividing by vector lengths prevents a vector from scoring higher merely because all its weights are proportionally larger. The score is not a probability.

< Similarity in search >

In TF-IDF search, each component represents a term's weight. Cosine similarity compares the query's pattern of weights with each document's pattern. Because these weights are nonnegative, the scores range from 0 to 1; zero means no shared terms with nonzero weights.

Cosine similarity also compares learned embeddings, where the meaning of similarity depends on what the model learned. Google's Machine Learning Crash Course explains how vector representations capture relationships between items.

Comparison

< Sparse vs. dense vectors >

Aspect	Sparse vector	Dense vector
Values	Mostly zero	Mostly nonzero
Typical examples	One-hot encoding, TF-IDF	Neural embeddings, many numerical feature vectors
Dimension	Can be small or large; often vocabulary-sized in text search	Can be small or large; chosen by the model for embeddings
Suitable storage	Nonzero positions and values, plus shape	An array containing every component
Similarity	Supports dot product and cosine similarity	Supports the same operations

Dot products and cosine similarities require compatible coordinates: the same dimension and the same feature or embedding space. A sparse vector and a dense vector can be compared when these conditions hold.

< Dot product, inner product, and cosine similarity >

• Example vectors: use $\mathbf{a}=[1,2]$ and $\mathbf{b}=[3,4]$ throughout. Their L2 norms are $\sqrt{5}$ and $5$.

  Concept	Meaning	Relationship	Example	Pros	Cons / limitations
  Dot product	Sum of corresponding component products for real vectors	The standard Euclidean inner product	$\mathbf{a}\cdot\mathbf{b}=1\times3+2\times4=11$	Simple to compute; preserves magnitude information when vector length is meaningful	Larger vector norms can dominate scores; sensitive to feature scales and has no fixed score range
  Inner product	A scalar-valued operation satisfying the inner-product properties	Includes the dot product and other choices, such as positive weighted products	With positive weights $[2,1]$, $\langle\mathbf{a},\mathbf{b}\rangle_W=2\times1\times3+1\times2\times4=14$	Can define a geometry that gives selected features more importance; provides consistent notions of length and angle	Requires a justified choice of geometry; weights must satisfy inner-product conditions, and scores depend on the chosen scaling
  Cosine similarity	Dot product divided by the two L2 norms	Measures directional alignment; equals the dot product when both vectors have unit length	$\cos(\mathbf{a},\mathbf{b})=11/(\sqrt{5}\times5)\approx0.9839$	Compares direction independently of vector magnitude; bounded between −1 and 1	Discards potentially useful magnitude information; undefined for zero vectors and still sensitive to unequal rescaling of individual features

  The weighted inner product uses $W=\operatorname{diag}(2,1)$; choosing the identity matrix instead gives the ordinary dot product, 11. The cosine example uses the standard Euclidean dot product and norms.

< Similarity and distance measures >

• Different data, different measures: numerical vectors, sets, and strings require different comparisons. A larger similarity score generally indicates a closer match; a smaller distance indicates a closer match.

  Method	What it measures	Typical use	Example
  Dot product	Vector alignment and magnitude	Embedding retrieval and recommendation models trained with dot-product scores	[1,2] and [3,4]: $1\times3+2\times4=11$
  Euclidean distance (L2)	Straight-line distance between vectors	Nearest-neighbor search	[1,2] and [4,6]: $\sqrt{3^2+4^2}=5$
  Manhattan distance (L1)	Sum of absolute coordinate differences	Comparing numerical feature vectors	[1,2] and [4,6]: $3+4=7$
  Jaccard similarity	Set intersection size divided by set union size	Comparing tags, categories, or sets of words	horror, drama vs. horror, comedy: one shared tag among three distinct tags, giving $1/3$
  Hamming distance	Number of differing positions	Binary codes or equal-length sequences	1011 vs. 1001: 1 differing position
  Edit distance (Levenshtein)	Minimum number of single-character insertions, deletions, and substitutions	Spelling comparison and name matching	cat → cart: 1 insertion
  Pearson correlation	Alignment after centering each vector by its mean	Comparing users' rating patterns on commonly rated items	[1,2,3] and [3,4,5]: correlation 1 despite different rating levels

  Hamming distance above is a count; some libraries return the fraction of positions that differ. Pearson correlation requires nonconstant vectors. Numerical distances depend on feature scales, so preprocessing should reflect the meaning of each coordinate.

• Normalized-vector equivalence: when the query and all candidate vectors have unit L2 norm, cosine similarity, dot product, and Euclidean distance produce the same nearest-neighbor ordering, with distance sorted ascending and similarity sorted descending. For unit vectors $\mathbf{u}$ and $\mathbf{v}$:

  $$\|\mathbf{u}-\mathbf{v}\|_2^2 =2-2\mathbf{u}^{\top}\mathbf{v} =2-2\operatorname{cosineSimilarity}(\mathbf{u},\mathbf{v})$$

  Without normalization, their rankings can differ because dot product and Euclidean distance depend on vector magnitude. Use the comparison intended by the embedding model's training objective; normalization can remove useful magnitude information.

• Embedding metric selection: see Choosing a similarity metric for Gemini embeddings for a table of cosine, L2, dot product, L1, and Hamming, including their pros, cons, and normalization requirements.

< linear_kernel() vs. cosine_similarity() >

• Different operations: both scikit-learn functions compare rows of two feature matrices. With X shaped (m, d) and Y shaped (n, d), the result has shape (m, n): one score per pair of rows.

  Function	Calculation for two rows	Effect of vector magnitude	Example: [1, 2] and [3, 4]
  linear_kernel(X, Y)	Dot product: $X Y^\top$	Retains magnitude	$1\times3+2\times4=11$
  cosine_similarity(X, Y)	Dot product divided by the two L2 norms	Removes magnitude for nonzero vectors	$11/(\sqrt{5}\times5)\approx0.9839$

  See the official linear_kernel and cosine_similarity documentation.

• When the results match: if all nonzero rows in both inputs have unit L2 norm, the dot product equals cosine similarity, up to floating-point rounding. linear_kernel does not normalize its inputs; it can avoid redundant normalization when vectors are already normalized.

• Choosing for TF-IDF: TfidfVectorizer defaults to norm="l2", so its document and query vectors can be compared with either function for equivalent scores. If normalization is disabled, changed to L1, or lost after later transformations, use cosine_similarity when directional similarity is intended. Use linear_kernel when the intended score is the unnormalized dot product.

• Sparse and zero vectors: both accept sparse matrices and default to dense output. With both inputs sparse, dense_output=False permits sparse output. Scikit-learn returns zero for cosine comparisons involving a zero vector, although mathematical cosine similarity is undefined there.

Implementation

< Inspect a tensor in PyTorch >

• Sample input: two examples, each represented by three numerical features. shape reports the axis sizes, ndim counts the axes, and numel() counts all stored elements.

  import torch
  
  x = torch.tensor([
      [0.2, 0.5, 0.8],
      [0.1, 0.4, 0.9],
  ])
  
  print("Shape:", tuple(x.shape))
  print("Number of axes:", x.ndim)
  print("Number of elements:", x.numel())
  print("First example shape:", tuple(x[0].shape))

• Verified output:

  Shape: (2, 3)
  Number of axes: 2
  Number of elements: 6
  First example shape: (3,)

• Interpretation: the full tensor has two axes: examples and features. Selecting x[0] returns one example as a vector with three components and one axis.

< Calculate matrix rank with NumPy >

• Runnable example: use numpy.linalg.matrix_rank() on the matrices above and the rank-1 product from the LoRA example. Install NumPy with pip install numpy if needed. The @ operator performs matrix multiplication.

  import numpy as np
  
  matrices = {
      "zero": [[0, 0], [0, 0]],
      "dependent": [[1, 2], [2, 4]],
      "full_rank": [[1, 0], [0, 1]],
      "column_sum": [[1, 0, 1], [0, 1, 1], [0, 0, 0]],
  }
  for name, values in matrices.items():
      matrix = np.array(values, dtype=float)
      print(f"{name}: shape={matrix.shape}, rank={np.linalg.matrix_rank(matrix)}")
  
  B = np.array([[1], [2], [3], [4]])
  A = np.array([[1, 0, 2, 0]])
  update = B @ A
  print(f"B @ A: shape={update.shape}, rank={np.linalg.matrix_rank(update)}")

• Verified output:

  zero: shape=(2, 2), rank=0
  dependent: shape=(2, 2), rank=1
  full_rank: shape=(2, 2), rank=2
  column_sum: shape=(3, 3), rank=2
  B @ A: shape=(4, 4), rank=1

• Numerical precision: NumPy estimates rank using Singular Value Decomposition (SVD), counting singular values above a tolerance. For nearly dependent columns, the tolerance affects the result. See the NumPy matrix_rank documentation. The matrix's array dimension, matrix.ndim, is 2 in all these examples; it counts array axes rather than independent directions.

< Compare scikit-learn similarity functions >

• Input and code: install scikit-learn with pip install scikit-learn. Each matrix below contains one vector; [0, 0] selects the single pairwise score.

  from sklearn.metrics.pairwise import linear_kernel, cosine_similarity
  from sklearn.preprocessing import normalize
  
  X = [[1.0, 2.0]]
  Y = [[3.0, 4.0]]
  print(f"Raw dot product: {linear_kernel(X, Y)[0, 0]:.4f}")
  print(f"Cosine similarity: {cosine_similarity(X, Y)[0, 0]:.4f}")
  
  X_unit, Y_unit = normalize(X), normalize(Y)
  print(f"Normalized dot product: {linear_kernel(X_unit, Y_unit)[0, 0]:.4f}")
  print(f"Normalized cosine: {cosine_similarity(X_unit, Y_unit)[0, 0]:.4f}")

• Expected output (calculated from the formulas; scikit-learn is not installed in the verification environment):

  Raw dot product: 11.0000
  Cosine similarity: 0.9839
  Normalized dot product: 0.9839
  Normalized cosine: 0.9839

< Dot product and weighted inner product in Python >

• Example: these functions operate on real vectors and require no external libraries:

  def dot_product(a, b):
      if len(a) != len(b):
          raise ValueError("Vectors must have the same dimension")
      return sum(x * y for x, y in zip(a, b))
  
  def weighted_inner_product(a, b, weights):
      if not (len(a) == len(b) == len(weights)):
          raise ValueError("Vectors and weights must have the same dimension")
      if any(w <= 0 for w in weights):
          raise ValueError("Weights must be positive")
      return sum(w * x * y for w, x, y in zip(weights, a, b))
  
  a, b = [1, 2], [3, 4]
  print("Dot product:", dot_product(a, b))
  print("Standard inner product:", weighted_inner_product(a, b, [1, 1]))
  print("Weighted inner product:", weighted_inner_product(a, b, [2, 1]))

  Expected output:

  Dot product: 11
  Standard inner product: 11
  Weighted inner product: 14

< Cosine similarity in Python >

• Example: this standard-library implementation shows the dot product and normalization directly:

  from math import hypot
  
  def cosine_similarity(a, b):
      if len(a) != len(b):
          raise ValueError("Vectors must have the same dimension")
      norm_a, norm_b = hypot(*a), hypot(*b)
      if norm_a == 0 or norm_b == 0:
          raise ValueError("Cosine similarity requires nonzero vectors")
      dot_product = sum(x * y for x, y in zip(a, b))
      return dot_product / (norm_a * norm_b)
  
  pairs = [
      ([1, 1], [2, 2]),
      ([1, 0], [0, 1]),
      ([1, 0], [-1, 0]),
      ([1, 0], [1, 1]),
  ]
  for a, b in pairs:
      print(f"{a} vs {b}: {cosine_similarity(a, b):.3f}")

  Expected output:

  [1, 1] vs [2, 2]: 1.000
  [1, 0] vs [0, 1]: 0.000
  [1, 0] vs [-1, 0]: -1.000
  [1, 0] vs [1, 1]: 0.707

Cosine similarity is mathematically undefined for a zero-length vector, so this example raises an error. Scikit-learn's cosine_similarity instead returns zero for comparisons involving a zero vector and supports comparing whole matrices, including sparse TF-IDF matrices.

Related ideas

• Low-Rank Adaptation (LoRA) uses low-rank matrix products to parameterize weight updates.
• Mathematical Foundations: Vector norms explains L1 and L2 norms.
• Term Frequency–Inverse Document Frequency (TF-IDF) demonstrates ranking documents by cosine similarity.
• Embeddings explains learned vector representations.

Reference

• PyTorch: Tensors
• Interactive Linear Algebra: The Rank Theorem
• Interactive Linear Algebra: Linear Independence
• NumPy: matrix_rank
• Google Machine Learning Crash Course: Embeddings
• SciPy: Sparse arrays
• Wolfram MathWorld: Dot Product
• Wolfram MathWorld: Inner Product
• Google Machine Learning Crash Course: Embedding space
• Google: Measuring similarity from embeddings
• Scikit-learn: cosine_similarity