Notes on PCA (principal component analysis).

0. What problem does PCA solve? (Why PCA)

High-dimensional data is everywhere, and it typically has:

1. High correlation between dimensions
2. Lots of noise, making the structure hard to observe
3. Downstream models (linear regression / classification) that suffer badly from redundant dimensions
4. A need for dimensionality reduction to lower complexity and improve visualization, computational efficiency and generalization

Goal:

Find a set of new orthogonal directions such that
1) each direction carries as much “information” as possible,
2) the directions are mutually uncorrelated,
3) a small number of directions is enough to approximate the original data.

These directions are the principal components.

1. PCA data preprocessing

Dataset:

Centering:

Reasons:

• It makes variance = second moment:
• The covariance matrix simplifies to
• Otherwise the mean complicates the entire analysis

Define the data matrix:

Covariance matrix (sample estimate):

Properties of the covariance matrix:

• is a symmetric matrix:
• is positive semi-definite: for any ,
• The eigenvalues of are all non-negative real numbers
• The eigenvectors of are mutually orthogonal

2. The core mathematical objective of PCA

2.1 Definition of the projection

Given a direction vector (a column vector, ), project the sample onto :

Convention: throughout this post we use the notation, i.e. a row vector times a column vector gives a scalar.

2.2 Why measure information by variance?

Intuition:

• If all points project to the same location (variance = 0), that direction cannot distinguish the data
• The more spread out the projections (the larger the variance), the more information the direction carries

Mathematical formulation: the information in direction = the variance after projection

Since the data is centered, , so:

2.3 Matrix form of the projected variance

Since is a constant, it can be pulled out of the expectation:

This is the core formula of PCA; every derivation that follows is equivalent to maximizing it.

2.4 The PCA optimization problem

Constraint: to prevent the variance from growing without bound as grows without bound, we need the constraint

So the PCA optimization problem is:

Subsequent principal components need an additional orthogonality constraint (keeping the directions mutually orthogonal):

3. The “three rigorous derivations” of PCA

3.1 Derivation 1: maximizing the projected variance (Rayleigh Quotient)

1) Detailed derivation of the one-dimensional principal component

Step 1: Set up the optimization problem

We want to find a unit vector such that the variance of the data projected onto that direction is maximized:

Step 2: Construct the Lagrangian

Introduce a Lagrange multiplier to handle the constraint:

Step 3: Differentiate and set to zero

Differentiating with respect to requires the following matrix calculus identities:

• (because is symmetric)

Therefore:

Simplifying:

This is exactly an eigenvalue problem! must be an eigenvector of , and is the corresponding eigenvalue.

Step 4: Determine which eigenvector is the optimum

Substitute into the objective function:

Therefore, projected variance = eigenvalue .

To maximize the variance, we should choose the eigenvector corresponding to the largest eigenvalue :

Step 5: Understanding the Rayleigh Quotient

Define the Rayleigh quotient:

For any non-zero vector :

• The maximum of = (the largest eigenvalue)
• The minimum of = (the smallest eigenvalue)
• The extrema are attained exactly at the corresponding eigenvectors

Intuition: the Rayleigh quotient measures how much the vector is “amplified” under the action of the matrix .

2) Derivation of the second principal component

Step 1: Add the orthogonality constraint

The second principal component must satisfy:

1. Unit length:
2. Orthogonality to the first principal component:

Optimization problem:

Step 2: Lagrangian

Step 3: Differentiate

Step 4: Use orthogonality to eliminate

Left-multiply by :

Since is symmetric and is an eigenvector:

We get , i.e. .

Step 5: Substitute back to obtain the eigen-equation

is also an eigenvector of ! Since it must be orthogonal to , must correspond to the second-largest eigenvalue .

3) General derivation for K principal components

The optimization problem in matrix form

Find a matrix satisfying the orthogonality constraint:

Objective function (total variance):

Why the trace?

Solving via the eigendecomposition

Let the eigendecomposition of be:

where is an orthogonal matrix, , and .

Key lemma: for any matrix satisfying :

Equality holds if and only if the columns of span the same subspace as .

Proof sketch:

Let , where satisfies (because is orthogonal).

Let ; then:

Since $Q^\top Q = IK\sum{j=1}^K q_{ij}^2 \leq 1\leq 1$).

To maximize the expression above, the associated with the large should be as large as possible, i.e. choose such that only the first rows are non-zero.

Final conclusion:

The optimal projection matrix consists of the eigenvectors corresponding to the largest eigenvalues.

3.2 Derivation 2: minimizing the reconstruction error (Reconstruction View)

This viewpoint interprets PCA as a best linear approximation problem: approximating high-dimensional data with a low-dimensional subspace.

(1) Detailed derivation of the one-dimensional case

Step 1: Define the reconstruction process

Given a unit direction , each data point is processed as follows:

1. Projection: compute the coordinate along , (a scalar)
2. Reconstruction:

Geometric interpretation: is the orthogonal projection of onto the line through the origin with direction .

Step 2: Compute the reconstruction error of a single point

Expanding this norm:

Step 3: Compute the average reconstruction error

Split it into two parts:

Therefore:

Step 4: Equivalence

Minimizing the reconstruction error maximizing the projected variance!

Intuition:

• Total energy = retained energy + lost energy
• Retaining more ( large) losing less ( small)

(2) Detailed derivation of the K-dimensional case

Step 1: Define the K-dimensional projection space

Let with .

The columns of span a -dimensional subspace.

Step 2: The projection matrix

The projection matrix onto the subspace spanned by is:

Properties of the projection matrix:

• (idempotence)
• (symmetry)

Step 3: The reconstruction process

where is the low-dimensional representation.

Step 4: Compute the reconstruction error

Let (the projection onto the orthogonal complement); then:

(using $P\perp^2 = P\perp, P\perp^\top = P\perp$)

Step 5: Matrix form of the average error

Using the linearity of the trace:

Therefore:

Step 6: The optimal solution

By the conclusion of Derivation 1, the optimal solution is:

Information retention ratio:

This ratio is commonly used to choose the size of (e.g. retaining 95% of the variance).

3.3 Derivation 3: the SVD view (the most practical and most stable)

This viewpoint starts directly from the data matrix and uses the singular value decomposition to establish the connection with PCA.

(1) SVD recap

Any matrix can be decomposed as:

where:

• : the matrix of left singular vectors,
• : a diagonal matrix (singular values )
• : the matrix of right singular vectors,

Economy SVD (when ):

where , , .

(2) Relationship between the SVD and the covariance matrix

Covariance matrix:

(using )

Let ; then:

where satisfies:

Core conclusions:

• The eigenvectors of = the left singular vectors of
• The eigenvalues of = the squared singular values divided by

Therefore, the PCA principal component directions = the left singular vectors of the SVD!

(3) Best low-rank approximation (the Eckart-Young theorem)

Theorem: under the Frobenius norm, the best rank- approximation of is:

where , , are the parts corresponding to the first singular values.

Approximation error:

Connection with PCA:

The PCA reconstruction:

One can verify that this is equivalent to the SVD low-rank approximation:

(4) Why is the SVD more stable than the eigendecomposition?

Dual PCA (when ):

Compute the eigendecomposition of (much smaller than ):

Then recover the left singular vectors:

(5) Algorithm for implementing PCA via SVD

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

输入: 数据矩阵 X ∈ R^{D×N}，目标维度 K

1. 中心化
   μ = mean(X, axis=1)           # 每个特征的均值
   X_centered = X - μ            # 广播减去均值

2. SVD 分解
   U, Σ, V^T = SVD(X_centered)   # 完整 SVD
   或
   U_K, Σ_K, V_K^T = truncated_SVD(X_centered, K)  # 截断 SVD（更高效）

3. 提取主成分
   W = U[:, :K]                  # 前 K 个左奇异向量

4. 投影（降维）
   Z = W^T @ X_centered          # Z ∈ R^{K×N}

5. 重建（可选）
   X_reconstructed = W @ Z + μ

输出: 
   - W: 主成分方向矩阵 (D × K)
   - Z: 低维表示 (K × N)
   - λ = σ²/N: 各主成分的方差（特征值）

4. Geometric interpretation of PCA

4.1 The ellipsoid and its principal axes

The covariance matrix defines an ellipsoid:

Principal axes of the ellipsoid:

• Directions = the eigenvectors of
• Lengths ∝ (the square roots of the eigenvalues)

The principal components found by PCA are exactly the principal axes of this ellipsoid!

4.2 Rotating into the principal-axis coordinate system

Covariance matrix of the original data:

After the projection , the covariance in the new coordinates is:

A diagonal matrix! This means:

• The principal components are uncorrelated with one another
• The variance of the -th principal component =

4.3 Geometric meaning of the maximum-variance direction

Imagine the data points forming a cloud:

• First principal component: the direction in which the cloud is “longest”
• Second principal component: the longest direction within the plane perpendicular to the first principal component
• And so on…

1
2
3
4
5
6
7
8
9

       PC1
        ↑
        |  . . .
    . . | . . . .
  . . . | . . . . .
. . . . * . . . . . → PC2
  . . . | . . . . .
    . . | . . .
        |

5. Practical applications of PCA

5.1 Python implementation

From scratch

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

import numpy as np

def pca_from_scratch(X, n_components):
    """
    X: 数据矩阵，形状 (n_samples, n_features)
    n_components: 目标维度
    """
    # 1. 中心化
    mean = np.mean(X, axis=0)
    X_centered = X - mean
    
    # 2. SVD 分解
    # X_centered: (N, D)，需要转置使用我们的符号约定
    U, s, Vt = np.linalg.svd(X_centered, full_matrices=False)
    
    # 3. 主成分方向（Vt 的前 K 行，即 V 的前 K 列）
    components = Vt[:n_components]  # (K, D)
    
    # 4. 投影
    X_transformed = X_centered @ components.T  # (N, K)
    
    # 5. 计算解释方差比
    explained_variance = (s ** 2) / (X.shape[0] - 1)
    explained_variance_ratio = explained_variance / explained_variance.sum()
    
    return {
        'components': components,
        'transformed': X_transformed,
        'explained_variance': explained_variance[:n_components],
        'explained_variance_ratio': explained_variance_ratio[:n_components],
        'mean': mean
    }

# 使用示例
np.random.seed(42)
X = np.random.randn(100, 5) @ np.array([[3, 0, 0, 0, 0],
                                         [0, 2, 0, 0, 0],
                                         [0, 0, 1, 0, 0],
                                         [0, 0, 0, 0.5, 0],
                                         [0, 0, 0, 0, 0.1]])
result = pca_from_scratch(X, n_components=2)
print("解释方差比:", result['explained_variance_ratio'])

Using scikit-learn

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np

# 数据预处理
X = np.random.randn(100, 10)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# PCA
pca = PCA(n_components=0.95)  # 保留 95% 方差
X_pca = pca.fit_transform(X_scaled)

print(f"原始维度: {X.shape[1]}")
print(f"降维后维度: {X_pca.shape[1]}")
print(f"各主成分解释方差比: {pca.explained_variance_ratio_}")
print(f"累计解释方差比: {np.cumsum(pca.explained_variance_ratio_)}")

5.2 Application scenarios

6. Limitations and extensions of PCA

6.1 Limitations of PCA

6.2 Extended algorithms

(1) Probabilistic PCA (PPCA)

The probabilistic model view:

Advantages:

• Can handle missing values
• Provides a probabilistic interpretation
• Can be solved with the EM algorithm

(2) Sparse PCA

Add a sparsity constraint:

Advantage: the principal components are more interpretable (each involves only a few original features)

1
2
3
4

from sklearn.decomposition import SparsePCA

spca = SparsePCA(n_components=2, alpha=0.1)
X_spca = spca.fit_transform(X)

(3) Incremental PCA

Handles large-scale data by computing in batches:

1
2
3
4
5
6

from sklearn.decomposition import IncrementalPCA

ipca = IncrementalPCA(n_components=2, batch_size=100)
for batch in data_generator:
    ipca.partial_fit(batch)
X_ipca = ipca.transform(X)

(4) Robust PCA

Decompose into low-rank + sparse:

Used to remove sparse noise / outliers.

7. Summary

7.1 Unifying the three derivations

All three are equivalent!

7.2 Quick reference for the core formulas

Translated from the Chinese original.