Principal Component Analysis (PCA)

A statistical technique that reduces dimensionality by linearly transforming high-dimensional data onto principal component axes along which the variance is maximized

Principal Component Analysis (PCA)

Overview

Principal Component Analysis (PCA) is a statistical technique that reduces dimensionality by linearly transforming high-dimensional data composed of many variables into a small number of mutually orthogonal principal component axes. It was first proposed from a geometric perspective by Karl Pearson in 1901, and established in its modern form by Harold Hotelling in 1933. Because it seeks directions that preserve the variance of the data as much as possible, it can simultaneously achieve visualization, noise removal, and improved computational efficiency while minimizing information loss, and it has become a standard tool in virtually every data-driven field, including statistics, machine learning, bioinformatics, and neuroscience.

Key Concepts

Definition and Intuition

The core idea of PCA is that "the direction in which the data spreads most widely carries the most information." For example, two strongly correlated variables such as height and weight can in effect be summarized as a single axis of information. PCA creates new axes (principal components) as linear combinations of the original variables, and arranges them as PC1, PC2, PC3… in order of decreasing variance. By retaining only the top few principal components, the number of variables can be greatly reduced while most of the structure of the original data is preserved.

Mathematical Principles

1. Standardization: If variables have different units, variables with large variance dominate the result, so data are normalized to mean 0 and standard deviation 1.

2. Covariance matrix computation: The covariance matrix Σ between variables is computed.

3. Eigenvalue decomposition: The eigenvalues λ and eigenvectors v satisfying Σv = λv are computed. The eigenvalue represents the amount of variance explained by that principal component, and the eigenvector represents the direction of the axis.

4. Projection: By selecting the k eigenvectors in order of decreasing eigenvalue and projecting the data onto them, the reduction to k dimensions is complete.

In practice, singular value decomposition (SVD) is often used instead of the covariance matrix. This is because SVD is numerically more stable and is applicable even when there are missing values or when the number of variables exceeds the number of samples.

Explained Variance and the Scree Plot

The proportion of the total variance accounted for by each principal component is called the explained variance ratio. In a scree plot, with the principal component number on the x-axis and the eigenvalue on the y-axis, the "elbow" point at which the curve flattens out serves as a criterion for the appropriate number of principal components. Setting a threshold of 80–90% cumulative explained variance is also widely used.

Main Fields of Application

Dimensionality reduction and visualization: Projecting tens to hundreds of dimensions onto a two-dimensional plane to check for cluster structure.
Noise removal: Since principal components with small variance are usually noise, discarding them raises the signal-to-noise ratio.
Feature extraction preprocessing: Mitigating multicollinearity and increasing training speed before feeding data into regression or classification models.
Bioinformatics: Diagnosing and correcting sample heterogeneity and batch effects in gene expression data (GTEx, TCGA).
Face recognition: It became the basis of the classical eigenface method.

Advantages and Disadvantages

Its advantages are that it is simple to implement, fast to compute, and that the resulting axes are mutually orthogonal, making interpretation clear. On the other hand, because principal components are linear combinations of the original variables, it is difficult to assign physical or biological meaning to them, and a large variance does not guarantee that the information is useful for prediction. PCA is also limited to linear transformations, is sensitive to outliers, and does not take class labels into account.

Recent Trends

Entering 2024–2025, PCA is being re-examined as a preprocessing and interpretation tool rather than as a standalone technique. First, interpretability research is active, using PCA to reduce the embedding vectors of large language models (LLMs) in order to visualize semantic space or to extract the direction of specific concepts within activation space. Second, in single-cell RNA sequencing (scRNA-seq) analysis pipelines (Seurat, Scanpy), PCA is still used as the standard step preceding UMAP and t-SNE, and variants such as augmented PCA and probabilistic PCA are used for batch effect correction. Third, nonlinear, sparse, and robust variants such as kernel PCA, sparse PCA (SPCA), and robust PCA are being combined with autoencoders and applied to high-dimensional financial and medical data. Fourth, in federated learning environments, distributed and privacy-preserving algorithms that perform PCA without sharing raw data are drawing attention. Meanwhile, as data scale has grown, scalability techniques such as randomized SVD and Incremental PCA have become practical standards.