EWC (Elastic Weight Consolidation)
Overview
EWC (Elastic Weight Consolidation) is a representative regularization-based method proposed in the field of continual learning. It measures and protects the importance of weights to maintain performance on previous tasks when a neural network learns new tasks. First introduced in a 2017 paper by James Kirkpatrick et al. at DeepMind, it is effective in addressing the catastrophic forgetting problem.
Main Content
1. Background: Catastrophic Forgetting Problem
When a neural network is trained on one task and then learns another, its performance on the previous task typically degrades sharply. This occurs because the network's weights are updated to fit new data, overwriting previously learned patterns. EWC addresses this by measuring how important each weight is to previous tasks and constraining changes to important weights.
2. Core Idea
EWC is inspired by Bayesian inference. When approximating the posterior, the parameter distribution of previous tasks is expressed using Laplace approximation. The importance of each weight is measured by the diagonal elements of the Fisher Information Matrix. The Fisher Information Matrix indicates the sensitivity of weights to the output; larger values mean the weight is more important for previous tasks.
3. Mathematical Formulation
The loss function of EWC is defined as follows:
L(θ) = L_new(θ) + (λ/2) Σ_i F_i (θ_i - θ_old_i)^2
Where:
- L_new(θ) is the loss for the new task
- λ is a hyperparameter controlling the regularization strength
- F_i is the diagonal element of the Fisher Information Matrix for the i-th weight
- θ_old_i is the weight value after learning the previous task
This loss function imposes constraints so that important weights do not change significantly while learning the new task.
4. Implementation Process
1. Learning the first task: Train the neural network in a standard manner, then compute the Fisher Information Matrix after training.
2. Learning the second task: Store the weights and Fisher information from the previous task, then train the new task using the EWC loss function.
3. Repetition: Accumulate and update the Fisher information as each new task is added.
5. Advantages and Limitations
Advantages:
- Relatively simple to implement, with no need to modify the existing neural network structure.
- Memory-efficient: Only weights and Fisher information are retained, without storing data from previous tasks.
- Proven effective across various domains (e.g., image classification, reinforcement learning).
Limitations:
- Uses only the diagonal elements of the Fisher Information Matrix, thus ignoring interactions between weights.
- As the number of tasks increases, the regularization term accumulates, potentially making it harder to learn new tasks.
- High sensitivity to the hyperparameter λ, making it difficult to find the optimal value.
6. Variants and Extensions
- Online EWC: Incrementally updates Fisher information in environments where the task order is not fixed.
- MAS (Memory Aware Synapses): Similar to EWC but uses importance measurement based on output changes.
- SI (Synaptic Intelligence): Tracks the amount of weight change during learning to dynamically compute importance.
Recent Trends
As of 2024–2025, EWC remains an important baseline in continual learning research. Recent studies are combining EWC with memory-based methods (e.g., replay buffers) to improve performance. Additionally, applying EWC to transformer-based models (e.g., GPT, BERT) to preserve pre-trained knowledge during fine-tuning is an active area of research. In particular, EWC has been shown to effectively mitigate catastrophic forgetting in continual learning of large language models (LLMs). In 2025, a new regularization technique called 'EWC++' was proposed, which partially reflects correlations between weights through block-diagonal approximation of the Fisher Information Matrix, drawing attention.
Related Topics
- [[Continual Learning]]
- [[Catastrophic Forgetting]]
- [[Fisher Information Matrix]]
- [[Regularization Techniques]]
- [[Memory-Based Continual Learning]]
---
AI-generated document · Community-driven improvements