Literature
Display
← All papers

Understanding LSTM Networks

Christopher Olah

Published · Christopher Olah’s blog ↗

Contents

Overview

This visual tutorial explains how an LSTM carries information through time and uses learned gates to write, retain, and expose it. It is an architectural explanation, not a new benchmark study.

Recurrent Neural Networks

Unrolling a recurrent network shows repeated copies of one module passing a state along a sequence. Each prediction can therefore depend on earlier inputs rather than only the current observation. [1]

A recurrent network with a hidden-state loop
Recurrent Neural Networks have loops.
Unrolling a recurrent network through time
An unrolled recurrent neural network.

The Problem of Long-Term Dependencies

Ordinary recurrent updates can learn short contextual dependencies, but learning signals become difficult to propagate over long gaps. LSTMs provide a state path that makes retaining information easier; this is a design advantage, not a guarantee of unlimited memory. [2] [3]

A short dependency between nearby sequence positions
A long dependency between distant sequence positions

LSTM Networks

The repeating LSTM module contains interacting layers rather than one tanh transformation. In the diagrams, arrows carry vectors, merging arrows concatenate them, and circular operators act element by element. [4]

A simple recurrent network using one tanh transformation per step
The repeating LSTM module and its interacting transformations
Diagram legend: layers, pointwise operators, vector transfer, concatenation, and copying

The Core Idea Behind LSTMs

A cell state carries memory across time. Sigmoid gates produce values between zero and one and regulate how much information is removed, added, or exposed. When gates permit it, the state can pass through many steps with limited modification.

The LSTM cell-state path through time
A sigmoid gate multiplying a vector element by element

Step-by-Step LSTM Walk Through

The forget gate scales the previous cell state. The input gate scales a candidate update, and their combination forms the new state. The output gate then selects which components of the transformed state become the hidden output. Each decision depends on the current input and previous hidden state; the original diagrams preserve the complete update equations.

LSTM forget gate and its update equation
LSTM input gate and candidate-state equations
LSTM cell-state update equation
LSTM output gate and hidden-state equations

Variants on Long Short Term Memory

Peephole connections expose the cell state to gates. Coupled gates tie forgetting to writing; a GRU combines these decisions and merges cell and hidden state. The cited comparisons show that performance depends on the task and architectural details, rather than establishing one universal winner. [5] [6] [7] [8] [9]

Peephole LSTM variant with cell-state connections to gates
LSTM variant coupling the input and forget gates
Gated recurrent unit and its update equations

Conclusion

Attention lets a recurrent model consult a larger collection of representations at each step. The tutorial closes by connecting gated memory to attention, grid architectures, and recurrent generative models, reflecting the research landscape in 2015. [10] [11] [12] [13] [14]

Acknowledgments

The acknowledgments credit feedback on the explanation and diagrams, along with earlier contributions to modern LSTM architectures.

Concept demonstration

Open image ↗