transformer
Sign in to saveAlso known as transformer model, transformer architecture, transformers
machine-learning model architecture first developed by Google Brain
Wikidata facts
- Has part
- decoder
- Follows
- long short-term memory
- Developer
- Noam Shazeer
- Has use
- automatic summarization
Show 4 more facts
- described by source
- Attention Is All You Need
- Stack Exchange tag
- ai.stackexchange.com/tags/transformer
- time of discovery or invention
- 2017-06-12
Sources (2)
via Wikidata · CC0
~40 min read
Encyclopedic overview
A standard transformer architecture. Many modern diagrams show the pre-layer normalization (pre-LN) convention, while the original 2017 paper used post-layer normalization (post-LN).
In deep learning, the transformer is a family of artificial neural network architectures built around the attention mechanism. Transformers were introduced to model sequential data without recurrence and without convolutions, allowing much more parallel computation during training. They are now a dominant architecture for natural language processing, computer vision, speech processing, multimodal learning, robotics, and many other sequence-modelling tasks.
Excerpted from Wikipedia’s “transformer” article, available under the CC BY-SA 4.0 licence.