> ## Content Index
> Fetch the complete content index at: https://www.anup.io/llms.txt
> Use this file to discover other available public pages before exploring further.

# LLM Transformer Overview
- URL: https://www.anup.io/llm-transformer-overview/
- Published: 2025-01-01T20:06:29.000Z
- Updated: 2025-08-31T13:59:22.000Z
- Description: ..for the busy AI engineer
- Author: Anup Jadhav
- Tags: LLM Fundamentals, Machine Learning

## Introduction

Large Language Models (LLMs) are built on the Transformer architecture introduced in [*Attention Is All You Need*](https://arxiv.org/abs/1706.03762?ref=anup.io) (AIAYN) paper in 2017.

Transformers are based solely on **attention mechanisms** and dispense with recurrence (as used in Recurrent Neural Networks or RNNs) and convolutions (as used in Convolutional Neural Networks or CNNs). RNNs are typically used in Natural Language Processing or NLP, and CNNs are used for Computer Vision.

Thanks for reading Anup Jadhav! Subscribe for free to receive new posts and support my work.

Before the Transformers architecture, the dominant machine learning (ML) architecture was sequence-to-sequence modeling (also referred to as sequence transduction models). Sequence-to-sequence modeling *transforms* input sequence to an output sequence. Examples include:

- Text Translation - Converting text between languages
- Text Summarisation - Condensing long text into shorter versions
- Conversation - Generating responses to questions

The Transformer described in the AIAYN paper is based on an encoder-decoder architecture, as seen in this image.

![](https://storage.ghost.io/c/b1/a4/b1a432f8-3f63-4839-8020-97b63b6056f7/content/images/2025/08/fed3d4dd-8f24-4fae-a3bf-fe38dcf56fde_850x1045-1.png)

Transformers operate on **tokens**, which are units of data like words, sub-words, or characters. For example:

- The string "tokenisation" is decomposed as "token" and "isation."
- A short and common word like "the" is represented as a single token.

> As a rule of thumb, 1 token is approximately *4 characters* or *0.75 words* for English text.

The three main variants of Transformers are:

- **BART** \- Bidirectional and Auto-Regressive Transformer

  - Uses both encoder and decoder components
  - Handles both understanding and generation tasks
- **BERT** \- Bidirectional Encoder Representations from Transformers

  - Uses encoder-only architecture
  - Specializes in understanding input text
- **GPT** \- Generative Pre-Trained Transformers

  - Uses decoder-only architecture
  - Excels at text generation tasks

## Key Architectural Concepts

**Bidirectional** means that the transformer *attends* to a single token by looking at tokens to the left (before) and to the right (after) to fully *understand* the sequence. Bidirectionality corresponds to the encoder stack and the multi-head attention layer in the Transformer architecture.

> Encoders that look bidirectionally are good at *understanding input*.

**Auto-Regressive** means that the value at a particular time (or position in a sequence) *depends* on its own previous values. Auto-Regressive predictions correspond to the decoder stack (and the Masked Multi-Head Attention layer).

> Decoders that mask all words after the current word in the sentence are good at being generative or generating one word (or more accurately, a token) at a time.

### Comparison of Transformer Variants:

![](https://storage.ghost.io/c/b1/a4/b1a432f8-3f63-4839-8020-97b63b6056f7/content/images/2025/08/4ad143b8-cbf5-4590-b842-97a14db31d9e_1356x522.png)

Modern LLMs are based on the **Decoder**\-only GPT architecture. Examples include:

- OpenAI's GPT-series models
- Anthropic's Claude-series models
- Meta's Llama-series models

These GPT-based LLMs power many current Generative AI (GenAI) applications.

## Original Transformer Processing Pipeline

What happens when you put a sequence of words into a transformer, as described in the AIAYN paper:

1. **Tokenization**: Splits input text into token units.
2. **Embedding**: Transforms tokens into a vector (or list) of numbers by creating an embedding representation.
3. **Positional Encoding**: Adds sequence position information to each token, i.e., keeps track of word positions.
4. **Residual Connection**: Remembers what you've already learned, i.e., maintains information flow through layers.
5. **Layer Normalization**: Stabilizes training and prevents overfitting.
6. **Multi-Headed Attention**: Processes input from multiple perspectives.
7. **Feed Forward Neural Network**: Provides additional input analysis, i.e., looks at the sequence from another angle.
8. **Encoder Block**: Processes input bidirectionally for understanding.
9. **Decoder Block**: Generates tokens based on the previous sequence (auto-regressive).
10. **Linear Projection**: Calculates raw scores (logits) for vocabulary tokens.
11. **Softmax**: Converts logits into probability distributions for token selection.

---

Transformers have revolutionised machine learning, laying the foundation for modern Generative AI and reshaping the future of AI-driven innovation. If you want to learn more, I’d highly recommend the following resources:

- [Attention in Transformers, Visually Explained](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=133s&ref=anup.io) by 3Blue1Brown. I’d recommend all videos in the ML series
- [Visual Transformer Explainer](https://poloclub.github.io/transformer-explainer/?ref=anup.io)
- [Hands on LLM](https://www.oreilly.com/library/view/hands-on-large-language/9781098150952/?ref=anup.io) book by Jay Allamar (I’m currently working my way through this book)

Thanks for reading Anup Jadhav! Subscribe for free to receive new posts and support my work.