Ask someone what model they’re running and they’ll usually give you a name, Llama, GPT-4, DeepSeek. Ask them what architecture that model actually uses and why it was picked, and the conversation usually stalls.
That gap matters. The architecture behind a model decides how it processes tokens, how much GPU memory it eats, how fast it responds, and what kind of hardware it wants. Two models with similar benchmark scores can have completely different cost and latency profiles once you look under the hood. If you work in AI infrastructure, this is the layer where your capacity planning, your hardware choices, and your cost per token all actually come from.
This post walks through the major architectures you’ll run into, what each one is actually doing differently, and when it earns its place over the alternatives. Here’s the map before we get into the detail.
flowchart TD
A[What are you optimizing for?] --> B{Choose a direction}
B -->|Best raw accuracy| C[Large Dense Model]
B -->|Lower inference cost at scale| D[Mixture of Experts]
B -->|Limited GPU memory| E[Small Language Model]
B -->|Images, PDFs, audio as input| F[Multimodal Model]
B -->|Multi-step reasoning| G[Reasoning Model]
B -->|Semantic search / RAG| H[Embedding Model]
B -->|Better retrieval precision| I[Reranker]
B -->|Lower decode latency| J[Speculative Decoding]
Dense Transformer: The Baseline
A dense transformer runs every input token through every layer and every parameter. Nothing is skipped. A 70B model means all 70 billion parameters participate in generating every single token, whether that token is part of a simple greeting or a hard reasoning step.
This is the architecture the field was built on. GPT-3, early LLaMA, and Mistral 7B are all dense.
Why it’s still relevant: predictable performance, simple to serve, and easy to reason about capacity planning since compute cost per token never changes.
Why it’s expensive: you pay full compute cost regardless of how hard the token actually is.
Mixture of Experts (MoE)
This is the biggest architectural departure from dense models, and it’s what powers Mixtral, DeepSeek, and reportedly GPT-4 class models.
Instead of one large feed-forward network, MoE splits it into many smaller networks called experts. A router picks a small subset of experts, typically 2 out of 8 or 2 out of 64, to handle each token.
flowchart LR
T[Incoming Token] --> R[Router]
R --> E1[Expert 1]
R --> E3[Expert 3]
R -.skipped.-> E2[Expert 2]
R -.skipped.-> E4[Expert 4]
E1 --> O[Output Token]
E3 --> O
DeepSeek-V3 has 671B total parameters but activates only 37B per token. You get large-model capability at a much lower compute cost per token.
The catch: all experts still need to sit in GPU memory even though only a couple fire per token. Memory footprint tracks total parameters, not active parameters, which makes MoE models memory-hungry relative to their compute demand. Serving one also requires expert parallelism, spreading experts across GPUs and routing tokens between them, which adds communication overhead a dense model never has to deal with.
Best fit: high-capability serving at scale where you can absorb the memory cost and amortize it across heavy traffic.
Small Language Models (SLMs)
Not everything needs a 70B model. SLMs, generally under 10B parameters, have gotten meaningfully more capable as training techniques improved. Phi-3, Gemma 2B, and Qwen2-1.5B can run on a single consumer GPU or even on-device.
They fall short on complex multi-step reasoning and broad world knowledge, but for narrow tasks, classification, short summarization, constrained code completion, or routing logic in front of a bigger model, they’re often enough.
Best fit: edge inference, high-volume low-complexity tasks, latency-sensitive apps where fast TTFT matters more than raw capability.
Multimodal Models
These take more than text: images, audio, video, documents. GPT-4o, Gemini, LLaVA, and Qwen-VL fall here. Architecturally, a vision encoder converts non-text input into embeddings the language model can consume alongside regular tokens.
flowchart LR
Img[Image] --> VE[Vision Encoder]
Txt[Text] --> Tok[Tokenizer]
VE --> Emb[Shared Embedding Space]
Tok --> Emb
Emb --> LLM[Language Model]
LLM --> Resp[Response]
Serving cost to know about: the vision encoder adds real prefill compute before the language model even starts, so TTFT on image-heavy requests runs noticeably higher than text-only requests of similar length.
Best fit: document understanding, visual QA, anything where the input isn’t purely text.
Reasoning Models
Reasoning models are trained to produce a chain of thought before the final answer. DeepSeek-R1, QwQ, and OpenAI’s o-series are examples. The thinking trace is often hundreds to thousands of tokens and usually hidden from the end user.
This matters operationally because reasoning models generate far more tokens per request. A task that produces 200 tokens from a standard model might produce 2000 from a reasoning model, mostly thinking tokens. That’s a direct multiplier on decode time, KV cache usage, and cost per request.
Best fit: math, logic, debugging, planning, anything where accuracy matters more than latency or cost.
Skip it for: simple, high-volume, or latency-sensitive workloads. You pay for the thinking whether the task needed it or not.
Embedding Models
Not generative at all. An embedding model takes text and returns a dense vector representing its meaning. Used for semantic search, RAG, clustering, classification. text-embedding-ada-002, BGE, and E5 are common examples.
Serving-wise these are a different beast entirely: there’s no decode phase, only prefill. Output is a fixed-size vector regardless of input length, the workload is compute-bound rather than memory-bandwidth-bound, and requests batch very aggressively since outputs don’t depend on each other.
Best fit: vector database construction, semantic search, document clustering.
Rerankers
A reranker sits after vector search in a retrieval pipeline. It scores each candidate document against the query directly, as a pair, rather than comparing independent vectors. That makes it more accurate than pure similarity search but more expensive, since every candidate needs its own scoring pass.
Best fit: production RAG where you use fast vector search for recall and a reranker for precision on the shortlist.
Speculative Decoding
Not a standalone model type, more of a serving technique that introduces a second model into the stack. A small, fast draft model proposes several tokens ahead. The large target model verifies all of them in a single forward pass. Matches get accepted in bulk; mismatches fall back to normal single-token generation.
sequenceDiagram
participant D as Draft Model (fast)
participant T as Target Model (large)
D->>T: Proposes tokens T1, T2, T3, T4, T5
T->>T: Verifies all 5 in one pass
T-->>D: Accepts T1, T2, T3 — rejects T4
Note over D,T: 3 tokens generated for ~1 forward pass cost
Works well when the draft model’s guesses are usually right. When acceptance rate is low, it adds overhead without paying off.
Best fit: latency-sensitive interactive apps where TPOT is the bottleneck. Less useful for throughput-optimized batch workloads.
Choosing Between Them
| Constraint | Architecture to consider |
|---|---|
| Need maximum capability | Large dense or MoE |
| Memory constrained | SLM or quantized dense |
| Cost per token matters | MoE or SLM |
| Input includes images or documents | Multimodal |
| Complex reasoning required | Reasoning model |
| Need semantic search | Embedding model |
| Retrieval precision matters | Reranker |
| Need lower TPOT on a large model | Speculative decoding |
In production, these rarely show up alone. A typical RAG pipeline runs an embedding model for retrieval, a reranker for precision, and routes between a small model and a large model depending on query complexity. Knowing what each architecture is actually good at is what makes those routing decisions sensible instead of arbitrary.
Where This Leaves You
None of these architectures are competing to be “the best model.” They’re solving different problems. A dense model is the safe, predictable default. MoE buys you capability without paying full compute for it, at the cost of memory. SLMs trade ceiling for speed and footprint. Reasoning models trade cost and latency for depth. Embedding models and rerankers aren’t even in the same category, they’re not generating anything, they’re organizing what a generative model gets to see.
The mistake to avoid is picking an architecture because it’s the newest or the most talked about, rather than because it matches what you’re actually optimizing for, cost, latency, capability, or memory footprint. Once you know what each one is built for, that choice stops being a guess.