ZeRO (Zero Redundancy Optimization): fully sharded data parallelism (FSDP) - eliminates memory redundancy by partitioning / sharding (instead of replicating) optimizer states (ZeRO-1), gradients (ZeRO-2), and parameters (ZeRO-3) across every GPU in the cluster, gathering them just-in-time.
HSDP (hybrid sharded data parallelism) - like FSDP, but shards within a group of GPUs (one node), and replicates across groups.
Pipeline Parallelism (PP) - if model too big for a GPU, split model layers into sequential stages across GPUs. GPU 0 => layers 1–10, GPU 1 => 11–20, etc. Activations flow down the pipeline. Naive version idles GPUs ("bubble"); fixed by micro-batches flowing thru pipeline (GPipe, PipeDream) - stages r busy,
Model / Tensor Parallelism (TP) - if layer too large for a GPU, split large matrix multiplications across GPUs, — shard the weight matrices across GPUs (e.g., split a matmul column-wise/row-wise). GPUs cooperate on a single forward pass with heavy communication. Needs fast interconnect (NVLink), so usually kept intra-node,
Context / Sequence Parallelism - splits a single sequence across GPUs when sequences are too long (128k+) for one GPU's activations to fit (e.g., tokens 1–8k on GPU 0, 8k–16k on GPU 1). But in attention - every token attends to every prior token => GPUs must exchange keys and values: Ring Attention rotates KV blocks around a ring of GPUs and DeepSpeed-Ulysses uses all-to-all communication to switch sharding strategies: tokens split across GPUs everywhere except inside attention, where it re-shards by attention head instead.
Expert Parallelism: distribute MoE layers across GPUs == route each token to a subset of "expert" sub-networks living on different GPUs. Scales parameter count without scaling per-token compute.
4D Parallelism = combining DP+TP+PP+CP in production setups for models with hundreds of billions of parameters - frontier LLMs. We use TP inside the node across NVLink, PP across nodes to stack the layers, and DP (or FSDP) on top of that architecture, CP across the sequence dimension when context lengths exceed what a single GPU's activations can hold — making million-token training feasible.
Trained LLM Uses
In-context learning - textual prompts incl. few-shot, CoT, RAG, Agents.
Embeddings in downstream tasks
Further fine-tuning on domain-specific data.
LLM Enterprise Use Cases
Summarization: product reviews, reports, articles, insights from unstructured data
Conversational AI – customer service bots, contact center solutions, enterprise Q&A,
Writing Assistant – writer’s block, writing assistant
Knowledge Mining – domain-specific research, social media trends, cross-functional insights
Software Development – faster coding, debugging, autocompletion, documentation
Image Generation – marketing, logotypes, images / videos for ads (happy cleaner)
𝗠𝗶𝘅𝘁𝘂𝗲 𝗼𝗳 Experts (MoE)
MoEs replace the FFN layers of transformer w/sparse MoE layers = each is a router + multiple experts (e.g., 8), each being an FFN w/own parameters. Router/gate network w/learned params and pretrained with rest of the network, selects which experts to send tokens to by taking each token as input and producing a proba distrib. over experts (softmax gating). Although a MoE might have many parameters, only some are used for inference => much faster inference compared to a dense model w/same # params + faster pre-training.
𝗠𝗶𝘅𝘁𝗿𝗮𝗹 𝟴𝘅𝟳𝗕
High-quality sparse mixture of experts model (SMoE), decoder-only model, has 8 models with 7B params (FFN layers = individual experts).
Outperforms Llama 2 70B and GPT-3.5, 6x faster inference, context window = 32K.
GPT-4 = MoE model w/16 expert models, each with around 111B params, total 1.76T
Mixture-of-Agents (MoA) - LLMs are better together
Multi-layer architecture - each layer = agent using multiple LLMs of varying strengths. Each new agent-layer uses outputs from prev. agent-layers to generate response. Aggregate-and-synthesize prompt to integrate responses from diff. agents. LLMs generate better responses when using outputs from other models. MoA superior performance with a 65.1% on AlpacaEval vs. 57.5% by GPT-4o.