Image and Vision Models
Image Captioning
Image encoder
CNN – outputs seq. of vectors: convolution filters, feat. vectors 3 x 3 x c., flatten and reshape them. Struggles with long range dependencies.
Transformers
Patchify – divide image into fixe-size patches (64 x 64), flatten, project into embedding space.
Positional encoding – 1D or 2D position (row, col). Fixed (sine-cosine), learnable, RoPe.
Transformer block
Text decoder – decoder-only transformer, input – seq. of vectors for image, output – caption, one token at a time. Training – unsupervised + SFT on image-caption pairs.
Sampling – beam search. Metrics – BLEU, ROUGE, METEOR, CIDEr.
Text to Image
Autoregressive model – decoder only transf-r, input – seq. of text tokens, output – seq. of image tokens. Image tokenizer decodes them. Simpler training, faster inference (one pass while diffusion – several iterative passes to transform noise into image).
Diffusion model – embed text prompt, sample random noise, iteratively denoise / transform into clear image. Better image quality, flexible in sampling (more iterations – better image quality and vice versa).
U-Net (CNNs)
Encoder - Downsampling blocks: Conv2D, BatchNorm2D, ReLu, MaxPool2D + Cross-Attention – attends to text prompt tokens (so that text influences pred. noise). Purpose – extract multi-scale image features.
Decoder - Upsampling blocks: TransposedConv, BatchNorm2D, ReLU, Cross-Attention. Reconstructs denoised image.
DiT (Diffusion Transformers) inspired by Vision Transformer (ViT). GREAT PROMISE
Patchify – divide image into fixe-size patches (64 x 64), flatten, project into embedding space.
Positional encoding – 1D or 2D position (row, col). Fixed (sine-cosine), learnable, RoPe.
Transformer block – processes seq. of embeddings, text prompt to predict noise for each patch
Unpatchify – convert seq. of predicted noise vectors into image with LayerNorm.
Training – forward process: add gradually noise until image completely noised; backward process: learn to reverse forward process by predicting noise until clear image. ML objective = minimize difference betw. true and pred; loss = MSE.
Sampling – start with random Gaussian noise. At each step – refine image by predicting noise.
Metrics – IS, FID, image-text alignment.
Text to Video
Diffusion model – embed text prompt, sample random noise, iteratively denoise / transform into clear image.
U-Net (CNNs) – Stable Video Diffusion
Same as Image above + temporal consistency added to each encoder / decoder block to generate a seq. of frames that are temporally consistent: a) temporal attention attending to relevant feats across frames, b) temporal convolution – 3D convolutions.
DiT (Diffusion Transformers) – performs better w/more data – OpenAI’s Sora.
Same as Image above except for using 3D patches.
Training – noise addition, noise prediction, loss calculation. ML objective = minimize difference betw. true and pred; loss = MSE. Since lack of data, train on both image and video data.
Heavy compute – train in lower dimensional latent space (LDM) + pre-compute image represent. in latent space before training. Then enhance image via pre-trained super-resolution model (pixels) & temporal super-resolution model (frames). Efficient architect., distributed training.
Sampling – start with random noise in latent space & refine latent representation.
Metrics – IS & FID averaged by frame + FVD for temporal consistency + video-text alignment.
Best multi-modal models
1. Understanding / Analysis of multimodal content
Models for understanding multimodal content.
1.1. General-purpose proprietary models (partially fine-tunable)
Google Gemini 3 / 3.1 Pro — native handling of text + image + audio + video in a 1M-token context. Currently the benchmark leader. Excels at video reasoning. Fine-tuning: no access to underlying weights; SFT or continuous training as a managed service through Vertex AI or Google AI Studio. You provide only structured datasets and Google handles compute and hosting of adapters.
Gemini 3 Flash: highly balanced model w/strong mix of speed and reasoning quality + lower cost than Pro.
Gemini 3.1 Flash-Lite: fastest and most cost-effective, optimized for high-volume, latency-sensitive tasks, and agentic workflows.
GPT-5.5: text + image + audio + video input, in one unified model. API-based SFT, RFT (reinforcement fine-tuning), and DPO on selected GPT-5 variants. Vision fine-tuning only on some checkpoints.
Anthropic Claude Opus 4.5 / 4.7 — text + image only, complex reasoning. Particularly strong on long, complex documents and code. No public API fine-tuning.
Grok 4.3 — text + image + video input natively; audio handled via separate xAI models (Grok Voice, STT/TTS). Limited public fine-tuning.
1.2. General-purpose open-source models (fully fine-tunable)
The open-source gap has closed faster than expected. Qwen3-VL-235B-A22B-Instruct rivals top-tier proprietary models Gemini Pro and GPT-5 for general Q&A, 2D/3D grounding, video understanding, OCR, and document comprehension. All output is text:
Gemma 4 (open-weight): text + image + audio + video; freely fine-tunable.
Qwen3-VL (Alibaba) —VLM frontrunner, text + image + video, with variants from 2B / 7B to 235B MoE.
GLM-4.5V / GLM-4.6V (Zhipu) —advanced 3D reasoning, text + image + video.
Llama 4 Scout (Meta) — 10M context (entire book, massive codebases & doc collections), text + image.
SmolVLM (256M / 500M / 2B) (on HF) —runs on CPU or laptop GPU - perfect for "edge VLM" project.
Moondream 2 (on HF) — ~2B params, very fast, good at visual Q&A. Popular for demos.
1.3. CLIP-style embeddings:
OpenAI CLIP or OpenCLIP —dual text + image encoder model; semantic search for both in common embedding space.
SigLIP / SigLIP 2 — Google's improved CLIP variant: shows you're current.
2. Generation / Creation of multimodal content
Models for creation rather than analysis.
Image Generation:
Nano Banana 2: text-to-image, image editing, multi-image composition.
Nano Banana Pro: upgraded version w/enhanced real-world knowledge and text rendering within images.
Veo - Video Generation: high-fidelity video + natively generated audio based on text prompts or reference images.
Lyria - Music Generation: highly advanced, generates professional-grade, 30-second tracks (realistic lyrics + vocals) based on text, images, or video inputs.
No fine-tuning for Google models - have to use Stable Diffusion or Flux for that.
Audio generation:
Qwen3-TTS
Whisper
VL-JEPA
Autoregressive LLMs learn surface-level language patterns instead of pure meaning. They don’t understand meaning first — they learn to speak fluently first by just generating the next token. LLMs taught machines to speak.
VL-JEPA is teaching machines to understand. Shift from token prediction to semantic prediction.
VL-JEPA is based on the Joint Embedding Predictive Architecture (JEPA) philosophy and extends it to vision + language. Instead of predicting text tokens, it predicts meaning - semantic embeddings. VL-JEPA ensures cleaner semantic understanding - optimizes for: “What does this mean?”. Instead of talking more, models will understand continuously and respond selectively.
VL-JEPA has four key components:
1. Vision Encoder – Converts images/videos into embeddings
2. Text Encoder – Converts target text into embeddings
3. Predictor – Predicts the text embedding from vision + query
4. Lightweight Decoder – Converts embeddings to text only when needed
No text decoding is required during training.
Everything lives in a shared embedding space => VL-JEPA naturally supports:
🔹Open-vocabulary classification (no pre-defined label set)
🔹Image/video ↔︎ text retrieval
🔹Visual Q&A
🔹Discriminative reasoning (model compares, or scores given options to decide which one best fits the input)
👉 No task-specific architectural changes required.
In experiments, VL-JEPA:
🔹Uses ~50% fewer parameters
🔹Matches or outperforms traditional VLMs
🔹Beats CLIP-style models in retrieval & classification
🔹Competes strongly on VQA tasks
Lower latency, reduced compute cost since VL-JEPA doesn’t decode tokens continuously and decodes text only on demand. It enables continuous real-time understanding and is perfect for: a) Video understanding, b) Robotics, c) AR/VR, d) Always-on AI agents.
VL-JEPA aligns perfectly with Agentic AI, edge & on-device AI, real-time multimodal reasoning