Microsoft Mage-VL: Video Codecs Meet Vision Transformers — A 4B Model That Reads Video Like a Codec

Most Vision-Language Models (VLMs) suffer from a modern Moravec's paradox: they excel at complex offline visual reasoning but fail at simple real-time streaming perception. Microsoft's Mage team took a radically different approach — instead of decoding video into uniformly-sampled frames and pushing dense patch grids through a frozen ViT, Mage-VL follows the structure of video codecs themselves. The result is a 4B-parameter model that matches Qwen3-VL-4B on static tasks while comprehensively beating it on video understanding, spatial reasoning, and proactive streaming.

We cloned microsoft/Mage at commit 76bec2bb (Aug 10, 2026), read the inference code on disk, and analyzed the full architecture. Here's what makes this model genuinely novel — and where it still falls short.

The Core Insight: Codec-Native Visual Tokenization

The central innovation is deceptively simple. Video codecs (H.264, HEVC) already solve the problem of efficient video representation: they split streams into anchor (I) frames (stored in full) and predicted (P) frames (stored as motion vectors + residuals from the anchor). Mage-VL borrows this exact structure for visual tokenization.

On a 16×16 patch grid, Mage-ViT keeps every patch from I-frames but retains only the motion-salient patches from P-frames — the regions where the codec actually spends bits. This cuts visual token consumption by over 75% (down to ~1/8 or less of dense frame sampling) while preserving spatio-temporal context.

The interface is codec-agnostic. The same encoder accepts:

• Traditional codecs (H.264/HEVC) — motion vectors + residual energy as the per-patch importance signal
• Neural codecs (DCVC-RT) — the learned rate map as the importance signal

No architecture change, no retraining. The on-disk inference code confirms this — inference_base.py passes a codec_config dict with engine: "hevc" or engine: "dcvc-rt" to the same processor, and the neural codec package is bundled directly in the checkpoint directory.

# From mage_vl/inference_base.py — codec config is a dict, not a model swap
codec_config = {
    "engine": "hevc" if args.codec_engine == "traditional" else "dcvc-rt",
    "target_canvas": args.num_frames,
    "patch": 16,
}
if args.codec_engine == "neural":
    codec_config["dcvc"] = {
        "pkg_dir": os.path.join(model_path, "neural_codec"),
        "device": str(model.device),
    }

This matters because it decouples the visual encoder from the codec implementation. As video codecs evolve (AV1, VVC, future neural codecs), the same model can adopt them without retraining — as long as the codec exposes a per-patch importance map.

Mage-ViT: A From-Scratch Visual Encoder

Mage-ViT is not a fine-tuned SigLIP or a distilled CLIP. It is trained entirely from scratch on only 560M unlabeled images and 100M video frames — no billion-scale image-text pair pretraining. Yet it matches SigLIP2 (pretrained on billions of pairs) on downstream benchmarks.

Two design choices enable this data efficiency:

1. Variable-resolution pretraining. Unlike fixed-resolution encoders that saturate or degrade as token budgets grow, Mage-ViT's quality improves monotonically with the visual token budget. At 676 tokens, it reaches >96.1% on Food-101 and >86.3% on ImageNet. The encoder never wastes capacity on padded empty regions.

2. 3D rotary position encoding (3D RoPE). A shared positional encoding scheme preserves spatio-temporal positions across the variable-length, codec-selected patches. This is critical — without it, the model couldn't distinguish "this patch is from frame 3, upper-left" from "this patch is from frame 47, center."

The standalone encoder is released as microsoft/Mage-ViT under MIT license, separate from the joint VLM training. You can drop it into your own multimodal pipeline.

System 1 & System 2: Proactive Streaming in a Single Model

The most architecturally interesting component is the dual-system streaming design. Mage-VL uses a single checkpoint — no multi-agent pipeline — to deliver event-gated commentary on live video.

• System 1 (cognition gate): A lightweight gate watches each rolling codec window and predicts p_speak = g(h_t) — the probability that the current segment contains a response-worthy event. It operates over a recurrent streaming memory maintained by an event-preserving feature extractor. When p_speak < τ (default 0.5), the model stays silent.

• System 2 (full VLM): Only when the gate fires does the full Qwen3-4B decoder generate a response, conditioned on a local sliding window of the most recent codec segments. A text query can be injected at any time.

The on-disk streaming code (inference_streaming.py) reveals the implementation:

# The gate is a single forward pass over all visual segments
logits = model.streammind_gate_forward_segments(visual_segments)[0]
# Extract per-segment boundary predictions
probabilities = torch.softmax(logits[boundaries].float(), dim=-1)[:, 1].tolist()

# Gate decides: speak or stay silent
for (start, end), inputs, probability in zip(timestamps, segments, probabilities):
    if probability < args.gate_threshold:
        print(f"[t={start:.1f}-{end:.1f}s] gate=silence (p={probability:.2f})")
        continue
    text = generate_current_segment(model, processor, inputs, args.max_new_tokens)
    print(f"[t={start:.1f}-{end:.1f}s] gate=response (p={probability:.2f}) -> {text}")

The gate is trained in Stage 5 of the curriculum on ~3.3M streaming samples, with the visual encoder and LLM frozen — only the gate weights are trained. This keeps the streaming capability from degrading the base model's understanding performance.

SoccerNet Streaming Results

Method	TriggerAcc	TimVal	F1	ROC-AUC	PR-AUC
StreamMind	52.18	47.36	–	–	–
JoyAI-VL-9B	97.98	19.25	3.55	56.26	1.68
Mage-VL-4B	79.21	55.54	16.35	83.14	9.30

JoyAI's higher trigger accuracy comes from predicting silence almost everywhere under SoccerNet's heavy class imbalance — it collapses on precision-sensitive metrics. Mage-VL wins on every metric that penalizes false triggers.

Five-Stage Training Curriculum

Mage-VL is trained through a progressive five-stage supervised curriculum. No RL post-training is used (though the paper explores a preliminary Zero-Vision SFT + RL study).

Stage	Data	Purpose
1. Multimodal alignment	~350M image captions + 4.2M short-video captions	Align visual and text spaces
2. Instruction tuning	~54M image-instruction + 3.4M 30–180s video captions	Task following + temporal grounding
3. Temporal-horizon expansion	Medium/long video (LLaVA-Video, TimeLens, VideoChat-Flash, Molmo2)	Long-context video reasoning
4. Codec-native adaptation	350K long videos as rolling codec windows (up to 384/768 frames)	Codec-native long-context
5. Streaming alignment	~3.3M streaming samples (encoder + LLM frozen)	Event-gated commentary

Stage 3 uses AI-guided diagnostics to determine which video categories, resolutions, and frame counts to train on — resulting in 384-pixel inputs, 384-frame temporal length, and RoPE base frequency θ=8M for stable long-context modeling.

The AI4AI Data Pipeline

Stage 1's captioning pipeline is itself an agentic system:

1. Frozen Qwen3-VL-32B captions images
2. GPT-5 rubric scorer grades each caption on completeness, redundancy, coherence, and OCR fidelity
3. GitHub Copilot agent co-designs the prompt and harness code — for example, rendering timestamp overlays onto frames so the captioner can anchor events via visual OCR
4. Human validation gate approves changes
5. Ten refine-verify iterations

This closed loop lifts every downstream benchmark: +5.6 InfoVQA, +3.8 OCRBench, +3.8 RealWorldQA. The timestamp-overlay trick is particularly clever — it converts implicit temporal tracking into a visual grounding task, reducing temporal hallucinations in long-video captions.

Performance: Where Mage-VL Wins and Loses

Video Understanding (selected benchmarks)

Benchmark	Mage-VL-4B	Qwen3-VL-4B	Phi-4-R-V-15B
VideoMME	64.0	59.7	55.3
MLVU-dev	68.7	61.5	51.8
LongVideoBench	61.3	57.7	51.2
VideoEval-Pro	45.2	20.7	16.8
NextQA	83.1	79.8	69.0

With the same 4B Qwen3 LLM backbone fixed, swapping in Mage-ViT beats Qwen3-VL-4B on every reported video and temporal-grounding benchmark.

Spatial Intelligence

Benchmark	Mage-VL-4B	Qwen3-VL-4B	Delta
VSI-Bench	64.3	53.3	+11.0
CrossPoint	80.00	26.90	+53.1
EmbSpatial	82.67	77.50	+5.2
CV-Bench-3D	94.75	92.30	+2.5

The spatial intelligence gains are striking. CrossPoint at 80.0 vs 26.9 is a 3× improvement. The paper attributes this to motion–spatial synergy — dynamic video training substantially improves static 2D/3D spatial reasoning, a counterintuitive finding (Finding 5).

Where Mage-VL Loses

Benchmark	Mage-VL-4B	Qwen3-VL-4B
TextVQA-val	77.28	80.55
CC-OCR Doc	32.25	39.69
CRPE-Relation	76.12	77.70
SAT	67.33	69.30
MMSI-Bench	28.20	31.00
ERQA	36.00	42.30

The losses cluster in text-heavy OCR and agentic spatial tasks. The paper acknowledges this gap: "Mage-VL still shows a gap to Qwen3-VL on complex agentic tasks," attributing it to tight compute budgets, the absence of joint text-multimodal mixed pretraining, and the omission of large-scale RL post-training.

The Seven Empirical Findings

Beyond the model itself, the paper distills seven findings that are arguably more valuable to the community than the checkpoint:

1. Data-efficient tokenizer. 560M images + 100M video frames is enough to train a competitive ViT from scratch. You don't need billion-scale image-text pairs.

2. Variable-resolution pretraining scales monotonically. Quality keeps improving with token budget. Fixed-resolution encoders saturate or degrade. This challenges the conventional wisdom of fixed input sizes.

3. Codec-native tokenization sets a better accuracy–efficiency frontier. Up to 3.5× wall-clock speedup over uniform frame sampling at matched accuracy. The codec path is the fastest of all compared models on most video benchmarks.

4. Explicit VideoQA SFT is redundant. Dense video captions + standard image SFT are sufficient for strong zero-shot VideoQA. You don't need task-specific video QA training data.

5. Motion–spatial synergy. Training on dynamic video substantially improves static 2D/3D spatial reasoning. This is counterintuitive — you'd expect video training to help video tasks, not static spatial tasks.

6. AI4AI data pipeline. Agentic closed-loop feedback (GPT-5 scorer + Copilot code agent) systematically lifts caption quality and downstream scores. The prompt and the harness code are co-designed, not just the model.

7. Zero-Vision SFT for multimodal RL. Skip visual SFT entirely. Use high-quality pure-text reasoning data before applying multimodal RL. On LLaVA-OV-1.5: +5.33% overall (54.28 vs 48.96), wins 19/24 tasks, 50.7% fewer RL steps. Reasoning (+7.92% avg.) and OCR/Chart (+9.52% avg.) benefit most. This suggests that preserving intrinsic textual reasoning during instruction tuning unlocks stronger multimodal RL.

Mage-Flow: The Generation Sibling

The repo also contains Mage-Flow, a 4B diffusion transformer for text-to-image generation and image editing. It shares the same efficiency philosophy applied to generation:

• Mage-VAE matches FLUX.2-VAE reconstruction fidelity at ~12×/22× fewer encode/decode MACs per pixel
• Native-resolution packing — one checkpoint generates 512 to 2048 at any aspect ratio (including 4:1), no buckets, no padding
• Turbo variant: 4 steps, 0.59 s/image at 1024² on a single A100, ~18-20 GB peak memory

On GenEval, Mage-Flow scores 0.90 — the best among all open-source models, beating FLUX.2-dev (32B, 0.87) and Qwen-Image (20B, 0.87). At 4B parameters. The editing model (Mage-Flow-Edit-Turbo) achieves 4.38 on ImgEdit-Bench at 4 steps in 1.02 seconds.

What's Released and What's Not

Released:

• microsoft/Mage-VL — single checkpoint for all capabilities (Apache-2.0)
• microsoft/Mage-ViT — standalone visual encoder, ViT pretraining only (MIT)
• 6 Mage-Flow checkpoints — Base/RL/Turbo × generation/editing (MIT)
• Clean inference scripts: offline, SGLang online, event-gated streaming

Not released:

• Training code (all five stages described but not reproducible from the repo)
• The AI4AI pipeline implementation
• The gate training code
• Upstream SGLang support (requires fork: kcz358/sglang, feat/mage-vl)

Responsible AI disclaimer: "Released for research purposes only and not intended for product or service deployment." Despite the open licenses, commercial use requires careful review.

Implications for the Field

Mage-VL represents a genuine paradigm shift in how VLMs process video. The codec-native approach is the kind of idea that seems obvious in retrospect — video codecs have spent decades optimizing for exactly the problem VLMs face (where to spend bits) — but nobody had cleanly applied it to visual tokenization before.

The seven findings are independently actionable. The Zero-Vision SFT result alone — that you can skip visual SFT and get better multimodal RL — challenges standard training recipes and offers a compute-efficient path for groups with limited GPU budgets.

The from-scratch encoder at 560M images validates a thesis we hold at SMF Works: well-designed compact models can compete with systems trained on orders of magnitude more data. The 4B budget is a feature, not a bug.

The question now is whether the community adopts codec-native tokenization as a standard, or treats it as a one-off architecture. The codec-agnostic interface makes adoption easy — but the lack of training code and upstream serving support creates friction. Watch for SGLang and vLLM integration PRs as the leading indicator.

---

Sources: github.com/microsoft/Mage @ 76bec2bb (Aug 10, 2026), read on-disk. arXiv: 2607.24904 (Mage-VL), 2607.19064 (Mage-Flow). HuggingFace: microsoft/Mage-VL, microsoft/Mage-ViT, microsoft/Mage-Flow-*.