Qwen2.5-VL video captioning, native on Apple Silicon with MLX

Cloud vision inference is fine until you’re calling it thousands of times. This project replicates the NemoStation/Marlin-2B temporal-captioning paradigm natively on Apple Silicon with MLX — ultra-fast video captioning and sub-second visual grounding, entirely inside the Mac’s unified memory.

Why unified memory changes the maths

On a discrete GPU you pay a tax every time weights and activations cross the PCIe bus. On Apple Silicon the CPU and GPU share the same physical memory, so a 2B VLM that would thrash a small VRAM budget simply… fits, and there’s no host↔device copy in the hot loop. That single fact is why so much local inference suddenly makes sense on a Mac.

The hard parts were sampling and the text encoder

Most of the porting effort wasn’t the transformer blocks — MLX handles those. It was:

• Temporal frame sampling — captioning quality lives or dies on which frames you feed. Uniform sampling misses the event; a cheap scene-change heuristic keeps the informative frames.
• The text encoder / tokenizer parity — matching the reference tokenization exactly, so embeddings line up with the vision tower.

frames = sample_keyframes(video, target=8)   # scene-aware, not uniform
feats  = vision_tower(frames)                 # stays in unified memory
caption = decoder.generate(feats, prompt)     # streamed, sub-second first token

Once frames are sampled sensibly, the model captions video in near real time on an M-series laptop — offline, private, and free per call. For anyone building on-device AI, MLX has quietly become the reason the Mac is a serious inference target.