PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference

Abstract

Vision‑Language Models (VLMs) exhibit strong visual reasoning capabilities, yet their inference costs rise sharply as visual tokens increase. Existing visual token pruning methods can partially reduce computational overhead, but suffer from two core limitations. First, most methods only perform pruning after the vision encoder, leaving the heavy latency of the visual encoding stage unoptimized. Second, under tight token budgets, they struggle to simultaneously preserve holistic visual context and fine‑grained details, which causes notable performance drops on information‑intensive tasks.To tackle these issues, this work proposes PACE (Pixel‑Adaptive Condense and Extract), a training‑free inference framework that accelerates both the vision encoder and Large Language Model (LLM) under a unified Condense‑and‑Extract paradigm. In the Condense phase, an Adaptive Pixel Compressor (APC) assesses visual information density before encoding, and adaptively downsamples redundant inputs into compact representations to reduce encoder computation while maintaining global context and key visual cues. In the Extract phase, a Dynamic Dual‑Attention Extractor (DDAE) keeps representative visual tokens through confidence‑weighted fusion of internal encoder visual signals and LLM semantic signals, so as to retain task‑relevant fine‑grained details. Comprehensive experiments verify that PACE attains an improved performance‑efficiency trade‑off. When equipped with PACE, Qwen2.5‑VL‑7B preserves 93.8% of the original model performance with merely 10% visual tokens, delivering a 3.1× end‑to‑end inference speedup.