arxiv:2607.09061

On Locality and Length Generalization in Visual Reasoning

Published on Jul 10

· Submitted by

M.Reza Ebrahimi on Jul 17

Qualcomm

Upvote

Authors:

Abstract

A striking feature of the human visual system is that it ingests visual information through a series of local foveated glimpses, rather than a single global computation. This makes human vision distinctly different from most popular computer vision models in use today, which input images globally and in a single shot. A natural question therefore is whether local, sequential vision models may provide any fundamental computational benefits in addition to being biologically more plausible than global models. In this work, we investigate this question from the perspective of visual state tracking and length generalization. Inspired by recent studies of length generalization in language models, we study the behavior of vision models trained on simple vision tasks that require the aggregation of local information across an image. Our experiments reveal that, similar to language models, vision models can learn to exploit global shortcuts and thereby fail to generalize over task length or complexity. We also show that recurrent vision policies based on strictly local perception can mitigate these failures, thereby allowing models to generalize on these tasks. Our results show that local attention may be an essential overlooked requirement for robust compositional generalization.

View arXiv page View PDF Add to collection

Community

rezaebrahimi

Paper submitter about 11 hours ago

•

edited about 11 hours ago

Your eyes don't see a whole scene at once — they dart around in a sequence of foveated glimpses. Modern vision models take in the whole image in one shot. That difference decides if a model can generalize to scenarios that are out-of-distribution.

We built simple visual puzzles (e.g. "read these switches, navigate, track the state") and made them longer at test time than in training. SOTA VLMs — GPT-5.4, Claude Sonnet 4.6, Qwen — ace the short ones… then fall off a cliff the moment the puzzle gets longer. A tiny recurrent agent (green) extrapolates the performance at test time. Even with task-specific training, the global model (Qwen) still breaks out-of-distribution.

The failure has a name: global shortcuts.
When a model sees the whole image at once, it can memorize a parallel "trick" that works for the lengths it saw — but doesn't actually implement the step-by-step computation. So it breaks out-of-distribution. Same thing happens to LLMs on parity/state-tracking. We show vision has the same problem.

You might think making the model recurrent (giving it a memory that updates step by step) is the fix. It isn't — on its own. Same recurrent LSTM, three ways of seeing the image:

Global — sees the whole image at once → collapses
Local + Global — adds small high-res crops but keeps the global view → still collapses
Foveated — only small local glimpses, no global view → generalized out-of-distribution.
Hand a recurrent net the whole image and it still memorizes a shortcut. Locality is a necessary ingredient.

Now flip it: fix the local glimpses, swap only the backbone. Strict recurrent nets (LSTM/GRU/RNN) generalize. Transformers, Mamba, xLSTM — all degrade. So the recipe is both: local perception + recurrence. Neither alone is enough. Together they're sufficient. 🔑

There's a real trade-off. Small/low-res glimpses → generalize but slow to explore. Big/high-res glimpses → easy to explore but invite shortcuts. With the right glimpse settings, we show that FoveAgentLSTM holds its accuracy at resolutions well beyond training — a global model only works in/near the resolution it was trained at.

But locality + recurrence isn't a free win everywhere — and knowing when it helps is the point. On a recall task the global VLM wins and our local agent lags. State-tracking wants recurrence + locality. Pure retrieval doesn't. The two split exactly like they do in language models.

The synthetic tasks isolate the mechanism. Does it carry over to a real task? We tested reasoning over math plots — finding a function's roots. At the same visual-compute budget, a foveated Qwen adds +29 pts (~100%) of accuracy over the global baseline. Uniformly cranking resolution 10× buys almost nothing (+3.8 pts). Same lesson as the synthetic tasks: locality + recurrence beats brute-force scaling. How you spend visual compute > how much visual compute you throw at it.