Multimodal Models Are Learning When to Reach for Code, Not Just How to Look Harder

Multimodal models have spent years getting better at looking. AIR is interesting because it asks them to get better at deciding when looking is no longer the hard part.

The paper, Adaptive Interleaved Reasoning with Code in MLLMs, trains multimodal large language models to use code during complex numerical visual reasoning. That is a different skill from the usual visual-tooling playbook. Zooming, cropping, rotating, OCR, and segmentation help when the bottleneck is perception. They do not solve a chart, geometry problem, table, or visual math prompt where the model can see the relevant facts but still needs to compute with them.

AIR’s central move is to make executable reasoning part of the multimodal trajectory. The model is not merely prompted to use Python whenever a benchmark author feels like it. It is trained to interleave visual reasoning and code use, then reinforced to invoke code when the task actually benefits from it. That “when” is the product skill.

The next MLLM should not stare harder at the chart

The authors evaluate AIR on Qwen2.5-VL-7B. The cold-start dataset is built from MMK12 and contains about 1.5K high-quality interleaved reasoning samples. Training uses lightweight fine-tuning for one epoch through LlamaFactory, followed by reinforcement learning with a group-constrained reward function. Benchmarks include MathVista, MathVision, MathVerse, DynaMath, WeMath, and LogicVista.

The reported gains are solid for a method this targeted. AIR improves average benchmark accuracy by 6.1 percentage points after RL training. The interleaved-reasoning subset improves by 9.9 points, and the paper reports overall tool-use success above 95%. Per-benchmark gains versus Qwen2.5-VL-7B include +8.1 points on MathVista, +3.8 on MathVision, +7.7 on MathVerse, +4.0 on DynaMath, +9.9 on WeMath, and +3.4 on LogicVista.

The 95% tool-use success number deserves as much attention as the accuracy lift. Tool use that fails to execute is not intelligence; it is latency with punctuation. A multimodal agent that produces broken code, calls Python for every trivial arithmetic step, or computes numbers disconnected from the visual evidence can look impressive in a demo and become expensive nonsense in production. AIR is valuable because it treats code execution as part of the reasoning behavior to train and evaluate, not as a magical appendix.

This is where multimodal tool use starts looking less like image manipulation and more like actual analysis. A model helping with a school geometry problem, a sales chart, a scientific plot, a financial table, or an operations dashboard should not only identify shapes and text. It should decide when to extract quantities, write a small program, check the calculation, and connect the result back to the visual premise. The useful assistant is not the one that narrates the image more fluently. It is the one that knows when the problem has crossed from seeing into computing.

Adaptive tool use beats compulsive tool use

The practitioner trap is to optimize for “uses tools” instead of “uses tools appropriately.” Many agent systems already have this disease. A tool-enabled model calls search for obvious facts, calls Python for arithmetic it could do mentally, or shells out because the harness rewarded activity. In multimodal settings, that failure gets worse because every tool call adds coordination risk: did the OCR read the chart correctly, did the generated code encode the right values, did the execution result answer the actual question, and did the model notice if any step was suspect?

AIR’s method points in the right direction: build verified code-augmented traces, filter RL data by whether tool use is appropriate, and reward interleaved trajectories rather than final answers alone. For product teams, the evaluation harness should mirror that. Track whether the model invoked the right tool, whether code executed, whether the code matched visual evidence, whether tool use was necessary, and whether the final explanation preserved the connection between image and computation.

There are real caveats. The evaluation is concentrated on mathematical reasoning benchmarks. Qwen-Max is used as a judge model for scoring outputs, which means judge calibration deserves scrutiny. The cold-start set is only about 1.5K samples, and broader multimodal tool-use generality remains unproven. A method that improves visual math may not automatically transfer to UI automation, medical images, engineering diagrams, robotics, or messy business documents.

Still, the direction is strategically important. Multimodal models are becoming agents not because they can describe images, but because they can decide what operation should happen next. Sometimes that operation is visual. Sometimes it is symbolic. Sometimes it is executable. The control problem is choosing the right mode and making the transition auditable.

LGTM’s take: AIR is a useful signpost for multimodal reasoning. The next strong MLLM is not the one that stares harder at a chart. It is the one that knows when to write a small program, verify the math, and stop calling tools once the work is done.

Sources: arXiv, AIR code/data repository, Qwen2.5-VL, MathVista, VLMEvalKit