Robot Foundation Models Can Manipulate Objects and Still Forget What Those Objects Mean

The phrase “robot foundation model” carries an implied promise: if the model learned enough from the web-scale visual-language world, then fine-tuning it into a policy should give the robot useful common sense with hands attached. Act2Answer is a useful bucket of cold water on that assumption. It asks a simple question the physical-AI hype cycle keeps skating past: after a vision-language model becomes a vision-language-action policy, does it still know what the objects mean?

The benchmark design is deliberately modest. Act2Answer takes existing VLM knowledge questions and turns them into tabletop robot episodes. Instead of answering in text, the policy must place a cube on the correct visual answer. This avoids the usual escape hatch where a robot model looks capable because it can manipulate objects, while nobody checks whether it can connect those actions to basic knowledge about color, state, time, public information, living things, or social cues.

The suite contains 1,720 unique binary questions across 12 knowledge categories, expanded into 3,440 evaluation episodes through left/right answer swapping. The authors evaluate 7 VLA models and 9 VLM baselines. Categories include emotion, attribute, state, color, shape, symmetry, counting, time, public information, traffic, celebrity, and living-world knowledge. This is not dexterous manipulation, household navigation, or long-horizon planning. It is “move the cube to the right answer.” Which is precisely why the failures matter.

Good hands, weak semantics

The project page gives the cleanest version of the problem. Some models that perform well on LIBERO-style manipulation collapse on Act2Answer’s action-grounded knowledge test: SpatialVLA: 76% LIBERO vs 5% Act2Answer, π0: 94% LIBERO vs 8% Act2Answer, and InternVLA-M1: 95% LIBERO vs 11% Act2Answer. The exact benchmark-to-benchmark comparison should not be overread — LIBERO and Act2Answer measure different things — but that is the whole point. A robot can become good at practiced manipulation routines while being terrible at expressing knowledge through action.

VLM text probes remain much stronger on many categories. InternVL3.5-38B, for example, is listed at 99% emotion, 100% color, 100% time, and 96% living world on the project table. That creates an uncomfortable gap: the knowledge may exist in the model family, but the robot policy may not preserve it, access it, or translate it into the correct motor decision after adaptation.

Magma is the interesting exception among the VLA models, scoring strongly in several categories: 72% emotion, 77% time, 88% public information, 94% celebrity, and 77% living world. That suggests the failure is not inevitable. Training choices, VQA retention, multimodal objectives, or architecture decisions can preserve more semantic capability. But it also means “we started from a strong VLM” is not enough evidence. You have to test what survived the conversion into a policy.

Fine-tuning can move capability out of reach

The paper’s layerwise probing result is the detail builders should keep. Answer-relevant signals appear to peak in middle VLA layers and attenuate in upper layers. In plain English: the model may still internally represent the right answer, but the final action pathway fails to use it. That is a different failure than “the model never knew.” It is closer to a bad interface between perception, semantics, and action.

This matters because physical AI is full of transfer assumptions. We talk as if a robot policy inherits object knowledge, commonsense priors, and web-scale visual grounding from the VLM underneath. Act2Answer says: maybe, but prove it after the policy is trained. Fine-tuning is not a lossless adapter. It can optimize for task completion while washing out, suppressing, or failing to expose capabilities that are critical in open-world environments.

For robotics teams, the practical lesson is to add knowledge-retention probes to the eval stack. If you fine-tune a VLM into a policy, do not only rerun manipulation success rates. Test whether the system still understands states, attributes, quantities, hazards, object affordances, social cues, and safety-relevant categories — and test those through the same action channel the robot will use in deployment. A model that can verbally answer “which object is dangerous for a child?” but cannot act accordingly is not safe. It is a fluent commentator strapped to unreliable hands.

The benchmark also argues for more careful post-training objectives. If semantic information is present in middle layers but does not influence the action head, then future VLA training needs auxiliary losses, probes, VQA co-training, or correction objectives that explicitly reward action-relevant knowledge. Manipulation success alone is too weak a signal. A robot that moves the cube perfectly to the wrong answer has not solved the problem; it has made the failure visually crisp.

Physical AI needs benchmark humility

Act2Answer should not be treated as the final word on robot intelligence. The tasks are binary-choice, tabletop, and intentionally constrained. But constrained tests are useful when they isolate a failure. If a model cannot express simple world knowledge by moving an object, giving it a more complex environment will not magically make the missing capability safer.

There is a product lesson here too. Any team building robots for homes, labs, warehouses, hospitals, restaurants, or farms should separate motor benchmarks from semantic benchmarks. “Can manipulate” and “can choose wisely” are different claims. The first can be measured with success rates on known tasks. The second requires adversarial, knowledge-grounded, safety-relevant evaluation. Physical AI will not earn trust by showing smoother demos; it will earn trust by failing less mysteriously when the object has meaning.

The broader model-evaluation lesson travels beyond robotics. Foundation-model adaptation always risks capability drift. A model can become better at a downstream interface while becoming worse at the knowledge behavior that made it attractive in the first place. The only adult response is to measure the retained capability after adaptation, not admire the base model’s résumé.

My take: Act2Answer is a benchmark-trust story disguised as a robotics paper. Manipulation success is table stakes. Robot foundation models need to prove they still know what they are manipulating.

Sources: arXiv, Act2Answer project page, Act2Answer code, LIBERO, OpenVLA