What problem am I trying to solve?
I investigated whether there is an instruction following pathway for the instructions given to Vision Language Action (VLA) models and if it can be edited to influence real world behavior. This model biology question was inspired by Othello (Nanda et al.), especially using a model native coordinate system to solve for a hidden world state (latent space or variable). While the original paper intervened in the residual stream of a custom Transformer to predict the next move on boards, we edited the VLA’s image token positions to redirect a robot’s simulated behavior.
I hypothesized that fine-grained LLM style steering can successfully be applied to VLAs to change high level robot actions.
Why is this important?
Fine-grained control of models is increasingly important to solve alignment problems that reinforcement learning has limitations on. Models can behave in a black box and exhibit evolutionary greedy algorithm-like behavior when objective hacking (Hugging Face incidents). This is especially salient as we scale physical AI in the real world with VLAs and world models, some of which operate in latent space (JEPA, DINO).
These steering methods have seen early results that are promising with CAFT in LLMs and COAST in VLAs (Casademunt et al, Miao et al). Contributing to the applied interpretability toolkit for models that can take actions with real consequences is critical. This knowledge can help diagnose unsafe decisions and know which internal signals can control the models’ actions.
Most of all, LLMs often support cleaner token level forensics. I was interested in a less developed problem (VLAs) that had an ambiguous problem space and systematically use clean, mechanistic interpretability techniques to make a contribution to that space.
Finding 1: A broad image-state edit changed the robot’s action
I found that instructions did control actions (18/20 of the episodes succeeded). However, we had to replace all 512 Transformer positions in layers 12–17 to successfully intervene on robot behavior. This suggests the instruction effect is spread broadly across the model’s layers instead of in a few select positions. We found no evidence that fine-grained ablations at the token position level can successfully steer π0.5 in simulated robotics tasks, which responded only to broad edits. One possibility is that semantic representations of high-level instructions ("pick this object up") are spread across multiple token positions and layers, and may change during action generation.
As shown in the first graph above, we did find a pathway when giving the model the wrong instruction in π0.5 (and patching in the image position activations from the correct instruction scenes). However, this only had a significant effect after ablating much of the broader image token layers. This test assessed whether the internal visual signal could override the wrong instructions. This is interesting especially since vision is thought of as a “second class citizen” in VLAs by some robotics researchers (Jim Fan 2026).
High-resolution reruns of the illustrated case: matching task outcomes, with up to one simulator step of timing variation.
Code found here
Finding 2: Readable language was not controlling behavior
Conversely, editing text token states barely affects actions. It only has a 1% effect. In early layers, the model may keep the copy of the instructions in the text token states. However, these instructions may no longer directly control a robot’s actions after the model processes a scene.
Instruction classification
Action progress after replacement
Model biology for physical AI
Dissecting the model biology of models that operate with real world consequences is essential. It has practical safety implications, particularly since fields such as robotics are traditionally more closed-source when it comes to benchmarks, data, and model weights.
Representation geometry
However, VLAs and world models have a tricky problem. They have longer causal chains than LLMs. Visual concept, language, and geometry can be entangled. And especially for JEPA family architecture, the model may learn its own private coordinate system in latent state space. This can mean mapping pathways or mechanisms that can be more easily found in LLMs, e.g. those associated with safety or obedience, may involve more complex, representational geometry questions.
Select a label to highlight its points.
Future directions
The experiments used a limited set of simulated pick-and-place tasks, so they do not establish how broadly this intervention pattern generalizes. Perhaps the highest ROI intervention would have been to choose a stimuli set that specifically involved this stop/go mechanism to directly address the obedience mechanism that we were looking for.
Supporting figures
Code and further reading
- Experiment code and research archive
- Emergent Linear Representations in World Models of Self-Supervised Sequence Models — Nanda et al.
- Toy Models of Superposition — Elhage et al.
- Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning — Casademunt et al.
- Contrastive Conceptor Activation Steering (COAST) — Miao et al.
- Mechanistic Interpretability for Steering Vision-Language-Action Models — Häon et al.
- Not All Features Are Created Equal — Grant, Zhao, and Wang.
- Action Atlas — Interactive mechanistic interpretability explorer for vision-language-action models.
Related work already demonstrates VLA steering. This report focuses on the text/image cache comparison, the breadth of the tested intervention, and its behavioral validation in this checkpoint.