ARPGScaling Dexterous Manipulation
via Action Representation Pretraining

REAL-WORLD DEMONSTRATIONS

Dexterity, in motion.

Three tasks. Coordinated hands. Watch ARPG in the real world.

Each method is evaluated over 30 trials per task on the physical robot.

Figure 3 · Full-task success. Videos are edited demonstrations; rates summarize all evaluation trials.
Task	ARPG	π0.5

ARPG improves success on all three evaluated tasks, with the largest gain on pouring coffee beans: 33.3 percentage points.

VISUAL INTUITION · 26 SECONDS

Change the space.
Change the learning problem.

The same actions can be easier to model in a learned representation. Watch the data change coordinates before the generator learns their structure.

Raw action space

Learned action space

Points · action sequencesMesh · learned structure

00:00 / 00:26

How this relates to ARPG

A decoder converts generated representations into robot actions.

Conceptual animation, not a projection of measured embeddings or a training simulation. A useful representation need not be lower-dimensional or flat. Its benefit depends on the pretraining objective, motion information retained, and decoder quality; the ablation does not establish manifold flattening as the cause.

Read the abstract

THE METHOD

Learn the action representation space.
Then learn to generate representations.

Scroll to explore the three stages

Original ARPG framework. Stage 1a aligns a trainable action encoder with frozen V-JEPA 2.1 on human data. Stage 1b trains an action decoder. Stage 2 trains DiT to generate in the frozen action space, conditioned on vision and robot state. — Figure 2. Original ARPG framework. Stage 1 learns a structured, decodable action space; Stage 2 learns conditional generation in that frozen space. The action encoder constructs training targets and is not used at inference.

DEXJOCO BENCHMARK

Stronger dexterous manipulation.

Evaluated on DexJoCo.

+34.6percentage points

Bimanual success rises from 10.1% to 44.7% under multi-task training. This is the clearest gain in the benchmark: coordinating two hands across multiple tasks.

Lower deployment costs.

Peak inference VRAM2.63 GB

π0.5: 9.05 GB

Inference latency50.8 ms

π0.5: 84.9 ms

Active inference parameters0.46 B

π0.5: 3.35B · Backbone: 130M

Explore the full comparison · Table 1

DexJoCo performance and inference costs · Table 1
Setting	Method	Params (B)	VRAM (GB)	Latency (ms)	Single-arm (%)	Bimanual (%)	Overall (%)

Source: Table 1. Baseline benchmark results are reported by DexJoCo. Inference costs follow the measurement configuration in the paper; they are not hardware-independent guarantees.

SCALING PRETRAINING

More diverse human actions.
Better action generation.

Can a better pretrained action space improve manipulation without more robot demonstrations or a larger generative backbone? We vary three pretraining dimensions separately.

ABLATIONS & ANALYSIS

The action space matters.

With robot demonstrations, observations, and the downstream generator fixed, visual alignment produces the strongest representation for generation among the evaluated objectives.

Action representation

Table 2(a) · DexJoCo overall success (%)

Additional ablations

Multi-layer aggregation

Final encoder layers	Success (%)
1	52.5
2	57.8
3	51.6

Auxiliary generation objectives

Objective	Success (%)
Without REPA alignment	54.85
Without action decoding loss	56.0
Full objective	57.8

ARPGScaling Dexterous Manipulationvia Action Representation Pretraining