CLAP: Cross-Embodiment Action-Conditioned Video World Models are Zero-Shot Physical Simulators

Abstract

Overview

CLAP unifies human and robot action spaces via end-effector poses, natural language, and learned latent actions. A two-stage curriculum first learns unsupervised physical priors from (large-scale) unlabeled videos using latent actions, then grounds them in end-effector space for direct, zero-shot deployment to real-world tasks, e.g., inference-time planning and RL fine-tuning for robot policies such as \(\pi_{0.5}\) and MolmoAct-2. This recipe establishes a novel paradigm for training high-fidelity single-embodiment video world models via sample-efficient, few-shot adaptation of cross-embodiment models to target robot platforms.

Motivation

How CLAP Works

CLAP is grounded in the insight that universal physical laws govern spatiotemporal dynamics regardless of the actor. For example, a bottle tips, a towel folds, and a drawer slides according to the same physics whether the actor is a Franka arm, a WidowX gripper, or a human hand, underscoring that the appropriate response to heterogeneity is not to restrict training to one embodiment. CLAP harmonizes the disparate action spaces of diverse embodiments using end-effector poses, natural language instructions, and learned latent action representations.

Action-conditioning spaces

Curriculum-based recipe

Harmonized action representations have native limitations. For example, while latent actions enable learning from unlabeled videos, they do not facilitate zero-shot real-world deployment. Conversely, while end-effector actions address this drawback, they preclude training on unlabeled video data. CLAP introduces a curriculum-based cross-embodiment learning framework to overcome these limitations.

Architecture

CLAP is architecture-agnostic, integrating seamlessly with U-Nets, diffusion transformers, decoder-only transformers, and hybrid designs. It is similarly flexible across training objectives, supporting diffusion, flow-matching, and next-token prediction losses. Enabled by a Stable Video Diffusion (SVD) backbone, we exhaustively evaluate diverse conditioning schemes and data mixtures at a fraction of the computational cost. Moreover, unlike many newer architectures, SVD enables local inference on budget-friendly consumer hardware: CLAP requires under 12 GB of VRAM and runs comfortably on an RTX 3060. While CLAP currently relies on the diffusion framework for seamless integration, future iterations will adopt flow-matching to accelerate inference.

CLAP Models

Trained on a mix of Open X-Embodiment datasets and EgoDex (for 100K steps), CLAP yields a broad suite of cross-embodiment video models across diverse action spaces with checkpoints available at omni-CLAP/CLAP. Via sample-efficient finetuning, we adapt these models to pretrained and novel embodiments, including the bimanual YAM and G1 humanoid.

Cross-embodiment models

Novel-embodiment adaptation

Single-platform post-trained models

The full list of checkpoints, including additional single-platform variants, is in the repo README.

01Can cross-embodiment video world models outperform single-embodiment models?

We train CLAP (end-effector, language, and latent models) for 100K steps , benchmarking against state-of-the-art (SOTA) single-embodiment baselines: Ctrl-World on DROID and a new SOTA Bridge baseline.

Surprisingly, despite sharing identical architectures and training-step counts and using fewer domain-specific samples, CLAP matches the SOTA DROID baseline and achieves high-accuracy predictions on Bridge. This challenges the prevailing wisdom that cross-embodiment models require increased capacity or massive single-domain data unions, suggesting that CLAP learns useful generalizable physical priors from heterogeneous data. Ultimately, this paradigm has the potential to transform how action-conditioned video models are trained, mirroring recent breakthroughs in multi-modal foundation models for vision and language generation. Although CLAP does not quite match the Bridge baseline, the qualitative differences are mostly marginal. (See the paper for more detailed results.)

02Zero-Shot Generalization to Real-World Robotics

Beyond offline rollouts, CLAP's cross-embodiment models generalize zero-shot to real-world robot hardware with no target-embodiment fine-tuning, powering real-world inference-time cross-policy planning and reinforcement-learning fine-tuning for robot policies such as \(\pi_{0.5}\) and MolmoAct-2 on the DROID and Bimanual YAM platforms. While inference-time planning robustifies baseline policies against semantic confusion and jittery trajectories, the planner remains vulnerable to world model hallucinations. (See the paper for more results.)

03Sample-Efficient
Novel-Embodiment Adaptation

Beyond zero-shot transfer, CLAP establishes a paradigm for training high-fidelity single-embodiment video world models via sample-efficient, few-shot adaptation to target robot platforms it was never trained on. We adapt the cross-embodiment model clap-curr to the 14-dimensional Bimanual YAM and the 26-dimensional G1 Humanoid. Within a single gradient step, our adapted models generate meaningful video predictions, unlike models trained from raw SVD backbones.

Conclusion

CLAP is a cross-embodiment, action-conditioned video modeling framework that learns generalizable physical priors from heterogeneous human and robot data. By bridging disparate action spaces through a curriculum combining latent and end-effector actions, CLAP enables effective data scaling on unlabeled videos with no action-space mismatch in deployment. CLAP approaches or surpasses single-embodiment baselines and achieves zero-shot generalization and sample-efficient adaptation, enabling effective inference-time cross-policy planning and RL finetuning.

Future work will focus on mitigating video hallucinations (e.g., via uncertainty quantification) and extending our pipeline to bimanual and humanoid data to further scale diversity and adaptation efficiency. Additionally, balancing these scaling efforts with efficient model architectures will be essential to manage computational costs.

Ultimately, our work establishes cross-embodiment learning as a foundational catalyst for breakthroughs in video world modeling.

Acknowledgments

BibTeXComing soon

@misc{clap2026crossembodiment,
  title={CLAP: Cross-Embodiment Action-Conditioned Video World Models are Zero-Shot Physical Simulators},
  author={Coming soon: author list},
  year={2026},
  eprint={Coming soon: arXiv ID},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={Coming soon},
}