VPP2

Run VPP2 action and video inference with CFG and tensor parallelism.

The vpp2 plugin runs VPP2 RoboDojo and LIBERO action policies, plus standalone video prediction.

Use the PhyAI CUDA environment and the complete VPP2 release directory, including config.json, Wan assets, tokenizer, weights, and dataset_stats.json. Run the commands below from the PhyAI repository root.

​

Action inference

uv run python examples/vpp2/run_vpp2.py \
  --checkpoint /path/to/VPP2 \
  --variant robodojo_joint2b_100k \
  --image /path/to/composed-observation.png \
  --task "Pick up the object." \
  --proprio 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 \
  --steps 10 --seed 1 \
  --output action.pt

Replace the zero state with your measured robot state. Pass one composed image, or separate camera paths after --image in the order below:

Variant	Camera order	State values	Action shape
robodojo_joint2b_100k	Head, left wrist, right wrist	16	[32, 16]
libero_video10k_action30k	External, wrist	8	[32, 7]

The .pt file contains CPU tensors: action in the checkpoint’s action units and normalized_action. Add --return-video for a full decoded video in video_full. Action inference supports guidance scale 1 only.

For history conditioning, use --condition-history and --condition-observation; --help lists their tensor formats. Your application manages observation history and action execution.

​

Video prediction

uv run python examples/vpp2/run_vpp2.py \
  --checkpoint /path/to/VPP2 \
  --variant video_stage2_17f \
  --image /path/to/first-frame.png \
  --task "The robot picks up the object." \
  --guidance 4 --steps 30 --seed 3402 \
  --output video.pt

Use video_stage1_49f for 49 frames instead of 17. Both variants default to 30 steps and CFG scale 4. The saved output includes video_latents and decoded video, shaped [1, 3, T, H, W] with values in [-1, 1].

​

Multiple GPUs

Replace uv run python with uv run torchrun --standalone --nproc_per_node=N, then append:

Mode	N	Additional arguments
Two-way TP, action or video	2	--tp 2
Parallel CFG, video only	2	--cfg-parallel 2
TP × CFG, video only	4	--tp 2 --cfg-parallel 2

The process count must equal tp × cfg-parallel. DP, CP, and sequence parallelism are not supported.

​

Configuration

The example automatically selects a kernel policy. Override it with --kernel-config /path/to/policy.yaml.

Policy in examples/vpp2/	DiT attention	GEMM
kernels_action.yaml	FlashInfer	Torch
kernels_video.yaml	SDPA	Engine default

Both YAML files also select the fused normalization, modulation, and RoPE kernels. Keep the policy matched to the inference mode.

CUDA graphs are enabled by default; --no-cuda-graph disables them. First requests include graph capture and compilation costs. For repeated inference, reuse one Engine; see examples/vpp2/run_vpp2.py for the Python API and cleanup. When constructing the engine yourself, pass the matching YAML through KernelConfig(config_path=...) in EngineConfig.kernel.

​

Performance

RoboDojo measurements use one NVIDIA H200 GPU, 240 × 416 resolution, 17 video frames, 32 actions, and 10 Euler steps. With FlashInfer 0.7.4 and Humming 0.1.19:

Request scope	Warmed latency
Encoded conditions → GPU actions	128.4–129.3 ms
Preprocessed CPU images/token IDs → encoders → CPU actions	145.4–145.8 ms

Ranges cover run medians, each from ten requests after three warmups. Timing excludes image preprocessing, tokenization, checkpoint I/O, video decoding, and first-request capture.

The line chart shows the optimization stages measured with FlashInfer 0.6.17 and Humming 0.1.16, before the dependency upgrade: