> ## Documentation Index
> Fetch the complete documentation index at: https://phyai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# VPP2

> Run VPP2 action and video inference with CFG and tensor parallelism.

The `vpp2` plugin runs [VPP2](https://github.com/roboterax/video-prediction-policy-2)
RoboDojo and LIBERO action policies, plus standalone video prediction.

Use the PhyAI [CUDA environment](https://phyai.mintlify.site/index.md#setup) and the complete VPP2 release
directory, including `config.json`, Wan assets, tokenizer, weights, and
`dataset_stats.json`. Run the commands below from the PhyAI repository root.

## Action inference

```bash theme={null}
uv run python examples/vpp2/run_vpp2.py \
  --checkpoint /path/to/VPP2 \
  --variant robodojo_joint2b_100k \
  --image /path/to/composed-observation.png \
  --task "Pick up the object." \
  --proprio 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 \
  --steps 10 --seed 1 \
  --output action.pt
```

Replace the zero state with your measured robot state. Pass one composed image,
or separate camera paths after `--image` in the order below:

| Variant | Camera order | State values | Action shape |
| - | - | -: | - |
| `robodojo_joint2b_100k` | Head, left wrist, right wrist | 16 | `[32, 16]` |
| `libero_video10k_action30k` | External, wrist | 8 | `[32, 7]` |

The `.pt` file contains CPU tensors: `action` in the checkpoint's action units
and `normalized_action`. Add `--return-video` for a full decoded video in
`video_full`.
Action inference supports guidance scale `1` only.

For history conditioning, use `--condition-history` and
`--condition-observation`; `--help` lists their tensor formats. Your application
manages observation history and action execution.

## Video prediction

```bash theme={null}
uv run python examples/vpp2/run_vpp2.py \
  --checkpoint /path/to/VPP2 \
  --variant video_stage2_17f \
  --image /path/to/first-frame.png \
  --task "The robot picks up the object." \
  --guidance 4 --steps 30 --seed 3402 \
  --output video.pt
```

Use `video_stage1_49f` for 49 frames instead of 17. Both variants default to
30 steps and CFG scale `4`. The saved output includes `video_latents` and decoded
`video`, shaped `[1, 3, T, H, W]` with values in `[-1, 1]`.

## Multiple GPUs

Replace `uv run python` with
`uv run torchrun --standalone --nproc_per_node=N`, then append:

| Mode | N | Additional arguments |
| - | -: | - |
| Two-way TP, action or video | 2 | `--tp 2` |
| Parallel CFG, video only | 2 | `--cfg-parallel 2` |
| TP × CFG, video only | 4 | `--tp 2 --cfg-parallel 2` |

The process count must equal `tp × cfg-parallel`. DP, CP, and sequence
parallelism are not supported.

## Configuration

The example automatically selects a kernel policy. Override it with
`--kernel-config /path/to/policy.yaml`.

| Policy in `examples/vpp2/` | DiT attention | GEMM |
| - | - | - |
| `kernels_action.yaml` | FlashInfer | Torch |
| `kernels_video.yaml` | SDPA | Engine default |

Both YAML files also select the fused normalization, modulation, and RoPE
kernels. Keep the policy matched to the inference mode.

CUDA graphs are enabled by default; `--no-cuda-graph` disables them. First
requests include graph capture and compilation costs. For repeated inference,
reuse one `Engine`; see `examples/vpp2/run_vpp2.py` for the Python API and cleanup.
When constructing the engine yourself, pass the matching YAML through
`KernelConfig(config_path=...)` in `EngineConfig.kernel`.

## Performance

RoboDojo measurements use one **NVIDIA H200** GPU, `240 × 416` resolution,
17 video frames, 32 actions, and 10 Euler steps. With FlashInfer **0.7.4** and
Humming **0.1.19**:

| Request scope | Warmed latency |
| - | -: |
| Encoded conditions → GPU actions | 128.4–129.3 ms |
| Preprocessed CPU images/token IDs → encoders → CPU actions | 145.4–145.8 ms |

Ranges cover run medians, each from ten requests after three warmups. Timing
excludes image preprocessing, tokenization, checkpoint I/O, video decoding, and
first-request capture.

The line chart shows the optimization stages measured with FlashInfer **0.6.17**
and Humming **0.1.16**, before the dependency upgrade:

![VPP2 optimization stages: encoded-input latency and a separate observation-to-action series.](https://mintcdn.com/phyai/c9OJUHlGWSO5eEZ2/images/models/vpp2/acceleration.png?fit=max\&auto=format\&n=c9OJUHlGWSO5eEZ2\&q=85\&s=5e72dc9e98c80177ee32fa7f68b892aa)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.