> ## Documentation Index
> Fetch the complete documentation index at: https://phyai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Single-GPU inference for GR00T-N1.7

> How PhyAI runs GR00T-N1.7 inference on a single GPU

export const ModelCard = ({title, subtitle, icon, rows = {}}) => {
  const entries = Object.entries(rows);
  const renderValue = value => {
    if (value === null || value === undefined) {
      return <span className="text-sm text-zinc-400 dark:text-zinc-600">—</span>;
    }
    if (Array.isArray(value)) {
      return <div className="flex flex-wrap gap-1.5">
                    {value.map((v, i) => <span key={i} className="inline-flex items-center px-2 py-0.5 rounded-md text-[11.5px] font-medium bg-[#003399]/[0.06] text-[#003399] ring-1 ring-inset ring-[#003399]/15 dark:bg-[#60A5FA]/[0.10] dark:text-[#60A5FA] dark:ring-[#60A5FA]/20">
                            {v}
                        </span>)}
                </div>;
    }
    if (typeof value === "string" || typeof value === "number") {
      return <span className="text-sm text-zinc-800 dark:text-zinc-100 break-words">
                    {value}
                </span>;
    }
    return value;
  };
  const hasHeader = title || subtitle || icon;
  return <div className="not-prose my-6 overflow-hidden rounded-xl bg-white dark:bg-zinc-900 ring-1 ring-zinc-200 dark:ring-zinc-800 shadow-[0_1px_2px_rgb(15_23_42_/_0.04),0_4px_16px_-4px_rgb(15_23_42_/_0.06)] dark:shadow-[0_1px_0_rgb(255_255_255_/_0.04)_inset,0_8px_24px_-8px_rgb(0_0_0_/_0.5)]">
            {hasHeader && <div className="flex items-center gap-3.5 px-5 py-4 bg-zinc-50/60 dark:bg-zinc-800/20 border-b border-zinc-200/80 dark:border-zinc-800/80">
                    {icon && <div className="flex h-10 w-10 shrink-0 items-center justify-center rounded-[10px] bg-gradient-to-br from-[#003399] to-[#2563EB] text-white text-lg font-semibold ring-1 ring-inset ring-white/10 shadow-[0_1px_2px_rgb(0_51_153_/_0.25),0_3px_6px_-2px_rgb(0_51_153_/_0.18)]">
                            {icon}
                        </div>}
                    <div className="min-w-0">
                        {title && <div className="text-[15px] font-semibold tracking-tight text-zinc-900 dark:text-zinc-50">
                                {title}
                            </div>}
                        {subtitle && <div className="mt-0.5 text-xs text-zinc-500 dark:text-zinc-400">
                                {subtitle}
                            </div>}
                    </div>
                </div>}

            <div>
                {entries.map(([key, value], i) => <div key={key} className={`flex items-stretch ${i < entries.length - 1 ? "border-b border-zinc-100 dark:border-zinc-800/60" : ""}`}>
                        <div className="w-44 shrink-0 flex items-center px-5 py-3 text-[13px] font-medium text-zinc-500 dark:text-zinc-400">
                            {key}
                        </div>
                        <div className="flex-1 flex items-center px-5 py-3 min-w-0">
                            {renderValue(value)}
                        </div>
                    </div>)}
            </div>
        </div>;
};

<ModelCard
  title="GR00T-N1.7"
  subtitle="Vision-Language-Action · Single-GPU Inference"
  icon="G"
  rows={{
"Model Type": "VLA",
"Weights": <a href="https://huggingface.co/collections/nvidia/gr00t-n17" target="_blank" rel="noreferrer" className="text-sm text-[#003399] dark:text-[#60A5FA] underline underline-offset-2 hover:opacity-80 break-all">NVIDIA GR00T-N1.7 collection</a>,
"Tags": ["VLA", "Cosmos-Reason2-2B", "Qwen3-VL", "flow-matching", "single-GPU"],
"Inputs": "RGB cameras · robot state · language",
"Entry Point": <code className="px-2 py-0.5 rounded bg-[#003399]/10 dark:bg-[#60A5FA]/15 text-[#003399] dark:text-[#60A5FA] text-xs font-mono">GR00TN17WS1Scheduler</code>,
"Plugin": <code className="px-2 py-0.5 rounded bg-[#003399]/10 dark:bg-[#60A5FA]/15 text-[#003399] dark:text-[#60A5FA] text-xs font-mono">gr00t_n17</code>,
"Param Precision": "bf16",
"Paper": <a href="https://arxiv.org/abs/2503.14734" target="_blank" rel="noreferrer" className="text-sm text-[#003399] dark:text-[#60A5FA] underline underline-offset-2 hover:opacity-80 break-all">arxiv.org/abs/2503.14734</a>,
}}
/>

# Overview

[GR00T-N1.7](https://github.com/NVIDIA/Isaac-GR00T) is a vision-language-action (VLA) model. Its [Cosmos-Reason2-2B](https://huggingface.co/nvidia/Cosmos-Reason2-2B) backbone, based on the Qwen3-VL architecture, encodes camera images and the language instruction. A flow-matching action head then denoises an action chunk using those features and the robot state.

The embodiment ID is an explicit routing input rather than a sequence token passed to the action transformer. PhyAI uses it to select embodiment-specific state and action encoders and the action decoder. The action transformer receives the resulting encoded features, so the embodiment affects its inputs indirectly, but the raw ID is not appended to the token sequence.

PhyAI's `ws1` path runs the backbone and action head on one GPU. The engine receives model-ready tensors and returns a normalized action chunk. `GR00TProcessor` handles image transforms, tokenization, state normalization, embodiment metadata, and action decoding outside the engine.

<Note>
  The examples on this page were validated with the official `nvidia/GR00T-N1.7-LIBERO` weights and its `libero_10` checkpoint directory. Use model and processor files from the same compatible checkpoint bundle so the model geometry, modality definitions, normalization statistics, and embodiment routing stay aligned.
</Note>

# Architecture

The `gr00t_n17` plugin follows PhyAI's <Tooltip headline="Engine + plugin" tip="Engine resolves an Entry by plugin name. The Entry builds the model, loads weights, creates the scheduler, and forwards each request to scheduler.step().">engine + plugin contract</Tooltip>:

<Tree>
  <Tree.Folder name="phyai/src/phyai/models/gr00t_n17" defaultOpen>
    <Tree.File name="main_gr00t_n17.py" />

    <Tree.File name="scheduler_ws1_gr00t_n17.py" />

    <Tree.File name="model_runner_gr00t_n17.py" />

    <Tree.File name="modeling_gr00t_n17.py" />

    <Tree.File name="qwen3_vl_adapter.py" />

    <Tree.File name="configuration_gr00t_n17.py" />
  </Tree.Folder>

  <Tree.Folder name="phyai-utils-tools/src/phyai_utils_tools/models/gr00t" defaultOpen>
    <Tree.File name="processor_gr00t.py" />

    <Tree.File name="ops_gr00t.py" />
  </Tree.Folder>
</Tree>

| Component                  | Responsibility                                                                                                                                              |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GR00TN17Entry`            | Registers the `"gr00t_n17"` plugin, loads the checkpoint, and creates the single-GPU scheduler                                                              |
| `GR00TN17BackboneRunner`   | Runs the Cosmos-Reason2/Qwen3-VL vision and language backbone; owns shape-keyed CUDA graphs                                                                 |
| `GR00TN17ActionHeadRunner` | Selects embodiment-specific state/action encoder and action-decoder projections, samples or accepts action noise, and runs the flow-matching denoising loop |
| `GR00TN17WS1Scheduler`     | Prepares and dispatches request tensors to the backbone and action-head runners, then returns the normalized action chunk                                   |
| `GR00TProcessor`           | Converts raw observations to model tensors and decodes normalized chunks into the selected embodiment's action fields in physical units                     |

The request path is:

```text theme={null}
raw cameras + state + instruction
  -> GR00TProcessor.process_observation(...)
  -> Qwen3-VL image/text tensors + normalized padded state + embodiment ID
  -> GR00TN17BackboneRunner
  -> vision-language token features
  -> action head:
       embodiment ID -> state encoder
       noise + timestep + embodiment ID -> action encoder
       state/action features attend to vision-language features in AlternateVLDiT
       embodiment ID -> action decoder -> velocity update (repeated)
  -> masked normalized action chunk
  -> GR00TProcessor.decode_action(..., raw_state=...)
  -> selected embodiment's action fields in physical units
```

# Running GR00T-N1.7

<Steps>
  <Step title="Prepare the checkpoint">
    Choose a compatible checkpoint from NVIDIA's [GR00T-N1.7 collection](https://huggingface.co/collections/nvidia/gr00t-n17). GR00T-N1.7 uses the gated `nvidia/Cosmos-Reason2-2B` backbone metadata for tokenization and image preprocessing, so request access and authenticate before the first run:

    ```bash theme={null}
    uv run hf auth login
    ```

    The LIBERO checkpoint stores model files under `libero_10`. Download the files consumed by PhyAI:

    ```bash theme={null}
    uv run hf download nvidia/GR00T-N1.7-LIBERO \
      --include "libero_10/config.json" \
      --include "libero_10/embodiment_id.json" \
      --include "libero_10/model-*.safetensors" \
      --include "libero_10/model.safetensors.index.json" \
      --include "libero_10/processor_config.json" \
      --include "libero_10/statistics.json" \
      --local-dir checkpoints/GR00T-N1.7-LIBERO
    ```
  </Step>

  <Step title="Construct the processor">
    Use the same checkpoint directory for both. `--online` in the bundled example allows the first run to fetch uncached Cosmos-Reason2 tokenizer and preprocessor files.

    ```python theme={null}
    import torch
    from pathlib import Path

    from phyai.engine import Engine, EngineArgs
    from phyai.engine_config import DeviceConfig, EngineConfig, RuntimeConfig
    from phyai.models.gr00t_n17.configuration_gr00t_n17 import GR00TN17Config
    from phyai.models.gr00t_n17.main_gr00t_n17 import GR00TN17Args
    from phyai.utils import load_config
    from phyai_utils_tools.models.gr00t import GR00TProcessor

    checkpoint_dir = Path("checkpoints/GR00T-N1.7-LIBERO/libero_10")
    loading_kwargs = {"trust_remote_code": False, "local_files_only": False}
    cfg = load_config(checkpoint_dir, GR00TN17Config)

    processor = GR00TProcessor.from_pretrained(
        checkpoint_dir,
        embodiment_tag="LIBERO_PANDA",
        model_name=cfg.backbone.model_name,
        transformers_loading_kwargs=loading_kwargs,
    )

    ```

    Set `local_files_only=True` after the Cosmos-Reason2 tokenizer and preprocessor files are present in the local Hugging Face cache.
  </Step>

  <Step title="Prepare a request and construct the engine">
    Build `GR00TObservation` with the camera, state, and language keys listed in the checkpoint's modality config. The processor validates these keys and their history lengths before inference.

    | Field            | Value shape       | Dtype        |
    | ---------------- | ----------------- | ------------ |
    | `video[view]`    | `(B, T, H, W, 3)` | `np.uint8`   |
    | `state[name]`    | `(B, T, D)`       | `np.float32` |
    | `language[name]` | nested `(B, T)`   | `str`        |

    After constructing `observation` from this contract, continue with:

    ```python theme={null}
    from phyai.models.gr00t_n17.scheduler_ws1_gr00t_n17 import GR00TN17Request

    prepared = processor.process_observation(observation)
    request = GR00TN17Request(tensors=prepared.tensors)

    engine = Engine(
        EngineArgs(
            plugin="gr00t_n17",
            plugin_args=GR00TN17Args(
                checkpoint_dir=checkpoint_dir,
                max_batch_size=1,
                capture_profiles=(request,),
            ),
            config=EngineConfig(
                device=DeviceConfig(
                    target="cuda",
                    params_dtype=torch.bfloat16,
                ),
                runtime=RuntimeConfig(use_cuda_graph=True),
            ),
        )
    )

    normalized_action = engine.step(request)
    action = processor.decode_action(
        normalized_action,
        raw_state=prepared.raw_state,
    )
    ```

    The processor reads the required camera views, state fields, language key, and history lengths from the selected checkpoint. Keep `prepared.raw_state` for checkpoints that use relative actions; the decoder needs it to reconstruct actions in the robot's reference frame.

    With CUDA graphs enabled, `capture_profiles` are stabilized and captured while the engine is constructed. Profiles with the same complete Backbone and Action Head Graph structure are deduplicated before warmup, their outputs are discarded, and matching runtime requests only replay those graphs. The Action Head reuses the Backbone sequence bucket, so prompts in the same bucket share both graphs when their remaining Graph-key fields also match. Pass prepared requests covering every input structure this engine will serve; the fixed LIBERO example needs only the profile shown above. Add profiles if the same engine serves other sequence-length buckets, image-grid or camera layouts, batch shapes, embodiment categories, action shapes, or mask structures. An unlisted graph-compatible structure raises an error instead of capturing or replacing graphs at runtime. Fixed Graph mode spans both runners: if either runner does not support capture, such as an Action Head configured with FlashInfer attention, the scheduler disables CUDA Graphs for both and runs the complete request eagerly.

    `max_batch_size` is the scheduler's upper bound. In CUDA Graph mode, every runtime batch shape must also match a setup profile; a smaller but uncaptured batch does not reuse a larger-batch graph. Rebuild the engine with the required profiles if the served batch shapes change.

    `GR00TN17Request.noise` is optional. Leave it unset to sample Gaussian noise, or provide a fixed tensor for deterministic regression checks.
  </Step>

  <Step title="Close the engine">
    ```python theme={null}
    engine.close()
    ```

    This releases the runner-owned CUDA graph registries and model references.
  </Step>
</Steps>

# End-to-end example

`examples/gr00t/run_gr00t.py` builds a checkpoint-shaped synthetic observation, runs it through the processor and engine, and decodes the result. To run it:

```bash theme={null}
uv run python examples/gr00t/run_gr00t.py \
  --checkpoint checkpoints/GR00T-N1.7-LIBERO/libero_10 \
  --embodiment-tag LIBERO_PANDA
```

The script prints `engine.step()` latency statistics (mean / median / std / min / max over 3 untimed runs + 30 timed runs). CUDA Graph capture happens during engine construction; the untimed steps only stabilize the latency measurement. Observation preprocessing, input transfer to the GPU, and action decoding are outside the timed loop.

The synthetic inputs only test the execution path; their predicted actions have no task-level meaning. The command assumes the Cosmos-Reason2 tokenizer and preprocessor are already cached. Add `--online` to the first run only if those files are missing, then omit it for later runs. To use a specific local tokenizer/preprocessor snapshot, pass `--processor-model-name-or-path <path>`; this option does not override the engine's Backbone config or weights. The built-in Cosmos-Reason2 processor does not require remote code. Only pass `--trust-remote-code` when using a custom processor repository whose code you trust.

# Current limitations

* The scheduler supports one GPU. Tensor parallelism, continuous batching, preemption, and a network policy server are outside this path.
* `max_batch_size` is fixed when the engine is built. CUDA Graph mode also requires every served batch shape to be represented in `capture_profiles`.
* The engine returns normalized, padded actions. Use the matching processor and `raw_state` to recover the checkpoint's physical action fields.

# Full example

```python theme={null}
from pathlib import Path

import numpy as np
import torch

from phyai.engine import Engine, EngineArgs
from phyai.engine_config import DeviceConfig, EngineConfig, RuntimeConfig
from phyai.models.gr00t_n17.configuration_gr00t_n17 import GR00TN17Config
from phyai.models.gr00t_n17.main_gr00t_n17 import GR00TN17Args
from phyai.models.gr00t_n17.scheduler_ws1_gr00t_n17 import GR00TN17Request
from phyai.utils import load_config
from phyai_utils_tools.models.gr00t import GR00TObservation, GR00TProcessor

CHECKPOINT_DIR = Path(
    "/path/to/GR00T-N1.7-LIBERO/libero_10"
)  # change to your local checkpoint folder
BATCH_SIZE = 1
IMAGE_SIZE = 256
EMBODIMENT_TAG = "LIBERO_PANDA"

cfg = load_config(CHECKPOINT_DIR, GR00TN17Config)
loading_kwargs = {"trust_remote_code": False, "local_files_only": True}

processor = GR00TProcessor.from_pretrained(
    CHECKPOINT_DIR,
    embodiment_tag=EMBODIMENT_TAG,
    model_name=cfg.backbone.model_name,
    transformers_loading_kwargs=loading_kwargs,
)
embodiment_tag = processor.embodiment_tag

# Build a synthetic observation from the checkpoint's modality contract.
modality_cfg = processor.modality_config
video = {}
for key in modality_cfg["video"].modality_keys:
    history = len(modality_cfg["video"].delta_indices)
    video[key] = np.random.randint(
        0,
        256,
        size=(BATCH_SIZE, history, IMAGE_SIZE, IMAGE_SIZE, 3),
        dtype=np.uint8,
    )

state = {}
for key in modality_cfg["state"].modality_keys:
    history = len(modality_cfg["state"].delta_indices)
    state_dim = int(processor.norm_params[embodiment_tag]["state"][key]["dim"])
    state[key] = (
        np.random.rand(BATCH_SIZE, history, state_dim).astype(np.float32) * 2 - 1
    )

language_key = modality_cfg["language"].modality_keys[0]
language_history = len(modality_cfg["language"].delta_indices)
observation = GR00TObservation(
    video=video,
    state=state,
    language={
        language_key: [
            ["pick up the object" for _ in range(language_history)]
            for _ in range(BATCH_SIZE)
        ]
    },
)
prepared = processor.process_observation(observation)
request = GR00TN17Request(tensors=prepared.tensors)

engine = Engine(
    EngineArgs(
        plugin="gr00t_n17",
        plugin_args=GR00TN17Args(
            checkpoint_dir=CHECKPOINT_DIR,
            max_batch_size=BATCH_SIZE,
            capture_profiles=(request,),
        ),
        config=EngineConfig(
            device=DeviceConfig(target="cuda", params_dtype=torch.bfloat16),
            runtime=RuntimeConfig(use_cuda_graph=True),
        ),
    )
)

try:
    # Run one model step and decode the physical action fields.
    normalized_action = engine.step(request)
    action = processor.decode_action(
        normalized_action,
        raw_state=prepared.raw_state,
    )

    print(f"normalized action shape={tuple(normalized_action.shape)}")
    print(f"decoded action keys={sorted(action)}")
finally:
    engine.close()
```