Benjamin Satzger

Principal Software Engineer

HGX is NVIDIA’s server-side reference platform for dense GPU compute. Instead of using PCIe cards connected through the host’s PCIe bus, HGX systems use SXM modules – GPUs mounted directly to a shared baseboard. NVIDIA’s earlier generation GPUs like Hopper came in both SXM and PCIe versions, but the B200 ships only with the SXM version.

Also, even when H100 GPUs use SXM modules, their HGX baseboard layouts look different than the B200s.

Within an HGX system, GPUs communicate through NVLink, which provides high-bandwidth GPU-to-GPU connectivity. NVSwitch modules merge these connections into a uniform all-to-all fabric, so every GPU can reach every other GPU with consistent bandwidth and latency. This creates a tightly integrated multi-GPU module rather than a collection of independent devices.

In short, the B200 HGX platform’s uniform, high-bandwidth architecture is excellent for performance – but less friendly to virtualization than discrete PCIe GPUs.

Full Passthrough Mode

In Full Passthrough Mode, a VM receives direct access to the GPUs it is assigned. For multi-GPU configurations, the VM also takes ownership of the associated NVSwitch fabric, running both the NVIDIA driver and Fabric Manager inside the guest. On an HGX B200 system, this results in two configurations:

• Single 8-GPU VM: Pass all 8 GPUs plus the NVSwitches to one VM. The guest owns the entire HGX complex and runs Fabric Manager, with full NVLink connectivity between all GPUs.
• Multiple 1-GPU VMs: Disable NVLink for the GPU(s) and pass through a single GPU per VM. Each GPU then appears as an isolated PCIe-like device with no NVSwitch participation and no NVLink peer-to-peer traffic. 

Shared NVSwitch Multitenancy Mode

GPUs are grouped into partitions. A partition acts like an isolated NVSwitch island. Tenants can receive 1, 2, 4, or 8 GPUs. GPUs inside a partition retain full NVLink bandwidth, while GPUs in different partitions cannot exchange traffic. Fabric Manager manages routing and enforces isolation between partitions.

vGPU-based Multitenancy Mode

vGPU uses mediated device slicing to allow multiple VMs to share a single physical GPU. The GPU’s memory and compute resources are partitioned, and NVLink/NVSwitch are not exposed to the guest. This mode is optimized for light compute workloads rather than high-performance inference or training workloads.

Why Ubicloud Uses “Shared NVSwitch Multitenancy”

While the B200 GPUs are SXM modules, the Linux kernel still exposes them as PCIe devices. The procedure for preparing them for passthrough is similar: detach the GPUs from the host’s NVIDIA driver and bind them to the vfio-pci driver so that a hypervisor can assign them to a VM.

You can inspect the B200 GPUs via PCI ID 10de:2901:

During development, it’s common to switch between using the GPUs locally on the host and passing them through to a guest. The nvidia driver lets the host OS use the GPU normally, while vfio-pci isolates the GPU so a VM can control it. When a GPU is bound to vfio-pci, host tools like nvidia-smi won’t work. So switching drivers lets you alternate between host-side development and VM passthrough testing.

You can dynamically rebind the GPUs between the nvidia and vfio-pci drivers using their PCI bus addresses:

1. Configure IOMMU and VFIO PCI IDs in GRUB

Enable the IOMMU in passthrough mode and instruct the kernel to bind 10de:2901 devices to vfio-pci:

# Edit /etc/default/grub to include:
GRUB_CMDLINE_LINUX_DEFAULT="... intel_iommu=on iommu=pt 
vfio-pci.ids=10de:2901"

Then apply the changes:

update-grub

2. Preload VFIO Modules

To guarantee the VFIO driver claims the devices before any other driver can attempt to initialize them, we ensure the necessary kernel modules are loaded very early during the boot process.

tee /etc/modules-load.d/vfio.conf <vfio
vfio_iommu_type1
vfio_pci
EOF

3. Blacklist Host NVIDIA Drivers

To prevent any potential driver conflicts, we stop the host kernel from loading the standard NVIDIA drivers by blacklisting them. This is essential for maintaining vfio-pci ownership for passthrough.

tee /etc/modprobe.d/blacklist-nvidia.conf <blacklist nouveau
options nouveau modeset=0
blacklist nvidia
blacklist nvidia_drm
blacklist nvidiafb
EOF

4. Update Initramfs and Reboot

Finally, apply all the module and driver configuration changes to the kernel’s initial ramdisk environment and reboot the host system for the new configuration to take effect.

update-initramfs -u
reboot

After the reboot, verification is key. Running lspci -k -d 10de:2901 should show all 8 GPUs are now correctly bound to the vfio-pci driver, confirming the host is ready for passthrough.All GPUs should show Kernel driver in use: vfio-pci.

Once the host’s GPUs are configured for being passed through, the next critical requirement is ensuring that the NVIDIA driver stack on the host and inside each VM are aligned. Unlike full passthrough mode – where each VM initializes its own GPUs and NVSwitch fabric – Shared NVSwitch Multitenancy places Fabric Manager entirely on the host or a separate service vm. The host (or the service vm) is responsible for bringing up the NVSwitch topology, defining GPU partitions, and enforcing isolation between tenants.

Because of this architecture, the VM’s GPU driver must match the host’s Fabric Manager version exactly. Even minor mismatches can result in CUDA initialization failures, missing NVLink connectivity, or cryptic runtime errors.

A second important requirement for the B200 HGX platform is that it only supports the NVIDIA “open” driver variant. The legacy proprietary stack cannot operate the B200. Both host and guest must therefore use the nvidia-open driver family.

Host Configuration

On the host, after enabling the CUDA repository, install the components that bring up and manage the NVSwitch fabric:

apt install nvidia-fabricmanager nvlsm

You can verify the installed Fabric Manager version with:

dpkg -l nvidia-fabricmanager
Name            Version
===============-==================
nvidia-fabricmanager 580.95.05

Boot Image Requirements

Our VM images begin as standard Ubuntu cloud images. We customize them with virt-customize to install the matching nvidia-open driver:

dpkg -l nvidia-open
Name            Version
===============-==================
nvidia-open     580.95.05

For a PCIe GPU, this would be the end of the story.

But on the B200, CUDA initialization consistently failed, even though nvidia-smi reported no issues:

The Topology Mismatch

A critical difference emerged when comparing the PCI tree on the host to the PCI tree inside the VM. Inside the VM, the GPU sat directly under the PCI root complex:

lspci -tv -d 10de:2901 
-[0000:00]---04.0  NVIDIA Corporation Device 2901

But on the host, the B200 GPU sits several levels deep behind PCIe bridges and root ports:

lspci -tv -d 10de:2901
-[0000:00]-+-[0000:14]---02.0-[15-1a]----00.0-[16-1a]----00.0-[17]----00.0  NVIDIA Corporation Device 2901

Switching to QEMU for Custom PCI Layouts

Launching the VM with QEMU using a plain VFIO device still produced the same flat topology:

qemu-system-x86_64 
 ... # CPU/disk/network params omitted
-device vfio-pci,host=0000:17:00.0

But QEMU allows you to insert PCIe root ports and attach devices behind them, recreating a realistic hierarchy:

qemu-system-x86_64 
  ... # CPU/disk/network params omitted
  -device pcie-root-port,id=rp1 
  -device vfio-pci,host=0000:17:00.0,bus=rp1

Inside the VM, the topology now looked like this:

lspci -tv
-[0000:00]-+-04.0-[01]----00.0  NVIDIA Corporation Device 2901

This layout mirrors the host’s structure: the GPU sits behind a root port, not directly under the root complex. With that change in place, CUDA initializes normally:

cuInit -> 0 no error

Now we’re in business!

With the PCI topology corrected, GPU passthrough worked reliably once the VM was up. However, a new issue emerged when passing through multiple B200 GPUs – especially 4 or 8 at a time. VM boot would stall for several minutes, and in extreme cases even over an hour before the guest firmware handed off to the operating system.

After investigating, we traced the issue to the enormous PCI Base Address Registers (BARs) on the B200. These BARs expose large portions of the GPU’s memory aperture to the host, and they must be mapped into the guest’s virtual address space during boot.

You can see the BAR sizes with:

Solution 1: Upgrade to QEMU 10.1+

QEMU 10.1 includes major optimizations for devices with extremely large BARs. With these improvements, guest boot times return to normal even when passing through all eight GPUs.

Solution 2: Disable BAR mmap (x-no-mmap=true)

If upgrading QEMU or reserving large amounts of memory is not feasible, you can instruct QEMU not to mmap the large BARs directly, dramatically reducing the amount of virtual memory the guest must reserve:

qemu-system-x86_64 
  ... # CPU/disk/network parameters omitted
  -device pcie-root-port,id=rp1 
  -device vfio-pci,host=0000:17:00.0,bus=rp1,x-no-mmap=true

With x-no-mmap=true, QEMU avoids mapping the BARs into the guest’s virtual address space and instead uses a slower emulated access path. In practice:

• Virtual memory consumption becomes small and constant
• Guest boot times become fast and predictable
• Most real-world AI training and inference workloads show little to no measurable performance impact, since they do not heavily exercise BAR-access paths

Only workloads that directly access the BAR region at high rates may observe reduced performance.

With passthrough and PCI topology resolved, the final piece of Shared NVSwitch Multitenancy is partition management. In this mode, the host’s Fabric Manager controls how the eight B200 GPUs are grouped into isolated NVSwitch “islands”, each of which can be assigned to a VM.

Fabric Manager operates according to a mode defined in:

Predefined HGX B200 Partitions

For an 8-GPU HGX system, NVIDIA defines a set of non-overlapping partitions that cover all common VM sizes (1, 2, 4, and 8 GPUs). Fabric Manager only allows one active partition per GPU; attempting to activate an overlapping partition will fail.

These predefined layouts ensure that GPU groups always form valid NVSwitch “islands” with uniform bandwidth.

GPU IDs Are Not PCI Bus IDs

nvidia-smi -q

Example:

GPU 00000000:17:00.0
  Product Name                  : NVIDIA B200
  ...
  Platform Info
      Peer Type                 : Switch Connected
      Module Id                 : 1

This Module ID (1–8) is the index used by partition definitions, activation commands, and NVSwitch routing logic. When passing devices to a VM, you must map Fabric Manager GPU Module IDs → PCI devices, not assume PCI order.

Interacting with the Fabric Manager API

fmpm -l
# lists all partitions, their sizes, status (active/inactive)

fmpm -a 3
# activate partition ID 3

fmpm -d 3
# deactivate partition ID 3

Provisioning Flow

Putting everything together, the high-level flow for provisioning a GPU-enabled VM looks like this:

• A user requests a VM with X GPUs.
• The management system selects a free partition of size X.
• It activates the partition: fmpm -a .
• Fabric Manager configures NVSwitch routing accordingly.
• The system passes through the GPUs corresponding to the Module Ids of that partition into the VM.
• The VM boots; inside it, nvidia-smi topo -m shows full NVLink connectivity within the partition.
• After VM termination, the system calls fmpm -d to release the partition.

This workflow gives each tenant access to high-performance GPU clusters with full bandwidth and proper isolation, making the B200 HGX platform viable for multi-tenant AI workloads.

Getting NVIDIA’s HGX B200 platform to behave naturally in a virtualized, multi-tenant environment requires careful alignment of many layers: PCI topology, VFIO configuration, driver versioning, NVSwitch partitioning, and hypervisor behavior. When these pieces fit together, the result is a flexible, high-performance setup where tenants receive full-bandwidth NVLink inside their VM while remaining fully isolated from other workloads.

A final note we care about: everything described in this post is implemented in the open. Ubicloud is a fully open-source cloud platform, and the components that manage GPU allocation, activate NVSwitch partitions, configure passthrough, and launch VMs are all public and available for anyone to inspect, adapt, or contribute to.

If you’d like to explore how this works behind the scenes, here are good entry points: