Using VM Pools

What is a VM Pool?

A pre-allocated VM Pool is a set of virtual machines (VMs) reserved for your organization in advance and held ready for code runs. Instead of provisioning a machine when a run starts, the run claims a slot on a machine that already exists.

---

Why use a VM Pool?

• Guaranteed availability

  • High-demand workloads — GPU-intensive AI training and inference in particular — do not compete with other workloads for capacity that may not be there when you need it.

• Predictable cost

  • Capacity is a fixed reservation rather than a variable, per-run provisioning charge.

The tradeoff is that a VM pool is a fixed size. A pool's configuration (vCPUs, RAM, disk, and GPUs per VM) is set when the pool is created and cannot be overridden by an individual code object or code run.

See Compute Options to explore alternatives.

---

Prerequisites and Availability

Pre-allocated VM Pools are currently available for any code object that run on Google Cloud Platform (GCP) Rhino-hosted agents, and are supported for Generalized Compute, Python Code, and NVIDIA FLARE code objects.

If you do not see a Use VM Pool option, either no pools have been configured for your organization or your code object is not running on a GCP Rhino-hosted agent.

Pre-allocated VMs must be configured by a Rhino administrator before use. Contact your administrator or your Rhino representative for more information.

---

How Agents Claim Pool Capacity

A federated run executes at more than one Rhino Client, and each participating agent needs a slot. How those slots are claimed depends on the code object type:

• Generalized Compute and Python Code — each agent claims its slot independently. An agent that finds capacity begins work without waiting for the others.

• NVIDIA FLARE — a two-phase claim. All agents must secure pending slots before any job dispatches. A FLARE job therefore starts only when every participating site has capacity reserved.

For FLARE, a single site short on pool capacity will hold up the whole job.

When sizing pools for a FLARE collaboration, size for simultaneous participation rather than average use.

---

Selecting VM Pools for a Code Object

A code object can be associated with more than one pre-allocated VM pool. Where you maintain separate pools — CPU and GPU variants, or pools dedicated to particular projects or sites — a single code object can target each pool that suits it, without duplicating the code object per pool.

When a VM pool is selected, you cannot manually set individual compute resource specifications (CPU, RAM, GPU), because these are governed by the pool's fixed configuration.

System validation will prevent the creation of a Code Object if the requirements do not match the selected pool's capabilities (for example, requesting a non-GPU job on a GPU-only pool).

1

Open the compute resources section

In the Required Compute Resources section of the code object or code run, choose Use VM Pool.

2

Review the available pools

Each pool is listed with its name and current available resources:

• Number of VMs (if any)

• Number of vCPUs per VM

• Amount of RAM per VM

• Disk size per VM

• Whether GPUs are available, and how many and their specifications

3

Select the pools you want

Select one or more pools.

4

Create the code object or start the run

Select Create New Code Object or start a Code Run. The code object or code run shows an icon indicating that it will run on a pre-allocated VM pool.

If you hover over the icon, the name of the selected VM pool is shown.

---

Tracking VM Pool Status

You can track the performance, health, and availability of your VM Pool project resources in the Virtual Machine Pools tab. This dedicated subpage provides a comprehensive view of your environment, including total pool capacity, current utilization (number of running VMs), and queue length.

To learn more about the status of the VM Pools in your project, do the following.

1. Select Code from the main menu, then select the Virtual Machine Pools tab at the top of the page.

1. Review the information about the pools.

• Total Pools - Number of VM Pools available across the project.

• Total Capacity - Number of VMs, including how many are in use and how many are idle.

• Queued Jobs - Number of jobs waiting for processing across all pools.

• Health - How many VM pools are active, as well as a brief description of the inactive VM Pools.

• Running Jobs - There is also a listing that contains pool information for running jobs. This includes the pool name, compute specs (number of CPUs, amount of ram, GPUs), utilization, how many jobs are in the queue, and the status of the job.

If a pool undergoes maintenance, a banner will appear on the Code Run page indicating that jobs will remain queued until the maintenance is complete.

---

Confirming Which Resources a Run Used

Code run details show the VM pools a run requested alongside the resources it actually claimed. This is the authoritative record of what a run executed on, and the first place to look when:

• A run started more slowly than expected — it may have queued or fallen back.

• You are reconciling compute cost against reserved capacity.

• A FLARE job did not dispatch — check whether every agent secured a slot.

The Details tab of a code run's logs also displays the compute resource mode and the requested specifications.

---

VM Execution & Capacity Exhaustion

When a job is triggered against a pre-allocated pool, it follows a more efficient execution path compared to standard auto-scaling. Jobs are submitted to the chosen pool and transition rapidly from a "Queued" phase directly into a "Running" state as resources are already primed for use.

If a pool's capacity is exhausted, jobs are placed into a First-in-First-Out (FIFO) queue rather than failing. You can monitor the "Queued (VM Pool)" status and your job's position on the Code Run page. Once a slot frees, the job is automatically assigned and dispatched.

---

Troubleshooting

Symptom	Likely Cause	What to Check
Use VM Pool is not offered	No pools configured, or the code object is not on a GCP Rhino-hosted agent	Confirm with your administrator
Code object will not save with a pool selected	Validation mismatch between the code object's requirements and the pool's configuration	Compare the code object's GPU and resource requirements against the pool's fixed configuration
Job sits in Queued (VM Pool)	Pool capacity exhausted	Check queue position on the Code Run page and pool utilization on the Virtual Machine Pools tab
A FLARE job does not dispatch while some agents appear ready	Two-phase claim — not all agents have secured pending slots	Identify which participating site lacks capacity
A run used different resources than the pool provides	The run fell back to an ad hoc VM	Check requested pools against claimed resources in the run details