> For the complete documentation index, see [llms.txt](https://docs.rhinofcp.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.rhinofcp.com/creating-and-running-code-objects/autoscaling-and-compute-resources/using-vm-pools.md). # Using VM Pools ## What is a VM Pool? A pre-allocated **VM Pool** is a set of virtual machines (VMs) reserved for your organization in advance and held ready for code runs. Instead of provisioning a machine when a run starts, the run claims a slot on a machine that already exists. *** ## Why use a VM Pool? * **Guaranteed availability** * High-demand workloads — GPU-intensive AI training and inference in particular — do not compete with other workloads for capacity that may not be there when you need it. * **Predictable cost** * Capacity is a fixed reservation rather than a variable, per-run provisioning charge. {% hint style="info" %} The tradeoff is that a VM pool is a **fixed** size. A pool's configuration (vCPUs, RAM, disk, and GPUs per VM) is set when the pool is created and cannot be overridden by an individual code object or code run. {% endhint %} See [Compute Options](/creating-and-running-code-objects/autoscaling-and-compute-resources.md) to explore alternatives. *** ## Prerequisites and Availability Pre-allocated VM Pools are currently available for any code object that run on Google Cloud Platform **(GCP) Rhino-hosted agents**, and are supported for Generalized Compute, Python Code, and NVIDIA FLARE code objects. If you do not see a Use VM Pool option, either no pools have been configured for your organization or your code object is not running on a GCP Rhino-hosted agent. {% hint style="info" %} Pre-allocated VMs **must be configured by a Rhino administrator** before use. Contact your administrator or your Rhino representative for more information. {% endhint %} *** ## How Agents Claim Pool Capacity A federated run executes at more than one Rhino Client, and each participating agent needs a slot. How those slots are claimed depends on the code object type: * **Generalized Compute and Python Code** — each agent claims its slot independently. An agent that finds capacity begins work without waiting for the others. * **NVIDIA FLARE** — a two-phase claim. All agents must secure pending slots before any job dispatches. A FLARE job therefore starts only when every participating site has capacity reserved. {% hint style="info" %} For FLARE, a single site short on pool capacity will hold up the whole job. When sizing pools for a FLARE collaboration, size for **simultaneous** participation rather than average use. {% endhint %} *** ## Selecting VM Pools for a Code Object A code object can be associated with **more than one** pre-allocated VM pool. Where you maintain separate pools — CPU and GPU variants, or pools dedicated to particular projects or sites — a single code object can target each pool that suits it, without duplicating the code object per pool. {% hint style="info" %} When a VM pool is selected, you cannot manually set individual compute resource specifications (CPU, RAM, GPU), because these are governed by the pool's fixed configuration. {% endhint %} {% hint style="warning" %} System validation will prevent the creation of a Code Object if the requirements do not match the selected pool's capabilities (for example, requesting a non-GPU job on a GPU-only pool). {% endhint %} {% stepper %} {% step %} ### Open the compute resources section In the Required Compute Resources section of the code object or code run, choose Use VM Pool.
{% endstep %} {% step %} ### Review the available pools Each pool is listed with its name and current available resources: * Number of VMs (if any) * Number of vCPUs per VM * Amount of RAM per VM * Disk size per VM * Whether GPUs are available, and how many and their specifications {% endstep %} {% step %} ### Select the pools you want Select one or more pools. {% endstep %} {% step %} ### Create the code object or start the run Select Create New Code Object or start a Code Run. The code object or code run shows an icon indicating that it will run on a pre-allocated VM pool. If you hover over the icon, the name of the selected VM pool is shown. ![](https://2949362692-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FydySyCBmy6F7NnGn4bPa%2Fuploads%2Foj6FqJhR18ybIjNr5FCI%2Fimage.png?alt=media\&token=ba5a8628-5e06-451a-9beb-d8c8c9a014d4) {% endstep %} {% endstepper %} *** ## Tracking VM Pool Status You can track the performance, health, and availability of your VM Pool project resources in the **Virtual Machine Pools** tab. This dedicated subpage provides a comprehensive view of your environment, including total pool capacity, current utilization (number of running VMs), and queue length. To learn more about the status of the VM Pools in your project, do the following. 1. Select **Code** from the main menu, then select the **Virtual Machine Pools** tab at the top of the page.
2. **Review the information** about the pools. * Total Pools - Number of VM Pools available across the project. * Total Capacity - Number of VMs, including how many are in use and how many are idle. * Queued Jobs - Number of jobs waiting for processing across all pools. * Health - How many VM pools are active, as well as a brief description of the inactive VM Pools. * Running Jobs - There is also a listing that contains pool information for running jobs. This includes the pool name, compute specs (number of CPUs, amount of ram, GPUs), utilization, how many jobs are in the queue, and the status of the job. {% hint style="info" %} If a pool undergoes maintenance, a banner will appear on the Code Run page indicating that jobs will remain queued until the maintenance is complete. {% endhint %} *** ## Confirming Which Resources a Run Used **Code run details** show the VM pools a run requested alongside the resources it actually claimed. This is the authoritative record of what a run executed on, and the first place to look when: * A run started more slowly than expected — it may have queued or fallen back. * You are reconciling compute cost against reserved capacity. * A FLARE job did not dispatch — check whether every agent secured a slot. The **Details** tab of a code run's logs also displays the compute resource mode and the requested specifications. *** ## VM Execution & Capacity Exhaustion When a job is triggered against a pre-allocated pool, it follows a more efficient execution path compared to standard auto-scaling. Jobs are submitted to the chosen pool and transition rapidly from a "**Queued**" phase directly into a "**Running**" state as resources are already primed for use. If a pool's capacity is exhausted, jobs are placed into a First-in-First-Out (**FIFO**) queue rather than failing. You can monitor the "Queued (VM Pool)" status and your job's position on the Code Run page. Once a slot frees, the job is automatically assigned and dispatched. *** ## Troubleshooting
SymptomLikely CauseWhat to Check
Use VM Pool is not offeredNo pools configured, or the code object is not on a GCP Rhino-hosted agentConfirm with your administrator
Code object will not save with a pool selectedValidation mismatch between the code object's requirements and the pool's configurationCompare the code object's GPU and resource requirements against the pool's fixed configuration
Job sits in Queued (VM Pool)Pool capacity exhaustedCheck queue position on the Code Run page and pool utilization on the Virtual Machine Pools tab
A FLARE job does not dispatch while some agents appear readyTwo-phase claim — not all agents have secured pending slotsIdentify which participating site lacks capacity
A run used different resources than the pool providesThe run fell back to an ad hoc VMCheck requested pools against claimed resources in the run details

--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.rhinofcp.com/creating-and-running-code-objects/autoscaling-and-compute-resources/using-vm-pools.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.