Creating and Running a New Generalized Compute Code Object
Explains what a generalized compute code object is, how to create and run it, and how to adapt your code to run as a generalized code object.
What is a Generalized Compute Code Object?
In the Rhino Federated Computing Platform (FCP), the Generalized Compute (GC) Code Object represents a versatile and powerful way to execute pre-built container images within the FCP environment. This Code type enables you to run custom code, computations, or processes that are encapsulated within container images. With GC, you can harness the full potential of distributed computing while tailoring your computations to suit your specific needs.
Key Features of Generalized Compute
Flexible Container Execution: GC empowers you to execute container images containing diverse computational tasks, irrespective of programming language or complexity. This allows you to leverage existing tools, libraries, and applications within the FCP ecosystem.
Batch Processing: Ideal for tasks that do not require real-time interactivity, GC excels at batch processing scenarios. You can execute computationally intensive tasks, data transformations, or complex analyses efficiently across distributed datasets.
Secure Data Access: GC can only access data provided via the FCP, with no external internet connectivity. GC accesses input Datasets located under the
/inputdirectory and generates output Datasets under the/outputdirectory. This ensures secure data access and maintains the privacy of sensitive healthcare information.
Use Cases for Generalized Compute
Data Transformation: Execute data transformation and normalization tasks across distributed healthcare datasets.
Custom Analytics: Run complex statistical analyses, machine learning processes, or custom algorithms on distributed data.
Scientific Simulations: Perform large-scale scientific simulations or simulations requiring substantial computational resources.
Resource-Intensive Tasks: Process resource-intensive tasks, such as image processing, signal analysis, or simulations.
Benefits of Generalized Compute
Adaptability: Leverage various programming languages and tools within your container, ensuring flexibility in your computational tasks.
Distributed Processing: Distribute computation tasks across multiple sites, enhancing efficiency and reducing processing time.
Customization: Craft computations tailored to your project's unique requirements, enhancing the depth of insights you can derive.
In summary, Generalized Compute empowers you to harness the versatility and power of containerized computations within the Rhino FCP. By executing custom code across distributed datasets, you can streamline tasks, gain valuable insights, and contribute to advanced healthcare research in a secure and privacy-preserving manner
In the Rhino Federated Computing Platform (FCP), the Generalized Compute (GC) Code Object represents a versatile and powerful way to execute pre-built container images within the FCP environment. This Code type enables you to run custom code, computations, or processes that are encapsulated within container images. With GC, you can harness the full potential of distributed computing while tailoring your computations to suit your specific needs.
Key Features of Generalized Compute
Flexible Container Execution: GC empowers you to execute container images containing diverse computational tasks, irrespective of programming language or complexity. This allows you to leverage existing tools, libraries, and applications within the FCP ecosystem.
Batch Processing: Ideal for tasks that do not require real-time interactivity, GC excels at batch processing scenarios. You can execute computationally intensive tasks, data transformations, or complex analyses efficiently across distributed datasets.
Secure Data Access: GC can only access data provided via the FCP, with no external internet connectivity. GC accesses input Datasets located under the
/inputdirectory and generates output Datasets under the/outputdirectory. This ensures secure data access and maintains the privacy of sensitive healthcare information.
Use Cases for Generalized Compute
Data Transformation: Execute data transformation and normalization tasks across distributed healthcare datasets.
Custom Analytics: Run complex statistical analyses, machine learning processes, or custom algorithms on distributed data.
Scientific Simulations: Perform large-scale scientific simulations or simulations requiring substantial computational resources.
Resource-Intensive Tasks: Process resource-intensive tasks, such as image processing, signal analysis, or simulations.
Benefits of Generalized Compute
Adaptability: Leverage various programming languages and tools within your container, ensuring flexibility in your computational tasks.
Distributed Processing: Distribute computation tasks across multiple sites, enhancing efficiency and reducing processing time.
Customization: Craft computations tailored to your project's unique requirements, enhancing the depth of insights you can derive.
In summary, Generalized Compute empowers you to harness the versatility and power of containerized computations within the Rhino FCP. By executing custom code across distributed datasets, you can streamline tasks, gain valuable insights, and contribute to advanced healthcare research in a secure and privacy-preserving manner.
Prerequisites
If your code will use external files, complete the following steps before you create your python code object.
Your workgroup has a unique cloud storage (an S3 bucket on Rhino Orchestrator) to upload non-sensitive data that might be needed during code run, e.g., open-source large LLM model weights.
Cloud storage (specified in Workgroup code and model artifacts storage section in the instructions that follow) is owned by Rhino. Please DO NOT upload sensitive information such as proprietary datasets. For sensitive data, please use the instructions in Importing, Viewing a Dataset's Configuration, and Exporting Datasets instead.
Overview
For your code to be able to access files while running in your agent we need do 4 steps:
Find the bucket to upload your files
In your project, select the settings button (it looks like a sprocket at the bottom left side of the page.
In the Settings menu, which is near the top of the screen on the left side of the window, select Containers & Artifacts.
Your storage bucket and bucket prefix appear in the Workgroup code and model artifacts storage part of the page.
Upload files to your designated bucket
AWS offers many alternative ways to upload files to an S3 bucket. We've also provided a script in our user_resource repository you can use: upload-file-to-s3.sh.
You need to define your S3 credentials and then you can call the script like this:
For the example above, if you wanted to upload files in a local folder (called "local_folder" in the below example) and store them in a folder in FCP called "my_files", the command would be:
Reference your files in your code
When creating your code object, you can reference your external files by referencing them in the following path:
For the example above, assuming you uploaded model_params.txt file under my_files, the file would be accessible during runtime in
An example of Python code to read your file data into a text variable would look like as follows:
Use your files in a specific run
Once your code points to your external files, and the code object is created, you can run the code and specify the files you want to use in your code run. In the UI, you can simply select them from a dropdown
Alternatively, you can also use the SDK. See this notebook to use external files via SDK for more details.
Creating a New Generalized Compute Code Object
Follow these steps to use the UI to create a new Generalized Compute Code Object.
Go to the main project's page and select your project.
Select Code from the menu on the left to open the Code Objects page.
Select the Create New Code Object button to open the Create New Code Object page.
Select Generalized Compute as the Code Object Type.
The page changes to show the fields needed for this object type.
Enter information in the following fields.
Before you enter information in the fields (see table below), read the following about input and outputs
Multiple input and output datasets are supported in Python, Generalized, and Interactive Container Code Objects. While NVFlare Code Objects do not support multiple datasets, they can still be used as before for learning and inference across multiple sites.
Code objects usually require that you know the number of inputs and outputs your code needs. Those inputs and outputs are provided as data schemas. For example, if your code splits one dataset into two (like a train/test split), you will have to create a code object with one input schema and two output schemas. For more detail into how to structure your code to run on FCP, see Accessing your datasets in Code Objects.
Enter your information in the following fields.
Name
Indicates the name you want to provide for the code object.
Description
Provides a brief summary of what the code object does.
Input (including three-dot menu items)
Indicates the file(s) needed for the code object to run properly. Your selection here will affect which Dataset(s) can be selected as input when triggering a Code Run with this Code Object You can enter one or more input files, such as schemas, configuration files, or others. File(s) can be in DICOM, JSON, and or text formats. - If your input doesn't require a schema, select Any. - Select the Add button to the right of the input entry to add another input. You can add as many as you like. - If your input includes more than a single dataset, select the three dot menu to the right of the Input field, then select Input is a list. - If the input is optional, select the three-dot menu to the right of the Input field, then select Optional Input.
Output
Indicates the output file(s) produced when the code object run. You can enter one or more output files. Files can be in DICOM, JSON, and or text formats. - You can also select the option to [ Auto-generate Data Schema from Data ]. For more information about Auto-generating Data S chema s, please refer to Auto-Generated Data Schema s. - Select the Add button to the right of the output entry to add another output. You can add as many as you like. - If your output includes more than a single dataset, select the three dot menu to the right of the Output field, then select Output is a list. - If the output is optional, select the three-dot menu to the right of the Output field, then select Optional Output.
Container
Specifies the name of the Docker container you would like to execute when running this Code Object. This will either be a container image created by someone from your workgroup that is available on your ECR or an image provided by Rhino. If you need a refresher on pushing containers to your ECR, please refer to Pushing Containers to the ECR.
After you've completed your details, you can choose to set required compute resources.
If you want to do this, go to the Setting Required Compute Resources (Auto-scaling) section on this page.
When complete, select the Create New Code button to create the new Code Object.
The code object appears on the Code Object page.
Setting Required Compute Resources (Autoscaling)
Autoscaling allows Rhino to run Code Runs on additional compute resources when the default Rhino Client capacity is not sufficient. This capability is designed to support resource‑intensive, "bursty"(processing tasks occur in short, intense or irregular spurts), or GPU‑based workloads while improving isolation, reliability, and cost efficiency. With autoscaling enabled, Rhino provisions temporary cloud VMs on demand, executes workloads on those VMs, and automatically terminates them when execution completes. This can help you make more efficient use of your computing resources and save on costs.
In this section of the screen there are three options: Use Client Resources, Set Minimum, and Use VM Pool.
Use Client Resources - Uses the available client settings and resources.
Set Minimum - Allows you to set the minimum number of resources that must be available. If you choose the Set Minimum option, you can indicate the minimum number of CPUs, amount of RAM, the disk size (VRAM), as well as the type and number of GPUs. You can adjust these settings when you create a code object and also when you run your code. Note that adjustments do not persist and will need to be reset with each run.
Use VM Pool - Allows you to select a Pre-allocated Virtual Machine (VM) Pool. Pre-allocated VM Pools allow you to indicate which dedicated compute resource you want to use for your code run, ensuring guaranteed availability for high-demand workloads such as GPU-intensive AI training and inference. This feature provides availability of critical compute resources and offers predictable costs through fixed capacity reservations.
Each option is detailed below.
Use Client Resources
To use client resources, complete the following steps.
Select the Use Client Resources button. It should be selected by default.
When complete, select Create New Code Object or start a Code Run.
Set Minimum
To set the minimum required compute resources, complete the following steps.
In the Required Compute Resources section, select Set Minimum.
Next, make changes to the settings as needed. Note that these setting changes do not persist the next time you run the code object or code run. The settings are explained below.
Setting
Description
Usage Notes
CPU MIN
The minimum number of CPU cores on which your code will be run.
If the CPU MIN is too low, you might see slow code execution or throttling.
RAM MIN
The minimum amount of RAM (main memory) that you want to allocate to running your code.
If this is too low, you might see an out-of-memory crash or slowness in execution due to disk swapping.
DISK SIZE
Amount of storage allocated to the instance.
Increasing this setting increases capacity (not necessarily speed)
GPU TYPE
The model of GPU hardware. The number of GPUs indicate the number of GPUs attached to the instance.
If your code isn’t written to use multiple GPUs, extra GPUs might not be used. Note that GPU availability depends on the selected cloud region, Project‑level GPU quotas, and the current capacity at the cloud provider.
Use VM Pool
You can select a VM pool to indicate which dedicated compute resource you want to use for your code run, ensuring guaranteed availability for high-demand workloads such as GPU-intensive AI training and inference. This feature provides faster job startup times and offers predictable costs through fixed capacity reservations. To select a VM Pool, complete the following steps.
Choose Use VM Pool to see the available pools. For each pool you will see a name, as well as the current resources available. For each pool you will see the following information.
Number of VMs (if any)
Number of vCPUs per VM
Amount of RAM per VM
Disk size per VM
Whether there are GPUs available, and how many and their specifications
If you don't know which VM Pool to choose, see Learning More About Available VM Pools.
Select the pool of your choice.
When complete, select Create New Code Object or start a Code Run. The new code object (or code run) has an icon that indicates that the code object will run on a pre-allocated VM pool. If you hover over the icon, the name of the selected VM pool is shown.
VM Pool Statuses and Capacity
This section explains how to track VM Pool statuses across your project. It also provides information on VM Pool job execution and on what happens when a VM Pool capacity has been exhausted.
Tracking VM Pool Statuses
You can track the performance, health, and availability of your VM Pool project resources in the VM Pools Tab. This dedicated subpage provides a comprehensive view of your environment, including total pool capacity, current utilization (number of running VMs), and queue length.
To learn more about the status of the VM Pools are available in your project, do the following.
Select Code from the main menu, then select the Virtual Machine Pools tab at the top of the page.
Review the information about the pools.
Total Pools - Number of VM Pools available across the project.
Total Capacity - Number of VMs, including how many are in use and how many are idle.
Queued Jobs - Number of jobs waiting for processing across all pools.
Health - How many VM pools are active, as well as a brief description of the inactive VM Pools.
Running Jobs - There is also a listing that contains pool information for running jobs. This includes the pool name, compute specs (number of CPUs, amount of ram, GPUs), utilization, how many jobs are in the queue, and the status of the job.
VM Pool Job Execution and VM Pool Capacity Exhaustion When a job is triggered against a pre-allocated pool, it follows a more efficient execution path compared to standard auto-scaling. Jobs are submitted to the chosen pool and transition rapidly from a "Queued" phase directly into a "Running" state as resources are already primed for use.
If a pool's capacity is exhausted, jobs are placed into a First-in-First-Out (FIFO) queue rather than failing. You can monitor the "Queued (VM Pool)" status and your job's position on the Code Run page. Once a slot frees, the job is automatically assigned and dispatched.
Viewing a Code Object's Configuration
To get an object's configuration, make sure you are on the page containing the original object you would like to view the configuration of. In the box where your original object is, there should be a row for each version. Navigate to the version you would like to view the configuration for and select the three-dot menu button, shown below:
The menu button is on the right-hand side of the row, click it to view a new menu. The new menu should have an option to show that object's configuration, click on that to see the object's configuration.
For Code Objects and Code Runs, if you chose the Use VM Pool option, you will use the pool that was indicated during the Code Object or Code Run configurations. Selecting this option shows the resources, including the number of VMs and vCPUs that will be allocated, as well as the amount of RAM, the size of the disk, the number of GPUs, the GPU type/model, and the amount of VRAM.
Creating a New Generalized Compute Code Object Version
To create a new version, complete the following steps.
Go to the page for the original object you want to version.
In the upper right corner of the box where your original object is, select the + New Version button.
A new window appears that allows you to create a new version of your object.
Removing a Code Object
To remove a code object, complete the following steps.
Open the Code Objects page
Go to the page that lists your code objects.
Find the code object version
If needed, use the search fields at the top right.
Use Search Name to search by code object name.
Use Search Description to search by code object description.
Each code object appears with a row for each version. Find the version you want to delete.
Open the menu for the version
Select the three-dot menu on the right side of the version row, shown below.
Remove the code object version
Select Remove code object.
Repeat these steps for each version you want to remove. When you delete the last version, the code object is fully deleted.
Running a Generalized Compute Code Object
Create a Generalized Compute code object before you complete these steps.
Go to your project and open Code Objects.
Go to the main project's page and select your project.
Select Code from the menu on the left to open the Code Objects page.
Select the code object to run.
Select the Run button for the code object you'd like to run.
Review the Run Settings page.
In the Run Settings page, do the following.
Fill in the Code Run fields.
Fill in the following fields using the table below for reference.
Input Dataset(s)
One or more Datasets to be used as input to your Code Run. If multiple Datasets are selected, each will be run separately. If a Dataset happens to be a collaborator's Dataset, the code will be run on that collaborator's Rhino Client. In other words, the data never moves outside your collaborator's Rhino Client.
Output Dataset Name Suffix
A suffix that is appended onto the name of each input Dataset to define the name of the output Dataset that will be created during your Code Run.
Timeout in Seconds
The number of seconds that must elapse before a Code Run is automatically halted. This is to avoid zombie tasks that run perpetually within a Rhino client.
Run Parameters (Optional)
Here you can paste a JSON object that defines additional parameters to be provided to your Code Run. This can be used for specifying hyper-parameters for model validation or any parameter provided to the code. Run parameter s are made available to the container code in a file located at /input/run_params.json
To add run time files to your code run, select the run time files from the list of the workgroup run files that appears. You will want to only include the ones you need for the code run, like in the example that follows.
Selecting Run Time Files During Code Run Set Ups
If you are part of a workgroup that has collaborators, you will be able to see files that are 1) in your active workgroup that are part of the project, whether they are published or not and 2) from a collaborator’s workgroup that are part of the project only if they are published.
To better understand this, let's take a look at the following scenario. Project PRIME is led by workgroup Alpha. Workgroup Beta has been added as a collaborator. Each workgroup has uploaded run time files.
If YOUR active workgroup is Alpha, here is what you see when you open the Run Time Files screen in Project PRIME.
All Workgroup Alpha's files are shown. This is because Project PRIME is led by workgroup Alpha, so all of workgroup Alpha’s files that are part of the project are shown whether they are published or not.
Only Workgroup Beta’s published files are shown. Workgroup Beta’s files that are not part of the project or that are unpublished are not shown.
If YOUR active workgroup is Beta, here is what happens when you open the Run Time Files screen in Project PRIME.
Workgroup Beta's files are all shown. This is because Beta members can see their own workgroup’s files that are part of the project, whether there are published or not.
Only Workgroup Alpha’s files that have been published to PRIME are shown. Workgroup Alpha’s unpublished files that are not part of the project are not shown.
See Managing Run Time Files to learn how to manage, publish and unpublish run time files.
Set the required compute resources (if needed).
When you have completed adding all your Code Run details, either set the required compute resources by following the instruction in Setting Required Compute Resources (Autoscaling), or click the Run button to run your code.
Reviewing Code Run Logs
Code run logs provide details about what happened during a run, as well as the outcome.
To view these logs, complete the following steps.
Select Code Run.
In the Code Runs page, select the code run that you want to see logs for.
The logs page for the Code Run is shown. The start and finish time and date of the run, as well as the status of the run appears. Logs can be accessed while the Code Run is active and after the Code Run has completed. While the code is running, an auto-refresh option is available. Details are shown under the details tab; reports (if any) are shown under the reports tab. Reports are generated using the Rhino SDK.
Select the Details Tab.
Select the Details tab if needed (it should be selected by default).
The Details tab provides the following:
General Info - information about the Code Run, including
Code Object name and type
Code Object version
Code Object description
Input and Output Datasets
Compute Resources Mode including the requested resources at run time (if applicable).
Show Code Object Configuration button. Click this button to see the code object configuration. Show Code Run Configuration button. Click this button to see the code run configuration.
View in TensorBoard button. Click this button to view results in TensorBoard.
System Logs. These can include errors or warnings that occurred from the FCP side, e.g. Dataset import errors.
FL server logs (for NVIDIA FLARE runs)
FL client logs (for NVIDIA FLARE runs)
Each federated client's logs will appear separately
Client-specific logs (for all other run types).
Troubleshooting Code Runs
Error Message that Usage Limit Has Been Exceeded
If you get an error message indicating that the usage limit has been exceeded, your administrator has set usage limits on the number of input rows a workgroup can process per model within a certain timeframe. Getting this message means that the entire run has been cancelled. Please contact your administrator for more details and help with resolving the issue.
Viewing a Code Run's Configuration
To get an object's configuration, make sure you are on the page containing the original object you would like to view the configuration of. In the box where your original object is, there should be a row for each version. Navigate to the version you would like to view the configuration for and select the three-dot menu button, shown below:
The menu button is on the right-hand side of the row, click it to view a new menu. The new menu should have an option to show that object's configuration, click on that to see the object's configuration.
For Code Objects and Code Runs, if you chose the Use VM Pool option, you will use the pool that was indicated during the Code Object or Code Run configurations. Selecting this option shows the resources, including the number of VMs and vCPUs that will be allocated, as well as the amount of RAM, the size of the disk, the number of GPUs, the GPU type/model, and the amount of VRAM.
Deleting a Code Run
Follow these steps to delete a code run.
Open the Code Runs page
Go to the page that lists your code runs.
Find the code run
If needed, use the search fields at the top right.
Use Search Name to search by code run name.
Use Search Description to search by code run description.
Find the code run you want to delete.
Open the menu for the code run
Select the three-dot menu on the right side of the row, shown below.
Remove the code run
Select Remove code run.
Adapting your code to run as a Generalized Code Object
For your code to successfully run on the FCP do the following things.
Modify Where You Are Reading and Writing From in Your Code
All inputs should be read from the /input folder.
Datasets will be zero-indexed within the /input folder (i.e. Input Dataset 1 - /input/0/, Input Dataset 2 - /input/1/, etc.)
For Tabular Data:
/input/0/dataset.csv. If you would like to work with the data as a Pandas DataFrame you can load the Dataset using the following example:
For file data (data that has been defined in the Data Schema as type Filename): If there is any file data associated with the Dataset, it will be under the
/input/file_data/directory, structured in the same subdirectory structure as was specified when importing the Dataset. Example:
/input/0/file_data/
For DICOM data (data that has been defined in the Data Schema as type DicomStudyUID, DicomSeriesUID, or DicomInstanceUID.): If there is any DICOM data associated with the Dataset, it will be under the
/input/dicom_data/directory, structured in the same subdirectory structure as was specified when importing the Dataset. Example:
/input/0/dicom_data/
All output should be written to the /output mount.
For Tabular Data:
/output/0/dataset.csv. To write your data as a Pandas DataFrame you can use the following example:
For file data:
/output/0/file_data/For DICOM data:
/output/0/dicom_data/
[If applicable] Modify where you are reading your runtime parameters
Any run parameters that have been specified shall be in a file named /input/run_params.json
Create the Dockerfile
Create a Dockerfile with the configuration necessary for your code. You may use the example provided in the user-resources github repository as a template and make changes as needed where there are comments beginning with "# !! EDIT THIS:"
Ensure any files that might be downloaded at runtime are pre-downloaded and copied to the container in the Dockerfile
Copy files that might need to be downloaded at runtime as pre-downloaded copy statements within the Dockerfile
Optional recommendations
We recommend that you build and test your container locally. There is a utility script called docker-run.sh in the gc-docker-utils folder to assist with this. It can be used by running:
Last updated
Was this helpful?