Creating and Running a Python Code Object
How to create, configure, run, and remove python code objects.
This article explains how to create and run Python Code Objects. Python code objects allow you to provide your code to FCP without having to containerize it.
What is a Python Code Object?
In Rhino Federated Computing Platform (FCP), Python Code Objects offer a powerful way to execute Python code scripts effortlessly, eliminating the need to locally containerize your code. Building and deploying code becomes a seamless process, with FCP handling container image creation and storage on your workgroup's ECR (Elastic Container Registry).
With Python Code Objects, FCP streamlines code deployment process, allowing you to concentrate on code development and analysis. Whether you're performing sophisticated data manipulations or running custom algorithms, Python Code Objects empower you to leverage the full capabilities of Python while benefiting from FCP's containerization and deployment efficiency.
Python Code Objects on FCP bring simplicity and power to your code development endeavors. By offering options for both code snippets and standalone files, FCP adapts to your needs, allowing you to focus on creating impactful software. Embrace the convenience of executing Python code directly within the platform, while FCP takes care of the intricate details of containerization, image creation, and deployment. Elevate your workflow and unlock new dimensions of development possibilities with Rhino FCP.
Prequisites
If your code will use external files, complete the following steps before you create your python code object.
Your workgroup has a unique cloud storage (an S3 bucket on Rhino Orchestrator) to upload non-sensitive data that might be needed during code run, e.g., open-source large LLM model weights.
Cloud storage (specified in Workgroup code and model artifacts storage section in the instructions that follow) is owned by Rhino. Please DO NOT upload sensitive information such as proprietary datasets. For sensitive data, please use the instructions in Importing, Viewing a Dataset's Configuration, and Exporting Datasets instead.
Overview
For your code to be able to access files while running in your agent we need do 4 steps:
Find the bucket to upload your files
In your project, select the settings button (it looks like a sprocket at the bottom left side of the page.
In the Settings menu, which is near the top of the screen on the left side of the window, select Containers & Artifacts.
Your storage bucket and bucket prefix appear in the Workgroup code and model artifacts storage part of the page.
Upload files to your designated bucket
AWS offers many alternative ways to upload files to an S3 bucket. We've also provided a script in our user_resource repository you can use: upload-file-to-s3.sh.
You need to define your S3 credentials and then you can call the script like this:
For the example above, if you wanted to upload files in a local folder (called "local_folder" in the below example) and store them in a folder in FCP called "my_files", the command would be:
Reference your files in your code
When creating your code object, you can reference your external files by referencing them in the following path:
For the example above, assuming you uploaded model_params.txt file under my_files, the file would be accessible during runtime in
An example of Python code to read your file data into a text variable would look like as follows:
Use your files in a specific run
Once your code points to your external files, and the code object is created, you can run the code and specify the files you want to use in your code run. In the UI, you can simply select them from a dropdown
Alternatively, you can also use the SDK. See this notebook to use external files via SDK for more details.
Creating a Python Code Object
To create a new code object, follow these steps.
Go to your project
Go to the main project's page and select your project.
Open the Code Objects page
Select Code from the menu on the left to open the Code Objects page. To search for code objects, use the text boxes located at the top right corner of the page. To search by name enter the full name or part of the name in the Search Name text box. To search by description, enter part of the description in the Search Description text box.
Create a new code object
Select the Create New Code Object button to open the Create New Code Object page. You might need to scroll down to see the entire page.
Enter the code object details
Before you enter information in the fields (see table below), read the following about input and outputs
Multiple input and output datasets are supported in Python, Generalized, and Interactive Container Code Objects. While NVFlare Code Objects do not support multiple datasets, they can still be used as before for learning and inference across multiple sites.
Code objects usually require that you know the number of inputs and outputs your code needs. Those inputs and outputs are provided as data schemas. For example, if your code splits one dataset into two (like a train/test split), you will have to create a code object with one input schema and two output schemas. For more detail into how to structure your code to run on FCP, see Accessing your datasets in Code Objects.
Enter the following:
Ensure that Python Code is selected as the Code Object Type.
Enter Details about this code object, including the name you would like to provide for your code object and optionally, a description. The description that will help you and others understand the nature of this Code Object.
Add the inputs, which are what the Data Schema(s) for the Datasets this Code Object will accept as input. Your selection here will affect which Datasets can be selected as input when triggering a Code Run with this Code Object.
Enter the outputs for the Data Schema(s) for the Datasets that will be created as output when triggering a Code Run with this Code Object. You can also select the option to [Auto-generate Data Schema from Data]. For more information about Auto-generating Data Schemas, please refer to Auto-Generated Data Schemas.
Enter the container details. This is where you can specify parameters that will define your Code Object container:
Python and CUDA Versions, or conversely
Pre-existing Container Base Image (from Dockerhub). By default, the FCP will use the
python:3.9.7-slim-bullseyedocker image. This means your code will run within a Python 3.9 environment. You are free to select whichever docker base image makes the most sense for your use case (i.e. an image that supports GPU operations).
Provide your code
Your code can be provided as a code snippet, standalone file, or an upload file. The information about each appears below.
Code Snippet - Ideal for relatively straightforward scripts that primarily utilize basic Python packages, along with Pandas and Numpy. FCP automatically provides access to your cohort data as a Pandas Dataframe named df. Additionally, it generates the output cohort of your model from this same Dataframe. This setup facilitates elementary Pandas-based operations on your raw input data, such as feature extraction and value normalization.
As an example, consider this Python code snippet:
Executing this code snippet yields an output cohort with z-normalized numerical features, all without requiring any additional code.
Standalone File - This option grants you the freedom to execute varied Python code with custom dependencies, extending beyond Pandas and Numpy. The Standalone File option lets you specify your code's prerequisites, which will be automatically installed within the image generated by FCP. If your code necessitates a specific environment – for instance, to support GPU operations - you can define the Container Base Image that supports it. Use this option for more complex code, which can still be run as a single file. Unlike the Code Snippet, no additional "hidden" functionality is included here.
Upload File(s) - This option is similar to "Standalone File", but allows you to provide your code in more than one file, e.g., multiple Python files and/or shell scripts. Files can include non-Python files of any format, such as configuration files, model parameter files, and so forth. You just have to upload the files you need, or a folder(s), and select the entry point in the text box that appears when this option is selected. The entry point is the file that you'd like your container to run. Be sure to specify code requirements and the container environment as needed.
Enter requirements
Enter the requirements. This field enables you to specify Python libraries that are required to run your code. This capability is enabled for all Python Code types except for Code Snippet which has two locked dependencies—Pandas and NumPy. Paste all Python dependencies that are necessary for running your previously specified Python code. These dependencies will automatically be installed by the Rhino FCP when it builds your container in the background. Alternatively, click Upload File to load a requirements file from your local computer. The Rhino FCP supports both Pip and Conda as package management frameworks.
Set compute resources if needed
If you will be running your code object on the Google Cloud Platform (GCP) and you want to set minimum CPU, RAM, VRAM, and GPU resources, complete the steps in the Setting Required Compute Resources (Autoscaling) section of this document. If not, go to the next step.
Finish
When complete, click Create New Code Object or start a Code Run.
Setting Required Compute Resources (Autoscaling)
Autoscaling allows Rhino to run Code Runs on additional compute resources when the default Rhino Client capacity is not sufficient. This capability is designed to support resource‑intensive, "bursty"(processing tasks occur in short, intense or irregular spurts), or GPU‑based workloads while improving isolation, reliability, and cost efficiency. With autoscaling enabled, Rhino provisions temporary cloud VMs on demand, executes workloads on those VMs, and automatically terminates them when execution completes. This can help you make more efficient use of your computing resources and save on costs.
In this section of the screen there are three options: Use Client Resources, Set Minimum, and Use VM Pool.
Use Client Resources - Uses the available client settings and resources.
Set Minimum - Allows you to set the minimum number of resources that must be available. If you choose the Set Minimum option, you can indicate the minimum number of CPUs, amount of RAM, the disk size (VRAM), as well as the type and number of GPUs. You can adjust these settings when you create a code object and also when you run your code. Note that adjustments do not persist and will need to be reset with each run.
Use VM Pool - Allows you to select a Pre-allocated Virtual Machine (VM) Pool. Pre-allocated VM Pools allow you to indicate which dedicated compute resource you want to use for your code run, ensuring guaranteed availability for high-demand workloads such as GPU-intensive AI training and inference. This feature provides availability of critical compute resources and offers predictable costs through fixed capacity reservations.
Each option is detailed below.
Use Client Resources
To use client resources, complete the following steps.
Select the Use Client Resources button. It should be selected by default.
When complete, select Create New Code Object or start a Code Run.
Set Minimum
To set the minimum required compute resources, complete the following steps.
In the Required Compute Resources section, select Set Minimum.
Next, make changes to the settings as needed. Note that these setting changes do not persist the next time you run the code object or code run. The settings are explained below.
Setting
Description
Usage Notes
CPU MIN
The minimum number of CPU cores on which your code will be run.
If the CPU MIN is too low, you might see slow code execution or throttling.
RAM MIN
The minimum amount of RAM (main memory) that you want to allocate to running your code.
If this is too low, you might see an out-of-memory crash or slowness in execution due to disk swapping.
DISK SIZE
Amount of storage allocated to the instance.
Increasing this setting increases capacity (not necessarily speed)
GPU TYPE
The model of GPU hardware. The number of GPUs indicate the number of GPUs attached to the instance.
If your code isn’t written to use multiple GPUs, extra GPUs might not be used. Note that GPU availability depends on the selected cloud region, Project‑level GPU quotas, and the current capacity at the cloud provider.
Use VM Pool
You can select a VM pool to indicate which dedicated compute resource you want to use for your code run, ensuring guaranteed availability for high-demand workloads such as GPU-intensive AI training and inference. This feature provides faster job startup times and offers predictable costs through fixed capacity reservations. To select a VM Pool, complete the following steps.
Choose Use VM Pool to see the available pools. For each pool you will see a name, as well as the current resources available. For each pool you will see the following information.
Number of VMs (if any)
Number of vCPUs per VM
Amount of RAM per VM
Disk size per VM
Whether there are GPUs available, and how many and their specifications
If you don't know which VM Pool to choose, see Learning More About Available VM Pools.
Select the pool of your choice.
When complete, select Create New Code Object or start a Code Run. The new code object (or code run) has an icon that indicates that the code object will run on a pre-allocated VM pool. If you hover over the icon, the name of the selected VM pool is shown.
VM Pool Statuses and Capacity
This section explains how to track VM Pool statuses across your project. It also provides information on VM Pool job execution and on what happens when a VM Pool capacity has been exhausted.
Tracking VM Pool Statuses
You can track the performance, health, and availability of your VM Pool project resources in the VM Pools Tab. This dedicated subpage provides a comprehensive view of your environment, including total pool capacity, current utilization (number of running VMs), and queue length.
To learn more about the status of the VM Pools are available in your project, do the following.
Select Code from the main menu, then select the Virtual Machine Pools tab at the top of the page.
Review the information about the pools.
Total Pools - Number of VM Pools available across the project.
Total Capacity - Number of VMs, including how many are in use and how many are idle.
Queued Jobs - Number of jobs waiting for processing across all pools.
Health - How many VM pools are active, as well as a brief description of the inactive VM Pools.
Running Jobs - There is also a listing that contains pool information for running jobs. This includes the pool name, compute specs (number of CPUs, amount of ram, GPUs), utilization, how many jobs are in the queue, and the status of the job.
VM Pool Job Execution and VM Pool Capacity Exhaustion When a job is triggered against a pre-allocated pool, it follows a more efficient execution path compared to standard auto-scaling. Jobs are submitted to the chosen pool and transition rapidly from a "Queued" phase directly into a "Running" state as resources are already primed for use.
If a pool's capacity is exhausted, jobs are placed into a First-in-First-Out (FIFO) queue rather than failing. You can monitor the "Queued (VM Pool)" status and your job's position on the Code Run page. Once a slot frees, the job is automatically assigned and dispatched.
Viewing a Code Object's Configuration
To get an object's configuration, make sure you are on the page containing the original object you would like to view the configuration of. In the box where your original object is, there should be a row for each version. Navigate to the version you would like to view the configuration for and select the three-dot menu button, shown below:
The menu button is on the right-hand side of the row, click it to view a new menu. The new menu should have an option to show that object's configuration, click on that to see the object's configuration.
For Code Objects and Code Runs, if you chose the Use VM Pool option, you will use the pool that was indicated during the Code Object or Code Run configurations. Selecting this option shows the resources, including the number of VMs and vCPUs that will be allocated, as well as the amount of RAM, the size of the disk, the number of GPUs, the GPU type/model, and the amount of VRAM.
Creating a New Version of a Python Code Object
To create a new version, complete the following steps.
Go to the page that has the original object you want to create a new version of.
In the upper right corner of the box where your original object is, select the + New Version button.
A new window appears that allows you to create a new version of your object.
Removing a Code Object
To remove a code object, complete the following steps.
Open the Code Objects page
Go to the page that lists your code objects.
Find the code object version
If needed, use the search fields at the top right.
Use Search Name to search by code object name.
Use Search Description to search by code object description.
Each code object appears with a row for each version. Find the version you want to delete.
Open the menu for the version
Select the three-dot menu on the right side of the version row, shown below.
Remove the code object version
Select Remove code object.
Repeat these steps for each version you want to remove. When you delete the last version, the code object is fully deleted.
Running a Python Code Object
After you've created a Python code object, complete the following steps to run it.
Go to the main project's page
Select your project.
Open the Code Objects page
Select Code from the menu on the left to open the Code Objects page.
Select the Run button
Select the Run button for the code object you'd like to run. The Run Model Training page opens.
Fill in the Code Run configuration fields
Fill in the following fields within the new Code Run configuration window. These steps are the same for the Python Snippet, Standalone File, and Upload File(s) options:
Enter the Input Dataset(s). This is one or more Datasets to be used as input to your Code Run. If multiple Datasets are selected, each will be run separately. If a Dataset happens to be a collaborator's Dataset, the code will be run on that collaborator's Rhino Client. In other words, the data never moves outside your collaborator's Rhino Client
Enter the Output Dataset Name Suffix. A suffix that is appended onto the name of each input Dataset to define the name of the output Dataset that will be created during your Code Run
Adjust the Timeout in Seconds if needed. This is the number of seconds that must elapse before a Code Run is automatically halted. This is to avoid zombie tasks that run perpetually within a Rhino client.
Add Run Parameters (Optional). Here you can paste a JSON object that defines additional parameters to be provided to your Code Run. This can be used for specifying hyper-parameters for model validation or any parameter provided to the code. Run parameters are made available to the container code in a file located at
/input/run_params.json
To add run time files to your code run, select the run time files from the list of the workgroup run files that appears. You will want to only include the ones you need for the code run, like in the example that follows.
Selecting Run Time Files During Code Run Set Ups
If you are part of a workgroup that has collaborators, you will be able to see files that are 1) in your active workgroup that are part of the project, whether they are published or not and 2) from a collaborator’s workgroup that are part of the project only if they are published.
To better understand this, let's take a look at the following scenario. Project PRIME is led by workgroup Alpha. Workgroup Beta has been added as a collaborator. Each workgroup has uploaded run time files.
If YOUR active workgroup is Alpha, here is what you see when you open the Run Time Files screen in Project PRIME.
All Workgroup Alpha's files are shown. This is because Project PRIME is led by workgroup Alpha, so all of workgroup Alpha’s files that are part of the project are shown whether they are published or not.
Only Workgroup Beta’s published files are shown. Workgroup Beta’s files that are not part of the project or that are unpublished are not shown.
If YOUR active workgroup is Beta, here is what happens when you open the Run Time Files screen in Project PRIME.
Workgroup Beta's files are all shown. This is because Beta members can see their own workgroup’s files that are part of the project, whether there are published or not.
Only Workgroup Alpha’s files that have been published to PRIME are shown. Workgroup Alpha’s unpublished files that are not part of the project are not shown.
See Managing Run Time Files to learn how to manage, publish and unpublish run time files.
Run the code
When you have completed adding all your Code Run details, either set the required compute resources by following the instruction in Setting Required Compute Resources (Autoscaling), or click the Run button to run your code.
Reviewing Code Run Logs
Code run logs provide details about what happened during a run, as well as the outcome.
To view these logs, complete the following steps.
Select Code Run.
In the Code Runs page, select the code run that you want to see logs for.
The logs page for the Code Run is shown. The start and finish time and date of the run, as well as the status of the run appears. Logs can be accessed while the Code Run is active and after the Code Run has completed. While the code is running, an auto-refresh option is available. Details are shown under the details tab; reports (if any) are shown under the reports tab. Reports are generated using the Rhino SDK.
Select the Details Tab.
Select the Details tab if needed (it should be selected by default).
The Details tab provides the following:
General Info - information about the Code Run, including
Code Object name and type
Code Object version
Code Object description
Input and Output Datasets
Compute Resources Mode including the requested resources at run time (if applicable).
Show Code Object Configuration button. Click this button to see the code object configuration. Show Code Run Configuration button. Click this button to see the code run configuration.
View in TensorBoard button. Click this button to view results in TensorBoard.
System Logs. These can include errors or warnings that occurred from the FCP side, e.g. Dataset import errors.
FL server logs (for NVIDIA FLARE runs)
FL client logs (for NVIDIA FLARE runs)
Each federated client's logs will appear separately
Client-specific logs (for all other run types).
Troubleshooting Code Runs
Error Message that Usage Limit Has Been Exceeded
If you get an error message indicating that the usage limit has been exceeded, your administrator has set usage limits on the number of input rows a workgroup can process per model within a certain timeframe. Getting this message means that the entire run has been cancelled. Please contact your administrator for more details and help with resolving the issue.
Viewing a Code Run's Configuration
To get an object's configuration, make sure you are on the page containing the original object you would like to view the configuration of. In the box where your original object is, there should be a row for each version. Navigate to the version you would like to view the configuration for and select the three-dot menu button, shown below:
The menu button is on the right-hand side of the row, click it to view a new menu. The new menu should have an option to show that object's configuration, click on that to see the object's configuration.
For Code Objects and Code Runs, if you chose the Use VM Pool option, you will use the pool that was indicated during the Code Object or Code Run configurations. Selecting this option shows the resources, including the number of VMs and vCPUs that will be allocated, as well as the amount of RAM, the size of the disk, the number of GPUs, the GPU type/model, and the amount of VRAM.
Deleting a Code Run
Follow these steps to delete a code run.
Open the Code Runs page
Go to the page that lists your code runs.
Find the code run
If needed, use the search fields at the top right.
Use Search Name to search by code run name.
Use Search Description to search by code run description.
Find the code run you want to delete.
Open the menu for the code run
Select the three-dot menu on the right side of the row, shown below.
Remove the code run
Select Remove code run.
Last updated
Was this helpful?