> For the complete documentation index, see [llms.txt](https://docs.rhinofcp.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.rhinofcp.com/frequently-asked-questions/frequently-asked-questions-faqs.md).
# Frequently Asked Questions (FAQs)
## Setup and Configuration
Which hardware configurations are compatible with the Rhino FCP Platform?
The RH client has been successfully installed on a wide range of hardware configurations, including dedicated servers (Rhino Boxes), VMs provisioned on existing servers on-prem, and cloud-based instances on various clouds (AWS, GCP, Azure).
We have minimal requirements for hardware specifications, but they heavily depend on the kind of computation task the user is looking to perform (i.e. GPU is not required for a Rhino Client installation).
How can I setup or change my ECR credentials?
Before starting, please ensure you have completed all the steps within [FCP Client Installation and Environment Configuration](/getting-started/quick-start-guide/fcp-client-installation-and-environment-configuration.md). This will ensure that your dependencies have been installed. It will also show you how to retrieve your ECR credentials. To setup or re-configure your ECR your will need your **AWS Access Key ID** and your **AWS Secret Access Key**.
#### Configuring your AWS CLI with your ECR Credentials
{% stepper %}
{% step %}
#### Open a terminal or command prompt.
Type in the command:
```bash
aws configure
```
{% endstep %}
{% step %}
#### Enter your credentials.
The code block below shows what you will need to enter at each step of the prompt.
```bash
> aws configure
AWS Access Key ID [None]: YOUR_AWS_ACCESS_KEY_ID
AWS Secret Access Key [None]: YOUR_AWS_SECRET_ACCESS_KEY
Default region name [None]: us-east-1
Default output format [None]: json
```
As a reminder of what each value should be:
* **AWS Access Key ID** - The AWS Access Key ID from your FCP profile
* **AWS Secret Access Key** - The AWS Secret Access Key from your FCP profile
* **Default region name** - us-east-1
* **Default output format** - json
{% endstep %}
{% endstepper %}
## Importing and Exporting Data
How can I import data in my local environment onto my Rhino FCP client using SFTP?
The process of moving data from your local environment to your Rhino FCP client using SFTP depends on the operating system of your local machine.
#### Connecting to your Rhino FCP Client via SFTP from MacOS, Linux & Windows 10+
{% stepper %}
{% step %}
Open a terminal or command prompt on your respective operating system, and navigate to the folder where the data that you would like to upload are located.
{% endstep %}
{% step %}
Connect to your Rhino FCP Client via SFTP
* Retrieve your SFTP server IP and credentials from your Rhino FCP profile page. If you need a quick refresher on how to do that, see [Viewing and Adjusting Your Settings](/settings/viewing-and-adjusting-your-settings.md).
* Connect to your Rhino FCP client via SFTP, by entering the following.
Note: Ensure to replace `RHINO_CLIENT_IP_ADDRESS` in the below command with the credentials found in your profile.
```bash
> sftp rhinosftp@RHINO_CLIENT_IP_ADDRESS
```
{% endstep %}
{% step %}
Next, you will be prompted to enter your SFTP password, and if everything is entered properly you will be connected to your Rhino FCP client and your command line should change to look like the following:
```bash
sftp>
```
**For additional help with common commands within the SFTP terminal interface, check out the section** [**Common Commands when SFTP'ing Data**](#01H7BJ37P6X1GA3C4DXHDHEFB0) **.**
{% endstep %}
{% endstepper %}
#### Connecting to your Rhino FCP Client via SFTP from Other Operating Systems
{% stepper %}
{% step %}
To SFTP data from other Operating Systems to your Rhino FCP client, you will need to first install an SFTP client, like [Cyberduck](https://cyberduck.io/), [WinSCP](https://winscp.net/eng/index.php), etc.
{% endstep %}
{% step %}
Once your SFTP client has been installed, you can configure a new SFTP connection.
* Retrieve your SFTP server IP and credentials from your Rhino FCP profile page. If you need a quick refresher on how to do that, see [Viewing and Adjusting Your Settings](/settings/viewing-and-adjusting-your-settings.md).
* If you are successfully connected to your SFTP client, you should see two separate panels, one displaying your local machines file system and a second showing your Rhino FCP client's filesystem.
{% endstep %}
{% step %}
Using the STFP client to upload your data:
* On the local machine file system panel, navigate to the folder where your data is located.
* On the Rhino FCP client file system panel, navigate to where you would like to place the data.
Note: It is recommended that you create folders for your datasets to keep things organized.
* Drag the files from the local machine file system panel to the Rhino FCP client file system panel to upload your data to the Rhino FCP client server.
{% endstep %}
{% endstepper %}
#### Common Commands when SFTP'ing Data
Use the following commands to create folders, navigate to different folders, view folder contents, and upload data.
**Create a new folder called DIRECTORY\_NAME within your Rhino FCP client**
```bash
sftp> mkdir DIRECTORY_NAME
```
**Change to the specified directory, DIRECTORY\_NAME within your Rhino FCP client**
```bash
sftp> cd DIRECTORY_NAME
```
**List all files within the current directory on your Rhino FCP client**
```bash
sftp> ls
```
**Upload the specified file FILE\_NAME on your local machine to the current directory of your Rhino FCP client**
```bash
sftp> put FILE_NAME
```
**Upload the folder and all of its contents on your local machine to the current directory of your Rhino FCP client**
```bash
sftp> put -r DIRECTORY_NAME
```
**Exit out of the SFTP connection**
```bash
sftp> exit
```
## Writing and Running Code
How can I pass and use parameters in my code?
If you want to re-use your code, but need to make some small changes without hardcoding them, **Run Parameters** and **Run Secrets** can be very helpful.
#### Passing Run Parameters
When you are ready to run your code, you'll be able to provide parameters in either **RUN PARAMETERS** or **RUN SECRETS**, see a screenshot below and more detail in [What is a Code Object?](/creating-and-running-code-objects/what-is-a-code-object.md).
#### Loading/Using Run Parameters
To use the parameters, simply access them by reading the JSON file in which they are stored. The parameters will be stored in "/input/run\_params.json" and in "/input/secret\_run\_params.json" for **RUN PARAMETERS** and **RUN SECRETS PARAMETERS** respectively. For example, let's say you want to pass "my\_parameter" = 55555, in RUN PARAMETERS. In the UI, you'll set the following RUN PARAMETER:
```json
{"my_parameter": 55555}
```
If you were to write Python code, in your code first you'd read /input/run\_params.json:
```python
parameter_key = "my_parameter"
run_params = {}
run_params_filename = "/input/run_params.json"
try:
with open(run_params_filename, "r") as run_params_file:
run_params = json.load(run_params_file)
except IOError:
logging.error("No params file found")
sys.exit(1)
except json.JSONDecodeError:
logging.error("Invalid JSON found in params file")
sys.exit(1)
if parameter_key not in run_params:
logging.error(f"Key '{parameter_key}' not found in run parameters")
sys.exit(1)
```
and last you'd access the value of your parameter as follows:
```python
my_parameter_value = run_params[parameter_key]
```
For RUN SECRETS PARAMETERS the process would be almost identical except that you would set the parameters in the RUN SECRET PARAMETERS input form, and you would read them from /input/secret\_run\_params.json.
How do I find a UID for an object (data schema, dataset, code object, or code run)?
To find and copy an object's UID, open the page for that object. If the object has versions, go to the version you want. Then open the three-dot menu for that row, as shown below:
The menu is on the right side of the row. Select it, then choose **Copy UID**.
Projects and collaborators do not have versions. Open the object, then use the same three-dot menu to copy its UID.
When should I use a Python Code Snippet vs. a Python Code Standalone File?
* **Code Snippet** - Ideal for relatively straightforward scripts that primarily utilize basic Python packages, along with Pandas and Numpy. FCP automatically provides access to your cohort data as a Pandas Dataframe named **df.** Additionally, it generates the output cohort of your model from this same Dataframe. This setup facilitates elementary Pandas-based operations on your raw input data, such as feature extraction and value normalization.
As an example, consider this Python code snippet:
```python
normalized_df = (df-df.mean())/df.std()
df = normalized_df
```
Executing this code snippet yields an output cohort with z-normalized numerical features, all without requiring any additional code.
* **Standalone File** - This option grants you the freedom to execute varied Python code with custom dependencies, extending beyond Pandas and Numpy. The Standalone File option lets you specify your code's prerequisites, which will be automatically installed within the image generated by FCP. If your code necessitates a specific environment – for instance, to support GPU operations - you can define the Container Base Image that supports it. Use this option for more complex code, which can still be run as a single file. Unlike the ***Code Snippet***, no additional "hidden" functionality is included here.
* **Upload File(s)** - This option is similar to "Standalone File", but allows you to provide your code in more than one file, e.g., multiple Python files and/or shell scripts. Files can include non-Python files of any format, such as configuration files, model parameter files, and so forth. You just have to upload the files you need, or a folder(s), and select the entry point in the text box that appears when this option is selected. The entry point is the file that you'd like your container to run. Be sure to specify code requirements and the container environment as needed.
How can I use external files at run time?
Your workgroup has a unique cloud storage (an S3 bucket on Rhino Orchestrator) to upload non-sensitive data that might be needed during code run, e.g., open-source large LLM model weights.
{% hint style="warning" %}
Cloud storage (specified in *Workgroup code and model artifacts storage* section in the instructions that follow) is owned by Rhino. **Please DO NOT upload sensitive information such as proprietary datasets.** For sensitive data, please use the instructions in [Importing, Viewing a Dataset's Configuration, and Exporting Datasets](/datasets/importing-viewing-a-datasets-configuration-and-exporting-datasets.md) instead.
{% endhint %}
#### Overview
For your code to be able to access files while running in your agent we need do 4 steps:
#### Find the bucket to upload your files
The bucket is created as part of the onboarding process into the FCP by client request, you can find it in your settings page under Containers & Artifacts. Here is how to find it.
{% stepper %}
{% step %}
In your project, select the settings button (it looks like a sprocket at the bottom left side of the page.
{% endstep %}
{% step %}
In the Settings menu, which is near the top of the screen on the left side of the window, select Containers & Artifacts.
{% endstep %}
{% step %}
Your storage bucket and bucket prefix appear in the *Workgroup code and model artifacts storage* part of the page.
{% endstep %}
{% endstepper %}
#### Upload files to your designated bucket
AWS offers many alternative ways to upload files to an S3 bucket. We've also provided a script in our user\_resource repository you can use: [upload-file-to-s3.sh](https://github.com/RhinoHealth/user-resources/blob/main/rhino-utils/upload-file-to-s3.sh).
You need to define your S3 credentials and then you can call the script like this:
```bash
./upload-file-to-s3.sh the_folder_to_upload storage_bucket bucket_prefix your_path_in_bucket
```
For the example above, if you wanted to upload files in a local folder (called "local\_folder" in the below example) and store them in a folder in FCP called "my\_files", the command would be:
```bash
./upload-file-to-s3.sh ./local_folder external-files-rhino-health-dev my_files
```
#### Reference your files in your code
When creating your code object, you can reference your external files by referencing them in the following path:
```
/external_data/the_folder_in_s3_you_want_to_upload_to/filename
```
For the example above, assuming you uploaded `model_params.txt` file under `my_files`, the file would be accessible during runtime in
```
/external_data/my_files/model_params.txt
```
An example of Python code to read your file data into a `text` variable would look like as follows:
```python
from pathlib import Path
text = Path('/external_data/my_files/model_params.txt').read_text()
```
#### Use your files in a specific run
Once your code points to your external files, and the code object is created, you can run the code and specify the files you want to use in your code run. In the UI, you can simply select them from a dropdown
Alternatively, you can also use the SDK. See this [notebook to use external files via SDK](https://github.com/RhinoHealth/user-resources/blob/main/examples/rhino-sdk/runtime_external_files.ipynb) for more details.
How Do I Retrieve Weight Files?
There are a couple of ways to retrieve your weight files depending on the model type:
* If you are running a training with NVFlare, the model weights are retrievable from the Code Run page. Click on three dot menu in the desired NVFlare Code Run, you will be given the option to download the model weights. See screenshot below:
* For Interactive Containers and Generalized Compute, you will need to define a variable within the schema of type filename and direct the dataset.csv to the model weights that you have output.
How can I share my NVFlare model so that my collaborator can fine-tune it?
There are several ways to share your model with a collaborator so that they can fine-tune it. Here is one simple way to do this.
1. Create and train your model. To do this, you’ll need to create an NVFlare code object and run it using these instructions. These articles provide details on how to do this: [Creating and Running NVFlare Code and Running Inference](/creating-and-running-code-objects/creating-and-running-nvflare-code-and-running-inference.md), [Creating and Running NVFlare Code and Running Inference](/creating-and-running-code-objects/creating-and-running-nvflare-code-and-running-inference.md), [Managing Run Time Files](/creating-and-running-code-objects/managing-run-time-files.md).
2. Once the model has been trained, publish the model so that your collaborators can see it. See [Publishing and Unpublishing Code Objects and Code Runs](/creating-and-running-code-objects/publishing-and-unpublishing-code-objects-and-code-runs.md) for more details.
3. Finally, create a code object for fine-tuning, then publish the code object.
4. Your collaborator then runs the fine-tuning code object, and selects the runtime model that you published.
5. The fine-tuned model is generated and is available in runtime files for the collaborator’s workgroup. Note that you cannot access the collaborator’s fine-tuned model unless they publish it so that you can see it.
#### Protecting Your Model
If you need to protect your model, consider doing one or more of the following.
* You can use permissions to prevent model downloads. The Download Model Parameters, Manage Code Objects policies, and View Code Objects and Code Runs settings should be set properly. For more information, see [Persona and Permissions Policies Reference](/projects/persona-and-permissions-policies-reference.md) and [Creating a New Project](/projects/creating-a-new-project.md).
* Encrypt the fine-tuning code object that you supply using a secret manager integration, which pulls secrets from your secrets manager. You can then decrypt it at run-time so that your collaborator can use it. See [Model and Code Encryption for Federated Computing Scenarios](/tutorials-recipes-and-demos/model-and-code-encryption-for-federated-computing-scenarios.md) for more details.
* Use confidential computing to encrypt memory. See [Confidential Computing Supported Environments and Permission Settings](/settings/confidential-computing-supported-environments-and-permission-settings.md).
**Related Topics**
* [Creating and Running NVFlare Code and Running Inference](/creating-and-running-code-objects/creating-and-running-nvflare-code-and-running-inference.md)
How does the VM Pool option work?
When you target a pre-allocated VM pool, your jobs follow specific execution patterns:
* **Job Queueing:** If the VM pool has exhausted its current capacity, your job will be placed in a pool-level queue.
* **Queue Status:** Jobs waiting for capacity will display a **"Queued (VM Pool)"** status on the Code Run page. The system uses a First-in-First-Out (FIFO) policy for these jobs.
* **Performance:** Jobs typically transition rapidly from "Queued" to "Running" because the VMs are pre-warmed, eliminating standard provisioning delays.
* **Ad-Hoc Fallback:** If a VM pool is misconfigured or lacks reservations in a specific region, the system will automatically fall back to standard on-demand auto-scaling to ensure your job still executes. If this occurs, a warning banner will appear on the Code Run status page.
#### Monitoring Pool Metrics
You can track the performance and availability of your resources via the project dashboard.
* **VM Pools Subpage:** A dedicated "VM Pools" subpage is available on the project dashboard.
* **Real-time Metrics:** Use this page to view:
* Total pool capacity.
* Current utilization (number of running VMs).
* Current queue length.
* **Maintenance Mode:** If a pool is undergoing maintenance, a banner will appear on the Code Run page indicating that jobs will remain queued until the maintenance is complete.
#### Understanding Job Execution States: Pre-allocated VM Pools
When executing jobs using a Pre-allocated VM Pool, the platform manages job states to prioritize performance and reliability. Because these pools utilize pre-warmed Virtual Machines, you will notice significantly reduced initialization times compared to standard Rhino Client auto-scaling.
Below is an overview of how jobs behave when targeted at a pre-allocated VM pool.
#### The Job Lifecycle
When you trigger a Code Run against a pre-allocated VM pool, the platform follows a streamlined execution path:
* **Submission:** Your job is submitted to the specific VM pool you selected during Code Object configuration.
* **Provisioning:** Unlike standard auto-scaling, which requires time to provision new VMs, jobs on a pre-allocated pool skip the standard startup delay.
* **Execution:** Jobs transition rapidly from the "Queued" phase directly into a "Running" state, as the VMs are already "warm" and ready for your workloads.
#### About "Queued (VM Pool)"
If the VM pool has reached its total capacity, your job will not fail. Instead, it enters a specialized state:
* **State:** Queued (VM Pool)
* **How it works:** When capacity is exhausted, incoming jobs are placed into a pool-level job queue. This queue strictly follows a **First-in-First-Out (FIFO)** policy.
* **Monitoring your position:** On the Code Run status page, you can view your job's current position within this dedicated queue.
* **Transitions:** Once a running job completes, the VM is returned to the pool, re-warmed, and immediately assigned to the next job in the queue, moving it from Queued (VM Pool) to Running.
#### Ad-Hoc Fallback Behavior
In cases where a VM pool is misconfigured, or a reservation is unavailable in the specific zone/region where your Rhino Client is located, the system ensures your work is not interrupted.
* **Automatic Fallback:** The system will automatically revert to standard on-demand (ad-hoc) auto-scaling compute to ensure your job executes.
* **User Notification:** If this occurs, a prominent warning banner will appear on the Code Run status page, for example: *"Note: Standard on-demand compute was used for Agent \[X] due to reservation unavailability in its region."*
* **Audit Trail:** This fallback event is logged in the job’s audit trail to assist in troubleshooting and capacity planning.
## Other Questions
Why are the timestamps on the platform different from my local time?
To ensure consistency across users in different locations, all timestamps on the platform are displayed in Coordinated Universal Time (UTC). This allows users from various time zones to have a standardized reference point.
Why are GPU metrics not present or updating within the Collaborators page?
* **If no GPU metrics are present** - this could have been a conscious decision to not include a GPU within the hardware provisioned for the Rhino Client. A GPU in a Rhino Client is use-case-dependent; therefore, not all Rhino Clients will have GPUs attached. If one of your collaborators doesn't have a GPU attached and you require GPUs for the execution of your project, please reach out to for help.
* **If GPU metrics are not updating when you refresh the page** - this is likely due to an incompatibility with the GPU drivers and how the Rhino FCP platform measures GPU metrics. We strive to support all GPU drivers, but in rare cases, certain GPUs are not yet supported or are simply incompatible with our metrics measurements. Note: Just because metrics cannot be displayed does not mean that the GPU can't be used; it is only related to polling for GPU metrics.