> For the complete documentation index, see [llms.txt](https://docs.rhinofcp.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.rhinofcp.com/tutorials-recipes-and-demos/fhir-etl-via-the-rhino-data-harmonization-engine-rhinodhe.md).

# FHIR ETL via the Rhino Data Harmonization Engine (RhinoDHE)

The Rhino Data Harmonization Engine (RhinoDHE) simplifies the transformation of electronic health record (EHR) data into the FHIR data standard.

{% hint style="info" %}
Example data files used in this example can be downloaded from [Rhino user resources tutorials](https://github.com/RhinoHealth/user-resources/tree/main/tutorials/data-harmonization) to replicate the steps described below.
{% endhint %}

## Overview

FHIR harmonization on the Rhino Federated Computing Platform (FCP) involves translating data into a structured, standardized format through schema transformations (syntactic mapping) and aligning data values with standardized terminologies (semantic mapping). This ensures interoperability across healthcare systems.

1. **Create Syntactic Mappings to FHIR Schemas:** Map source tables to their corresponding FHIR resource schemas.
2. **Create Semantic Mappings to Custom Vocabularies:** Harmonize codes such as race, ethnicity, encounter types, and conditions to standardized FHIR ValueSets.
3. **Execute the Data Harmonization:** Apply mappings to convert your data into FHIRPath, which is a tabular representation of FHIR data.
4. **Generate FHIR Resources using a CodeObject:** A code object is used to transform the tabular representation of your FHIR data into nested JSON objects that are valid FHIR resources.

## Step 1: Review FHIR Profiles & Source Data

First, familiarize yourself with the FHIR profiles that are to be created through the project You will likely benefit from [learning more about the FHIR format for clinical data](https://hl7.org/fhir/overview-dev.html) and subsequently familiarizing yourself with the implementations of the FHIR format called *Profiles* that are specific to given countries or jurisdictions.

* **What FHIR resources am I creating?** Prior to starting any project to transform local data into FHIR resources, is it important to identify the set of FHIR resources to be created.
* **What data is required within each FHIR resource?** Each FHIR profile has several data elements that are required, whereas others are not required. [Simplifier.net](https://simplifier.net/search?Projects=) is a fantastic resource used by many FHIR developers to store their Profiles and the site offers an interface to review Profile specifications and the fields required therein.

Second, review your source data to better understand which data will be necessary to transform into your desired FHIR resources. The Rhino FCP offers a useful interface for this exploration:

![](/files/5be4669f276b514830a9f5b75b0a113cde659a2a)

* **Which source tables contain relevant information?** In some cases, creating a FHIR resource will require data from only a single source data table. However, it is more common that the data required for a single FHIR resource will be distributed across multiple tables.
* **Which source columns contain relevant data?** Focus on columns required by the FHIR resource schemas.
* **What transformations are required?** Identify proprietary codes requiring semantic mapping to standard FHIR ValueSets.

## Step 2: Create Syntactic Mapping

{% hint style="info" %}
**Tip:** Create 1 syntactic mapping for each source table. For example, if data for the {Encounter} FHIR resource is to be derived from two tables (ex. Visits and Hospitalizations), create 2 syntactic mappings.
{% endhint %}

Syntactic mappings serve as a blueprint for transforming your source data into the FHIR model. This process will create a tabular representation of your desired FHIR data; the actual FHIR data in JSON format will be created in a subsequent step. Follow these steps to create a Syntactic Mapping to FHIR:

1. Navigate to **Data Mappings** → **Syntactic Mappings**.
2. Click **Create Syntactic Mapping**.
3. Select **Custom** as the target data model.
4. Choose **Manually Configure** to use the graphical interface.
5. Select the relevant **Source Data Schemas**
   * If you'll use the VLOOKUP transformation in your syntactic mapping, select the data schema of the reference table *in addition* to the primary source data schema.
6. Select the relevant **Target Data Schema** that represents the FHIR Profile to be created.

![](/files/558fee801d6d3104a6ba40f7e19d9cb223ae91a3)

**Graphical Interface for Mapping**

Once the Syntactic Mapping has been created, an interface to map source fields to target FHIR fields will appear. The following three columns exist in this interface:

* **Source Fields:** This column enables the selection of the relevant source fields that will be used to populate a given FHIR field.
* **Target Field:** This column is automatically populated with each possible field associated with a given FHIR profile. In the case that a corresponding Source Field is not selected, the Target Field will remain empty in the resulting FHIR data.
* **Transformation**: This column enables users to define how the source field should be transformed into compliant FHIR data. Navigate below to learn more about Transformations.

**Column Transformations for FHIR:**

Transformations are the 'workhorse' of the RhinoDHE and are the primary means of standardizing your data into FHIR format.

* **No Transformation** In some cases, your source data column will require no transformation to be included in a FHIR resource. In this case, simply select the relevant source field and leave 'Transformations' empty. Examples include encoding a patient's identifier into an identifier.value field, or inserting an ICD code into a Condition FHIR resource, as ICD codes are typically valid per FHIR ValueSets.

![](/files/0cae2fd01ec390d290c7b33d7fb7d611cac872bc)

* **Semantic Mapping:** Map source values to standardized terms and is recommended for large ValueSets or for small ValueSets when there is a large number of corresponding source terms to be mapped.

![](/files/8a67f44979116854b4d08ced39cf4eb2673d5ba9)

* **Custom Mapping:** Like Semantic Mapping, Custom Mapping maps source values to standardized terms but is recommended for small ValueSets when there is a low number of corresponding source terms to be mapped. This exists as an alternative to Semantic Mapping.
  * In the following example, a Custom Mapping has been defined that maps three source does to three standard codes:

    ```auto
    male, Male
    female, Female
    unk, Unknown
    ```
* **Set Value:** Assign a constant value across all rows. This transformation is often used for [**system** codes in FHIR](https://simplifier.net/packages/hl7.fhir.r4.core/4.0.1/files/82704), which require a constant *uri* (e.g. <https://fhir.hl7.org.uk/CodeSystem/UKCore-AddressKeyType>). Note that any arbitrary Source Field can be selected for Set Value transformations without impact to the generated output.

![](/files/b6c7096eb3388b3012d825b8eb13800fb80bef27)

* **Convert Date:** Reformat date columns to [FHIR-required formats.](https://build.fhir.org/datatypes.html)
* **Stable UUID:** Generate consistent identifiers; these are transformed into FHIR-compliant identifiers in the post-processing FCP CodeObject (see Section 4).
* **VLookUp:** Retrieve data from another table based on a foreign key; approximates a JOIN operation.
  * For example, the creation of an Encounter FHIR Profile may require populating Encounter.diagnosis.condition with a diagnosis code that is in a table named Diagnoses, with **visit\_id** existing as a foreign key to link the tables. Your transformation would look like this:

![](/files/3c5c70afb15f7ac7637b5a8f690a52f261616a48)

* **Table Level Code**: Execute any arbitrary Python code across all available tables and rows. This is the appropriate transformation type when generating lists of multiple data elements, which is common in FHIR arrays (eg. a Patient with multiple phone numbers or an Encounter with multiple diagnoses).
  * In the following example, I am attempting to create an list of diagnosis codes for each Encounter. This can be accomplished by performing a groupby() operation in Pandas and then merge that dataset with my original source table.
  * ```auto
    # generate lists of diagnoses for each VisitID
    array_df = my_diagnoses_dataset_schema_v0.groupby('VisitID')['ICDCode'].apply(lambda x: x.unique().tolist()).reset_index()

    # merge list of diagnoses into original dataframe
    output = my_encounters_table_schema_v0.merge(array_df, on = 'VisitID', how = 'left')['ICDCode']
    ```

![](/files/5de0c45a9f2bdd968e5412066f8dd189ffc68e8f)

{% hint style="info" %}
**Helpful Tips for Creating Valid FHIR Data:**

1. All references to other tables (e.g. Procedure.subject.reference) should be populated with the identifier used for the table to which they relate (e.g. Patient.id). If a transformation was applied to the referent identifier (e.g. Stable UUID on Patient.id) then the same transformation should be applied to the reference column.
2. For all coding systems, (e.g. Patient.maritalStatus.coding.system), the value should be a URL to a code system, not a valueset (eg. <http://terminology.hl7.org/CodeSystem/v3-MaritalStatus>)
3. Always include a default value when creating semantic mappings to handle missing values. For example, the default value for Patient.gender should be 'unknown'. If this is neglected, the resulting FHIR data will be invalid.
   {% endhint %}

## Step 3: Create & Review Semantic Mappings

In the context of FHIR, the creation of Semantic Mappings on the FCP entails the translation of source data to FHIR ValueSets. A semantic mapping object should be created for each Semantic Mapping transformation specified in the Syntactic Mapping created in Step 2.

1. Navigate to **Data Mappings** → **Semantic Mappings**.
2. Click **Create Semantic Mapping**.
3. Select the **Dataset** and **Field to Map** from the relevant source dataset and column, respectively.
4. Select **Custom** **Vocabulary** as the target.
5. Select the relevant FHIR Value Set to be mapped via the Custom Vocabulary dropdown.

![](/files/3d17b944b153509090002bd27de8e90bac4ab068) You can follow [these instructions](/data-harmonization/creating-reviewing-editing-and-removing-a-semantic-mapping.md) on how to review and approve the AI-generated recommendations for semantic mappings.

## Step 4: Execute Data Harmonization via the User Interface

Syntactic mappings serve as a blueprint for transforming your source data into the FHIR model. This process will create a tabular representation of your desired FHIR data; the actual FHIR data in JSON format will be created in a subsequent step. Follow these steps to create a Syntactic Mapping to FHIR:

1. Navigate to **Code**
2. Select **Run** on the relevant Data Harmonization Code Object. Repeat this process on all Data Harmonization Code Objects related to FHIR data. Depending on the size of your source dataset, the amount of time required for the Code Run to complete will vary.

![](/files/164e8d23ec76fb18450138a94f78b82f0d549988)

After the Code Runs complete, you can navigate to the **Datasets** section to review your newly harmonized dataset. The name of each harmonized dataset will reflect the target schema (eg. 'patient\_fhir\_v0' will be created from a schema called Patient (FHIR)).

## Step 5: Generate FHIR resources via FCP CodeObject

Once your source data has been transformed into a tabular representation of FHIR data (per Step 4), you'll run an FCP Code Object specifically design to generate valid FHIR data. This Code Object can take multiple harmonized datasets as input and will create a JSON file for each row in the input datasets.

![](/files/e879d2396e6574b439513f9117480a3f422ef9a6)

Once the Code Object completes it's run, you can inspect the contents of the dataset, which will show the files created by the code run:

![](/files/d43b25df6c45b4002b1b9d5a788822f6fc2835fb)

## Step 6: Export Dataset

The final step to perform is to export your dataset, which will save all of the newly-generated JSON files onto a specified directory on your Rhino client. To do so, simply click on the three dots in the Datasets interface.

![](/files/bc36eb4c42c02a97260fd9bd62680879cc47db24)

## Automating the FHIR Pipeline

It is possible to automate the entire process outlined above using the [Rhino SDK](https://rhinohealth.github.io/rhino_sdk_docs/html/index.html), which is critical for compliance with requirements in some jurisdictions to update clinical data on an ongoing basis. The code sample below demonstrates how the [Rhino SDK](https://rhinohealth.github.io/rhino_sdk_docs/html/index.html) can be used to transform messy local clinical data and turn it into valid FHIR resources formatted as JSON files:

```python
import getpass

from pprint import pprint

from rhino_health.lib.endpoints.code_object.code_object_dataclass import CodeObject, CodeTypes, CodeObjectRunInput
from rhino_health.lib.endpoints.syntactic_mapping.syntactic_mapping_dataclass import DataHarmonizationRunInput

# Rhino FCP authentication
my_username = ""  # Replace with your email
session = rh.login(username=my_username, password=getpass.getpass())

# Project configuration
project_uid = '2d32128d-3a27-408a-9315-e7e37c458718'
workgroup_uid = 'c50eb65a-7c61-422f-84a5-dd515fff5c24'

# Execute data harmonization to FHIR
print("Transforming Data to FlatFHIR")

code_run = session.code_object.run_data_harmonization(
    code_object_uid='120569bb-6c9e-4760-9a1d-1151d6e57ca4',
    run_params=DataHarmonizationRunInput(
        input_dataset_uids=['1f124ad2-6a14-4bcd-ba00-9951cdf33758'],
        semantic_mapping_uids_by_vocabularies={}
    )
)

run_result = code_run.wait_for_completion()
output_dataset_uid = run_result.output_dataset_uids.root[0].root[2].root[0]

# Generate FHIR resources
print("Generating FHIR Resources")

code_run = session.code_object.run_code_object(code_object_params)

run_result = code_run.wait_for_completion(CodeObjectRunInput(
    code_object_uid='b54d8b44-3b10-4a64-85e0-767df07aa40b',
    input_dataset_uids=[[output_dataset_uid]],
    timeout_seconds=8600
))

fhir_dataset_uid = code_run.code_run.output_dataset_uids.root[0].root[0].root[0]

# Export results as CSV
print("Saving FHIR JSON To Rhino Client")

export_response = session.dataset.export_dataset(
    dataset_uid=fhir_dataset_uid,
    output_location='/rhino_data/fhir_data/',
    output_format='csv'
)
```