Benchmark Performance

Semantic (Terminology) Mapping Benchmark Performance

Rhino's self-hosted terminology mapping pipeline was benchmarked against frontier hosted LLMs across a 2,312-query biomedical test set spanning nine cross-vocabulary mapping tasks.

Running entirely inside the customer's firewall, the self-hosted pipeline (Rhino OSS) achieved 67.5% top-1 accuracy and 84.9% top-5 accuracy (the correct concept appears among the first five suggestions), at roughly 7.8x the speed of frontier hosted LLMs on the same task.

Against hosted frontier models using the same candidate lists and prompts, Rhino OSS lands within about five points of the strongest configurations tested: Rhino + Claude Opus (72.1% top-1 / 90.0% top-5) and Rhino + GPT (71.3% top-1 / 89.3% top-5). Frontier models performed comparably only when paired with Rhino's retrieval pipeline — on their own, general-purpose LLMs struggled with this task. This gives customers a self-hosted option that closes most of the gap to frontier-assisted mapping without sending terminology data outside the site, at a fraction of the per-query cost.

A follow-on evaluation added a MedGemma-27B classifier reranking stage over embedding-retrieved candidates (4-way fusion plus synonym expansion, k=30), scored across the same nine datasets. Accuracy varies meaningfully by vocabulary pair: strongest on ICD-O-3→SNOMED and OPCS-4→SNOMED (93–95% top-5/top-10), and weakest on ICD-10-CM→SNOMED (63.7% top-5, 68.5% top-10) and Sheba Labs→LOINC (76.4% top-5, 80.3% top-10). Across all nine sets, unweighted average accuracy was 83.4% top-5 and 86.6% top-10.

Methodology note: both benchmarks score 1:1 mappings only. The first benchmark held candidate lists and prompts identical across all three model configurations; the second reflects classifier-stage accuracy on top of embedding retrieval rather than a standalone LLM comparison. Because accuracy is pair-dependent, results should be cited per-vocabulary-pair rather than as a single blended figure when responding to RFP evaluation criteria.