Why This Guide Matters
If you're a partner exploring the concept of LLM routing through SAP AI Core, a common question is how foundation model requests should be routed and what integration options are available. This article provides an overview of the available approaches and key considerations when getting started.
The content is intended as architectural guidance and reference material for solution architects and development teams.
What Is LLM Routing?
LLM routing describes the path taken by prompts and responses between an application and a foundation model.
| Direct Provider Access | Routing Through SAP AI Core |
Request Flow | Partner Application → Model Provider | Partner Application → SAP AI Core / Generative AI Hub → Foundation Models |
Integration Approach | Direct integration with the provider's APIs and services. | Integration through SAP AI Core and the Generative AI Hub
|
Provider Management | Application teams manage provider-specific integrations. Typically requires separate integrations per provider. | SAP AI Core provides a centralized access layer for supported providers. Provides a consistent approach for accessing supported foundation models. |
Getting Started with LLM Routing
Prerequisites
Before implementing LLM routing through SAP AI Core, ensure the following are available:
- SAP BTP Global Account
- SAP BTP Subaccount
- SAP AI Core (Extended Plan)
- Generative AI Hub enabled
High-Level Reference Architecture
This reference architecture illustrates how applications can securely and efficiently access generative AI capabilities, providing a centralized approach to consuming foundation models while supporting trust, governance, and operational control.
Learn More: Access to Generative AI Models | SAP BTP AI Best Practices
Choosing an Integration Approach
When implementing LLM routing through SAP AI Core, partners can choose from two primary integration patterns for consuming foundation models. While each approach serves a different purpose, both leverage SAP AI Core as the access layer for foundation model interactions and can be combined to meet specific solution requirements.
Option A: SAP Cloud SDK for AI
For applications primarily focused on consuming foundation models, the SAP Cloud SDK for AI offers a simple and consistent integration experience. By abstracting provider-specific APIs and providing support for Java, JavaScript, and Python, it enables developers to interact with supported foundation models through a unified interface, helping reduce implementation effort while accelerating development.
Learn More:
Option B: SAP AI Core Orchestration Service
The SAP AI Core Orchestration Service is designed for scenarios where interactions with foundation models require additional processing beyond basic prompt execution. It extends model access with capabilities such as prompt templating, grounding, content filtering, and data masking, helping standardize and govern AI interactions across applications. By centralizing these functions, the service can reduce repetitive implementation effort while providing greater consistency and control over how prompts and responses are handled.
Learn More:
Validating LLM Routing
After implementing LLM routing, it is often useful to validate that foundation model interactions are being processed through SAP AI Core as intended. The following examples highlight several ways to confirm and demonstrate the routing flow across infrastructure, configuration, implementation, and runtime layers.
Infrastructure Validation
- SAP AI Core instance provisioned
- Extended Plan enabled
Configuration Validation
- AI Core service bindings
- AI Core credentials
- Generative AI Hub configuration
Implementation Validation
- SAP Cloud SDK for AI usage
- SAP AI Core Orchestration Service usage
Runtime Validation
- Application logs
- SAP AI Launchpad screenshots
- Generative AI Hub screenshots
- Network traces showing SAP AI Core endpoints
Final Thoughts
Getting started with LLM routing through SAP AI Core can be simpler than it may first appear. With flexible integration patterns and access to a broad range of foundation models, partners can adopt the capabilities that best support their AI-powered scenarios.
Additional References
