Providers
An LLM Surface’s Upstream node holds one provider configuration. Every provider type carries a common set of sampling parameters; the fields below list what else each one adds once you select it, and where to find the values it asks for.
Common sampling parameters
Every provider accepts these optional fields, unless noted otherwise:
| Field | What it does |
|---|---|
| Temperature | Sampling temperature, as a number box. Leave unset to use the model’s own default; lower it for more deterministic output, raise it for more varied output. |
| Max Tokens | Maximum completion tokens. |
| Top P | Nucleus sampling threshold. |
| API Key Secret | Secret-store ID holding the provider API key. Never a raw key. Create the secret first on the Secrets page, pasting in the API key from the provider’s own dashboard, then reference it here. |
Provider types
Choosing a provider type on the Upstream node sets the wire format, the credentials you need, and which extra fields that upstream expects, such as a deployment name, a cloud project ID, or a region. Get it right and the surface speaks the format your provider actually understands, with cost and telemetry attributed to the right vendor instead of lumped in as something generic.
- OpenAI, Anthropic, Google Gemini, AWS Bedrock, and Azure OpenAI are first-class types, each with all the fields that provider’s own console will hand you. Pick whichever one you already have an account and billing relationship with.
- For an OpenAI-compatible provider (Groq, xAI, OpenRouter, Fireworks, Cerebras, DeepSeek, Moonshot, SambaNova, Hyperbolic, NVIDIA NIM, Nebius), pick its own preset rather than the generic OpenAI type: you still get that provider’s pricing, rate limits, and model lineup, with cost and telemetry attributed to it by name.
- Custom only makes sense when your endpoint isn’t in the catalogue at all. It asks for your own auth header and secret, since there’s no known schema to assume.
- Azure OpenAI is the one type that identifies the model by Deployment Name rather than Model, since Azure’s own portal names deployments rather than models directly.
| Provider type | Extra fields | Where these values come from |
|---|---|---|
| OpenAI | Organization ID, Base URL | Organization ID (optional) comes from your OpenAI account’s Organization settings in the OpenAI platform dashboard; set it only if your account belongs to more than one organization. Leave Base URL unset unless you’re routing through a proxy in front of the OpenAI API. Also the fallback shape for every OpenAI-compatible preset (see below). |
| Anthropic | API Version | Anthropic Messages API. The version string comes from Anthropic’s own API documentation; most surfaces can leave this at its default. |
| Google Gemini | Project ID | Your Google Cloud project ID, from the Google Cloud Console. Uses Gemini’s generateContent API. |
| AWS Bedrock | Region, Base URL | Region is a fixed AWS region code (e.g. us-east-1), not free text. Set it to wherever your Bedrock access is provisioned; it derives a working default endpoint for that region unless Base URL is also set, in which case Base URL takes precedence. |
| Azure OpenAI | Deployment Name (required), Endpoint (required), API Version (required) | All three come from your own Azure OpenAI resource in the Azure Portal: Deployment Name and Endpoint from that resource’s Deployments page, API Version from the version Azure has published for it (Azure versions the API independently of the model). Unlike other providers, identifies the model by Deployment Name rather than Model. |
| Cohere | — | No provider-specific fields; the common sampling parameters and Model are all that apply. |
| Mistral | Endpoint | Point this at your Mistral API base URL. |
| Together AI | — | No provider-specific fields; the common sampling parameters and Model are all that apply. |
| Replicate | — | No provider-specific fields; the common sampling parameters and Model are all that apply. |
| Hugging Face | Endpoint | Point this at your Hugging Face Inference Endpoint URL. |
| Ollama (local) | Base URL, API Schema | Base URL should point at your own locally or privately hosted Ollama instance rather than a hosted API. |
| Custom | Name (required), Endpoint (required), Auth Header, Auth Secret | For any custom OpenAI-compatible endpoint not in the catalogue. Endpoint is your own service’s base URL; set Auth Header and Auth Secret to whatever header name and credential that endpoint itself requires. Identified by Name rather than Model. |
| Microsoft 365 Copilot | Base URL | Delegated-only: the caller’s Entra bearer token is forwarded (identity pass-through), so answers are grounded on the signed-in user’s M365 content under their own access control. Uses a Copilot-specific wire format, not OpenAI. Sampling parameters are accepted for consistency but ignored by the Copilot Chat API. Defaults to https://graph.microsoft.com and never appends a path suffix. |
OpenAI-compatible presets
Groq, xAI, OpenRouter, Fireworks, Cerebras, DeepSeek, Moonshot, SambaNova, Hyperbolic, NVIDIA NIM, and Nebius are stored as the OpenAI provider type with a preset base URL. Picking one of these when you already have an account with that specific provider gets you their pricing, rate limits, and model lineup, with cost and telemetry attributed to that provider by name rather than lumped in as generic “OpenAI.” Each reuses the OpenAI schema, headers, and path handling under the hood, so no extra fields beyond the common OpenAI ones are needed. An explicit Base URL still overrides the preset if you need to point at a different endpoint for that provider.
Custom catalogue providers
A provider type that is not one of the built-ins above is resolved against the merged model catalogue (built-in catalogue entries plus any custom entries an operator has added). The catalogue entry’s wire-format setting (Anthropic, Gemini, Copilot, or anything else, which falls through to OpenAI) determines which concrete request/response shape the surface actually uses, while the operator-chosen provider identifier is preserved for pricing and telemetry. The catalogue’s default base URL is used when the surface itself did not set one.
Parameter override policy
The parameter override policy decides, per parameter, whether the surface’s configured value or the caller’s request value wins. This lets one surface be strict about some parameters and flexible about others: pinning the model and capping Max Tokens for cost control, say, while still letting callers tune their own temperature.
- Gateway Wins guarantees cost or behavior regardless of what a caller sends, useful for pinning Model or capping Max Tokens.
- Use Client Wins for a pure passthrough surface where callers should control their own sampling.
- Leave the default Inject If Missing when you just want sensible defaults for callers who don’t specify a value.
| Value | Meaning |
|---|---|
| Gateway Wins | The surface’s configured value always overrides the caller’s. |
| Client Wins | The caller’s value is authoritative; the surface never injects its own, even when the caller omits the parameter (true passthrough). |
| Inject If Missing (default) | The surface’s value is injected only when the caller did not provide one. |
Overridable parameters, each taking one of the values above: Temperature, Max Tokens, Reasoning Effort, Top P, Frequency Penalty, Presence Penalty, Model, API Key. A ninth setting, Reasoning Effort Value (minimal/low/medium/high, default medium), names the effort level injected when the Reasoning Effort policy applies. This is what lets a $high/$low surface variant send a distinct effort level per variant.
Token pricing resolution
Cost per call is computed from token pricing resolved in this order, most specific first:
- A per-surface input/output price override on cost tracking, used only when both fields are present and strictly positive. A missing, zero, or negative override falls through to the catalogue.
- The matched endpoint’s catalogue pricing: the most specific catalogue-level override, for example a discounted batch endpoint.
- The model’s catalogue pricing.
- A small built-in fallback table covering OpenAI GPT-4/GPT-4 Turbo/GPT-3.5 Turbo and Anthropic Claude Opus/Sonnet/Haiku (including Claude models served through Bedrock).
When no pricing is known at any level, the computed cost is 0.0 rather than an error.
Related
- Core configuration: Where surface type and the parameter override policy sit on the surface.
- Cost and usage limits: Cost tracking, usage limits, and usage alerts.
- Resilience and caching: Failover and load-balancing targets, which reuse the same provider shape.
- Guardrails: The Judge, Jury, and Decider LLM targets, which use this same provider/model/endpoint shape.
- Surfaces concept model: Client wire format independent of provider, and the full supported-provider list.
Glad to hear it! Please tell us how we can improve more.
Sorry to hear that. Please tell us how we can improve.
Thank you for sharing your feedback so we can improve your experience.