> ## Documentation Index
> Fetch the complete documentation index at: https://phaseo.app/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Phaseo Gateway base URL: https://api.phaseo.app/v1. Send `Authorization: Bearer $PHASEO_API_KEY` with every request.
> Model IDs look like `provider/model`, for example `openai/gpt-6-astra`. Do not guess IDs: list them with `GET /v1/models` or check https://phaseo.app/models. `phaseo/free` routes to an available free model and needs no wallet balance.
> For new text integrations prefer `POST /v1/responses`. `POST /v1/chat/completions` (OpenAI format) and `POST /v1/messages` (Anthropic format) suit existing code.
> The full API contract is https://phaseo.app/docs/openapi/v1/openapi.yaml. Official SDKs: `@phaseo/sdk` (TypeScript) and `phaseo` (Python), plus Go, C#, PHP, Ruby and Rust.
> Search these docs over MCP at https://phaseo.app/docs/mcp, and install Phaseo skills with `npx skills add https://phaseo.app/docs`. If a page is wrong or out of date, report it with the MCP `submit_feedback` tool.
> Manage Phaseo accounts with the `phaseo` CLI (`npm install -g @phaseo/cli`, then `phaseo login` or `phaseo login --device-code`), the `/v1` Management API or the MCP server at `https://mcp.phaseo.app/mcp?access=manage`. Do not send users to the dashboard for account tasks. A person is still needed for sign-up, approving a sign-in, paying the first top-up on the checkout link that `phaseo credits add` returns when no card is saved, confirming each saved-card top-up from the emailed link, and account security (password, MFA, passkeys). Create API keys with `phaseo keys create --name <name> --json`; never guess one. Credit top-ups need `phaseo login --scopes billing:write` first. Task map: https://phaseo.app/docs/v1/developers/manage-from-an-agent.md.

# Routing and Fallbacks

> How the Gateway selects providers and keeps requests reliable.

The Phaseo Gateway routes each request to a provider that can serve your chosen model. When a provider is slow, rate-limited, or returning errors, the Gateway can attempt fallbacks so your requests still complete.

## Choose a routing mode

Start with `balanced` unless one production requirement clearly matters more than the others.

| Mode | Use it when |
| - | - |
| `balanced` | You want a practical mix of price, latency, throughput, and availability. |
| `price` | Reducing provider cost matters more than minimizing response time. |
| `latency` | Fast response start is the primary requirement. |
| `throughput` | Sustained token generation speed matters most. |

Configure the workspace default in **Dashboard -> Settings -> Routing**, with `phaseo settings update --routing-mode <mode>`, or by sending `routing_mode` to `PATCH /v1/settings`. Use presets when one workflow needs a narrower provider or model policy than the workspace default.

### Model routing suffixes

Append a routing suffix when the model ID itself should make the optimization mode authoritative for that request:

| Suffix | Routing mode |
| - | - |
| `:nitro` | `throughput` |
| `:cheap` | `price` |
| `:fast` | `latency` |

For example, `openai/gpt-5-mini:nitro` uses throughput-first routing. A recognized routing suffix takes precedence over request-level `routing.mode` or `provider.sort`, preset and workspace routing modes. Other constraints such as provider allow lists, regional requirements, guardrails, and price ceilings still apply.

## How routing works at a high level

* You send a request with a model id.
* The Gateway evaluates provider health, latency, and capability coverage.
* A provider is selected and the request is executed.

Inspect request outcomes through your activity logs and response metadata when debugging routing behavior.

## Select a model with the auto-router

Auto Routing is currently an Alpha available to selected workspaces.

Use `phaseo/auto` when the model should change with the workload. Phaseo starts
from every eligible production text model, then applies workspace constraints
and builds a workload-specific shortlist:

1. Open **Dashboard -> Settings -> Routing -> Auto Routing**.
2. Choose whether to optimize for balanced performance, quality, cost, or
   latency.
3. Select an Economy, Standard, Premium, or unrestricted spend profile. These
   profiles apply fixed input and output price ceilings before scoring.
4. Optionally narrow eligible models with patterns such as `anthropic/*`,
   `openai/gpt-5.*`, or an exact model ID.
5. Choose whether Phaseo may try the remaining ranked models after a retryable
   failure.
6. Save the configuration.

To set the same options without the dashboard, run `phaseo routing auto update` with `--objective`, `--spend-profile`, `--allowed-patterns` and `--allow-fallbacks`, or call `PATCH /v1/routing/auto`. `phaseo routing auto get` shows the current configuration.

Applications then opt in without copying workspace routing policy into every
request:

```json theme={null}
{
  "model": "phaseo/auto",
  "input": "Review this TypeScript function for correctness."
}
```

The request cannot change the workspace objective, spend profile, or model
patterns. Fixed model requests continue to bypass the auto-router.

The objective changes the relative weight of model quality, provider
reliability, latency, and price:

| Objective | Use it when |
| - | - |
| `balanced` | You want a practical default across all four signals. |
| `quality` | Relevant benchmark performance matters most. |
| `cost` | Lower estimated input and output token price matters most. |
| `latency` | Lower recent provider latency matters most. |

Spend profiles are hard standard-tier price ceilings in USD per million text
tokens:

| Spend profile | Maximum input price | Maximum output price |
| - | -: | -: |
| Economy | \$0.10 | \$0.50 |
| Standard | \$0.30 | \$1.50 |
| Premium | \$1 | \$5 |
| Any price | No ceiling | No ceiling |

Custom limits set the input and output ceilings directly. Models without known
standard text pricing are not included in the managed candidate universe.

For each `phaseo/auto` request, Phaseo creates a normal child gateway request
to a pinned low-cost classifier model. The child request uses the same
workspace and billing identity, appears separately in request logs, and is
labelled with `purpose=auto_routing_classifier` and the parent request ID.
The classifier returns a structured workload mix, complexity score, and
confidence value; it never selects a model directly.

Hard request facts such as tools and structured output are included as trusted
metadata. If the classifier request fails, times out, or returns invalid data,
Phaseo falls back to the local deterministic code, reasoning, tool use,
structured output, translation, summarization, or general classifier rather
than failing the generation request.

Classifier complexity represents the minimum model capability likely to
produce a reliably acceptable answer. Phaseo applies a capability margin, then
combines benchmark fit with the workspace objective, price, latency, provider
health, and reliability. The classifier request and selected generation
request are metered independently through the normal gateway pipeline.

Before scoring, every allow-listed model must pass the normal endpoint,
workspace model, provider, privacy, guardrail, and circuit-breaker checks. The
router then combines relevant non-self-reported Phaseo catalogue benchmarks
with current Phaseo provider health, latency, and pricing. Missing benchmark or
operational data is neutral; it does not widen the allow-list.

The request's limits on where its data may go apply to Auto too. When a
request sets `provider.require_zero_data_retention: true`, or the workspace or the
key's guardrail allows only providers with zero data retention, the router
shortlists only models with a provider that offers it. Required regions are
checked against each candidate's providers in the same way. The classifier
request carries the same limits, along with the `only` and `ignore` providers;
if the classifier model can't meet them, Phaseo uses the local classifier
instead. When the request's residency requirements are what leaves Auto no
model (every model it considered failed them), the request fails with
`403 model_region_unavailable` and `unmet_requirements`, the same error a fixed
model gets; if models were also ruled out for other reasons, it fails with
`400 unsupported_model_or_endpoint`.

The response `model` field identifies the selected model. Request details show
the workload, objective, selected model, fallback order, candidate scores,
factor scores, benchmark IDs, exclusions, and algorithm version. Routing traces
contain no prompt or response content.

If model fallbacks are enabled in the workspace, Phaseo retries the remaining ranked models on
`429`, `500`, `502`, `503`, or `504`. Each fallback passes the complete policy
and provider-selection pipeline again. Client errors do not switch models.

A fixed model selected by an attached dynamic route takes precedence. In that
case, request details record the override and auto-router model fallbacks are
disabled for the request.

> [!WARNING]
>
> Spend profiles and model patterns are eligibility controls, not quality
> guarantees. Validate the resulting model mix against your own workloads.

## Explain a routing decision

Open **Dashboard -> Settings -> Usage -> Request logs**, select a request, then expand **Routing observability** under **Provider Responses**.

The request log shows:

* every ranked provider and its final score
* the provider Phaseo selected and any providers it attempted
* providers excluded before ranking, with the recorded reason
* providers that were deranked by their rollout or routing status
* the score inputs, weights, contributions, and multipliers used for each ranked provider

Score factors are separated from recorded context. A score factor changes the final score for the active routing mode. Recorded context helps explain the decision but does not necessarily affect the score.

For `balanced` text requests (Chat Completions, Responses and Messages), Phaseo ranks eligible providers by their expected cost per successful answer. That cost adds what a typical request for the model costs on the provider, using the model's measured prompt and output sizes and the provider's measured cache-hit rate, to the time spent waiting for the first token and the full response. It is then divided by the provider's recent success rate. Provider errors add a further penalty, and rate limits a smaller one. Providers close to the lowest cost share most of the traffic, and every provider keeps a small share so a recovery is noticed. To rank by price alone, send `provider.sort: "price"`. Other endpoints are scored on reliability, latency, tail latency, throughput, price, and token fit. The displayed calculation shows how each factor contributed to the final score instead of presenting every recorded metric as equally important.

Pick a scenario to see how the cost per successful answer splits into tokens, waiting and failures, and how that sets each provider's share of traffic. The providers are examples.

(

*

) ÷
\+

\=

:

### Reliability and provider uptime

The reliability sample is the value used by the score. It is calculated from provider outcomes; the success rate is shown as supporting context.

These outcomes reduce provider uptime:

* authentication failures (`401`)
* payment failures (`402`)
* model-not-found responses (`404`)
* server errors (`500` and above)
* errors after a response stream starts
* successful HTTP responses that finish with an error reason

These outcomes do not reduce provider uptime:

* bad requests (`400`)
* geographic restrictions (`403`)
* oversized payloads (`413`)
* rate limits (`429`)

Geographic restrictions and rate limits are tracked separately because they do not show that the provider itself is unavailable.

### Trace availability and privacy

Complete routing traces are available for requests made after routing observability was enabled. Older requests may show a partial trace or no routing details.

Routing traces are bounded and content-free. They contain the numbers and statuses needed to explain provider selection, but do not copy prompts, messages, or generated content into the trace.

## Choose an exact provider in the model id

Use `<provider-id>:<canonical-model-id>` when a request must use one specific provider-model pair:

```json theme={null}
{
  "model": "baseten:thinking-machines/inkling-small",
  "input": "Hello"
}
```

The qualifier disables cross-provider fallback for that request. Model suffixes remain part of the canonical model ID, including in identifiers such as `baseten:google/gemma-4-26b-a4b:free`.

Read [Provider-qualified model IDs](https://phaseo.app/docs/v1/guides/provider-qualified-models.md) for complete syntax, free-route validation, routing precedence, aliases, error codes, and request examples.

## Control routing and fallbacks

The current public fallback and routing controls are intentionally explicit:

### Presets constrain the fallback pool

In **Dashboard -> Settings -> Presets**, you can define:

* allowed models
* provider allow lists
* provider ignore lists
* default prompt and parameter behavior

Those constraints are applied before provider selection, so a preset can intentionally narrow which providers are eligible for retries and failover.

### Routing mode changes provider ranking

In **Dashboard -> Settings -> Routing**, workspaces can tune how the Gateway ranks compatible providers:

* `balanced`
* `price`
* `latency`
* `throughput`

From the CLI, run `phaseo settings update --routing-mode <mode>`.

The same page also exposes beta and alpha channel toggles so preview traffic can be introduced intentionally instead of appearing as an untracked routing side effect.

### BYOK fallback is explicit

In **Dashboard -> Settings -> BYOK**, teams can choose whether a failed BYOK (Bring your own key: Phaseo routes the request with your own provider API key, and your provider bills the model usage directly.) request is allowed to fall back to Phaseo credits. This is the current public control for the common "my own key failed, should the request still complete?" decision. Agents can set it with `byok_fallback_enabled` on `PATCH /v1/settings`, for example `phaseo settings update --body-json '{"byok_fallback_enabled":true}'`, and add provider keys with `phaseo byok create --provider <id> --name <name> --key-stdin`.

### Dynamic routes attach policy to API keys

In **Dashboard -> Settings -> Routing**, create a dynamic route when different
API keys or request classes need different provider behavior. From the CLI, use
`phaseo routing dynamic-routes create`, `set-keys` and `deploy`, or the
`/v1/routing/dynamic-routes` API. A route can:

* branch on nested request-body fields, request headers, custom metadata,
  endpoint, model, or session ID
* split traffic by percentage for A/B tests and gradual rollouts
* enforce daily, weekly, or monthly request and cost limits using the
  authenticated key's usage buckets
* call a different model and choose its routing mode, provider preference, and
  fallback policy
* enable cache-aware and session-aware provider affinity
* attach to one or more inference API keys

Conditions expose true and false outputs. Rate and budget nodes expose within
and exceeded outputs. Percentage selection is deterministic for a session or
prompt-cache key, so a rollout does not randomly move the same cached
conversation between branches.

Saving creates an immutable draft version. Deploying a selected version copies
that snapshot to the Gateway and invalidates policy caches for attached keys.
Earlier versions remain available for rollback. Operational provider-health
recommendations appear under **Insights**, separately from the flow editor.

Custom metadata is available on the OpenAI-compatible text inference surfaces:

```json theme={null}
{
  "model": "openai/gpt-5-mini",
  "metadata": {
	    "customer_plan": "pro",
	    "workspace": "acme"
  }
}
```

## Fallback behavior

If a provider returns errors or rate limits, the Gateway can retry or route to another provider that supports the same model. You should still handle 429 and 5xx responses with exponential backoff.

```mermaid theme={null}
flowchart TD
  req["Request"] --> pol["Apply presets, guardrails and provider policy"]
  pol --> rnk["Rank eligible providers by routing mode"]
  rnk --> prov["Call the top-ranked provider"]
  prov -->|"Success"| resp["Return the response"]
  prov -->|"Provider error or rate limit"| nextProv{{"Another eligible provider?"}}
  nextProv -->|"Yes"| prov
  nextProv -->|"No"| fallbackCheck{{"Retryable status and a fallback model left?"}}
  fallbackCheck -->|"Yes: rerun with the next model"| pol
  fallbackCheck -->|"No"| err["Return the error"]
```

Model nodes in a dynamic route can also define an ordered list of fallback
models. Phaseo first exhausts the eligible provider attempts for the selected
model. If the resulting response is retryable (`429`, `500`, `502`, `503`, or
`504`), the Gateway reruns the complete policy and provider-selection pipeline
for each fallback model in order. Client errors are returned immediately and do
not switch models.

Each fallback is independently checked against workspace model restrictions,
guardrails, provider policy, pricing, and capability support. A route can store
up to eight fallback models.

### Request-level model fallbacks

`model` is required and is always the primary model. Add the optional
`models` array to list fallback models for one request. Phaseo tries `model`
first, then each `models` entry in order:

```json theme={null}
{
  "model": "openai/gpt-5-mini",
  "models": ["anthropic/claude-sonnet-5-5", "google/gemini-2.5-flash"],
  "input": "Summarize this incident report."
}
```

A request that sends `models` without `model` is rejected with `400`.
Request-level fallbacks use the same machinery as dynamic-route fallbacks:

* **Triggers.** The next model is tried only when the previous attempt, after
  its own provider retries, ends with `429`, `500`, `502`, `503`, or `504`.
  Client errors such as `400`, `401`, `402`, `403`, `404`, `408`, `413`, and
  `422` are returned without switching models. A `429` from a Phaseo budget or
  free-model quota reached while executing the request also moves to the next
  model, because a different model may fit within the remaining limit.
  When no provider serves the primary model at all, for example because it is
  [retired](#retired-models), Phaseo also moves to the next model.
  Authentication, API key rate limits, and other checks that fail before the
  primary model runs return immediately.
* **Limits.** `models` accepts 1 to 25 entries. Duplicates and entries equal to
  `model` are ignored. Dynamic routes keep their own limit of eight fallbacks.
* **Validation.** Each entry can be a model ID, a provider-qualified model ID,
  or a preset such as `@support`. Each attempt is validated as its own request:
  endpoint and modality support, requested parameters, workspace model
  restrictions, guardrails, presets, pricing, and BYOK credentials. A fallback
  that cannot serve the request is skipped and recorded in request logs; it
  never replaces the failure you would otherwise receive. A fallback that
  resolves to an attempt already made, with the same model and eligible
  providers, is skipped.
* **Logging and billing.** Every attempt is a separate request in your logs,
  with its own usage and billing record, so each attempt is billed exactly as
  it would be without fallbacks.
* **Which model served the request.** The response `model` field shows the
  model that served the request. Its routing trace lists the primary request
  ID, the fallback order, and each earlier attempt with its status.
* **Streaming.** Fallbacks happen only before the first byte is sent. An error
  after a stream has started is returned in that stream.
* **Video and music.** These requests fall back only while submitting the job.
  Once a provider accepts a job, a later failure does not start a new job with
  another model.

`models` interacts with other routing controls as follows:

* **Dynamic routes.** Fallbacks configured on an attached dynamic route take
  precedence over `models`. A fixed model selected by a dynamic route still
  applies to each attempt.
* **Presets.** A preset can be the primary `model` or a `models` entry. Each
  attempt applies its own preset constraints, so a preset can narrow the
  providers a fallback may use.
* **Auto Routing.** `phaseo/auto` already manages ranked fallbacks, so it
  cannot be combined with `models` and cannot appear inside `models`.

| Endpoint | `models` support |
| - | - |
| Chat Completions, Responses, Messages | Supported, including streaming before the first byte |
| Embeddings, Moderations, Rerank | Supported |
| Image generations and edits | Supported, including multipart uploads |
| Speech, transcriptions, and translations | Supported, including multipart uploads |
| OCR, Parse, Decisions | Supported |
| Video and music generation | Supported at job submission |
| Batches | Rejected with `400`; each batch row targets one model |
| Realtime and live sessions | Rejected with `400`; a session binds one model |
| Council runs | Rejected with `400`; models come from the run configuration |

Multipart endpoints accept repeated `models` fields, `models[]` fields, or one
JSON array string.

Read more in:

* [Rate limits](https://phaseo.app/docs/v1/api-reference/limits.md)
* [Error handling](https://phaseo.app/docs/v1/api-reference/errors.md)

## Retired models

A retired model is still served while any provider serves it. Once none does,
requests fail with `410 model_retired`, unless the workspace has turned on
retired model routing to send them to a replacement. Your own `models` list is
always tried before a replacement. See
[Model Deprecation and Retirement](https://phaseo.app/docs/v1/guides/model-retirement.md).

## Cache-aware and session-aware affinity

Cache-aware routing is enabled by default on text generation endpoints. Requests
that send `prompt_cache_key` or `session_id`, or opt into caching with
`cache_control` (top-level, `provider_options.anthropic.cache_control`, or a
`cache_control` or `prompt_cache_breakpoint` mark on a message block or tool),
are pinned to the provider and route that served their first successful request.
Other requests are pinned for 15 minutes once a provider reports a prompt-cache
read or write for the same opening context, as long as that provider's cached
input is cheaper than its standard input.

Pins are kept per workspace and last from 15 minutes to 24 hours, following the
provider's cache retention. A pin moves only when its provider's circuit breaker
opens, its recent error rate is high, Phaseo's managed quota on it runs out, or
the active route, preset, guardrail, or request policy no longer allows it.

When `session_id` is present, the cache affinity is keyed to the session instead
of only the opening context, and is retained for up to 24 hours of session
activity. See [Provider affinity](https://phaseo.app/docs/v1/guides/prompt-caching.md#provider-affinity) for pin
lifetimes and the affinity hints Phaseo sends to each provider.

Opt out for one request without changing workspace or dynamic-route defaults:

```json theme={null}
{
  "model": "openai/gpt-5",
  "session_id": "support-session-42",
  "provider": {
    "cache_aware_routing": false
  }
}
```

To retain context cache affinity but ignore the session identifier for one
request, use:

```json theme={null}
{
  "model": "openai/gpt-5",
  "session_id": "support-session-42",
  "routing": {
    "session_affinity": false
  }
}
```

## BYOK considerations

Use BYOK when you want Phaseo routing and observability while the selected provider bills model usage to your own account.

* The first 250,000 completed BYOK requests in each UTC calendar month have no Phaseo service fee.
* After that allowance, Phaseo charges 2.5% of provider-equivalent cost.
* Keep at least \$1 of Phaseo credit available. This protects managed fallback and post-allowance fee collection; provider charges still go directly to your provider account.
* Your provider's quotas, data policy, model access, and account restrictions still apply.

Provider credentials are encrypted with AES-256-GCM and bound to their workspace and provider before storage. Scope each key to the models and Phaseo API keys that need it, and disable managed fallback if a request must never use Phaseo credits.

## What to log

For production workloads, log request ids, response status codes, and model ids so you can correlate failures and confirm routing behavior when debugging.

## Related guides

* [Presets](https://phaseo.app/docs/v1/guides/presets.md)
* [Model Deprecation and Retirement](https://phaseo.app/docs/v1/guides/model-retirement.md)
* [Feature Parity Matrix](https://phaseo.app/docs/v1/migration-guides/feature-parity-matrix.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.