AI Gateway

The AI Gateway is an OpenAI-compatible HTTP layer that sits in front of every LLM provider you’ve configured. Your applications, workflows, and agents call one endpoint, and the gateway routes each request to the right provider — falling back to alternates when one fails, enforcing rate limits, and emitting usage events for billing and analytics.

How it works

The gateway is embedded in the main proxifai binary at internal/llmgateway/embedded. It’s mounted under /api/v1/llm on the same HTTP server that serves the rest of the API — there’s no separate gateway service to deploy. Every org configures providers in the model_provider table; the gateway resolves them at request time, caches the resolution for 60 seconds, and dispatches.

┌──────────────┐   POST /api/v1/llm/v1/chat/completions   ┌─────────────────┐
│ your client  │ ───────────────────────────────────────► │   AI Gateway    │
│ (curl, SDK,  │                                          │  (chi sub-router│
│  workflow,   │ ◄─────────────────────────────────────── │   in proxifai)  │
│  agent)      │   200 OK · streamed tokens               │                 │
└──────────────┘                                          └────────┬────────┘
                                                                   │
                                                                   ▼
                                              ┌──────────┬──────────┬──────────┐
                                              │  OpenAI  │ Anthropic│  Gemini  │
                                              │   API    │   API    │   API    │
                                              └──────────┴──────────┴──────────┘

When a request lands:

1. Auth — Authorization: Bearer <key> (or Anthropic-style x-api-key) is verified against the configured static keys, or against an HMAC-signed pfai_<execID>_<sig> token issued by the workflow runtime.
2. Org resolution — the org is read from the X-Org-Id header (set upstream by the main API) or defaults to "default".
3. Provider lookup — providers + model-to-provider mappings are loaded from the database (cached 60s).
4. BYO override — if the user has personal keys in user_provider_key, those get prepended to the routing chain so they’re tried before org-level keys.
5. Routing — for the requested model, the circuit breaker filters out providers currently in the open state; the rest are tried in order.
6. Retry — failed attempts retry up to 2 times (500 ms initial wait, 3 s max) before falling back to the next provider.
7. Usage event — on success, an event is published to the LLM_USAGE NATS stream for downstream tracking.

Supported providers

Four provider types are supported by the gateway runtime (internal/llmgateway/dbprovider/resolver.go):

Default model catalog

Out of the box that’s 19 models across three first-party providers. Adding an openai-compatible row lets you plug in any other model your endpoint exposes — Llama, Mistral, DeepSeek, Qwen, local Ollama instances, OpenRouter, etc. — under whatever model IDs you choose.

Configuring a provider

Each row in model_provider represents one provider. Add them through the Settings → Model Providers UI or via the management API; the API key is encrypted at rest using the workspace’s encryption key.

After a write, call the management API’s invalidate endpoint or wait up to 60 seconds for the resolver cache to expire.

Two API surfaces

The gateway accepts requests in either OpenAI or Anthropic format, regardless of which provider ultimately handles them. Pick the surface that matches the SDK you already have.

curl http://localhost:3000/api/v1/llm/v1/chat/completions \
  -H "Authorization: Bearer $PROXIFAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4.6",
    "messages": [
      {"role": "user", "content": "Explain the CAP theorem in two sentences."}
    ],
    "stream": true
  }'

curl http://localhost:3000/api/v1/llm/v1/messages \
  -H "x-api-key: $PROXIFAI_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4.6",
    "max_tokens": 256,
    "messages": [
      {"role": "user", "content": "Explain the CAP theorem in two sentences."}
    ],
    "stream": true
  }'

Both surfaces accept any model from any configured provider — you can ask for gpt-4o through the Anthropic-format endpoint and the gateway will translate. Streaming uses Server-Sent Events on both surfaces.

Authentication

Workflow keys carry the executing workflow’s identity, so usage events are attributed to the right run, and credit checks (Enterprise) can deduct against the right org.

Resilience

The defaults wired up in internal/llmgateway/embedded/embedded.go:

If every provider for a model trips its circuit, the gateway returns 503 Service Unavailable with {"error":{"type":"gateway_error"}}. This is the only place a request fails outright — partial failures within the fallback chain are transparent to the caller.

Bring your own key (BYOK)

BYOK is two-layered:

• Workspace-level — keys configured in model_provider apply to everyone in the org.
• User-level — a row in user_provider_key for (user_id, org_id, provider_type) overrides the workspace key for that user, and is prepended to the routing chain so it’s tried first.

This lets individual contributors use their own enterprise/Anthropic/OpenAI accounts (volume discounts, separate quotas, evaluation credits) while the team’s shared key remains the fallback.

Usage tracking

Successful requests publish a usage.Event to the LLM_USAGE JetStream stream:

{
  "executionId": "exec_…",
  "provider": "anthropic_…",
  "model": "claude-sonnet-4.6",
  "promptTokens": 412,
  "completionTokens": 128,
  "totalTokens": 540,
  "streaming": true,
  "cacheHit": false,
  "estimatedCostUsd": 0,
  "timestamp": "2026-05-04T22:14:00Z"
}

A consumer in internal/llmusage writes these to the database. The management API exposes them at /api/v1/gateway/usage*:

The Settings → AI Gateway page in the web UI is built on these endpoints.

Pricing and cost reporting

In the OSS build, pricing.CalculateCost returns 0 because no cost calculator is registered (internal/llmgateway/pricing/pricing.go). Token counts are accurate; the estimatedCostUsd field will be zero. The Enterprise build registers a calculator with current per-model rates so the same field reflects real USD.

To add custom pricing in OSS, register a calculator at startup:

import "github.com/proxifai/proxifai-oss/internal/llmgateway/pricing"

pricing.CostCalculator = func(model string, prompt, completion int) float64 {
    // your rate table
}

Configuration reference

Provider keys, model lists, and rate limits live in the database, not env vars — so configuration changes don’t require a restart.

Endpoint reference

See also