Self-hosted enterprise AI gateway for LLMs

• Three LLM APIs on nine provider types →

  Clients call Chat Completions, Responses or Messages on any configured provider: native where the provider offers the API, bridged through chat completions elsewhere.

• Keys, teams and budgets →

  Virtual keys keep provider credentials server-side. Teams share a USD budget alongside each member's own, and limits can key on identity-provider claims.

• Guardrails on streamed replies →

  DLP and the prompt guard inspect prompts and, with reply scanning on, every chunk of a streamed reply. Each attempt is signed after redaction.

• Routing and failover →

  Fallback, round-robin, weighted and semantic strategies, key pools that rest a key on 429, and a 502 that says why each target failed.

• Agent actions held at the gateway →

  MCP tool allowlists, REST and OpenAPI as MCP tools, A2A routing, and human approvals that deny by default after 45 seconds.

• GenAI telemetry →

  OpenTelemetry spans with GenAI conventions, Langfuse export, Prometheus metrics and a usage API for tokens and spend.

teams:  - name: search    budget:      limit: 2000000000        # $2,000 a month, shared      period: calendar_monthconsumers:  - name: alice                # her own $500 a month    team: search    plugins:      - name: ai-cost-budget        config:          cumulative: { limit: 500000000, period: calendar_month }          alerts: { webhook_url: "${BUDGET_WEBHOOK_URL}" }

Virtual keys

A consumer's gateway key is bound to provider keys the operator holds. Clients never see a provider credential, and upstream capacity is attributed to each consumer.

Keys issued in one call

One admin API call mints a key, shown once, with a USD budget that resets daily or monthly or runs for a lifetime, request and token limits per minute, allowed models, an expiry and a team.

Teams with shared budgets

A team carries a shared USD budget enforced alongside each member's own, so a request goes through only when both allow. Usage and spend roll up per team.

Per-user limits from your IdP

Verified JWT claims or forward-auth headers can key rate limits, token limits and budgets, so each person your identity provider signs in gets their own allowance without a gateway consumer per user.

Windows, caps and alerts

Token limits and USD budgets run over fixed, sliding, rolling-anchor or token-bucket windows, and their counters can be shared across replicas with Redis. Daily, monthly and lifetime caps per consumer and team persist in PostgreSQL when a database is configured. Slack-compatible webhooks fire at thresholds you set, 50, 80 and 95% by default.

80-model pricing catalog

A bundled catalog prices 80 models per token, including cached-input rates where the vendor publishes them, so budgets work without hand-written pricing. Your own prices always take precedence.

Semantic cache

Answers a similar prompt from cache by embedding similarity above a cosine threshold, using an OpenAI-compatible embeddings endpoint or an opt-in local ONNX embedder, stored in memory, Redis or pgvector. Every lookup makes an embedding call, and streaming, tool-calling, image or file, and embeddings requests bypass the cache.

Semantic routing

Classifies each prompt into a complexity tier that can map to a cheaper model. It runs in shadow mode by default, serving the requested model and recording the choice it would have made and the projected saving.

Selection strategies

Pick a provider by model, or give a route a fallback, round-robin, weighted or semantic strategy. Targets that cannot serve the requested API are excluded before selection.

Provider key pools

A provider can hold several API keys. A key that gets a 429 is rested. Quota-aware rotation reads rate-limit headers from OpenAI, Azure OpenAI, Anthropic and Groq, and from OpenAI-compatible endpoints when enabled; other providers rotate on status codes.

Failover that explains itself

Fail over on the status codes you list or on connect failures, and rest a failed provider for a cooldown. When every target fails, the 502 lists each one and why: its status, a connect failure or a refused body.

Long completions

Each provider gets its own response-header timeout, 600 seconds by default to match the official OpenAI and Anthropic SDKs, and transport timeouts return 504 rather than 502.

Autopilot (opt-in)

A control loop tunes weights across the targets you declare toward lower cost, lower latency or more resilience, within limits you set. By default each change waits for human approval; in auto mode it applies changes and rolls back on regression. It runs in the all-in-one role, not in a split control-plane and data-plane deployment.

MCP tools in LLM requests

A route can add an MCP toolset's tools to outgoing LLM requests. If a name collides, the client's own tool wins, and the client still executes the tool calls.

GenAI OpenTelemetry spans

With the OpenTelemetry plugin attached, each AI request gets a span with gen_ai.* attributes: provider, requested and resolved model, token counts including cached and reasoning tokens, finish reasons, time to first chunk and cost. Prompt and completion capture is opt-in, and redacted when the DLP plugin is attached.

Langfuse export

Each AI request produces one record with the model, each provider it tried, timing including time to first token, and, for metered calls, token counts and cost. The Langfuse plugin exports it as a trace and generation.

Prometheus metrics

Requests, tokens, cost, time to first token, failovers, key cooldowns, provider quota, semantic-cache lookups, and DLP and prompt-guard counters.

Usage and spend API

Admin endpoints report requests, tokens and realized cost per consumer, team, provider and model. With PostgreSQL, the data survives restarts and aggregates across data planes.

LLM-as-judge

Grade a deterministic sample of replies against a rubric with a judge model, after the response has been sent. It only observes and never blocks.

Topology

One binary, or separate control-plane and data-plane roles that sync configuration over NATS. Redis shares counters, approval queues and provider quota state across replicas.

Packaging

The source includes a Dockerfile for a distroless, non-root image and a Docker Compose reference stack with NATS, a control plane and two data planes. No image is published yet, and there is no Helm chart.

Optional PostgreSQL

PostgreSQL keeps durable budgets and usage history, and append-only records of approval decisions, autopilot changes and, in control-plane mode, config revisions. It is not the config store, and the gateway runs without it.

Operations

Linux and macOS. Zero-downtime binary upgrades hand the listening sockets to the new process. On SIGTERM, /healthz returns 503 so load balancers deregister the node before in-flight work drains.

Where inference runs

Private VPC with external providers

Host the gateway in your cloud account and allow only the provider endpoints you approve. Requests routed to an external model still leave your network; review provider retention, processing region and logging settings.

On-premises with local inference

Route to an OpenAI-compatible model endpoint on your own infrastructure, and keep inference, gateway logs, caches and telemetry inside the same approved boundary.

Fully air-gapped

Keep model serving, identity, dependencies and observability reachable locally; the model pricing catalog can load from a file. External model APIs cannot be used from an environment with no outbound connectivity.