One Gateway for APIs,
LLMs & AI Agents

Tygress is an open-source AI gateway and API gateway built on Rust and Cloudflare's Pingora framework. One gateway routes, guards, and observes traditional HTTP and gRPC APIs, LLM provider traffic, MCP tool servers, and A2A AI agents — with sub-millisecond latency and memory safety.

An enterprise API gateway is the single entry point that routes, secures, and observes all API traffic across an organization — adding authentication (SSO, OIDC, mTLS), rate limiting, high availability, audit logging, and governance on top of basic request routing. Tygress is an open-source enterprise API gateway that extends those same controls to AI traffic, governing LLM, MCP, and AI agent calls alongside REST and gRPC APIs in one Rust data plane.

An AI gateway sits between your applications and LLM providers, adding routing, failover, authentication, cost controls, guardrails, caching, and observability to AI traffic. Tygress is an enterprise AI gateway that combines those AI features with a full API gateway: one self-hosted Rust binary governs LLM, MCP, and AI agent traffic alongside your REST and gRPC APIs.

An LLM gateway (sometimes called an LLM proxy) routes requests to large language model providers through one unified API, handling provider failover, key management, token-aware rate limiting, cost budgets, and response caching. Tygress is a self-hosted LLM gateway written in Rust — the same capabilities LLM-only proxies offer, at reverse-proxy speed, plus the API gateway features they lack.

No — that's the point of Tygress. Most stacks run an API gateway (Kong, Apigee) for REST and gRPC plus a separate AI gateway or LLM proxy (LiteLLM, Portkey) for model traffic, then duplicate auth, rate limiting, and audit across both. Tygress is one data plane that does both jobs, so a single set of policies governs every API call, LLM request, MCP tool call, and agent interaction.

In preliminary benchmarks on identical hardware (4 vCPU, 8 GB RAM), Tygress achieves 98K requests/sec with 0.12ms p99 latency and 18 MB memory usage, compared to NGINX at 65K rps / 0.45ms / 38 MB, Envoy at 72K rps / 0.31ms / 52 MB, and Kong at 34K rps / 1.2ms / 128 MB. Beyond speed, AI gateway features are native in Tygress rather than bolted on as plugins.

LiteLLM and Portkey focus on AI traffic only; Tygress is a full production gateway where AI features are native. You get Rust proxy performance plus auth (JWT, OIDC, HMAC, API keys), distributed rate limiting, circuit breaking, and observability on the same routes that serve your LLM, MCP, and A2A traffic — one binary instead of an API gateway plus a separate Python proxy.

Yes. The open source edition includes the full proxy engine, the 45+ plugin suite, the AI gateway, load balancing, and the plugin SDK — free forever. Cloud, self-managed cloud, and enterprise editions are also available for teams that need managed infrastructure and premium support.

Yes. Tygress runs as a fully air-gapped API gateway when needed: self-host on Docker, Docker Compose, or Kubernetes, entirely inside your own VPC or on-premises. Prompts, API keys, and data never leave your network, so data residency and sovereignty requirements are met by design. A split control plane and data plane sync configuration over NATS JetStream with last-known-good caching for high availability.

Yes. Tygress ships a full enterprise authentication stack: OIDC single sign-on, LDAP and Active Directory, mutual TLS client certificates, OAuth2 introspection with required scopes, JWE decryption, HMAC, and API keys. Auth plugins can be chained per route, and any verified credential maps to a consumer for multi-tenant policy enforcement.

Tygress supports OpenAI, Anthropic, Google Gemini, Azure OpenAI, AWS Bedrock, Mistral, and Groq natively, plus any OpenAI-compatible endpoint. All are reachable through one unified, OpenAI-compatible API for both chat completions and embeddings, with virtual keys, quota-aware key pooling, and automatic cross-provider failover.

Tygress centralizes AI governance at the gateway: PII and secret redaction (AI DLP), prompt-injection and jailbreak guardrails, per-consumer model and tool allowlists, human-in-the-loop approvals for sensitive calls, immutable audit logging to Postgres, and role-based admin access — giving security and compliance teams a single, defensible control point for all AI and API traffic.

Yes. Tygress is an AI-native gateway: a unified multi-provider API for OpenAI, Anthropic, AWS Bedrock, and others with automatic provider failover; prompt-injection and jailbreak guardrails; PII and secret redaction (AI DLP); semantic caching for LLM responses; token-aware rate limiting; and per-route USD cost budgets.

Yes. Tygress natively proxies Model Context Protocol (MCP) servers — JSON-RPC 2.0 over HTTP and SSE — with the full plugin chain in front: authentication, rate limiting, and audit logging, plus a dedicated mcp-guard plugin that enforces per-consumer tool allowlists on tools/call invocations.

Yes. Tygress includes an A2A (Agent2Agent) gateway that federates JSON-RPC agents with skill-affinity, round-robin, or fallback routing, and a unified human-in-the-loop approval system that can gate any MCP or A2A call behind an admin decision with a shared queue, audit log, and webhooks.

Four layers: semantic caching replays answers to similar prompts without contacting the provider; token-aware rate limiting meters usage by tokens instead of requests; per-route USD cost budgets cap spend in dollars using a per-model price table; and automatic failover routes around degraded providers.

Yes. Tygress ships with post-quantum cryptography out of the box. Every TLS connection can negotiate a NIST-standardized hybrid key exchange (X25519 + ML-KEM-768, FIPS 203), defending your traffic against harvest-now, decrypt-later attacks where adversaries store encrypted data today to break it once quantum computers mature. You can enforce post-quantum handshakes per-route or fleet-wide, and generate an on-demand crypto-posture report to prove your quantum-readiness for audits.

Pingora is Cloudflare's open-source Rust framework for building fast, reliable, and programmable network services. It powers Cloudflare's global edge network handling trillions of requests daily. Tygress uses Pingora for its async-first architecture, intelligent connection pooling, and battle-tested reliability at scale.