# Tygress — Enterprise API Gateway & AI Gateway in Rust

> Tygress is an open-source enterprise API gateway and AI gateway built on Rust and Cloudflare's Pingora framework. Self-host one gateway to route, govern, and observe traditional HTTP/gRPC APIs, LLM provider traffic, MCP (Model Context Protocol) tool servers, and A2A (Agent2Agent) AI agents — replacing an API gateway (Kong, Apigee, NGINX, Envoy), an LLM proxy (LiteLLM, Portkey), and custom agent middleware with a single fast data plane that your platform, security, and FinOps teams control.

## What is Tygress?

Tygress is a high-performance gateway for the agentic era, designed for enterprises and platform teams running both classic microservices and AI workloads in production. It is built entirely in Rust on Cloudflare's open-source Pingora proxy framework — the same technology that powers Cloudflare's global edge network, handling trillions of requests per day.

Unlike gateways written in C (NGINX), C++ (Envoy), Go/Lua (Kong), Java (Apigee), or Python (LiteLLM), Tygress benefits from Rust's zero-cost abstractions, compile-time memory safety, and absence of garbage collector pauses — deterministic, predictable performance at any scale.

## Why enterprises choose Tygress

- **Self-hosted & sovereign**: Run entirely inside your own VPC or on-premises (Docker, Compose, Kubernetes), including air-gapped environments. Prompts, keys, and data never leave your network — data residency by design.
- **One control point for AI governance**: PII/secret redaction, prompt-injection guardrails, model and tool allowlists, human-in-the-loop approvals, and immutable audit logging — a single, defensible record across all AI and API traffic.
- **Enterprise identity**: OIDC SSO, LDAP/Active Directory, mutual TLS, OAuth2 with required scopes, JWE, HMAC, and API keys — chained per route and mapped to consumers.
- **High availability**: Split control plane and data plane sync over NATS JetStream with last-known-good caching; push config to fleets with zero-downtime reloads and binary upgrades.
- **No vendor lock-in**: Open-source core, OpenAI-compatible API, runs on any cloud or bare metal.

## Key Capabilities (API Gateway)

- **Sub-millisecond routing**: Trie-based route matching on host, path, method, and headers; first-class gRPC service/method matching.
- **Memory safety**: Rust's ownership model eliminates buffer overflows, use-after-free, and data races at compile time.
- **Quantum-safe TLS**: Post-quantum cryptography out of the box — every TLS connection can negotiate a NIST-standardized hybrid key exchange (X25519 + ML-KEM-768, FIPS 203), enforceable per-route or fleet-wide, with on-demand crypto-posture reports. Mitigates harvest-now, decrypt-later attacks.
- **Zero-downtime operations**: Graceful shutdown with connection draining and zero-downtime binary upgrades.
- **Full protocol support**: HTTP/1.1, HTTP/2, gRPC, gRPC-Web, WebSocket, and gRPC-JSON transcoding (call native gRPC backends with JSON / REST bindings).
- **45+ built-in plugins**: JWT (with remote JWKS), OIDC, LDAP, mTLS, HMAC, JWE, OAuth2 introspection, basic/API-key/multi-auth, OPA and forward-auth external authorization, ACL, IP restriction, bot detection, CSRF, security headers, body threat detection and validation, compression, redirects, request/response rewriting, fault injection, and more.
- **Traffic management & resilience**: Distributed rate limiting (Redis), concurrency limits, per-upstream bulkheads, request size limits, circuit breaking, status-based retry with retry budgets, outlier detection, traffic splitting (canary/blue-green), and sticky sessions.
- **Load balancing**: Round-robin, weighted, and least-request balancing with passive and active health checks (HTTP/TCP and gRPC health probes), plus DNS-based discovery.
- **Multi-tenancy**: First-class consumers, groups, and tenants with per-consumer plugin overrides and a usage/analytics admin API.
- **Observability**: Prometheus metrics, OpenTelemetry tracing, and structured HTTP access logging out of the box.
- **Extensible**: Native Rust plugins, sandboxed WebAssembly (proxy-wasm — Rust, Go/TinyGo, AssemblyScript), or inline Rhai/JavaScript serverless scripts with no rebuild.
- **Architecture**: Declarative YAML/JSON config; split control plane / data plane with NATS JetStream snapshots; optional Postgres for control-plane persistence and audit logging; etcd-backed config with multi-instance live sync.

## AI Gateway Capabilities

Tygress is AI-native and includes first-class support for proxying, governing, and optimizing LLM and agent traffic:

- **Unified multi-provider API**: One OpenAI-compatible endpoint for OpenAI, Anthropic, Google Gemini, Azure OpenAI, AWS Bedrock (OpenAI-compatible and native Converse API with SigV4), Mistral, and Groq — plus any OpenAI-compatible endpoint. Chat completions and embeddings.
- **Virtual keys**: Map per-team consumer keys to upstream provider keys server-side; rotate, revoke, and meter centrally so raw provider keys never leave the gateway.
- **Quota-aware failover & key pooling**: Pool multiple keys per provider with round-robin rotation and 429 cooldowns; fail over across providers on 5xx/429/connect errors using upstream rate-limit headers.
- **Autopilot cost routing**: A self-tuning control loop continuously shifts traffic toward the cheapest model meeting your quality bar by adjusting routing weights from live cost and latency.
- **Prompt injection guard**: Block prompt-injection and jailbreak attempts with built-in heuristics plus an optional external classifier; flag completions that leak the system prompt.
- **AI DLP**: Scan prompts and responses for PII and secrets; redact in place, block, or annotate — matched text never reaches the provider.
- **Semantic caching**: Cache LLM responses by prompt similarity (embeddings); similar prompts replay prior answers without contacting the provider.
- **Token-aware rate limiting & USD cost budgets**: Meter by tokens and cap spend in dollars with per-route, per-model budgets, including cached-input, cache-creation, and reasoning-token accounting.
- **RAG injection**: Ingest documents and inject retrieved context into prompts as a managed plugin.
- **MCP gateway**: Proxy Model Context Protocol servers (JSON-RPC 2.0 over HTTP+SSE) with the full plugin chain in front; per-consumer tool allowlists; generate MCP toolsets from OpenAPI 3.x specs; aggregate multiple MCP servers behind one virtual endpoint; serve REST upstreams as MCP tools.
- **A2A (Agent2Agent) gateway**: Federate JSON-RPC agents with skill-affinity, round-robin, or fallback selection and a merged agent card at /.well-known/agent.json.
- **Human-in-the-loop approvals**: Gate sensitive MCP and A2A calls behind an admin decision with a shared queue, audit log, and webhooks.
- **Self-healing incident response**: Auto-detect floods, abuse, and provider outages and auto-remediate (shed load, trip breakers, fail over).
- **AI observability**: AI-specific Prometheus series — token usage, provider latency, TTFT, cost, quota state, semantic-cache hits, and failover transitions.

## LLM Provider Support

OpenAI, Anthropic, Google Gemini, Azure OpenAI, AWS Bedrock, Mistral, Groq, and any OpenAI-compatible endpoint. Bedrock supports both the OpenAI-compatible endpoint and the native Converse API (Claude, Llama, Mistral, Cohere, Nova) with AWS SigV4 auth. Embeddings supported across OpenAI, Azure, Mistral, Google, and Bedrock.

## Deployment

Self-hosted via Docker (distroless image), Docker Compose (with NATS and optional Postgres audit tier), or Kubernetes (split control plane / data plane Deployments with health probes and graceful termination). Zero-downtime binary upgrades via socket handoff. Optional dependencies: PostgreSQL (control-plane persistence + audit), NATS JetStream (control/data-plane sync), Redis (distributed rate limiting).

## How Tygress Compares to Alternatives

| Capability | Tygress | Kong | Apigee | LiteLLM | Portkey |
|------------|---------|------|--------|---------|---------|
| Language | Rust | Lua/Go | Java | Python | Node/TS |
| API gateway | Full | Full | Full | No | No |
| AI / LLM gateway | Native | Plugin | Limited | Native | Native |
| Self-hosted / air-gapped | Yes | Yes | Hybrid | Yes | Partial |
| Multi-provider failover | Yes | No | No | Yes | Yes |
| Semantic caching | Yes | Plugin | Yes | Via Redis | Yes |
| AI governance & DLP | Native | No | Partial | Partial | Partial |
| Enterprise SSO (OIDC/LDAP/mTLS) | Yes | Enterprise | Yes | Partial | Partial |
| MCP gateway | Native | Enterprise | Managed | Native | Yes |
| A2A agent gateway | Yes | No | No | Yes | No |
| Human-in-the-loop approvals | Yes | No | No | Partial | No |
| Post-quantum TLS (hybrid PQC) | Native | No | No | No | No |
| Open source | Full-featured | Limited | No | Yes | Partial |

## Performance (Preliminary Benchmarks)

On identical hardware (4 vCPU, 8 GB RAM): Tygress (Rust/Pingora) 98K rps / 0.12ms p99 / 18 MB; Envoy (C++) 72K / 0.31ms / 52 MB; NGINX (C) 65K / 0.45ms / 38 MB; Kong (Lua/NGINX) 34K / 1.2ms / 128 MB; Traefik (Go) 28K / 1.8ms / 86 MB.

## Editions

| Edition | Description | Price |
|---------|-------------|-------|
| Open Source | Full-featured gateway and AI gateway, self-hosted, free forever. Community supported. | Free |
| Cloud | Fully managed with global edge deployment, auto-scaling, 99.99% SLA. | Starting at $49/mo |
| Cloud (Self-Managed) | Run on your own AWS/GCP/Azure with the Tygress management plane, SSO, data residency, and audit logging. | Starting at $99/mo |
| Enterprise | Air-gapped/on-prem, AI governance, multi-region HA, advanced RBAC + audit, custom SLA. | Custom pricing |

## Frequently Asked Questions

**What is Tygress?**
An open-source enterprise API gateway and AI gateway built on Rust and Pingora. One self-hosted gateway for APIs, LLMs, MCP tool servers, and AI agents.

**What is an enterprise API gateway?**
The single entry point that routes, secures, and observes all API traffic across an organization — authentication (SSO, OIDC, mTLS), rate limiting, high availability, audit logging, and governance on top of request routing. Tygress extends those controls to LLM, MCP, and AI agent traffic in one Rust data plane.

**Can I self-host Tygress or run it air-gapped?**
Yes — Docker, Compose, or Kubernetes, inside your own VPC or on-prem including air-gapped. Prompts, keys, and data never leave your network.

**Does Tygress support enterprise SSO, LDAP, and mTLS?**
Yes — OIDC SSO, LDAP/Active Directory, mutual TLS, OAuth2 with required scopes, JWE, HMAC, and API keys, chainable per route.

**Which LLM providers does Tygress support?**
OpenAI, Anthropic, Google Gemini, Azure OpenAI, AWS Bedrock, Mistral, Groq, and any OpenAI-compatible endpoint — through one unified API with virtual keys and failover.

**How does Tygress help with AI governance and compliance?**
PII/secret redaction (AI DLP), prompt-injection guardrails, model/tool allowlists, human-in-the-loop approvals, immutable audit logging, and role-based admin access — one control point for all AI and API traffic.

**How does Tygress compare to LiteLLM or Portkey?**
They focus on AI traffic only; Tygress is a full production gateway where AI features are native, with Rust performance plus enterprise auth, rate limiting, circuit breaking, and observability on the same routes.

**Is Tygress quantum-safe?**
Yes — Tygress ships post-quantum cryptography out of the box. Every TLS connection can negotiate a NIST-standardized hybrid key exchange (X25519 + ML-KEM-768, FIPS 203), enforceable per-route or fleet-wide, with on-demand crypto-posture reports — defending against harvest-now, decrypt-later attacks.

**What is Pingora?**
Cloudflare's open-source Rust framework for building network services, powering its edge network at trillions of requests daily.

## Status

Tygress is currently in development. Sign up at https://tygress.com to join the waitlist and get early access.

## Links

- Website: https://tygress.com
- Tygress vs LiteLLM comparison: https://tygress.com/vs/litellm