Skip to content
Help guide the future of agentgateway — Take the survey

For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

SDKs (Python, Node.js, Go)

Verified : Automated tests passed for: End-to-end run: Every step ran in a fresh test environment and the results were checked.
Page as Markdown

Use the OpenAI SDK in Python, Node.js, or Go to call configured LLM providers through agentgateway in Kubernetes, and migrate from the LiteLLM SDK.

Call configured LLM providers from your application code through agentgateway in Kubernetes with the OpenAI SDK for Python, Node.js, or Go.

Agentgateway exposes an OpenAI-compatible API. Point the SDK’s base URL at the gateway and send OpenAI Chat Completions requests. Agentgateway forwards each request to the configured provider and translates it when the provider uses a different API, such as Anthropic, Amazon Bedrock, or Vertex AI. For per-provider API support, see About LLMs.

Before you begin

  1. Set up an agentgateway proxy.
  2. Set up access to the OpenAI LLM provider.

Get the gateway URL

Tip

Kind cluster? Kind does not support LoadBalancer services by default. To use this option with a Kind cluster, install and run cloud-provider-kind.

The following command reads the LoadBalancer IP address or hostname, whichever your cloud provider assigns.

export INGRESS_GW_ADDRESS=$(kubectl get svc -n agentgateway-system agentgateway-proxy \
  -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")

echo "Gateway address: $INGRESS_GW_ADDRESS"

If you use port-forwarding, keep it running and set the address in the terminal where you run the SDK examples:

export INGRESS_GW_ADDRESS=localhost:8080

Set the base URL

End the SDK base URL with /v1, such as http://$INGRESS_GW_ADDRESS/v1. The SDK appends the API path, such as /chat/completions, to the base URL.

If your HTTPRoute matches a path prefix, such as /openai, put the prefix before /v1: http://$INGRESS_GW_ADDRESS/openai/v1.

Python

  1. Install the OpenAI SDK in your Python project.

    pip install openai
  2. Create and run the following script to send a request through agentgateway.

    import os
    from openai import OpenAI
    
    gateway_address = os.environ["INGRESS_GW_ADDRESS"]
    
    client = OpenAI(
        base_url=f"http://{gateway_address}/v1",
        api_key="anything",  # placeholder if gateway has no auth
    )
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Hello from Kubernetes!"}],
    )
    print(response.choices[0].message.content)

Node.js

  1. Install the OpenAI SDK in your Node.js project.

    npm install openai
  2. Create and run the following script to send a request through agentgateway.

    import OpenAI from "openai";
    
    const gatewayAddress = process.env.INGRESS_GW_ADDRESS;
    
    const client = new OpenAI({
      baseURL: `http://${gatewayAddress}/v1`,
      apiKey: "anything",
    });
    
    const response = await client.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [{ role: "user", content: "Hello from Kubernetes!" }],
    });
    console.log(response.choices[0].message.content);

Go

  1. Add the OpenAI Go SDK to your module.

    go get github.com/openai/openai-go/v3
  2. Create and run the following program to send a request through agentgateway.

    package main
    
    import (
    	"context"
    	"fmt"
    	"os"
    
    	"github.com/openai/openai-go/v3"
    	"github.com/openai/openai-go/v3/option"
    )
    
    func main() {
    	client := openai.NewClient(
    		option.WithBaseURL("http://"+os.Getenv("INGRESS_GW_ADDRESS")+"/v1"),
    		option.WithAPIKey("anything"),
    	)
    
    	resp, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{
    		Model:    "gpt-4o-mini",
    		Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Hello from Kubernetes!")},
    	})
    	if err != nil {
    		panic(err)
    	}
    	fmt.Println(resp.Choices[0].Message.Content)
    }

Call more than one provider

Create an AgentgatewayModel for each provider with a provider prefix. The SDK code stays the same; only the model value changes.

  1. Allow the AgentgatewayModel kind on the Gateway listener. Keep HTTPRoute in the list to keep HTTPRoutes on other paths, such as /openai.

    Warning

    After you allow AgentgatewayModel, the listener serves LLM paths such as /v1/chat/completions from its models instead of from HTTPRoutes. A request for a model name with no matching model, such as gpt-4o-mini without a prefix, returns a model-not-found error.

    kubectl apply -f- <<EOF
    apiVersion: gateway.networking.k8s.io/v1
    kind: Gateway
    metadata:
      name: agentgateway-proxy
      namespace: agentgateway-system
    spec:
      gatewayClassName: agentgateway
      listeners:
      - name: http
        port: 80
        protocol: HTTP
        allowedRoutes:
          namespaces:
            from: All
          kinds:
          - group: gateway.networking.k8s.io
            kind: HTTPRoute
          - group: agentgateway.dev
            kind: AgentgatewayModel
    EOF
  2. Set ANTHROPIC_API_KEY to your Anthropic API key and create a secret for it. The OpenAI secret, openai-secret, is from the prerequisites.

    export ANTHROPIC_API_KEY='<your-anthropic-api-key>'
    kubectl create secret generic anthropic-secret -n agentgateway-system \
      --from-literal="Authorization=$ANTHROPIC_API_KEY"
  3. Create a model for each provider prefix.

    kubectl apply -f- <<EOF
    apiVersion: agentgateway.dev/v1alpha1
    kind: AgentgatewayModel
    metadata:
      name: openai
      namespace: agentgateway-system
    spec:
      parentRefs:
      - group: gateway.networking.k8s.io
        kind: Gateway
        name: agentgateway-proxy
        sectionName: http
      match:
        model: "openai/*"
      provider: OpenAI
      policies:
        auth:
          secretRef:
            name: openai-secret
        transformations:
        - field: model
          expression: 'llmRequest.model.stripPrefix("openai/")'
    ---
    apiVersion: agentgateway.dev/v1alpha1
    kind: AgentgatewayModel
    metadata:
      name: anthropic
      namespace: agentgateway-system
    spec:
      parentRefs:
      - group: gateway.networking.k8s.io
        kind: Gateway
        name: agentgateway-proxy
        sectionName: http
      match:
        model: "anthropic/*"
      provider: Anthropic
      policies:
        auth:
          secretRef:
            name: anthropic-secret
        transformations:
        - field: model
          expression: 'llmRequest.model.stripPrefix("anthropic/")'
    EOF
  4. Send requests with the prefixed model names.

    messages = [{"role": "user", "content": "Hello!"}]
    client.chat.completions.create(model="openai/gpt-4o-mini", messages=messages)
    client.chat.completions.create(model="anthropic/claude-sonnet-4-5", messages=messages)

Agentgateway strips the prefix and sends the request to the matching provider. For more about models, see Serve a model.

Migrate from the LiteLLM SDK

The example configuration in the previous section accepts openai/gpt-4o-mini and anthropic/claude-sonnet-4-5, so you can keep those LiteLLM model strings. Other model strings need matching gateway configuration. This example migrates Chat Completions calls. Review provider-specific parameters and other LiteLLM features separately. Replace the litellm call with the OpenAI SDK and move provider API keys from your application to Kubernetes secrets, as in Call more than one provider.

Before, with LiteLLM:

from litellm import completion

response = completion(
    model="anthropic/claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

After, with agentgateway:

import os
from openai import OpenAI

client = OpenAI(base_url=f"http://{os.environ['INGRESS_GW_ADDRESS']}/v1", api_key="anything")

response = client.chat.completions.create(
    model="anthropic/claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

For asynchronous calls, replace acompletion with AsyncOpenAI:

import asyncio
import os
from openai import AsyncOpenAI

async_client = AsyncOpenAI(base_url=f"http://{os.environ['INGRESS_GW_ADDRESS']}/v1", api_key="anything")

async def main():
    response = await async_client.chat.completions.create(
        model="anthropic/claude-sonnet-4-5",
        messages=[{"role": "user", "content": "Hello!"}],
    )
    print(response.choices[0].message.content)

asyncio.run(main())
LiteLLM SDKOpenAI SDK with agentgateway
completion(...)client.chat.completions.create(...)
await acompletion(...)await async_client.chat.completions.create(...)
stream=Truestream=True
embedding(...)client.embeddings.create(...)
num_retries=3OpenAI(..., max_retries=3) on the client, or a retry policy on the gateway
Router(..., fallbacks=[...])A virtual model with failover routing. The client sends the virtual model name.
completion_cost(...)Agentgateway prices each request from its model cost catalog and records the USD cost in access logs, traces, and metrics.
Provider keys, such as ANTHROPIC_API_KEY, in the application environmentA Kubernetes secret referenced by each AgentgatewayModel

With LiteLLM fallbacks, the failed request moves to the next model. With agentgateway failover, the request that triggers eviction returns an error, and later requests go to the next priority group. The OpenAI Python SDK retries that error twice by default (max_retries=2), and the retry reaches a model in that group. If you set max_retries=0, configure a retry policy on the gateway instead.

Authenticate with a JWT

Configure an AgentgatewayPolicy with jwtAuthentication on the gateway. Use mode: Strict to require a valid token on every request.

Obtain a JWT access token for your gateway API from your identity provider. The token must satisfy the policy’s validation rules, including any configured issuer and audience requirements. Set AGENTGATEWAY_JWT in your application’s environment to the raw token, without the Bearer prefix.

Pass that token as the SDK API key. The SDK sends Authorization: Bearer <token>.

import os
from openai import OpenAI

jwt_token = os.environ["AGENTGATEWAY_JWT"]
client = OpenAI(
    base_url=f"http://{os.environ['INGRESS_GW_ADDRESS']}/v1",
    api_key=jwt_token,
)

This token authenticates the caller to agentgateway. Keep the upstream provider API keys in the Kubernetes secrets referenced by your backend or model resources. Obtain a fresh token when it expires and update the SDK client before making more requests.

Was this page helpful?
Agentgateway assistant

Ask me anything about agentgateway configuration, features, or usage.

Note: AI-generated content might contain errors; please verify and test all returned information.

Tip: one topic per conversation gives the best results. Use the + button in the chat header to start a new conversation.

Switching topics? Starting a new conversation improves accuracy.
↑↓ navigate ↵ select esc dismiss

What could be improved?

Your feedback helps us improve assistant answers and identify docs gaps we should fix.

Need more help? Join us on Discord: https://discord.gg/y9efgEmppm

Want to use your own agent? Add the Solo MCP server to query our docs directly. Get started here: https://search.solo.io/.