OpenAI moderation

The OpenAI Moderation API detects potentially harmful content across categories including hate, harassment, self-harm, sexual content, and violence.

Before you begin

1. Set up an agentgateway proxy.
2. Set up access to the OpenAI LLM provider.

Block harmful content

1. Configure the prompt guard to use OpenAI Moderation:

   kubectl apply -f - <<EOF
   apiVersion: agentgateway.dev/v1alpha1
   kind: AgentgatewayPolicy
   metadata:
     name: openai-prompt-guard
     namespace: agentgateway-system
   spec:
     targetRefs:
     - group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: openai
     backend:
       ai:
         promptGuard:
           request:
           - openAIModeration:
               policies:
                 auth:
                   secretRef:
                     name: openai-secret
               model: omni-moderation-latest
             response:
               message: "Content blocked by moderation policy"
   EOF

2. Test with content that triggers moderation.

   Cloud Provider LoadBalancerPort-forward for local testing

   curl -i "$INGRESS_GW_ADDRESS/openai" \
     -H "content-type: application/json" \
     -d '{
       "model": "gpt-4o-mini",
       "messages": [
         {
           "role": "user",
           "content": "I want to harm myself"
         }
       ]
     }'

   curl -i "localhost:8080/openai" \
     -H "content-type: application/json" \
     -d '{
       "model": "gpt-4o-mini",
       "messages": [
         {
           "role": "user",
           "content": "I want to harm myself"
         }
       ]
     }'

   Expected response:

   HTTP/1.1 403 Forbidden
   Content blocked by moderation policy

Cleanup

kubectl delete AgentgatewayPolicy openai-prompt-guard -n agentgateway-system