Google Model Armor

Google Cloud Model Armor lets you create safety templates in the Google Cloud console and apply them to LLM traffic. Model Armor sanitizes both user prompts and model responses against your configured policies, blocking content that violates your templates.

Google Cloud Model Armor guardrails are model-agnostic and can be applied to any Large Language Model (LLM), whether it is hosted on Google Cloud, another cloud provider (like Amazon or Azure), or on-premises.

Before you begin

Install and set up an agentgateway proxy.
Set up access to the Gemini LLM provider.

Set up the Google Model Armor

Log in to the Google Cloud console and create a Model Armor template. For more information, see the Google Cloud documentation.
Note the template ID, project ID, and the region where the template is deployed. Alternatively, you can use the following command to retrieve this information.
```
gcloud model-armor templates list --location=<location>
```

Configure access to GCP

Set up your agentgateway proxy with the credentials so that you can access Google Model Armor guardrails. The setup varies depending on what type of cluster you run agentgateway in.

Create a service account with the required permissions to access Google Model Armor. Then mount a JSON key file to the agentgateway-proxy pod.

Set the Google Cloud project ID where you created the Google Model Armor template as an environment variable.
```
export PROJECT_ID=<google-cloud-project-ID>
```

Create a service account and assign permissions to access Google Model Armor. Then, create a local JSON key file with your credentials.

gcloud iam service-accounts create agw-sa-modelarmor-kind --project $PROJECT_ID

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:agw-sa-modelarmor-kind@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/modelarmor.user"

gcloud iam service-accounts keys create key.json \
  --iam-account=agw-sa-modelarmor-kind@$PROJECT_ID.iam.gserviceaccount.com

Optional: Review the local JSON key file.
```
cat ./key.json
```

Create a Kubernetes secret to store the details of your JSON key file.

kubectl create secret generic gcp-credentials \
  -n agentgateway-system \
  --from-file=key.json=./key.json

Patch the agentgateway-proxy deployment to mount the secret as a volume.

kubectl patch deployment agentgateway-proxy -n agentgateway-system --type=json -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/volumes/-",
   "value": {
      "name": "gcp-credentials",
      "secret": {"secretName": "gcp-credentials"}
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/volumeMounts/-",
    "value": {
      "name": "gcp-credentials",
      "mountPath": "/var/secrets/google",
      "readOnly": true
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/env/-",
    "value": {
      "name": "GOOGLE_APPLICATION_CREDENTIALS",
      "value": "/var/secrets/google/key.json"
    }
  }
]'

Verify that the agentgateway proxy picked up the credentials.

kubectl get deployment agentgateway-proxy -n agentgateway-system \
  -o jsonpath='{.spec.template.spec.containers[0].env}' | jq

Example output:

...
{
 "name": "GOOGLE_APPLICATION_CREDENTIALS",
 "value": "/var/secrets/google/key.json"
}
...

In your GKE cluster, set up workload identity.

Set the Google Cloud project ID where you created the Google Model Armor template as an environment variable.

export PROJECT_ID=<google-cloud-project-ID>
export ZONE=<gke-cluster-zone>
export CLUSTER=<gke-cluster-name>

Enable workload identity on your cluster.

gcloud container clusters update $CLUSTER \
 --workload-pool=$PROJECT_ID.svc.id.goog \
 --zone $ZONE

Create a GCP service account with Model Armor permissions.

gcloud iam service-accounts create agentgateway-sa \
  --project=$PROJECT_ID

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:agentgateway-sa@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/modelarmor.user"

Allow the agentgateway proxy’s Kubernetes service account to impersonate the GCP service account.

gcloud iam service-accounts add-iam-policy-binding \
  agentgateway-sa@$PROJECT_ID.iam.gserviceaccount.com \
  --role="roles/iam.workloadIdentityUser" \
  --member="serviceAccount:$PROJECT_ID.svc.id.goog[agentgateway-system/agentgateway-proxy]"

Annotate the Kubernetes service account.

kubectl annotate serviceaccount agentgateway-proxy \
  -n agentgateway-system \
  iam.gke.io/gcp-service-account=agentgateway-sa@$PROJECT_ID.iam.gserviceaccount.com

Restart the agentgateway proxy.

kubectl rollout restart deploy/agentgateway-proxy -n agentgateway-system

Apply guardrails

Configure the prompt guard and apply it to the Gemini route that you set up before you began. Add the location, project ID and template ID of your guardrail.

kubectl apply -f - <<EOF
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: google-prompt-guard
  namespace: agentgateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: google
  backend:
    ai:
      promptGuard:
        request:
        - googleModelArmor:
            templateId: <template-ID
            projectId: <project-ID>
            location: <location>
            policies:
              auth:
                gcp: 
                  type: AccessToken
        response:
        - googleModelArmor:
            templateId: <template-ID
            projectId: <project-ID>
            location: <location>
            policies:
              auth:
                gcp: 
                  type: AccessToken
EOF

Send a request to the Gemini provider that triggers the guardrail.

Cloud Provider LoadBalancer:

curl "$INGRESS_GW_ADDRESS/v1beta/openai/chat/completions" -H content-type:application/json  -d '{
  "model": "",
  "messages": [
   {"role": "user", "content": "I want to harm myself"}
 ]
}'

Localhost:

curl "localhost:8080/v1beta/openai/chat/completions" -H content-type:application/json  -d '{
  "model": "",
  "messages": [
   {"role": "user", "content": "I want to harm myself"}
 ]
}'

Cloud Provider LoadBalancer:

curl "$INGRESS_GW_ADDRESS/v1/chat/completions" -H content-type:application/json  -d '{
  "model": "",
  "messages": [
   {"role": "user", "content": "I want to harm myself"}
 ]
}'

Localhost:

curl "localhost:8080/v1/chat/completions" -H content-type:application/json  -d '{
  "model": "",
  "messages": [
   {"role": "user", "content": "I want to harm myself"}
 ]
}'

Cloud Provider LoadBalancer:

curl "$INGRESS_GW_ADDRESS/gemini" -H content-type:application/json  -d '{
  "model": "",
  "messages": [
   {"role": "user", "content": "I want to harm myself"}
 ]
}'

Localhost:

curl "localhost:8080/gemini" -H content-type:application/json  -d '{
  "model": "",
  "messages": [
   {"role": "user", "content": "I want to harm myself"}
 ]
}'

Example output:

The request was rejected due to inappropriate content

AWS Bedrock Guardrails Multi-layered guardrails

Google Model Armor

Before you begin

Set up the Google Model Armor

Configure access to GCP

Apply guardrails

What could be improved?