📝 AI Guardrails

Description

< What is it? >

Meaning: AI guardrails are checks and controls that keep an AI application within defined requirements. They can validate inputs, restrict data access and tool actions, check outputs, and determine what happens when a requirement is not met.
Scope: traditional machine-learning systems may use feature validation, prediction bounds, and fallback rules. Large Language Model (LLM), Retrieval Augmented Generation (RAG), and agentic applications also need controls around generated content, retrieved evidence, and tool execution.
Example: a support assistant may answer from documents available to the signed-in user, return a prescribed JSON structure, and submit a refund only after the backend verifies the user's permissions and the refund policy.
System responsibility: guardrails are usually implemented around the model in application code, middleware, or dedicated services. A guardrail can itself use a classifier or another model, but it need not involve machine learning. Google's production ML systems module explains why verification and monitoring extend beyond model code.

Key points

< Where guardrails run >

Placement: validate at the boundary where a decision takes effect. An output check after a tool has already transferred money cannot prevent that transfer.

Stage	Example control	Response when the check fails
Input	Validate required fields, size limits, and supported file types	Reject invalid input or request a correction
Retrieval	Enforce tenant and document permissions using authenticated identity	Exclude unauthorized records before they reach the model
Model response	Validate schema, applicable content policy, and citation references	Withhold, repair, or regenerate the response
Tool execution	Check allowed tool, arguments, ownership, and transaction limits	Deny the action or request approval according to policy
Agent loop	Enforce time, cost, and step budgets	Stop or route to a fallback
Production operation	Detect sustained quality degradation or anomalous behavior	Alert; apply a configured rollback or fallback

Illustrative RAG flow: the application controls whether content proceeds to the next stage.

Request → input checks → permission-filtered retrieval → model
                                                          ↓
                                                     draft answer
                                                          ↓
                                                  output validation
                                                   /             \
                                                pass             fail
                                                 ↓                ↓
                                              release      bounded retry / fallback

Proposed tool call → authorization + argument checks → execution

< Deterministic and model-based checks >

Deterministic checks: application code can enforce exact requirements, such as valid JSON, an allowed document ID, a maximum amount, or a bounded number of tool calls. These checks are predictable within their defined scope.
Model-based checks: a classifier or judge can estimate whether text violates a content policy or whether a claim is supported by retrieved passages. Such decisions can have false positives and false negatives; measure them on representative examples.
Layering: use each method for the requirement it can evaluate. A semantic judge does not replace backend authorization, and a schema validator does not establish that an answer is true.

< RAG and prompt injection >

Untrusted content: retrieved documents, webpages, and tool results can contain instructions intended to redirect the model. Treat that content as evidence to analyze, while keeping application instructions and permissions separate. OWASP's prompt-injection guidance describes direct and indirect injection and layered mitigations.
Access control: derive identity and permissions from trusted application state, not from a username or role suggested by the model. Apply document restrictions before retrieval results enter model context; propagate the authenticated user's scope into tool execution.
Citation checks: verify that cited records were retrieved and permitted for this request. Then separately assess whether the cited passages support the claims. A real source ID can accompany an unsupported statement.
Limits: prompt instructions, filtering, and fine-tuning can reduce particular failures, but none alone establishes complete protection against prompt injection. Restrict the consequences a compromised model response can trigger.

< Failure handling and streaming >

Explicit policy: define whether a failed check causes rejection, redaction, one or more bounded retries, human review, or a fallback answer. Retrying the same request indefinitely increases cost without ensuring success.
Unavailable validators: decide behavior by the requirement. An authorization failure or unavailable permission check should prevent the protected action; a noncritical formatting failure may use a simpler fallback.
Streaming: text already sent to the client has already been disclosed. If a requirement applies to the entire answer before release, buffer and validate the complete answer. Chunk-level screening can reduce delay, but phrases and instructions can span chunk boundaries.
Logging: record the check name, policy version, decision, latency, and relevant record IDs. Minimize sensitive text in traces and restrict access to logs.

< Guardrails for predictive ML >

Input contracts: validate types, required features, units, ranges, and missing-value handling before prediction. For example, an age feature measured in years should not silently receive a value measured in months.
Output constraints: reject non-finite predictions and enforce application-specific bounds where appropriate. A confidence-based fallback requires validation of the confidence measure; an arbitrary threshold is not a reliability guarantee.
Distribution changes: monitor shifts in incoming data and realized outcomes. A drift alert is evidence to investigate; it does not by itself prove that model accuracy has declined or authorize a particular recovery action.

Comparison

< Prompts, guardrails, evaluation, and observability >

Different responsibilities: these components support one another.

Component	Main purpose	Example
Prompt engineering	Instruct the model how to respond	Ask for an answer with source IDs
Guardrails	Check or constrain a live request, response, or action	Reject source IDs outside the request's allowed evidence
Evaluation	Measure performance on test cases or sampled traffic	Measure unsupported-claim rate across 500 answers
Observability	Record execution behavior for inspection	Trace retrieval hits, tool calls, and validation outcomes

Relationship: evaluation determines whether guardrails are effective, and observability helps explain their decisions. Monitoring becomes an active control when a defined policy connects a signal to an action, such as disabling a failing tool.

< Tools and implementation choices >

Choose by the check: no single library replaces the application's identity, permission, and business-rule enforcement.

Tool or approach	Useful for	Scope to keep in mind
Pydantic	Typed schemas and custom field validation	Valid structure does not imply factual correctness
NVIDIA NeMo Guardrails	Configurable input, retrieval, dialog, execution, and output controls	Configure and evaluate the rails needed by the application
Guardrails AI	Input/output validators and structured-data workflows	Validator coverage and accuracy depend on the selected checks
Application code and backend policy	Permissions, ownership, budgets, and business constraints	Enforce at the service that reads data or executes the action

Implementation

< Enforce JSON-only output >

Guardrail type: requiring a response to be valid JavaScript Object Notation (JSON) is an output-format guardrail. Requiring specific fields and types adds schema validation. The model produces text; the application validates that text and, when needed, saves it as a .json file.
Enforcement: a prompt such as “Return only JSON, without Markdown or explanation” communicates the requirement but does not enforce it. Use the provider's schema-constrained structured-output feature when available, then validate the completed response before releasing or saving it. For example, Gemini structured outputs accepts a JSON Schema. Handle refusals, interrupted generation, and API errors as separate failure paths.

Guarantee boundary: the application can enforce that only validated JSON is released through this output path. It cannot guarantee that every generation attempt succeeds or that the values are factually correct. On failure, withhold the raw response and return a defined error or make a bounded retry. For an API, serialize both success and error responses as JSON and use Content-Type: application/json.

Input → model with structured output → parse complete response → validate fields
                                                                   ↓
                                                        pass → serialize → result.json
                                                        fail → withhold / bounded retry

Concrete task: extract an order from Order 2 notebooks. The required result has exactly two fields: product, a nonempty string, and quantity, a positive integer. JSON syntax alone also permits arrays, strings, and other values; this application's schema requires an object.
```
{"product": "notebook", "quantity": 2}
```

Runnable example: fixed response strings simulate model output so the validation can be reproduced without an API key. The code parses the entire response, rejects nonstandard numeric constants and duplicate keys, checks the schema, and writes only an accepted result. It does not extract a JSON-looking substring from surrounding prose. See Python's JSON encoder and decoder documentation.

import json
from pathlib import Path

def reject_constant(value):
    raise ValueError(f"Nonstandard JSON constant: {value}")

def unique_object(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError(f"Duplicate key: {key}")
        result[key] = value
    return result

def validated_order_json(raw_response):
    data = json.loads(
        raw_response,
        parse_constant=reject_constant,
        object_pairs_hook=unique_object,
    )
    if not isinstance(data, dict) or set(data) != {"product", "quantity"}:
        raise ValueError("Expected exactly product and quantity")
    if not isinstance(data["product"], str) or not data["product"].strip():
        raise ValueError("product must be nonempty text")
    # Use type(...) is int because Python bool is a subclass of int.
    if type(data["quantity"]) is not int or data["quantity"] < 1:
        raise ValueError("quantity must be a positive integer")
    return json.dumps(data, ensure_ascii=False, allow_nan=False)

responses = {
    "valid": '{"product":"notebook","quantity":2}',
    "extra_prose": 'Here is your JSON: {"product":"notebook","quantity":2}',
    "markdown": '```json\n{"product":"notebook","quantity":2}\n```',
    "wrong_type": '{"product":"notebook","quantity":"two"}',
}

for name, raw_response in responses.items():
    try:
        output_json = validated_order_json(raw_response)
    except ValueError:
        # Log the decision; do not release or save the invalid response.
        print(f"{name}: BLOCK")
        continue
    Path("result.json").write_text(output_json + "\n", encoding="utf-8")
    print(f"{name}: SAVED result.json")

Verified console output: these diagnostic messages are separate from the JSON file.
```
valid: SAVED result.json
extra_prose: BLOCK
markdown: BLOCK
wrong_type: BLOCK
```
Verified contents of result.json: the file contains one JSON object, without Markdown fences or explanatory text. The blocked attempts do not overwrite it.
```
{"product": "notebook", "quantity": 2}
```
Production integration: pass the completed model response to validated_order_json() and release only its returned value. Do not stream unvalidated model tokens to the client through this path. Use a request-specific destination when saving concurrent outputs; the fixed filename above is for the local demonstration.

< Validate a RAG answer before release >

Concrete input: a signed-in user belongs to tenant shop-a. For this request, the application retrieves and authorizes document D17, which states: Headphones may be returned within 30 days. The model must return an object containing answer and a nonempty list of source_ids.

Runnable example: the standard-library code below validates a fixed response schema and checks that every cited ID belongs to the request's permitted retrieved evidence. The tenant identity and retrieved IDs represent trusted backend state, not fields supplied by the model. The examples use fixed model-response strings so the guardrail behavior is reproducible without an API call.

import json

documents = {
    "D17": {
        "tenant": "shop-a",
        "text": "Headphones may be returned within 30 days.",
    },
    "D99": {
        "tenant": "shop-b",
        "text": "Internal policy for another tenant.",
    },
}
authenticated_tenant = "shop-a"
retrieved_ids = {"D17"}  # Already filtered before model invocation.
allowed_source_ids = {
    doc_id for doc_id in retrieved_ids
    if documents[doc_id]["tenant"] == authenticated_tenant
}

def validate_answer(raw_response, allowed_ids):
    try:
        data = json.loads(raw_response)
    except json.JSONDecodeError:
        return "BLOCK: invalid JSON"

    if not isinstance(data, dict) or set(data) != {"answer", "source_ids"}:
        return "BLOCK: invalid schema"
    if not isinstance(data["answer"], str) or not data["answer"].strip():
        return "BLOCK: answer must be nonempty text"
    sources = data["source_ids"]
    if (not isinstance(sources, list) or not sources
            or any(not isinstance(value, str) for value in sources)):
        return "BLOCK: source_ids must be a nonempty list of strings"
    if not set(sources).issubset(allowed_ids):
        return "BLOCK: citation outside permitted retrieved evidence"
    return "PASS: structure and citation access checks"

cases = {
    "valid": '{"answer":"The return period is 30 days.","source_ids":["D17"]}',
    "unauthorized": '{"answer":"Another policy applies.","source_ids":["D99"]}',
    "malformed": 'The return period is 30 days.',
    "missing_sources": '{"answer":"The return period is 30 days.","source_ids":[]}',
    "unsupported_claim": '{"answer":"The return period is 90 days.","source_ids":["D17"]}',
}
for name, response in cases.items():
    print(f"{name}: {validate_answer(response, allowed_source_ids)}")

Verified output:

valid: PASS: structure and citation access checks
unauthorized: BLOCK: citation outside permitted retrieved evidence
malformed: BLOCK: invalid JSON
missing_sources: BLOCK: source_ids must be a nonempty list of strings
unsupported_claim: PASS: structure and citation access checks

Interpretation: the final case deliberately passes these checks even though 90 days contradicts D17. To validate that claim, add a separate comparison against an authoritative structured return-period field, or evaluate a groundedness checker for free-form answers. The function does not perform content moderation, personal-data detection, or semantic verification.
Integration: run the required checks before sending the answer. A failure should return a controlled fallback or trigger a bounded repair attempt. This output check supplements retrieval-time authorization; it cannot undo disclosure if unauthorized text was already sent to the model.

Troubleshoot

< Measure failures and improve coverage >

Valid requests are blocked: inspect false positives by language, topic, and input type. Adjust the relevant policy or detector using labeled cases rather than weakening unrelated permissions.
Unwanted responses pass: identify which requirement lacked a check. Passing a schema validator should not be reported as passing all safety and grounding requirements.
High latency or cost: measure each validator separately. Apply inexpensive deterministic checks early and use model-based checks where their added coverage is needed.
Repair loops repeat: set retry and time limits, preserve failure reasons for debugging, and choose a predictable fallback.
Regression after an update: rerun benign, boundary, and adversarial cases when prompts, models, tools, or policies change. Track violation pass-through rate, false-block rate, task completion, and added latency; report denominators and sample sizes.

Harness Engineering places guardrails around the agent's tools and execution loop.
Agentic AI System explains agent orchestration and tool use.
Retrieval Augmented Generation (RAG) supplies retrieved evidence to a language model.
Milvus includes metadata and permission filtering for retrieval.
Agentic RAG Evaluation measures application quality.
Agentic RAG Observability records execution traces and operational signals.

Description​

< What is it? >​

Key points​

< Where guardrails run >​

< Deterministic and model-based checks >​

< RAG and prompt injection >​

< Failure handling and streaming >​

< Guardrails for predictive ML >​

Comparison​

< Prompts, guardrails, evaluation, and observability >​

< Tools and implementation choices >​

Implementation​

< Enforce JSON-only output >​

< Validate a RAG answer before release >​

Troubleshoot​

< Measure failures and improve coverage >​

Related ideas​

Reference​