Prompt Shields in Microsoft Foundry

Prompt Shields detect attempts to manipulate your model's behavior through adversarial inputs. Prompt Shields, powered by Azure AI Content Safety, analyze supplied text for user prompt attack and document attack patterns. You can analyze text for user prompt attacks in the user prompt and analyze text for document attacks in the documents. Azure AI Content Safety doesn't independently verify the submitted content's provenance; customers define provenance through their integration and can interpret results in that application context.

Prompt Shields detect two types of attacks:

• User prompt attacks — Malicious prompts that attempt to bypass system instructions or safety training. Prompt Shields scan these attacks at the user input intervention point.
• Document attacks — Hidden instructions embedded in document content (documents, emails, web pages) that try to hijack the model session. For model API calls, Prompt Shields scan document content at the user input intervention point. For agents, Prompt Shields can also scan document attacks at the tool response intervention point for supported tools (Preview).

When you enable blocking in Foundry, Prompt Shields block inputs classified as attacks before those inputs are processed by the model. In Azure AI Content Safety APIs, the API doesn't itself block or route content. Customers decide how to use the results in their application.

In Foundry, Prompt Shields are part of the guardrails and controls system. User prompt attack detection is enabled by default. Document attack detection is off by default and must be enabled when you configure guardrail controls for your model deployments or agents. Each configured control returns annotation results with detected (true/false) and filtered (true/false) values.

The following example shows the annotation structure for a detected user prompt attack that wasn't filtered:

{
	"choices": [...],
	"prompt_filter_results": [{
		"prompt_index": 0,
		"content_filter_results": {
			"jailbreak": {
				"filtered": false,
				"detected": true
			}
		}
	}]
}

User scenarios

Detect user prompt attacks

An application submits text in user_prompts for Foundry or userPrompt for AACS APIs to analyze it for user prompt attack patterns. The submitted text can originate from a user, a model, or another source. In Foundry, if an attack is detected, the result can be filtered or allowed depending on user settings. In Azure AI Content Safety APIs, if an attack is detected, the application decides whether to allow, block, review, or otherwise handle the content.

Detect document attacks

An application submits text in the documents array to analyze it for document attack patterns, such as injected instructions that attempt to alter the intended behavior of an LLM system. The submitted text can include retrieved documents, emails, webpages, model-generated content, or content from other sources.

Prompt Shields returns detection results. In Foundry, if an attack is detected, the result can be filtered or allowed depending on user settings. In Azure AI Content Safety APIs, blocking content, excluding a document, suggesting alternate content, or escalating to a person requires application logic.

Types of input attacks

The types of input attacks that Prompt Shields detects are described in this table.

Prompt Shields for user prompts

Previously called Jailbreak risk detection, this shield detects user prompt attack patterns that attempt to circumvent or alter system instructions.

Examples

Subtypes of user prompt attacks

Prompt Shields for user prompt attacks recognizes the following classes:

Prompt Shields for documents

This shield detects document attack patterns in text submitted through the documents request field. These attacks use instructions embedded in document content to attempt to alter the intended behavior of an LLM-based system.

Examples

Subtypes of document attacks

Prompt Shields for document attacks recognizes the following classes of attacks:

Spotlighting (preview)

Spotlighting provides enhanced protection against indirect attacks when your application processes third-party documents that might contain embedded malicious instructions. Use Spotlighting when you need an extra layer of protection beyond standard document attack detection, especially for applications that handle user-uploaded files or external web content.

How it works

Spotlighting tags input documents with special formatting to indicate lower trust to the model. The service transforms document content by using base-64 encoding so the model treats it as less trustworthy than direct user and system prompts. This method helps prevent the model from executing unintended commands found in third-party documents.

Cost and limitations

There's no direct cost for spotlighting, but it increases document tokens due to base-64 encoding, which can increase total costs. Spotlighting can also cause large documents to exceed input size limits. Spotlighting is only available for models used through the Chat Completions API.

Enable Spotlighting

Spotlighting is turned off by default. You can enable it when you configure document attack controls in the Foundry portal or through the REST API by turning on the Spotlighting toggle when configuring document attack controls.

Configure Prompt Shields

Choose the configuration path for the service that your application uses.

Configure Prompt Shields in the Foundry portal

1. In the Foundry portal, go to your project.
2. Select Guardrails from the left navigation.
3. Select Create guardrail.
4. Choose User prompt attack or Document attack from the risk dropdown.
5. Select an available intervention point and action. User prompt attacks apply only to user input. For model API calls, document attacks apply to document content at user input. For Foundry Agent Service agents, document attacks can also apply to tool response for supported tools (Preview).
6. For Spotlighting, turn on the Spotlighting toggle when configuring document attack controls.
7. Assign the guardrail to your model deployments or agents.

For detailed configuration steps, see Configure guardrails and controls.

Configure Prompt Shields in the REST API

POST https://{endpoint}/openai/deployments/{deployment-id}/chat/completions?api-version=2024-10-01-preview
Content-Type: application/json
api-key: {key}

{
  "messages": [{"role": "user", "content": "Hello"}],
  "data_sources": [{...}],
  "prompt_shield": {
    "user_prompt": {
      "enabled": true,
      "action": "annotate"
    },
    "documents": {
      "enabled": true,
      "action": "block",
      "spotlighting_enabled": true
    }
  }
}

Configure the Azure AI Content Safety API

The standalone Azure AI Content Safety API analyzes a userPrompt and up to five documents. The response reports attackDetected for the prompt and each document. To try the API and integrate it into an application, see the Prompt Shields quickstart.

Troubleshooting

Prompt Shield doesn't detect expected attacks

• Verify the guardrail is assigned to your deployment or agent.
• Check intervention points match where attacks occur (user input vs tool response).
• Review annotation results to see detected vs filtered status.

False positives

• Adjust from block to annotate mode to log without filtering.
• Review specific attack subtypes triggering false positives.
• Consider exempting trusted input sources from document attack scanning.

Spotlighting causes encoding references in responses

• This effect occurs when Spotlighting is enabled.
• Consider disabling Spotlighting if encoding mentions disrupt user experience.
• Use system prompts to instruct the model to avoid mentioning encodings.

Related content

• Configure guardrails and controls in Microsoft Foundry
• Azure AI Content Safety REST API
• Azure AI Content Safety client library for Python
• Azure AI Content Safety client library for .NET
• Azure AI Content Safety samples