Create a test set for an agent (preview)

[This article is prerelease documentation and is subject to change.]

A test set is a collection of conversations you use to evaluate your agent's performance. Each conversation represents a scenario you want your agent to handle.

Prerequisites

An agent created and saved with the GitHub Copilot harness. Learn more in Create an agent (preview).

Start a new evaluation

To start a new evaluation:

1. Open your agent in Copilot Studio.
2. Select the Evaluate tab.
3. Select New evaluation.

Choose one of these options:

Option 1: Upload conversations from a CSV file

1. On the Evaluate tab, in the Data source section, drag a CSV file onto the upload area or select it to browse for a file.

   Tip

   Select the CSV link to download a template that shows the correct file format. The maximum file size is 5 MB.

2. Review the imported conversations and adjust them as needed.

3. In the Configure test set panel on the right, enter a Name for the evaluation.

4. Select Evaluate to run the evaluation, or select Save to save it without running.

Option 2: Generate conversations with AI

1. On the Evaluate tab, in the More ways to start section, select Quick conversation set.
2. Wait while the system generates 10 conversations based on the agent's description, instructions, and topics.
3. Review the generated conversations in the Review your test cases list.
4. In the Configure test set panel, enter a Name, and then select Evaluate or Save.

You can also generate more conversations from the evaluation view. Select Add conversations > Generate 10 conversations. After you run an initial evaluation, select Add conversations > Generate 25 conversations or Generate 50 conversations to generate more.

Option 3: Write conversations manually

1. On the Evaluate tab, in the More ways to start section, select Or, write some questions yourself.
2. In the Review your test cases list, select Add conversations > Write to manually create a test case.
3. Add a user question for each conversation. Optionally, add an expected agent response.
4. In the Configure test set panel, enter a Name and select Evaluate or Save.

Configure the test set

When you create or edit an evaluation, the Configure test set panel includes these options:

• Name: A required name for the evaluation.
• Test method: The method that scores responses. General quality checks whether responses meet quality standards like relevance and completeness.
• User profile: The authenticated profile that runs the evaluation and connects to tools. Evaluations use the profile of the signed-in user. You can't connect with a different user while you're signed in. To run an evaluation as a different user, sign out, sign in with that user's account, and then start the evaluation.

Edit conversations in a test set

To edit conversations in a test set:

1. On the Evaluate tab, open an existing test set from the Evaluation dropdown.
2. Select a conversation to edit its messages.
3. Use the Add conversations dropdown to add more conversations.
4. Select Save when you're done.

Test set limits

The following limits apply on the Evaluate tab when you create or edit a test set or run an evaluation.

The same limits apply when you upload conversations from a CSV file:

• Only the first 20 conversations are imported.
• Only the first six question-and-answer pairs of each conversation are imported.
• Conversations with a question or expected response longer than 1,000 characters aren't imported. A message tells you that the import skipped some conversations.

Related information

• Evaluate an agent
• Run an evaluation for an agent
• View evaluation results for an agent
• Preview and test an agent