Build a Simple Agent from Scratch

This tutorial will guide you through building a knowledge base-driven intelligent Q&A Agent from scratch, suitable for enterprise internal knowledge management scenarios. This Agent can retrieve information from the enterprise knowledge base and intelligently respond to user queries, thereby improving information retrieval efficiency.

Taking the daily work of an M365 operations engineer as an example, a large number of documents often need to be consulted during project operations. These documents are diverse and scattered, making manual searching time-consuming and labor-intensive. With the help of an intelligent Agent, the efficiency of consulting documents can be greatly improved, repetitive work reduced, and convenience brought to operations.

This tutorial will use this scenario as a background to demonstrate step by step how to build a practical knowledge Q&A Agent.

Prepare the Knowledge Base Required for the Agent

Before officially creating the Agent, you need to prepare and configure the knowledge base it depends on. The core capability of the Agent comes from the content of the knowledge base, so high-quality, well-structured knowledge documents are crucial.

In this case, we will use the existing knowledge base “Microsoft Learning Database” as the information source.

• Please ensure that documents, FAQs, operation manuals, etc. related to M365 operations have been uploaded to the knowledge base in advance.
• Supported file types generally include PDF, Word, TXT, web page copies, etc. It is recommended to process them as follows before uploading:
  • Categorize and organize the content for easier system retrieval;
  • Use standardized file naming to avoid ambiguity;
  • Remove redundant information to improve text readability.

✅ Tip: The richness of the knowledge base content, document quality, and clarity of structure directly affect the accuracy and usability of the Agent's answers. It is recommended that the operations team regularly update and maintain the knowledge base content.

Once the knowledge base is ready, you can connect it in the Agent configuration stage through the “Knowledge Source” feature to enable content-based intelligent Q&A.

Create the Basic “M365 Support” Agent

1. Go to the SERVICEME NEXT homepage, click the circular icon in the lower left corner to enter the Agent Q&A interface.
2. Click “More Agents” in the left menu to enter the Agent Plaza.
3. In the upper right corner of the Agent Plaza, click “Create Agent” and select “Basic Orchestration Creation”.
4. Fill in the following basic information:
   • Agent Name: Enter M365 Support
   • Agent Avatar: Select one from the system built-in avatars (custom uploads are not supported currently)
   • Model Group: Select the model group configured by the administrator, such as General Model Group
   • Category: Select the business category, such as IT
   • Description: For example, Provide knowledge base Q&A support for M365 operations engineers
5. After completing the information, click “Create” to successfully generate the basic Agent.

Configure the Basic “M365 Support” Agent

After creation, the system will automatically enter the Agent configuration page. Complete the following configuration items in order:

1. Prompt Configuration

• Enter a brief prompt in the Prompt input box.
• You can click “Generate Intelligently” to let the system call the model to automatically expand the prompt and generate a more complete version.
• The prompt for this example is:

## Role
You are an M365 Operations Support Agent, specializing in helping operations engineers analyze issues related to Microsoft 365 services and efficiently retrieving relevant resources from the knowledge base.

## Skills
1. Analyzing M365 Issues:
   - Identify and diagnose problems encountered in various M365 services such as Exchange Online, SharePoint, Teams, and OneDrive.
   - Gather relevant error messages, logs, and contextual information to accurately understand the root cause of operational issues.
   - Prioritize incidents based on severity and impact, providing clear and concise problem statements to streamline troubleshooting.

2. Retrieving Knowledge Base Resources:
   - Search and extract pertinent documentation, troubleshooting guides, and best practices from the M365 knowledge base according to the identified issue.
   - Summarize and present solutions, workarounds, and reference materials tailored to the engineer’s specific case.
   - Keep track of the most frequently used resources to optimize future query efficiency.

## Constraints
- Only address topics strictly related to M365 operations, issue analysis, and knowledge base lookup. Decline unrelated queries.
- All outputs must follow the prescribed structure and remain within the context of technical support for Microsoft 365.
- Do not provide generic advice or speculative troubleshooting steps without referencing official or knowledge base resources.

2. Greeting Configuration

• You can enter a custom greeting or click “Generate Intelligently” to automatically generate a welcome message.
• The greeting for this example is:

Hello, I am M365 Support, here to assist you with Microsoft 365-related questions and support needs.

[How can you help me troubleshoot issues with Microsoft 365?] [What are some common problems users face with M365 and how can I resolve them?] [Can you guide me on optimizing my use of Microsoft 365 tools and features?]

3. Model Group Settings

• The model group (such as General Model Group) was selected when creating the Agent and will be automatically displayed here.

• You can switch as needed, but note:

  • Model group content may differ in different environments;
  • Model group content is pre-configured by the administrator;
  • The General Model Group used in the example includes models: gpt-4.1, DeepseekR1-Ali, Qwen3.

  

4. Skill Configuration (Optional)

• Skills must be created in advance by the administrator in skill management before they can be used.
• Common skills include: web page reading, news retrieval, etc.
• No skills are configured in this example. It is recommended to add skills only when necessary, as too many skills may affect Agent performance.

5. Conversation Experience Settings

The conversation experience determines the details of the Agent's interaction with users. It is recommended to configure it reasonably according to the business scenario.

• Context Memory Count:
  Set to 5, meaning the Agent will retain the last 5 rounds of conversation for context understanding.
  • Advantage: Improves the coherence of multi-turn conversations and facilitates handling complex questions.
  • Note: Too much context may slow down responses or mix information; 5 is a commonly recommended value.

• Click the gear icon to enter advanced configuration items. Common settings are as follows:

  • Enable User Suggested Questions

    • Function: Automatically recommends follow-up questions to the user based on the current conversation context.
    • Scenario: Improves user guidance, suitable for business scenarios friendly to novice users.
    • Recommendation for this case: Can be disabled, as operations engineers usually have clear goals.

  • Enable Question Guidance ✅

    • Function: Displays sample reference questions in the input box to lower the difficulty of asking questions.
    • Recommendation for this case: Enable, as it helps users understand the scope of questions they can ask.

  • Enable Chat History ✅

    • Function: Retains the user's conversation history with the Agent for later review.
    • Scenario: Issue tracking, case reproduction, knowledge base optimization.
    • Recommendation for this case: Enable, to facilitate operations personnel in tracing issues.

  • Enable Chat Feedback ✅

    • Function: Allows users to like/dislike or comment on each answer to collect feedback.
    • Scenario: Monitor and continuously optimize answer quality.
    • Recommendation for this case: Enable, to collect user opinions and optimize Agent answer quality.

  • Enable Keyword Filtering

    • Function: Can intercept input/output containing sensitive words to ensure content compliance.
    • Recommendation for this case: Enable as appropriate according to internal compliance requirements.

6. Knowledge Source Configuration

The knowledge source determines the core information source for the Agent's answers. Configuring the correct knowledge base and retrieval strategy can significantly improve Q&A accuracy.

• Select Knowledge Base
  • Click the “+” on the right, and select the required knowledge base from the pop-up list, such as:
    Microsoft Learning Database (M365 operations-related document collection).

• Configure Knowledge Source (Gear Icon)
  Enter the knowledge base settings page for detailed parameter configuration:

  • Retrieval Strategy: Hybrid Retrieval

    • Hybrid retrieval combines semantic understanding and keyword matching, improving recall and accuracy.
    • Recommended to enable in this case, suitable for knowledge Q&A scenarios with a large number of documents and diverse question forms.

  • Maximum Recall Count: 5

    • The system recalls up to 5 most relevant contents from the knowledge base each time to participate in answer generation.
    • The recommended value is 5; too high will slow down response speed, too low may miss key information.

  • Force Private Domain Q&A: Enable ✅

    • When enabled, the Agent will only use the selected knowledge base content to generate answers, no longer referencing general model knowledge.
    • Advantage: Ensures answer content is real, accurate, and controllable.
    • Recommendation for this case: Enable, to avoid generating generalized content unrelated to M365.

  • Document Matching Similarity, QnA Matching Similarity: Keep Default

    • The default settings are suitable for most scenarios. If you encounter recall failures or misjudgments in actual use, you can adjust this value appropriately (generally 0.6 ~ 0.8).

⚠️ In this scenario, external data sources do not need to be configured. If you do not need to connect to real-time interfaces or databases, you can skip the “Data Source” configuration item.

7. Save and Test

• After configuration, the upper right will prompt “Agent has been edited, please click Sync.”
• Click “Sync”, then you can test Q&A on the right, for example, enter:
  • “How to create a Copilot?”
  • “How to configure Copilot?”
• Fine-tune the configuration according to the test results until the answers meet expectations.
• Finally, click “Save Configuration” in the upper right to complete the Agent configuration.

The complete configuration is as follows: