D365 Sales Copilot Optimization for a Customised Data Model

Views (136)

cemaltevrizci

Like(1)

Report

Understanding the Challenge

Data structures are often intricate, with custom tables. In a sample scenario where “Client Meeting Summary” (custom table) capturing customer communications, parented by an Opportunity table and “Client” (account table) layered on top of these tables. The goal is to leverage Copilot to provide actionable insights for users, focusing on large opportunities and account hierarchies. However, common issues include:

Name Confusion: Copilot may confuse custom tables (e.g., “Client Meeting”) with similarly named standard tables (e.g., “Appointment”), leading to inaccurate responses.
Low Query Accuracy: Natural language queries can produce errors or irrelevant results due to language ambiguity, algorithm limits, or incomplete datasets. Improving accuracy requires better parsing, broader domain knowledge, and regular model updates based on user feedback.
Performance Bottlenecks: Queries spanning long timeframes or involving complex filters can time out or fail to complete.

Key Learnings and Solutions

1. Precise Glossary Definitions

Customers will see significant improvements from refining the glossary within Copilot Studio. By clearly defining what each custom table represents, and explicitly stating distinctions (e.g., “Client Meeting is not an Appointment”), ambiguity for the language model is reduced. Glossary entries should:

Use clear, unambiguous language.
Reference schema names where possible. Glossary entries are essential when custom tables have names similar to standard tables, as this can cause confusion and incorrect data retrieval. Clearly stating what a table is and is not helps reduce ambiguity.
Explicitly state what a table is not, to avoid overlap.
Write the full context in the description to maximize effectiveness.
Double quotes are used for specific phrases, and single quotes are used for table names, to ensure clarity.

Copilot response before adding the glossary

Although the prompt requests a client meeting related to our custom entity in the app module, Copilot returns results from the appointment table, assuming appointments are being requested.
A key challenge encountered when using Copilot in our app module. Despite the prompt specifically requesting information about a client meeting related to our custom entity, Copilot returns results from the appointment table instead. This behaviour occurs because Copilot assumes that appointments are being requested, likely due to ambiguity or overlap in the naming conventions used within the system. As a result, rather than retrieving data from the intended custom 'client meeting' table, Copilot defaults to the standard appointments table, which does not reflect the precise requirements of the query. This highlights the importance of having a well-defined glossary and clear distinctions between custom entities and standard tables to ensure Copilot understands the context and provides accurate responses.
Adding the glossary item
In this example we will use a clarification prompt under the glossary item to clarify what a “Client Meeting” is and what it is not.
The prompt will be “Client Meeting” is referring to ‘cte_client_meeting’ table. “Client Meeting” is not an ‘Appointment’.

Copilot response after adding the glossary

Adding our instructions to the Glossary helped Copilot generate correct queries automatically. This improved Copilot's understanding of our needs, resulting in more accurate outputs and a smoother, more reliable workflow. It understood the context of the query and presented appropriate attributes in response.

2. Selective Use of Synonyms

While synonyms can help users refer to fields in familiar terms, overuse can confuse the model. Best practices include:

Synonyms are intended for columns only and cannot be used for table names.
Add synonyms only for columns where users consistently use alternative names.
Regularly review and prune synonyms to prevent regressions in response quality.

3. Custom Tables Need to be Part of the App Module in the Relevant App that Copilot is Enabled for

Including custom tables in the app module is beneficial because it ensures that Copilot can access and reference these entities when processing natural language queries, which is essential for accurate data retrieval and relevant responses, especially when business processes rely heavily on custom data structures. This inclusion also allows for the application of glossary terms and synonyms to those tables, further improving Copilot’s understanding and reducing confusion with similarly named standard entities.

4. Iterative Testing and Network Trace Analysis

Testing prompts in real time and analyzing diagnostics data in network traces helps traces helps identify where Copilot is misinterpreting queries. This allows teams to:

Adjust glossary and synonym entries based on observed behaviour.
Validate that changes are having the intended effect by checking which glossary terms are being applied to each query.

5. Managing Knowledge Source Limitations

Copilot Studio currently limits the number of tables by fifteen for which synonyms and descriptions can be defined. Prioritize the most critical custom tables and fields, ensuring that the most business-relevant data is optimized first. For broader coverage, rely on glossary entries, which are not subject to the same limits.
Important Note: Adding a new knowledge source into Sales Copilot is not supported.

6. Troubleshooting

Diagnosing Query Accuracy Issues

Test with a Range of Prompts: Use both simple and complex queries to identify patterns in Copilot’s misinterpretations.
Analyze Network Traces: Review which tables and fields Copilot is querying. This helps pinpoint where confusion or errors occur.
Iterate on Glossary Entries: Refine glossary definitions to clarify entity meanings and reduce ambiguity.

Understanding the Query Evaluation Process

Accuracy issues or performance issues, such as slow or incomplete responses, often arise when queries span large timeframes or involve complex filters. For instance, queries over a year’s worth of data may time out, while those limited to a month return results quickly.

Narrow Query Scope: Encourage users to limit queries to shorter timeframes or smaller datasets.
Monitor Query Execution: Use network trace tools to identify where delays or failures occur.
Check network traces to troubleshoot query evaluation.
- Find QueryStructuredData

- Navigate to response.

- Check if the request failed.
  - Failure may be caused by the complexity of the query and result in a timeout.
  - Query complexity may be due to confusion around the user intent i.e. Joining two tables when the query was only around a single table. These types of issues can be mitigated by adding relevant instructions into the glossary.
- Once the relevant glossary instructions are added run the same prompt again and navigate to Network Traces à QueryStructuredData à Responses and scroll all the way down where you will see the original prompt and how the query is paraphrased by Copilot which will give you a good understanding of whether Copilot is able to understand user’s intent or not. In addition, this will also show you if the relevant glossary items have been taken into account while the query is being paraphrased.

This trace shows how Sales Copilot in Dynamics 365 (D365) interprets a user query and turns it into a backend request.

User Query
Input: “List me all the client meetings”

Glossary Mapping
"client meeting" → mapped to the cr387_clientmeeting table
It is not related to appointments, activities, or Teams meetings
Other glossary terms (not used here but available):
“my call” → phonecall table
“my emails” → email table
“my activity” → activitypointer table

Paraphrased Query
Reformulated as: “List all records from the cr387_clientmeeting table”

Paraphrase Steps
Identified the topic: client meetings
Removed unnecessary words
Mapped “client meeting” to the correct table
Reformulated the query using that mapping
Verified glossary relevance
Finalized the backend query

Practical Recommendations

Start with Glossary: Define your custom tables and their relationships in the glossary before adding synonyms.
Test, Observe, Refine: Use real-world prompts and network trace tools to see how Copilot interprets your data model.
Limit Synonyms: Only add synonyms where there is a clear, recurring need.
Document and Share Learnings: Keep a record of glossary and synonym changes and share best practices across teams.

Conclusion
Optimizing Copilot for custom tables is an iterative process that requires clear definitions, careful management of synonyms, and close collaboration between business, engineering, and support teams. By applying these lessons, organizations can unlock more accurate, reliable insights from their data—empowering users and driving better business outcomes.

Community site session details