Microsoft's new AI assistants under the Microsoft Copilot brand are being hailed as game changers – and rightly so. However, in order for the technology to maximise its impact, it needs to be used correctly, and to do this you need to know how it works. In this article, we shed some light on the subject.
The answer seems simple: Microsoft uses the term "Copilot" to denote AI assistants designed to help users in their day-to-day work, anywhere possible: there are now Copilots for Office, sales, ERP and CRM or service, for developers, security specialists and more.
But not all Copilots are the same. What sounds confusing at first can be easily resolved if we take a look at the Copilot history. It all started with Microsoft's partnership with OpenAI.Since the AI company was founded in 2015 as a donation-funded non-profit organisation, Microsoft has supported OpenAI, primarily with computing resources. Then, in 2019 and 2023, Microsoft invested billions of dollars and entered into a partnership to jointly advance AI and cloud supercomputing technologies. OpenAI is best known for the performance of its generative language models (Large Language Models, LLMs) of the GPT family and for the AI chatbot ChatGPT (GPT stands for "Generative Pre-Trained Transformer").
As part of this partnership, Microsoft is integrating AI solutions from OpenAI into more and more of its products, on the one hand with Azure OpenAI Service and on the other with special tools – above all the Copilots. Microsoft has an exclusive licence to use the code from GPT (and not just the API) for its own solutions.
The first Microsoft-related "Copilot" product was the GitHub Copilot from 2021. However, this was not yet a real Copilot in the sense that Microsoft understands the concept today. In fact, the first version of GitHub Copilot ran with the GPT-3 derivative "Codex", which was not yet chat-enabled, and had no access to external data – both defining features of the "Copilot system" (source: Microsoft).
The first actual incarnation of the "copilot system" was Bing Chat, released in February 2023. Bing Chat was trained with an early version of GPT-4, which impressed the developers with its performance, but still showed the usual limitations of AI language models: Although they are trained with huge amounts of data, especially from the Internet, they only have a statistical and no semantic (meaning-related) understanding of it and also know very little about the context of a specific request. This is why they often give irrelevant, incorrect or overly "creative" answers (they "hallucinate"). And last but not least, LLMs know nothing about events that occur after their training has been completed (for OpenAI models up to and including GPT-4 this was September 2021, for the now newest model GPT-4-Turbo April 2023).
Microsoft's Copilot is designed to overcome these limitations.
To this end, Microsoft and OpenAI developed a technology called "Prometheus" for Bing Chat. Prometheus aimed to give the AI access to the current and comprehensive web search and ranking information in the Bing backend, thereby combining the performance of a leading LLM with the performance of a leading web index. Microsoft calls this "grounding" - the AI is "earthed" with current, correct and relevant data.
The next stage of the Copilot system – and this is what we want to talk about in the rest of the article – came with Copilot for Microsoft 365, which is available in all Microsoft 365 apps (and across applications in Microsoft 365 Chat). It combines these applications, such as Word or Teams, with the language models of OpenAI, with the Microsoft Graph data integration platform and via Microsoft Graph with the Semantic Index for Copilot. Connectors in Microsoft Graph can connect the AI securely and privacy-compliant with all potentially relevant data sources: company databases, external sources and the Internet. Microsoft's Copilot system takes care of orchestrating all these services to make the AI's response as useful as possible.
You may have heard of prompt engineering, one of the most important techniques for influencing the behaviour of AI models and improving their output. Background: Once the training of an AI base model has been completed, there are only two ways to provide it with new data and teach it new skills: firstly, fine-tuning and secondly, special input in the request (the "prompt"). The prompt is of particular importance, as this is the place where you can tell your model exactly what you need from it. Techniques for optimising the input are known as prompt engineering.
Anyone who has already gained experience with LLMs will have noticed that even minor changes to the prompt, such as clearer instructions, more precise descriptions of the expected behaviour and examples provided, can bring about considerable quality improvements for the model's response. The Copilot system perfects prompt engineering through grounding. For this, it can use not only the latest data from the Internet (as with Prometheus), but also all company data that the person making the enquiry is authorised to access, for example Office documents, emails, data from social networks or SharePoint sites (Microsoft Graph takes care of this).
And how does Copilot decide which of this wealth of data is really relevant to a specific query? With the help of the Semantic Index for Copilot: This solution uses a modern type of data management (vector database) and its own AI to weight semantic similarities between a search query and the content of documents and other files – all the data that Microsoft Graph offers in a search. The most relevant content parts for the context of a query are added to the prompt and this is sent to the LLM. (The principle behind this type of prompt engineering is called RAG - Retrieval Augmented Generation). The response from the AI is also processed again and checked in particular for compliance, data protection and responsible AI use (source: Microsoft).
There are various aspects to consider when using Copilot: Strategic, licensing, organisational and technical issues need to be clarified in order to make the most of opportunities and minimise risks (the SoftwareOne Copilot Advisory Service can support you with this). At this point, one particularly important technical aspect is to be focussed on: Data.
Data of the highest possible quality is crucial for the performance of any AI – here, too, applies the old computer scientist adage "garbage in, garbage out". After all, Microsoft Copilot has to work with what you make available to it, i.e. what it finds in the prompt of a user request. (Fine-tuning is not available for the Copilot LLM.) The good news is that you have various options for optimising Copilots for your requirements – we'll tell you the four most important ones:
Tip 1: Make relevant data accessible to Copilot. Make sure that Copilot, within the given user context, has access to exactly (and only) the data that is relevant to the work context of a request (you can read more about this in our blog post Getting ready for Microsoft 365 Copilot – It’s all about data).
Tip 2: Expand the knowledge base of your Copilot if required. By default, it can only access data in Microsoft 365 via Microsoft Graph. With the help of Microsoft Graph connectors, you can also make external data accessible, e.g. from local or Azure databases, wikis, Jira, Salesforce or customised sources.
Tip 3: Extend the skills of your Copilot with plugins. Plugins allow you to retrieve data from external services in real time directly via their API (Application Programming Interface). They can be easily created in various ways. For example, if you have set up message extensions in Microsoft Teams, you can use these as Copilot plugins.
Tip 4: Create customised Copilots for your usage scenarios. Microsoft Copilot Studio is available for this purpose in Copilot for Microsoft 365. Microsoft Copilot Studio is a low-code tool, which means you don't need any programming knowledge to create and customise copilots. Among other things, you can use it to define relevant topics for your Copilot, connect various data sources, use plugins and modify prompt instructions.
Under the hood of Microsoft Copilot, it’s all about prompt engineering. Once you understand the principle, there are numerous ways to customise your Copilots to your own requirements. At the same time, your users don't have to deal with the intricacies of prompt engineering – behind the scenes, the Copilot system ensures that they still get exactly the help they need from their "everyday AI companion".
AI applications such as Microsoft Copilot are a key element of the future workplace. But not the only one. Find out more in our whitepaper.
AI applications such as Microsoft Copilot are a key element of the future workplace. But not the only one. Find out more in our whitepaper.
