Gov Copilot Experiment Encounters Challenges Due to Usage Issues and Unfulfilled Anticipations
We independently review everything we recommend. When you buy through our links, we may earn a commission which is paid directly to our Australia-based writers, editors, and support staff. Thank you for your support!
Challenges in the Microsoft 365 Copilot Trial for Australian Government
Summary
- Two-thirds of the participants in the Microsoft 365 Copilot federal government trial engaged with the tool “a few times a week” or less.
- Only one-third of the participants accessed Copilot daily, mainly for summarising meetings and rewriting documents.
- Challenges included insufficient user interface clarity and outdated Microsoft Outlook versions.
- Expectations from participants were largely unfulfilled, leading to a decrease in positive perceptions regarding the tool’s efficiency in saving time.
- Training was linked to the frequency of tool usage, yet many staff members found it challenging to make time for this training.
- Concerns were expressed about AI-generated meeting transcriptions being exposed to Freedom of Information requests.
Limited Usage Despite High Hopes
A report from the Digital Transformation Agency (DTA) has provided insights into the six-month trial of Microsoft 365 Copilot within the Australian federal government. Out of around 5765 licences deployed, it was noted that only a third of the participants used Copilot on a daily basis, while two-thirds engaged with the tool “a few times a week” or less.
The main uses for Copilot included summarising meetings, rewriting content, and providing information. Despite the anticipation surrounding generative AI tools, the trial indicated that many participants felt their expectations were not met, resulting in diminished perceived value after the initial excitement.
Limitations of Self-Reporting and Executive Bias
A significant drawback of the evaluation was its dependency on user self-assessments, which may have influenced the results’ objectivity. Additionally, the trial saw a higher representation of executives, potentially skewing the overall findings.
Nevertheless, the report presents important insights regarding how federal agencies in Australia are incorporating AI tools like Copilot. Despite moderate usage, the findings indicate persistent barriers, particularly related to user interface and accessibility.
Link Between Training and Utilisation
The evaluation identified a distinct correlation between the level of training received and how frequently users engaged with Copilot. The more that training was specifically adapted for the Australian Public Service (APS) environment, the more the tool was utilised.
However, numerous staff found it difficult to carve out time for training amid their work schedules. For those lacking adequate training, the tool often appeared unwieldy as users realized that editing and validating Copilot’s results took longer compared to completing tasks manually.
User Interface Challenges Affect Adoption
One unexpected finding was the frequency with which trial participants overlooked the integration of Copilot within Microsoft 365 applications. For instance, at the CSIRO, many users simply forgot about Copilot’s existence due to the lack of clarity in the user interface.
Focus groups indicated that because the features of Copilot were not immediately apparent, users missed out on recording meetings for transcription or leveraging other beneficial functions. Internal CSIRO research highlighted that the integration with existing Microsoft workflows—considered one of the tool’s significant strengths—was largely undermined due to its lack of visibility.
Varied Experiences Across Microsoft Applications
Users’ experiences with Copilot varied across different Microsoft applications. Those anticipating smoother Excel analysis found themselves disappointed, while others hoping for better Outlook integration were let down when it became clear that their organization used an outdated version of Outlook incompatible with Copilot.
While these technical issues may not solely fall on Microsoft, they complicate the decision of whether Copilot represents a worthwhile long-term investment, particularly when updates are necessary for full functionality.
High Hopes, Disappointing Outcomes
The report emphasizes that participants began the trial with elevated expectations, influenced by marketing that implied Copilot would greatly decrease the time spent on emails and meetings. Unfortunately, these expectations were not realised.
A 32% decline was observed in the belief that Copilot helped save time on emails, alongside a 54% reduction in expectations regarding fewer meetings. Although initial sentiments were optimistic, it soon became evident that the tool did not meet the inflated expectations.
Worries About AI-Generated Meeting Records
A major concern that surfaced during the trial was the possible legal consequences of employing AI for transcribing meeting discussions. Some participants expressed apprehensions that comprehensive AI-produced meeting transcripts might be subject to Freedom of Information (FOI) requests, potentially hindering frank discussions in meetings.
The report advises federal agencies to carefully weigh the ramifications of using generative AI technologies like Copilot, especially within sensitive contexts where FOI requests could introduce risks.
Conclusion
The six-month examination of Microsoft 365 Copilot within the Australian government has displayed varied outcomes. While certain users recognized the platform’s utility, especially in meeting summaries, the trial fell short of the high initial expectations. Primary issues included insufficient user interface visibility, outdated software versions, and challenges in training participation. The evaluation also surfaced concerns regarding AI-generated meeting records being susceptible to Freedom of Information inquiries, which could restrict open communication. For now, it appears Copilot has yet to fully demonstrate its worth within the federal government framework.
Q: What was the primary goal of the Microsoft 365 Copilot trial?
A:
The trial aimed to evaluate the effectiveness and practical value of Microsoft 365 Copilot in assisting federal government employees with tasks such as summarising meetings, rewriting documents, and managing emails. It sought to determine whether Copilot could fulfill its marketing commitments to enhance productivity and save time.
Q: What led to the unmet expectations of the trial participants?
A:
Participants entered the trial with lofty expectations that Copilot, alongside broader generative AI tools, would considerably lessen the time spent on tasks like email management and meeting attendance. Nevertheless, many users found that the outputs required extensive verification and editing, and the tool’s presence within the Microsoft 365 suite was not always evident, resulting in decreased usage and unmet anticipations.
Q: Were there any technical complications that affected the trial?
A:
Yes, several technical challenges impacted the trial. For example, some users were dissatisfied with Copilot’s performance in Excel, while others couldn’t harness its full capabilities in Outlook due to their organization using a less current software version. Additionally, the tool’s lack of visibility within the Microsoft 365 suite was a critical issue, causing users to frequently overlook Copilot’s availability.
Q: How did training affect Copilot usage during the trial?
A:
A clear relationship was found between the level of training users completed and their frequency of Copilot usage. When the training was customized to the context of the Australian Public Service, usage levels improved. Nonetheless, many staff members found it difficult to dedicate time to training, which limited the potential benefits.
Q: What concerns were associated with AI-generated meeting records?
A:
Participants voiced concerns that AI-generated meeting transcripts could be subject to Freedom of Information (FOI) requests. This consideration could hinder candid conversations during meetings, as participants might be more guarded about their statements, aware that detailed transcripts could become public through FOI legislation.
Q: What are the forthcoming actions for the Australian government concerning Copilot?
A:
The report does not specify concrete recommendations for incorporating Copilot into regular government operations. However, it suggests federal agencies thoughtfully consider the consequences of using generative AI tools and provide clearer guidance on the potential legal and ethical issues, particularly concerning matters like FOI requests.