How Do You Measure the Productivity of AI Training?
In this age of data-driven decision-making, we’re more inclined than ever to seek metrics to steer the ship.
Accurately measuring AI productivity is crucial. This article explores effective strategies for evaluating AI tools’ effectiveness and assessing the impact of AI training programs.
A few days ago, a prospective client asked, “How will we measure productivity after our AI training?”
It’s a good but tough question. And as your trusted AI consultant and trainer, I have no intention of dodging it. Let’s see what we can do.
First up: Published AI productivity metrics
Although the generative AI revolution has just begun, a number of studies already demonstrate how AI-supported work increases productivity.
ChatGPT boosts productivity by 40% for professional writing tasks and reduces inequality between higher-skilled and lower-skilled workers.
An MIT team assessed speed, quality, and impact on work across 444 white-collar workers. The ChatGPT group was 37% faster (17 min. vs. 27 min.) with similar quality. As workers repeated tasks, the ChatGPT group’s quality improved significantly faster.
Science Magazine reported ChatGPT users are able to increase productivity and improve quality by 18%. They also enjoy the tasks more.
Lower-skilled workers benefited by with as much as a 35% increase in resolutions of customer support issues per hour, according to The National Bureau of Economic Research.
Understand: It’s a process, not a switch
You don’t just flick on the AI switch. Adoption takes place gradually. Becoming proficient takes practice.
This comes from Boston Consulting Group, Harvard Business School & MIT (Sep 2023)
A study into the impact of ChatGPT on efficiency and effectiveness, analyzed the workplace performance of 758 consultants from BCG’s offices globally, found that the use of ChatGPT:
- increased the quality of the work done by an average of 30%
- increased the quantity of the work done, increasing the rate of task completion by 25%
I’ve been introducing marketing professionals to these tools since February 2023 and have seen varying adoption rates—which makes measurement even more challenging.
Why hasn’t every marketer boarded the AI ship? Simple: they see this as another skill they have to learn. They’re busy. They feel they do well with their current processes and tools. They resist AI. The bottom line: not all marketers are early adopters.
Give AI adoption time
I believe we can and should track key performance metrics related to AI training and usage. I also recommend you give the process three to six months. Given the necessary ramp time, employees are far more likely to apply newly learned skills. Meaningful patterns will emerge and deliver useful data.
Measure what’s measurable
Let’s look at several example marketing use cases and KPIs you may try to track and align these KPIs with your business goals to ensure that AI integration is driving desired outcomes.
1. CONTENT CREATION
- Volume of Content: Compare the volume of new or upscaled sales-focused content produced before and after AI training.
- Quality: Assess quality through readability scores, SEO rankings, and engagement metrics such as email open rates, CTRs, and response rates to calls or messages including how prospects and customers engage with the sales content.
- Peer Review: Have a defined panel of peers, such as members from marketing or sales, rate the quality of AI-generated content versus human-written content using criteria like usefulness, accuracy, and readability.
2. SALES LEADS
- Lead Generation: For the sales reps using AI, compare the volume of leads generated with and without AI assistance.
- Sales Cycle Length: If your business has short sales cycles, compare the duration of these cycles with and without AI assistance. For longer cycles, consider a longitudinal study.
3. DATA MANIPULATION
- Efficiency: Compare time spent on data tasks like collection, preparation, analysis, and reporting with and without AI. Assess data volume and processing speed to gauge efficiency improvements.
- Accuracy: Evaluate the accuracy of data handling and reporting, considering false positives and false negatives when using AI for predictive modeling or anomaly detection.
For analyzing data, use ChatGPT’s Advanced Code Interpreter (paid Pro version) to interpret and summarize data from .txt, .csv and .xlsx files and plain text.
4. EMAIL CAMPAIGNS
- Open and Click-Through Rates: A/B test different versions of a campaign email – one version uses AI to generate subject lines/content, the other is human-created.
- Personalization: Evaluate the level of personalization and relevance of content to different segments of your audience.
- Automation Efficiency: Assess how efficiently campaigns are executed with AI aiding in segmentation, scheduling, and analytics.
5. ADVERTISING CAMPAIGNS
- Ad Copy & Display Ad Creation: Measure effectiveness in using ChatGPT for text ad creation and AI tools like MidJourney for display ads. Compare results with traditionally crafted materials.
- ROI and Ad Relevance: Use AI to boost ad relevance and lower CPC. Evaluate quality scores and interaction rates to measure AI’s impact on relevance and ROI.
- Time Efficiency: Analyze time saved in campaign management and ad optimization.
- Recommendations Optimization: Evaluate the effectiveness of AI tool recommendations in optimizing ad bids, placements, and targeting strategies.
6. CUSTOMER SUPPORT
- Inquiry Resolution: Compare customer satisfaction scores and inquiry resolution rates between customer support agents who are AI-supported and those who are not.
7. SOCIAL MEDIA POSTS
- Engagement: Compare engagement metrics—likes, shares, comments—on AI-generated and human-created posts targeting the same audience and topics.
8. GRAPHIC DESIGN
- Cost Savings: Assess cost savings using AI tools (e.g. MidJourney) vs stock imagery vs creative output
- Creativity: Evaluate the level of creativity in work produced, considering peer reviews or time to approvals as indicators.
- Speed: Analyze the time required to complete design projects with and without the use of AI tools.
9. TIME TRACKING FOR PRODUCTIVITY GAINS
- Time serves as a universal efficiency metric. Use time-tracking software and manually compare the data before and after AI integration in various departments.
- KPIs to Monitor:
Time Saved: Track man-hours saved post-AI.
Task Complexity: Assess if saved time led to tackling more complex tasks.
Employee Satisfaction: Measure if saved time led to increased job satisfaction.
Factor in the intangibles
Some things, like motivation, are downright hard to measure. But that doesn’t mean they’re not meaningful. Case in point:
It’s hard to pinpoint improvements just from AI training without something to compare it to. But using these three methods together gives us a good view of how AI training is helping with work performance.
- Supervisor & Work Quality Assessment: Request supervisor feedback on employees’ post-training effectiveness and assess accuracy and completeness improvements in work pre- and post-AI training.
- Peer & Self-Assessment: Collect peer evaluations for insights on collaborative performance changes and self-assessments from employees for personal perception insights post-AI training.
- Post-Training Surveys: Conduct surveys post-training to measure employees’ confidence and perceived performance improvement.
Upskilling is a superpower
The topic of training employees was a consistent thread at the recent GAI World conference. It not only makes employees feel valued. The organization thrives because happier teammates deliver greater performance.
Eighty-four percent of employees expect their employer to provide the training and education they need to stay up-to-date with changing skillsets in their industry, according to a recent survey of more than 800 C-suite executives and 800 employees, from online learning platform edX.
Fast-track AI adoption in your organization with these six straightforward next steps:
1. Set up an AI council
The Marketing AI Institute proposed a compelling concept: assemble a council of employees from marketing, sales, and customer support to work together to foster AI adoption. The council would accelerate adoption by focusing on the types of work that are well-suited to AI assistance. Note: the council should report regularly to senior management.
2. Create pilot programs
Start small and test what you suspect might be applications that are easy to put into play. For example, you could record and transcribe a week of sales calls and feed them into an AI platform. You’re likely to uncover valuable insights regarding pain points and go away with ideas for creating sales and marketing content to address them.
3. Establish a budget for AI tools
On multiple occasions, clients have told me they were denied a small pittance from the company (e.g. $20/month for ChatGPT Plus) to subscribe to generative AI tools. You can’t measure the productivity results of tools you don’t have.
It’s time to establish a budget for the employees’ use of AI. I’m hopeful the introduction of paid platforms such as ChatGPT Enterprise, Bing Chat Enterprise or Microsoft 365 Co-Pilot will drive progress on the adoption front.
4. Give employees time, too
You may remember when Google gave employees one day a week to work on their own projects. I propose that in addition to giving employees a small budget to use paid AI tools, give them the time to work with them. I estimate it takes 10 hours to work with AI tools to best understand how to use them effectively.
5. Get employee feedback
Early adopters on the teams I train are more than happy to share their positive experiences using AI tools. Recently, one person told me he used ChatGPT to translate one hundred pages of English content into Spanish, a task that would normally take months and a significant investment in a translation company.
My eager beaver client said,
“I put on my headset, sat down for a week, and created all the pages. The Spanish team claimed it was high quality.”
6. Celebrate wins
You can motivate those who aren’t using AI by giving employees a feedback loop to share their results. It’s also effective to highlight teams and individuals who have successfully adopted AI.
Use cases show peers the art of “the possible.” Make it happen in all-hands company meetings and at the department level.
We now have very strong evidence that AI elevates the skills of the lowest performers across a wide range of fields to, or even far above, what was previously average performance.
Source: Ethan Mollick
A little training can go a long way
If I could give you heaps of before-and-after scenarios to prove the worth of generative AI tools, I would. I have given you the results of some trustworthy studies, ideas to establish relevant metrics, and a list of ideas to measure results.
Nestle employees are saving an average of 45 minutes per week, creating better and faster content, and spending less time searching for information.
The impact of AI Tools
In a large-scale study involving over 6000 employees from more than 60 organizations, the impact of AI tools like Copilot was clear.
11% fewer emails read, 10% more documents edited, and up to 49% more key facts included in reports – these are just a few of the productivity gains observed in Microsoft’s latest 2024 research on generative AI in the workplace.
The Rise of AI Power Users
The survey showed that 78% of employees used AI tools, many of which were unsanctioned. Among them, 29% were classified as AI Power Users, saving more than 30 minutes a day through AI.
Acknowledging Microsoft's Role
This Microsoft-led study focuses on its own product, Copilot, but the findings offer valuable insights into the impact of generative AI on workplace productivity. Synthesizing over a dozen studies, including the largest randomized controlled trial of its kind, the report reveals significant but varied impacts of tools like Copilot across different roles and tasks.
- Generative AI Productivity Impact: Generative AI tools, such as Microsoft Copilot, have been shown to increase productivity in real-world settings, though the extent of the impact varies by role, function, and organization.
- Meeting Attendance: The effects on meeting attendance were mixed: some organizations saw an increase of 0.36 meetings per day (from 2 pre-Copilot meetings), while others saw a decrease of 0.39 meetings per day (from 3 pre-Copilot meetings).
- Employee Usage and AI Power Users:
- In a survey, 78% of respondents used unsanctioned AI tools.
- 29% of respondents were classified as “AI Power Users,” who saved more than 30 minutes a day using AI.
- Role-Specific Benefits:
- Customer service and sales professionals reported the highest productivity benefits from Copilot, with mean scores of 4.2 and 3.97 (out of 5) for increased productivity.
- Legal professionals reported the lowest productivity gains, with a mean score of 3.0.
- AI in Sales: In sales, using AI tools like a licensing chatbot improved task completion speed by 39% and accuracy by 25 percentage points.
I’ll leave you with a short story about the value of getting trained and getting going…
An associate of mine told me, “I can’t find the time to learn about AI design tools.” I gave him our AI design training module. He was blown away and instantly understood a simple (but amazing) generative AI design tool could save the department the $150K/year they spend on stock photography.
How much time and money can AI save you?
Let’s talk about it. No charge. 30 minutes.
Here’s a brief overview of ToTheWeb’s AI consulting and training offering.
Training starts at $3,000/session for Marketers.
Rosemary Brisco
AI Training Videos
AI Consulting and Training
Master today’s most effective productivity tools.
ToTheWeb ensures your route to capitalizing on the immense power of AI is fast and efficient. Our AI consulting and training programs will quickly enable your marketing team to adopt solutions to optimize operations and boost performance.
Learn about our generative AI and ChatGPT consulting and training programs