In this age of data-driven decision-making, we’re more inclined than ever to seek metrics to steer the ship.

A few days ago, a prospective client asked, “How will we measure productivity after our AI training?”

It’s a good but tough question. And as your trusted AI consultant and trainer, I have no intention of dodging it. Let’s see what we can do.

First up: Published AI productivity metrics

Although the generative AI revolution has just begun, a number of studies already demonstrate how AI-supported work increases productivity.

Understand: It’s a process, not a switch

You don’t just flick on the AI switch. Adoption takes place gradually. Becoming proficient takes practice.

This comes from Boston Consulting Group, Harvard Business School & MIT (Sep 2023)

I’ve been introducing marketing professionals to these tools since February 2023 and have seen varying adoption rates—which makes measurement even more challenging.

Why hasn’t every marketer boarded the AI ship? Simple: they see this as another skill they have to learn. They’re busy. They feel they do well with their current processes and tools. They resist AI. The bottom line: not all marketers are early adopters.

Give AI adoption time

I believe we can and should track key performance metrics related to AI training and usage. I also recommend you give the process three to six months. Given the necessary ramp time, employees are far more likely to apply newly learned skills. Meaningful patterns will emerge and deliver useful data.

Measure what’s measurable

Let’s look at several example marketing use cases and KPIs you may try to track and align these KPIs with your business goals to ensure that AI integration is driving desired outcomes.

1. CONTENT CREATION

• Volume of Content: Compare the volume of new or upscaled sales-focused content produced before and after AI training.
• Quality: Assess quality through readability scores, SEO rankings, and engagement metrics such as email open rates, CTRs, and response rates to calls or messages including how prospects and customers engage with the sales content.
• Peer Review: Have a defined panel of peers, such as members from marketing or sales, rate the quality of AI-generated content versus human-written content using criteria like usefulness, accuracy, and readability.

2. SALES LEADS

• Lead Generation: For the sales reps using AI, compare the volume of leads generated with and without AI assistance.
• Sales Cycle Length: If your business has short sales cycles, compare the duration of these cycles with and without AI assistance. For longer cycles, consider a longitudinal study.

3. DATA MANIPULATION

• Efficiency: Compare time spent on data tasks like collection, preparation, analysis, and reporting with and without AI. Assess data volume and processing speed to gauge efficiency improvements.
• Accuracy: Evaluate the accuracy of data handling and reporting, considering false positives and false negatives when using AI for predictive modeling or anomaly detection.

For analyzing data, use ChatGPT’s Advanced Code Interpreter (paid Pro version) to interpret and summarize data from .txt, .csv and .xlsx files and plain text.

4. EMAIL CAMPAIGNS

• Open and Click-Through Rates: A/B test different versions of a campaign email – one version uses AI to generate subject lines/content, the other is human-created.
• Personalization: Evaluate the level of personalization and relevance of content to different segments of your audience.
• Automation Efficiency: Assess how efficiently campaigns are executed with AI aiding in segmentation, scheduling, and analytics.

5. ADVERTISING CAMPAIGNS

• Ad Copy & Display Ad Creation: Measure effectiveness in using ChatGPT for text ad creation and AI tools like MidJourney for display ads. Compare results with traditionally crafted materials.
• ROI and Ad Relevance: Use AI to boost ad relevance and lower CPC. Evaluate quality scores and interaction rates to measure AI’s impact on relevance and ROI.
• Time Efficiency: Analyze time saved in campaign management and ad optimization.
• Recommendations Optimization: Evaluate the effectiveness of AI tool recommendations in optimizing ad bids, placements, and targeting strategies.

6. CUSTOMER SUPPORT

• Inquiry Resolution: Compare customer satisfaction scores and inquiry resolution rates between customer support agents who are AI-supported and those who are not.

7. SOCIAL MEDIA POSTS

• Engagement: Compare engagement metrics—likes, shares, comments—on AI-generated and human-created posts targeting the same audience and topics.

8. GRAPHIC DESIGN

• Cost Savings: Assess cost savings using AI tools (e.g. MidJourney) vs stock imagery vs creative output
• Creativity: Evaluate the level of creativity in work produced, considering peer reviews or time to approvals as indicators.
• Speed: Analyze the time required to complete design projects with and without the use of AI tools.

9. TIME TRACKING FOR PRODUCTIVITY GAINS

• Time serves as a universal efficiency metric. Use time-tracking software and manually compare the data before and after AI integration in various departments.
• KPIs to Monitor:
  Time Saved: Track man-hours saved post-AI.
  Task Complexity: Assess if saved time led to tackling more complex tasks.
  Employee Satisfaction: Measure if saved time led to increased job satisfaction.

Factor in the intangibles

Some things, like motivation, are downright hard to measure. But that doesn’t mean they’re not meaningful. Case in point:

It’s hard to pinpoint improvements just from AI training without something to compare it to. But using these three methods together gives us a good view of how AI training is helping with work performance.

• Supervisor & Work Quality Assessment: Request supervisor feedback on employees’ post-training effectiveness and assess accuracy and completeness improvements in work pre- and post-AI training.
• Peer & Self-Assessment: Collect peer evaluations for insights on collaborative performance changes and self-assessments from employees for personal perception insights post-AI training.
• Post-Training Surveys: Conduct surveys post-training to measure employees’ confidence and perceived performance improvement.

Upskilling is a superpower

The topic of training employees was a consistent thread at the recent GAI World conference. It not only makes employees feel valued. The organization thrives because happier teammates deliver greater performance.

Fast-track AI adoption in your organization with these six straightforward next steps:

1. Set up an AI council

The Marketing AI Institute proposed a compelling concept: assemble a council of employees from marketing, sales, and customer support to work together to foster AI adoption. The council would accelerate adoption by focusing on the types of work that are well-suited to AI assistance. Note: the council should report regularly to senior management.

2. Create pilot programs

Start small and test what you suspect might be applications that are easy to put into play. For example, you could record and transcribe a week of sales calls and feed them into an AI platform. You’re likely to uncover valuable insights regarding pain points and go away with ideas for creating sales and marketing content to address them.

3. Establish a budget for AI tools

On multiple occasions, clients have told me they were denied a small pittance from the company (e.g. $20/month for ChatGPT Plus) to subscribe to generative AI tools. You can’t measure the productivity results of tools you don’t have.

It’s time to establish a budget for the employees’ use of AI. I’m hopeful the introduction of paid platforms such as ChatGPT Enterprise, Bing Chat Enterprise or Microsoft 365 Co-Pilot will drive progress on the adoption front.

4. Give employees time, too

You may remember when Google gave employees one day a week to work on their own projects. I propose that in addition to giving employees a small budget to use paid AI tools, give them the time to work with them. I estimate it takes 10 hours to work with AI tools to best understand how to use them effectively.

5. Get employee feedback

Early adopters on the teams I train are more than happy to share their positive experiences using AI tools. Recently, one person told me he used ChatGPT to translate one hundred pages of English content into Spanish, a task that would normally take months and a significant investment in a translation company.

My eager beaver client said,

“I put on my headset, sat down for a week, and created all the pages. The Spanish team claimed it was high quality.”

6. Celebrate wins

You can motivate those who aren’t using AI by giving employees a feedback loop to share their results. It’s also effective to highlight teams and individuals who have successfully adopted AI.

Use cases show peers the art of “the possible.” Make it happen in all-hands company meetings and at the department level.

We now have very strong evidence that AI elevates the skills of the lowest performers across a wide range of fields to, or even far above, what was previously average performance.

Source: Ethan Mollick

A little training can go a long way

If I could give you heaps of before-and-after scenarios to prove the worth of generative AI tools, I would. I have given you the results of some trustworthy studies, ideas to establish relevant metrics, and a list of ideas to measure results.

I’ll leave you with a short story about the value of getting trained and getting going…

An associate of mine told me, “I can’t find the time to learn about AI design tools.” I gave him our AI design training module. He was blown away and instantly understood a simple (but amazing) generative AI design tool could save the department the $150K/year they spend on stock photography.

How much time and money can AI save you?

Let’s talk about it. No charge. 30 minutes.

Here’s a brief overview of ToTheWeb’s AI consulting and training offering.

Training starts at $3,000/session for Marketers.