INSIGHTS
Challenges and opportunities: Redefining metrics in modern L&D with AI

AI is no longer a looming concept in learning and development (L&D). It’s here. Fifty-eight percent¹ of HR teams report a revenue increase from AI adoption.

As we embrace tech evolution in the L&D space, AI-powered data analytics is becoming a major driver behind modern L&D initiatives. Today, analytics is no longer a mere way of measuring the impact of corporate learning (yet AI transforms learning analytics, too) but a basis for data-driven learning pathways.

Let’s delve into how AI-powered analytics is transforming the way we interpret and use data, and why it’s critical in 2024.

Challenges and limitations in traditional methods of measuring learning effectiveness

Traditional L&D analytics methods aren’t keeping up with the changing needs and goals of modern organizations, creating the need to redefine our strategies for measuring learning impact.

Metrics don’t match with modern L&D goals

Today’s L&D pros continue to use vanity metrics to measure the impact of employee training. Among the top indicators² used for evaluating L&D success are employee satisfaction, the number of employees involved in training, post-learning assessment scores, and the number of courses completed.

The problem is, that while learning analytics may help to grasp employee engagement in learning programs, it doesn’t reflect the contribution of L&D to your business performance.

Struggling to measure the impact of employee training, 95% of L&D leaders³ want to be able to track the correlation between employee improvement and business performance. Given that traditional data analysis methods aren’t helping to bridge the gap, AI-powered analytics has a strong potential to change our approach to measuring L&D impact.

“I feel the more important issue is how L&D appears to be approaching the potential of AI primarily as a mechanism to produce more formal learning content.

L&D would be better served by AI if we were to apply it to bringing learning and working together, and using business analytics as the driver – not learning analytics.

The huge potential for AI to drive ‘performance support on steroids’ and bring learning and continuous improvement into the workflow (adopting a performance mindset rather than a learning mindset) is enormous. It will be far greater than simply using it as a content/analytics engine in a 20th-century content-centric L&D model.”

Charles Jennings

Managing Director | Duntroon Consultants

Too much data, too few resources

Every day, we aggregate plenty of information from HR systems, employee surveys, and learning management system (LMS) reports. However, there’s little we can do with all this data; so, we let it gather dust while focusing on vanity metrics (because it’s easy to interpret them).

One of the most common ways to interpret large amounts of data is by building dashboards that must tell something about L&D performance. But what many fail to do is to turn data analytics into a story that will encourage action from stakeholders.

“Data and analytics teams have always created dashboards and visualizations, but many are unfamiliar with wrapping those artifacts into a narrative.”

James Richardson

Senior Director Analyst | Gartner⁴

Accessible data ≠ meaningful data

What prevents us from collecting the data that reflects the real results of the training delivered? It’s not that L&D teams intentionally select the wrong metrics. It’s data accessibility that drives us toward poor decisions.

“[…] when we rely on data that is easy to collect rather than data that is meaningful to collect, we head for trouble.

As an MBA student, I learned of the classic salesperson problem where we collect data on the amount of revenues a salesperson brings in, forgetting to measure the amount of profit the salesperson generates. By measuring one and not the other, we reward sales while hoping for profits. Would it be rude of me to point out that your compensation is likely tied to some things that are easier to measure (like current profits or stock price) in lieu of other things that are important to measure (like the strength and innovation of your R&D team)?”

Will Thalheimer

Consultant, Speaker, and Researcher | Work-Learning Research⁵

AI changes the way we educate and evaluate

As traditional solutions are incapable of solving major L&D challenges, AI brings in new means of data analytics (very timely).

Shifting from measuring learners’ engagement, AI-powered analytics unlocks access to complex datasets, big data, and machine learning algorithms. With these in our arsenal, we’ll minimize manual work, get meaningful insights into the effectiveness of our training efforts, and enhance the personalization of L&D programs.

Use cases of AI in learning and development in 2024

AI-powered analytics is transforming learning and development in 2024, and L&D professionals will be busy implementing these seven applications of AI within organizational learning. Here’s how.

1. Training content creation

Less than a year after the debut of generative AI technology, one-third⁶ of organizations report at least one of their business functions uses gen AI regularly.

Generative AI tools can analyze vast amounts of data to generate training materials in no time. From transforming PDFs to engaging content to translating training programs into multiple languages to AI voicing of written text — all these tasks are already handled by AI in Attensi products.

With the help of Attensi’s PROCESS solution, Equiniti has built 15 playable training modules and achieved a 3x return on investment in reduced training times and staff churn. Learn more in the case study.

2. Gen AI-based information search

Forty-seven percent⁷ of digital workers struggle to find the information necessary to perform their tasks successfully. In the abundance of information and applications used in the workplace, employees waste hours searching for the data they need. AI-powered search is going to fix that.

McKinsey has recently launched “Lili,”⁸ the company’s own generative AI solution that singles out the necessary information from the firm’s extensive knowledge and speeds up research. When a user types in a question, Lili scans McKinsey’s entire knowledge base and identifies from five to seven pieces of content. She then provides the key points, including links and even experts in the appropriate fields in the resulting file — eliminating the need to go through thousands of documents manually.

And yes, you can also access AI-generated answers in Attensi.

3. AI-powered skill gap assessment

Even though we’ve been using HR technology for skill gap analysis for quite some time, it still requires a lot of manual effort. AI analytics helps to automate manual skill mapping tasks and close these gaps more effectively.

By analyzing large datasets across various sources, AI ensures a comprehensive analysis of skills across different roles and functions, allowing L&D specialists to not only address skill gaps but anticipate those.

4. Data interpretation

With the rise of generative AI, L&D professionals access actionable insights without wading through graphs, pie charts, and scatter plots. Due to its ability to transform data into rich and helpful narratives, AI lets us finally turn large datasets into actionable plans.

5. Internal mobility and employee development

Modern AI solutions are capable of tailoring resources and learning paths to foster employees’ career growth and support internal mobility. Using historical data, AI can predict potential career paths based on an employee’s performance and aspirations. AI algorithms can also analyze job requirements and match them with employee profiles, suggesting suitable internal job openings to individuals who have the necessary skills.

For instance, to empower employees with growth opportunities, Spotify has launched the Internal Talent Marketplace, called Echo.⁹ The AI-powered technology helps employees develop their skills during their tenure at Spotify, build up their career portfolios, and find career advancement opportunities through the marketplace.

6. Feedback analysis

Learner satisfaction scores hardly help gauge the effectiveness of employee training. Instead of solely relying on subjective impressions, AI-driven feedback analysis delves deeper into the qualitative aspects of learner feedback.

Natural Language Processing (NLP) algorithms can automatically analyze learners’ responses, uncovering insight into not only what learners liked or disliked but also why certain aspects resonated positively or posed challenges.

7. Personalization of learning experiences

As traditional one-size-fits-all training methods are failing to meet individual learners’ needs, AI has come to the rescue.

AI’s distinctive capability to effortlessly adapt to the specific needs of each learner helps to create personalized learning paths, ensuring that each employee meets their own potential and interests. AI-driven personalization empowers learners to take ownership of their upskilling or re-skilling, exploring their own trajectories of lifelong learning.

Among the most common examples of AI-driven learning experiences in action are microlearning, AR and VR simulations, mini-games, and AI-driven personalities.

Attensi’s immersive learning scenarios encourage employees to learn by doing through realistic and impactful training tailored to their needs. By using AI, we create personalized learning paths that cater to individual needs, optimizing knowledge retention and facilitating the practical application of skills. The resulting learning experience immediately contributes to measurable improvements in performance metrics.

Considerations for implementing AI-powered analytics

When entrusting AI with data, mind ethical and regulatory responsibilities:

1. Ethical implications of AI usage

While AI holds tremendous potential for corporate learning and development, it also brings a threat of bias and ethical issues with its implementation. We need to remember that this technology is designed to complement human skills, not replace them.

AI is not here to make unsupervised decisions but to help us collect and optimize data that we can combine with ethical, moral, and other human considerations before taking any action.

Bryan Hancock, partner at McKinsey, has shared¹⁰ that in his role as a McKinsey evaluator, he combines AI and human intelligence for performance reviews:

“One of my personal favorite uses for generative AI on the people front is actually for performance reviews. Hear me out: I don’t want generative AI actually generating somebody’s performance review. That needs the human in the loop, needs human judgment, needs empathy.”

Hancock explains that generative AI helps him turn written feedback on each individual into a first draft, from where he starts a manual review process. Such a synthesis helps him save time on manual data organization and “get more quickly to what I really need to probe for that person’s development and growth.”

2. Privacy regulations and AI

Combining AI and human intelligence isn’t just beneficial but is soon going to be mandatory across the globe. For the first time in the world’s history, the use of AI will be regulated by law. The EU parliament has approved the AI Act establishing guidelines and transparency requirements for all parties involved in the development or use of AI.

The new regulation states that “AI systems should be overseen by people, rather than by automation, to prevent harmful outcomes.”

The EU is the first to announce a comprehensive AI law, but it looks like the rest of the world will join¹¹ in the next few years. And even though AI laws are yet to be enacted, we already have other privacy policies to adhere to:

“Asking Generative AI, for example, to summarize and perform statistics on private or sensitive data, may mean breaking the law (such as HIPAA laws for medical data or GDPR laws, for personal data, for example). At a minimum, sharing personal information is likely unethical. If your data is private or sensitive, you must find out if you can ethically use Generative AI for your desired purposes.”

Patti Shank

Author and Founder | Learning Peaks

AI enhances learning through data analytics

AI-powered analytics not only helps you measure the impact of L&D programs but also transforms vast datasets into actionable narratives, providing personalized learning paths for trainees.

As we navigate the benefits of AI, let’s not forget the importance of human judgment in the decision-making process. The balanced integration of AI and human expertise and empathy is the key to unlocking the full potential of learning and development.

Book a demo with Attensi to see how AI-powered learning scenarios will fit in your organization.