TIL: The AI Wall

Gen AI helps people in adjacent roles perform almost like experts, but it can't close the gap for people far from the domain. Here's what the AI wall means.

Very interesting HBR research/article on how Gen AI won't make employees experts in a domain or topic if they don't have the expertise or knowledge the task requires.

Gen AI Won’t Make Your Employees Experts
Generative AI can help workers perform unfamiliar tasks more quickly, but it doesn’t eliminate the performance gap between novices and experts. Researchers explored this dynamic by running a controlled writing experiment with employees at a fintech firm, dividing participants into three groups based on their level of relevant expertise. Each group conceptualized and drafted an article with and without AI assistance. Performance in the conceptualization stage was similar across groups when using AI, indicating that the technology helped people generate solid ideas even without deep experience. During the writing phase, however, differences emerged. The medium-expertise group nearly matched the experts when using AI, but the low-expertise group showed minimal improvement, suggesting that having experience in the task was crucial to interpreting and refining AI’s output. The findings imply that generative AI’s ability to upskill workers depends on the “expertise distance” between the person and the task. Organizations should pair AI tools with structured guidance, baseline training, and redesigned workflows, rather than assuming technology alone can turn novices into experts.

Key Takeaways:

  1. Expertise distance decides how much AI helps. The further someone is from the knowledge a task needs, the less AI closes the gap to real experts. People in adjacent roles (marketers writing articles) nearly matched the experts. People far from the domain (developers writing articles) got almost nothing from AI.
  2. "The AI wall." The researchers' term for the limit on how far gen AI can carry someone outside their expertise. It pushes back on the idea that AI will flatten skill hierarchies and let anyone do any task.
  3. Judging a task is easier than doing it. AI helped everyone come up with article ideas, because that only needs you to spot a good one. Writing the article needs craft. Their metaphor: imagining a marathon versus running one.
  4. Your judgement is what makes AI useful. You need enough domain knowledge to decide what to keep, change or throw away. People without it just copy and paste what the AI gives them.
  5. Don't overrate AI when it can't do the whole task. When it can't fully automate the work, it only narrows the gap to experts in some cases. It isn't a fix for everything.
  6. Redesign the work, not just the tools. AI can blur boundaries between nearby roles, like SEO and content strategy. Bridging distant functions like marketing, sales, and product is much harder because each has its own expertise, budget, and power. That takes structural and cultural change.
  7. Deploy with the human context in mind. Ask who is using the tool, what they know, and how well they can interpret and refine its output.
  8. AI speeds up the path to expertise but doesn't replace experience. In the interview, Olga Pirog (formerly of IG Group) says it only helps if you're "already in the neighbourhood of that domain."
  9. Hiring fewer juniors damages the talent pipeline. If you only hire experts to edit AI output, nobody builds the taste and judgement that come from doing the work.
  10. Teach judgement, not just execution. Training should focus on what makes work good, so people can tell whether the AI is right.
Join AI engineers getting weekly insights on agents, RAG & production LLM systems

No spam, no sharing to third party. Only you and me.

Member discussion