User Experience Design for Chatbots

Over the last year, I’ve seen many people fall into the same trap: They launch an AI-powered agent (chatbot, assistant, support tool, etc.)… But only track surface-level KPIs — like response time or number of users. That’s not enough. To create AI systems that actually deliver value, we need 𝗵𝗼𝗹𝗶𝘀𝘁𝗶𝗰, 𝗵𝘂𝗺𝗮𝗻-𝗰𝗲𝗻𝘁𝗿𝗶𝗰 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 that reflect: • User trust • Task success • Business impact • Experience quality    This infographic highlights 15 𝘦𝘴𝘴𝘦𝘯𝘵𝘪𝘢𝘭 dimensions to consider: ↳ 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆 — Are your AI answers actually useful and correct? ↳ 𝗧𝗮𝘀𝗸 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗶𝗼𝗻 𝗥𝗮𝘁𝗲 — Can the agent complete full workflows, not just answer trivia? ↳ 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 — Response speed still matters, especially in production. ↳ 𝗨𝘀𝗲𝗿 𝗘𝗻𝗴𝗮𝗴𝗲𝗺𝗲𝗻𝘁 — How often are users returning or interacting meaningfully? ↳ 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 𝗥𝗮𝘁𝗲 — Did the user achieve their goal? This is your north star. ↳ 𝗘𝗿𝗿𝗼𝗿 𝗥𝗮𝘁𝗲 — Irrelevant or wrong responses? That’s friction. ↳ 𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗗𝘂𝗿𝗮𝘁𝗶𝗼𝗻 — Longer isn’t always better — it depends on the goal. ↳ 𝗨𝘀𝗲𝗿 𝗥𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻 — Are users coming back 𝘢𝘧𝘵𝘦𝘳 the first experience? ↳ 𝗖𝗼𝘀𝘁 𝗽𝗲𝗿 𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻 — Especially critical at scale. Budget-wise agents win. ↳ 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻 𝗗𝗲𝗽𝘁𝗵 — Can the agent handle follow-ups and multi-turn dialogue? ↳ 𝗨𝘀𝗲𝗿 𝗦𝗮𝘁𝗶𝘀𝗳𝗮𝗰𝘁𝗶𝗼𝗻 𝗦𝗰𝗼𝗿𝗲 — Feedback from actual users is gold. ↳ 𝗖𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 — Can your AI 𝘳𝘦𝘮𝘦𝘮𝘣𝘦𝘳 𝘢𝘯𝘥 𝘳𝘦𝘧𝘦𝘳 to earlier inputs? ↳ 𝗦𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 — Can it handle volume 𝘸𝘪𝘵𝘩𝘰𝘶𝘵 degrading performance? ↳ 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 — This is key for RAG-based agents. ↳ 𝗔𝗱𝗮𝗽𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗦𝗰𝗼𝗿𝗲 — Is your AI learning and improving over time? If you're building or managing AI agents — bookmark this. Whether it's a support bot, GenAI assistant, or a multi-agent system — these are the metrics that will shape real-world success. 𝗗𝗶𝗱 𝗜 𝗺𝗶𝘀𝘀 𝗮𝗻𝘆 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗼𝗻𝗲𝘀 𝘆𝗼𝘂 𝘂𝘀𝗲 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀? Let’s make this list even stronger — drop your thoughts 👇

📊 What’s the right KPI to measure an AI agent’s performance? Here’s the trap: most companies still measure the wrong thing. They track activity (tasks completed, chats answered) instead of impact. Based on my experience, effective measurement is multi-dimensional. Think of it as six lenses: 1️⃣ Accuracy – Is the agent correct? Response accuracy (right answers) Intent recognition accuracy (did it understand the ask?) 2️⃣ Efficiency – Is it fast and smooth? Response time Task completion rate (fully autonomous vs guided vs human takeover) 3️⃣ Reliability – Is it stable over time? Uptime & availability Error rate 4️⃣ User Experience & Engagement – Do people trust and return? CSAT (outcome + interaction + confidence) Repeat usage rate Friction metrics (repeats, clarifying questions, misunderstandings) 5️⃣ Learning & Adaptability – Does it get better? Improvement over time Adaptation speed to new data/conditions Retraining frequency & impact 6️⃣ Business Outcomes – Does it move the needle? Conversion & revenue impact Cost per interaction & ROI Strategic goal contribution (retention, compliance, expansion) Gartner predicts that by 2027, 60% of business leaders will rely on AI agents to make critical decisions. If that’s true, then measuring them right is existential. So, here’s the debate: Should AI agents be held to the same KPIs as humans (outcomes, growth, value) — or do they need an entirely new framework? 👉 If you had to pick ONE metric tomorrow, what would you measure first? #AI #Agents #KPIs #FutureOfWork #BusinessValue #Productivity #DecisionMaking

AI products like Cursor, Bolt and Replit are shattering growth records not because they're "AI agents". Or because they've got impossibly small teams (although that's cool to see 👀). It's because they've mastered the user experience around AI, somehow balancing pro-like capabilities with B2C-like UI. This is product-led growth on steroids. Yaakov Carno tried the most viral AI products he could get his hands on. Here are the surprising patterns he found: (Don't miss the full breakdown in today's bonus Growth Unhinged: https://lnkd.in/ehk3rUTa) 1. Their AI doesn't feel like a black box. Pro-tips from the best: - Show step-by-step visibility into AI processes - Let users ask, “Why did AI do that?” - Use visual explanations to build trust. 2. Users don’t need better AI—they need better ways to talk to it. Pro-tips from the best: - Offer pre-built prompt templates to guide users. - Provide multiple interaction modes (guided, manual, hybrid). - Let AI suggest better inputs ("enhance prompt") before executing an action. 3. The AI works with you, not just for you. Pro-tips from the best: - Design AI tools to be interactive, not just output-driven. - Provide different modes for different types of collaboration. - Let users refine and iterate on AI results easily. 4. Let users see (& edit) the outcome before it's irreversible. Pro-tips from the best: - Allow users to test AI features before full commitment (many let you use it without even creating an account). - Provide preview or undo options before executing AI changes. - Offer exploratory onboarding experiences to build trust. 5. The AI weaves into your workflow, it doesn't interrupt it. Pro-tips from the best: - Provide simple accept/reject mechanisms for AI suggestions. - Design seamless transitions between AI interactions. - Prioritize the user’s context to avoid workflow disruptions. -- The TL;DR: Having "AI" isn’t the differentiator anymore—great UX is. Pardon the Sunday interruption & hope you enjoyed this post as much as I did 🙏 #ai #genai #ux #plg

In the rapidly evolving world of conversational AI, Large Language Model (LLM) based chatbots have become indispensable across industries, powering everything from customer support to virtual assistants. However, evaluating their effectiveness is no simple task, as human language is inherently complex, ambiguous, and context-dependent. In a recent blog post, Microsoft's Data Science team outlined key performance metrics designed to assess chatbot performance comprehensively. Chatbot evaluation can be broadly categorized into two key areas: search performance and LLM-specific metrics. On the search front, one critical factor is retrieval stability, which ensures that slight variations in user input do not drastically change the chatbot's search results. Another vital aspect is search relevance, which can be measured through multiple approaches, such as comparing chatbot responses against a ground truth dataset or conducting A/B tests to evaluate how well the retrieved information aligns with user intent. Beyond search performance, chatbot evaluation must also account for LLM-specific metrics, which focus on how well the model generates responses. These include: - Task Completion: Measures the chatbot's ability to accurately interpret and fulfill user requests. A high-performing chatbot should successfully execute tasks, such as setting reminders or providing step-by-step instructions. - Intelligence: Assesses coherence, contextual awareness, and the depth of responses. A chatbot should go beyond surface-level answers and demonstrate reasoning and adaptability. - Relevance: Evaluate whether the chatbot’s responses are appropriate, clear, and aligned with user expectations in terms of tone, clarity, and courtesy. - Hallucination: Ensures that the chatbot’s responses are factually accurate and grounded in reliable data, minimizing misinformation and misleading statements. Effectively evaluating LLM-based chatbots requires a holistic, multi-dimensional approach that integrates search performance and LLM-generated response quality. By considering these diverse metrics, developers can refine chatbot behavior, enhance user interactions, and build AI-driven conversational systems that are not only intelligent but also reliable and trustworthy. #DataScience #MachineLearning #LLM #Evaluation #Metrics #SnacksWeeklyonDataScience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gj6aPBBY    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gAC8eXmy

AI models like ChatGPT and Claude are powerful, but they aren’t perfect. They can sometimes produce inaccurate, biased, or misleading answers due to issues related to data quality, training methods, prompt handling, context management, and system deployment. These problems arise from the complex interaction between model design, user input, and infrastructure. Here are the main factors that explain why incorrect outputs occur: 1. Model Training Limitations AI relies on the data it is trained on. Gaps, outdated information, or insufficient coverage of niche topics lead to shallow reasoning, overfitting to common patterns, and poor handling of rare scenarios. 2. Bias & Hallucination Issues Models can reflect social biases or create “hallucinations,” which are confident but false details. This leads to made-up facts, skewed statistics, or misleading narratives. 3. External Integration & Tooling Issues When AI connects to APIs, tools, or data pipelines, miscommunication, outdated integrations, or parsing errors can result in incorrect outputs or failed workflows. 4. Prompt Engineering Mistakes Ambiguous, vague, or overloaded prompts confuse the model. Without clear, refined instructions, outputs may drift off-task or omit key details. 5. Context Window Constraints AI has a limited memory span. Long inputs can cause it to forget earlier details, compress context poorly, or misinterpret references, resulting in incomplete responses. 6. Lack of Domain Adaptation General-purpose models struggle in specialized fields. Without fine-tuning, they provide generic insights, misuse terminology, or overlook expert-level knowledge. 7. Infrastructure & Deployment Challenges Performance relies on reliable infrastructure. Problems with GPU allocation, latency, scaling, or compliance can lower accuracy and system stability. Wrong outputs don’t mean AI is "broken." They show the challenge of balancing data quality, engineering, context management, and infrastructure. Tackling these issues makes AI systems stronger, more dependable, and ready for businesses. #LLM

Product managers & designers working with AI face a unique challenge: designing a delightful product experience that cannot fully be predicted. Traditionally, product development followed a linear path. A PM defines the problem, a designer draws the solution, and the software teams code the product. The outcome was largely predictable, and the user experience was consistent. However, with AI, the rules have changed. Non-deterministic ML models introduce uncertainty & chaotic behavior. The same question asked four times produces different outputs. Asking the same question in different ways - even just an extra space in the question - elicits different results. How does one design a product experience in the fog of AI? The answer lies in embracing the unpredictable nature of AI and adapting your design approach. Here are a few strategies to consider: 1. Fast feedback loops : Great machine learning products elicit user feedback passively. Just click on the first result of a Google search and come back to the second one. That’s a great signal for Google to know that the first result is not optimal - without tying a word. 2. Evaluation : before products launch, it’s critical to run the machine learning systems through a battery of tests to understand in the most likely use cases, how the LLM will respond. 3. Over-measurement : It’s unclear what will matter in product experiences today, so measuring as much as possible in the user experience, whether it’s session times, conversation topic analysis, sentiment scores, or other numbers. 4. Couple with deterministic systems : Some startups are using large language models to suggest ideas that are evaluated with deterministic or classic machine learning systems. This design pattern can quash some of the chaotic and non-deterministic nature of LLMs. 5. Smaller models : smaller models that are tuned or optimized for use cases will produce narrower output, controlling the experience. The goal is not to eliminate unpredictability altogether but to design a product that can adapt and learn alongside its users. Just as much as the technology has changed products, our design processes must evolve as well.

Why would your users distrust flawless systems? Recent data shows 40% of leaders identify explainability as a major GenAI adoption risk, yet only 17% are actually addressing it. This gap determines whether humans accept or override AI-driven insights. As founders building AI-powered solutions, we face a counterintuitive truth: technically superior models often deliver worse business outcomes because skeptical users simply ignore them. The most successful implementations reveal that interpretability isn't about exposing mathematical gradients—it's about delivering stakeholder-specific narratives that build confidence. Three practical strategies separate winning AI products from those gathering dust: 1️⃣ Progressive disclosure layers Different stakeholders need different explanations. Your dashboard should let users drill from plain-language assessments to increasingly technical evidence. 2️⃣ Simulatability tests Can your users predict what your system will do next in familiar scenarios? When users can anticipate AI behavior with >80% accuracy, trust metrics improve dramatically. Run regular "prediction exercises" with early users to identify where your system's logic feels alien. 3️⃣ Auditable memory systems Every autonomous step should log its chain-of-thought in domain language. These records serve multiple purposes: incident investigation, training data, and regulatory compliance. They become invaluable when problems occur, providing immediate visibility into decision paths. For early-stage companies, these trust-building mechanisms are more than luxuries. They accelerate adoption. When selling to enterprises or regulated industries, they're table stakes. The fastest-growing AI companies don't just build better algorithms - they build better trust interfaces. While resources may be constrained, embedding these principles early costs far less than retrofitting them after hitting an adoption ceiling. Small teams can implement "minimum viable trust" versions of these strategies with focused effort. Building AI products is fundamentally about creating trust interfaces, not just algorithmic performance. #startups #founders #growth #ai

I spoke to a company last week that makes software for doctors. But sometimes patients - usually in crisis - create an account looking for their doctor. When this happens their current big-name AI solution just starts happily giving totally irrelevant (and dangerous) answers. Lorikeet's agent instead instantly disengaged and escalated the ticket to a human agent. This is a great illustration of how it's hard to build a truly good CX AI solution when you focus on containment or deflection. In fact I think the excessive focus on deflection is the Achilles' heel for a lot of the solutions in our space. Focusing on deflection weakens the product in five core ways: 1. Product architecture reflects different values - chatbots maximize engagement, agents know their limits 2. Self-awareness is a real technical challenge - most vendors avoid the hard engineering work 3. Bad metrics create bad feedback loops - you can't improve what you can't measure properly 4. Testing tools get built around the wrong goals - celebrating coverage instead of quality 5. Workflow design suffers - optimizing for engagement over effectiveness More on what we've learned about these trade offs in comments.

Sharing 10 personal learnings from studying Claude's system prompt. If you're building with LLMs - it's a goldmine of design principles and shaping its behavior to the intended user experience 👇 1️⃣ Define the assistant’s purpose clearly Key insight: Claude begins with a values-driven role: helpful, honest, harmless. Hook: What’s your AI for - and what’s off-limits? Most prompts never say. 2️⃣ Tone is a first-class citizen Key insight: Claude isn’t just accurate. It’s grounded, warm, and clear—by design. Hook: Your LLM doesn’t just need goals. It needs vibes. 3️⃣ Embrace ambiguity with options, not guesses Key insight: Claude offers multiple interpretations when users are unclear. Hook: Defaulting to “I’m not sure - did you mean X or Y?” beats hallucinating. 4️⃣ Use conditional logic for guardrails Key insight: Claude refuses dangerous content and redirects to constructive alternatives. Hook: “No” isn’t the end. It’s the start of a better direction. 5️⃣ Respect user preferences - but know when to ignore them Key insight: Claude applies preferences only when relevant. Hook: Over-personalization is just as risky as ignoring context. 6️⃣ Provide examples, not just instructions Key insight: Claude learns from examples in the prompt (e.g. bad vs. good clarifying questions). Hook: LLMs imitate better than they obey. Feed them patterns, not laws. 7️⃣ Explicitly define how to handle limits Key insight: Claude has phrasing for “I can’t help with that” - no awkward dead ends. Hook: Saying “no” is easy. Saying it gracefully takes practice. 8️⃣ Reflect before responding Key insight: Claude can “think silently” on complex questions before answering. Hook: Add a reasoning step. It’s like turning on a prefrontal cortex. 9️⃣ Use modular roles to scale capabilities Key insight: Claude uses role-based wrappers (e.g. writing coach, code reviewer) with consistent norms. Hook: Want reusable behavior? Prompt like you’re building Lego blocks. 🔟 Prompts shape behavior probabilistically Key insight: The system prompt nudges - not forces - behavior. Hook: You’re not programming. You’re parenting.

A year ago, for me, ChatGPT was just a work tool — a writing aid for social media posts. Today, it’s also crept into my personal life. That shift is showing up in the data too. According to Marc Zao-Sanders in Harvard Business Review, “therapy and companionship” is now the #1 use case for GenAI. People aren’t just using chatbots to get things done — they’re using them to feel better, find clarity, and connect emotionally. But is it working, and at what long-term cost? A recent RCT from AHA at MIT Media Lab and OpenAI offers some insight into what that kind of use actually does to us. Nearly 1,000 participants were asked to chat daily with ChatGPT for 4 weeks. Each was assigned to 1 of 9 combinations of modality (text, neutral voice, or emotionally expressive voice) and conversation type (personal prompts, non-personal prompts, or open-ended). *The researchers found that more frequent use—regardless of format or topic—was consistently associated with greater loneliness, stronger emotional dependence, and lower social interaction with real people.* Interestingly, text-based chats were more emotionally “sticky” than voice, prompting more self-disclosure and stronger attachment. And while personal prompts (like reflecting on values or gratitude) led to a slight uptick in loneliness, they were also linked to lower emotional dependence and less problematic use. On the other hand, non-personal prompts — the kind we often think of as purely practical — were more likely to foster emotional reliance over time. That nuance matters. The study didn’t suggest that emotionally expressive AI is inherently risky, or that personal conversations are always harmful. Instead, it showed how easily frequent, habitual use — even for neutral tasks — can shift from support to substitution. Over time, chatbots can become not just a tool, but a source of comfort, perspective, and emotional regulation. And that comes with tradeoffs. The takeaway? Overuse (even for neutral tasks) is the clearest risk factor for emotional dependence. But how we use GenAI matters too. Structured, self-reflective prompts may help users think without over-attaching, and voice-based interactions — often seen as more “human” — can actually be less emotionally sticky than text. As more people turn to GenAI for emotional support, this research is a reminder: design and intention matter. We can build AI that supports reflection without replacing relationships, but only if we design for that edge where helpful turns into habitual. This post is part of my Friday Findings series — curated research at the intersection of minds and machines. — Cathy (Mengying) F., Auren Liu, Valdemar Danry, Eunhae L., Samantha Chan, Pat Pataranutaporn, Pattie Maes, Jason Phang, Michael Lampe & Sandhini Agarwal (2025). How AI and Human Behaviors Shape Psychosocial Effects of Chatbot Use: A Longitudinal Randomized Controlled Study. arXiv preprint Nuance Behavior