Mike Gallers — Experience Design & Strategy
← All projects

Improving a customer chatbot with research & conversation design

A plan participant tries to roll an old 401(k) into their current plan, and struggles with the chatbot.

Four-panel storyboard. A plan participant asks a retirement chatbot to roll over an old 401(k), gets stuck in repeated 'I didn't understand' replies, abandons the chat and waits on hold with support, then succeeds after the bot is retrained.

EXPERIENCE GAP(s) and OBJECTIVE

A participant arrives with a clear, specific goal, such as rolling an old 401(k) into their current plan, and expects the site's assistant to help. The bot misses the intent, offers generic menus, and loops on "I didn't understand."

They abandon the chat and call support, the very outcome the assistant was meant to prevent. With the chatbot exposed to 3 million+ plan participants, every missed intent was a potential call to the support center.

Objective
Keep the chatbot's roadmap tied to business priorities while making it genuinely useful to the people asking:

  • Understand what participants mean the first time
  • Reduce unhandled utterances and escalations to live agents
  • Grow chatbot adoption and self-service, and reduce support center calls

Measuring Success
Success was measured by intent-recognition accuracy, unhandled utterances, and daily escalations to live agents.

The retirement plan website with the Retirement Chatbot open in a side panel, greeting the visitor and offering help with account questions and managing investments.
The Retirement Chatbot opens from the plan website, ready to answer questions about accounts and investments.

OUTCOMES and IMPACT

The retraining work and a weekly review loop moved the metrics that matter to both members and the business. A rollover question now gets a specific, guided answer instead of a loop of "I didn't understand."

7%
Higher intent-recognition accuracy
5%
Fewer unhandled utterances
100s
Fewer escalations to live agents every day

The weekly review also surfaced emerging member needs that shaped new intents, new content, and product decisions.

Chatbot conversation. A member types 'how do i get my retirement balance' and the bot explains how to log in and find Total plan balances and View balance details.
A plain-language question gets directions to the right place.
Chatbot conversation. A member types 'how do i do a rollover' and the bot answers with the Incoming Assets Form, who rollover checks are payable to, the mailing address, and a link to contact support.
The rollover question now gets the form, the check details, and a path to help.

APPROACH, EFFORT, and ARTIFACTS

As UX lead, I partnered with leadership, product, and engineering to keep the chatbot's roadmap aligned with corporate and departmental priorities, and I owned the conversation design and the research that fed it.

Listen: a weekly utterance review

Every week I reviewed real member questions alongside the intent the bot chose, its confidence score, the page the member was on, and whether they were signed in. Sorting by theme (no match, missed intent, low confidence, escalation to an agent) showed where the bot was failing, and the notes captured why.

A spreadsheet of chatbot sessions with columns for session ID, date and time, utterance, intent, slots, score, theme, notes, URL, and authentication status.
Illustrative sample of the weekly review view. All data shown is synthetic.

Tune: training phrases and intent scores

Findings became changes to the model: new and reworded training phrases for the questions members actually ask, checked with automated testing before release so a fix in one place didn't break another.

Conversational AI tooling showing an intent with example sentences and a good score, charts of intents by score range, and a trainer for adding new phrases to an intent.
Illustrative example of intent training and score review in the platform's tooling.

Design: flows before and after sign-in

I designed and maintained the conversation flows for both signed-out and signed-in members, so people could get answers on their own without contacting a service rep.

A conversation flow diagram with branches for a yes or no answer, a goodbye and feedback request, a rating step, and a fallback that asks how else the bot can help.
Illustrative example of conversation flow design, with feedback and fallback paths.
  • Weekly utterance-review practice and reporting
  • Reworded and new training phrases for key intents
  • Automated tests run before releases
  • Conversation flows for pre- and post-authentication experiences
  • New intents and content informed by emerging member needs

LEARNINGS

  • Real member language is the best source of truth. The questions people actually typed surfaced needs no roadmap had predicted, from new intents to content gaps.
  • A steady weekly habit beats a big cleanup. Small, repeated tuning compounded into measurable gains in accuracy and fewer escalations.
  • Tie the work to business measures. Framing improvements in terms of support calls, adoption, and self-service kept the chatbot a priority for leadership.