Conversation Engineering
The data science behind a live assistant that answers from what you actually know
Advisory Guide helps people during live work conversations: a customer escalation where the account is at risk, a discovery call where the prospect asks about something in the product brief, a client check-in where a specific number comes up.
We spent months on four problems: hearing the question correctly through a noisy transcript, answering from the material the user actually prepared, holding back when that material does not support a claim, and phrasing the reply so it can be said out loud within a few seconds. This post covers the data science behind them.
A dataset built from real jobs
The first training set has 8,000 examples. Each one pairs a persona, an objective, the preparation material that person brought, a messy speech-recognition transcript of the other side, and a reply written to be spoken.
Partway through, at 4,428 rows, an audit of who those personas were showed a problem. Seven senior desk titles, led by Head of Operations and Staff Engineer, made up 42.8% of the data, while most people who have difficult conversations at work hold other jobs. We rebalanced against US Bureau of Labor Statistics employment data, toward the jobs people actually hold: home health aides, retail workers, customer service representatives, nurses, drivers, teachers, office clerks. The finished set spans 3,233 distinct roles across more than 3,000 industries, and the share held by those seven titles fell to 23.7%.
A model trained on conversations takes on the voice of the people in its data, so the mix of personas is a product decision.
Replies that are easy to say
2,373 examples require simple phrasing: short sentences, common words, nothing you would trip over while speaking under pressure. Every reply in the set is written to be read once and said aloud, in the speaker's own register.
Systems the model has never seen
Real people work inside their employer's own tools, and no model has memorized your company's ticketing system or pricing sheet. So part of the data describes fictional employer systems that exist only in the preparation material. Because the model cannot have learned them in pretraining, the only way to answer well is to read the context it was given. We want that behavior everywhere: the user's material comes first.
Alongside those, a set of examples covers well-known platforms people use every day, such as HR, payroll, and accounting systems, so that the model handles real tools at the level a competent practitioner would.
Training aimed at failure modes
The second dataset adds 1,853 training sequences, organized into families. Each family targets a specific way a conversation assistant goes wrong:
- False premises. The question assumes something that is not true. The right reply corrects the assumption instead of building on it.
- Unsupported claims. The other person asks for something the preparation does not cover. The right reply says what is known and stops.
- Specificity traps. Questions that tempt a precise, confident, invented answer. An invented number that sounds plausible is hard for the speaker to catch in real time.
- Brevity. Short replies when the question only needs a short answer.
- Heavy recognition errors. Transcripts where speech recognition mangled names and terms, and the meaning has to be recovered from context.
- Long context and multiple turns. Staying consistent across a conversation that has gone on for a while.
- Several voices. Calls with more than one person on the other side, where the reply has to address the right one.
A frozen evaluation set of 344 sequences, drawn from the same families, is checksummed so it cannot quietly drift between experiments. If a training change helps one family and hurts another, we see it.
Evaluation against real context
Synthetic evaluations passed long before the product was ready. Testing against real preparation material and real transcripts surfaced problems the clean benchmark had hidden, from a chat template that broke when it received more than one system message to grounding that slipped once the context grew long. Each one was a launch blocker, and none of them appeared in the synthetic results.
Serving it ourselves
The model runs on cloud GPUs we provision and monitor, and speech recognition runs locally on the user's PC. Holding many long conversations resident in GPU memory at once turned out to be the hardest infrastructure problem in the stack. I wrote that up separately in Holding Long Conversations in GPU Memory, and what observability looks like for a service like this is in What Changes When the Thing You Monitor Is an LLM.
What it adds up to
In the product, all of this shows up as a few simple behaviors. Nothing appears until you press Suggest. The reply draws on the material you prepared. When that material runs out, the reply gets shorter. Replies are phrased for speaking out loud.