Support Chat Assistant at a Regional Broadband and Mobile Provider
An example project: a broadband and mobile provider with 1.2 million subscribers puts an assistant in front of its 9,000 support chats a day, aiming to resolve about a third of them without an agent, on an endpoint sized for outage peaks.
This is an example project, not a customer story. The company is made up, and every number on this page is an assumption stated so you can replace it with your own.
Resolving a third of chats without an agent saves about $5.6 million a year
Build cost assumes engineers at $180,000 a year fully loaded. Time from other staff, such as reviewers labeling examples, is not counted. Payback is counted from go-live. See the arithmetic.
Agents answer the same billing and outage questions all day
A regional broadband and mobile provider has about 1.2 million subscribers. Its support chat, on the website and in the app, takes about 9,000 chats a day, seven days a week. Contact-center agents, partly in house and partly outsourced, handle every one of them.
Most chats are routine. A customer asks why this month’s bill is higher, whether there is an outage at their address, how to move to a different plan, or when a promotional price ends. The agent verifies the customer, opens the billing system and the outage map, and reads back what is there. A chat costs the provider about $5 in agent time.
Outages are the hard case. When a fiber cut or a power failure takes down service in a town, chat volume can triple within an hour. Wait times climb past 20 minutes, customers give up and call instead, and the chats that matter most wait behind questions the outage page already answers.
The goal is to have an assistant answer every chat first, resolve the routine questions from the help center and the account systems, and hand the rest to an agent with the account already looked up and a short summary of the conversation.
The model answers, and the provider's code decides what it may do
The model writes replies and chooses which read-only lookups to call. Identity checks, permissions and hand-off routing stay in code the provider controls.
Chat session and identity
existing chat platformThe provider’s chat platform keeps running the widget, the queues and the agent desktop. The assistant joins each chat as the first responder. Customers who are logged in to the app are already verified; others get the same verification step agents use today, run by the platform, before any account tool is available.
Find the relevant help articles
provider's searchThe customer’s message goes to the search the help center already uses, and the top three or four articles are added to the prompt. If the provider wants embedding-based search, that embedding model runs separately; it is not part of this endpoint.
Answer and look things up
modelThe model writes the reply and can call a small set of read-only tools: current bill and line items, outage status for the customer’s address, the customer’s plan and the plans they can move to. Tool calling depends on the model and on the serving engine’s parser for it, so confirm it for your deployment on these exact tools.
Check every action in code
provider's codeThe tools enforce the rules, not the prompt. They only return data for the verified account in this chat. Anything that changes the account (credits, plan changes, payment arrangements, cancellations) is not a tool the model has; it is a reason to hand off. Card numbers typed into the chat are masked before they reach the model.
Hand off with a summary
provider's code and modelRules in code trigger the hand-off: the customer asks for a person, wants to cancel or dispute a charge, or the assistant has failed to answer twice. The model writes a three-line summary, and the chat platform routes the chat to the right agent queue with the summary and the account lookups attached.
Outages, not the daily total, set the number of GPUs
The arithmetic we would start from. Profiling on real traffic replaces these estimates before anything goes into a contract.
| Chats per day | 9,000 |
|---|---|
| Assistant replies per chat | 6 |
| Input tokens per reply (about 2,000 of system prompt and policy text, 1,500 of help articles, and 1,000 of conversation and tool results on average) | 4,500 |
| Output tokens per reply | 120 |
| Tokens per day | 243M in, 6.5M out |
| Usual busiest hour, if 10% of the day’s chats arrive in it | 900 chats, about 6,750 input and 180 output tokens per second |
| Outage hour, at three times the usual busiest hour | 2,700 chats, about 20,250 input and 540 output tokens per second |
| Starting configuration to profile | Qwen3-30B-A3B-Instruct-2507 on H200 (141 GB) GPUs |
| GPUs, if one GPU sustains 8,000 input tokens per second with p95 time to first token under one second | 2 minimum, 4 maximum |
The customer is watching the reply appear, so this endpoint is set to favor latency over throughput, and the target that matters is p95 time to first token. Input tokens dominate: every reply resends the system prompt, the policy text and the conversation so far. Most of that repeats from one reply to the next, and prefix caching lets the serving engine skip recomputing it. That makes 8,000 input tokens per second per GPU a cautious assumption, and it is the first number profiling checks.
At that rate one GPU covers the usual busiest hour. The minimum is two workers so that a restart does not stall live chats, and because a new worker takes minutes to start. Two workers already cover about 2,100 chats an hour, more than twice the usual busiest hour, which carries the first part of an outage while autoscaling adds workers up to the maximum of four. An outage hour needs three; the fourth is headroom for a larger event.
The 30B model’s weights are about 61 GB in 16-bit precision. On a 141 GB GPU that leaves about 75 GB for the cache that holds hundreds of open conversations.
How the numbers add up
| Chats a year, at 9,000 a day | 3,285,000 |
|---|---|
| Agent cost today, at $5.00 a chat fully loaded | about $16.4 million a year |
| Chats the assistant resolves without an agent (the target; the pilot tells you the real rate) | 35% |
| Agent cost for the other 65%, still at $5.00 a chat | about $10.7 million a year |
| Quality review of 3% of assistant-resolved chats, at $1.50 a chat | about $52,000 a year |
| Two H200 workers running all year, at our reference price of $3.80 per GPU-hour | $66,600 a year |
| Two more workers for about 150 outage hours a year | $1,100 a year |
| Ongoing cost with the assistant | about $10.8 million a year |
| Saving a year | about $5.6 million |
| One-time build: three engineers for five months, at $180,000 a year loaded | $225,000 |
| Payback on the build, from go-live | under 1 month |
At these assumptions the ongoing cost falls from about $16.4 million to about $10.8 million a year. The first year also carries the build, about $225,000 in engineering plus the support team leads' time. At 35% resolution the saving covers it in under a month.
The resolution rate drives the result. At 15% instead of 35%, the total is about $14.1 million and the saving about $2.4 million. The chats left for agents are also the harder ones, so their cost per chat will rise. If those cost $5.75 instead of $5.00, the saving at 35% is about $4.0 million. The GPUs cost under 1% of today’s agent cost at any of these rates, so the decision rests on resolution quality, not on serving cost.
A resolved chat only counts if the customer does not come back. Count a chat as resolved when there is no hand-off and no new chat or call on the same topic within seven days. An assistant that closes chats but sends customers back to the phone queue raises cost.
The saving becomes cash when staffing changes, usually through outsourced volume and attrition rather than layoffs. The benefit that does not show in the table is the outage queue: every customer gets an answer about their address within seconds, and agents are free for the customers who need one.
How the project would run
Build the evaluation set
weeks 1 to 3Pull 2,000 recent chat transcripts across billing, outages and plan changes, with the account data the agent saw. Support team leads mark the correct answer and whether the chat should have been handed off. Hold back 500 that no prompt or fine-tune ever sees.
Compare models on the held-out set
weeks 4 to 5Run three or four candidate models with the same prompt and tools on a trial endpoint. Score answer correctness, correct tool calls, and correct hand-off decisions, and measure time to first token on real prompt lengths. Pick the smallest model that meets the bar.
Shadow mode as agent assist
weeks 6 to 10The assistant drafts a reply for every chat, and agents see the draft and choose to send it, edit it or ignore it. Customers only ever see what the agent sends. How often agents send drafts unchanged is the first real estimate of the resolution rate, and their edits show which policies the prompt gets wrong.
Pilot on one topic
weeks 11 to 14The assistant answers 10% of billing chats directly. Compare resolution, seven-day repeat contacts and satisfaction scores against the 90% that agents handle. Run an outage drill against the endpoint before outage traffic is included.
Widen by topic
week 15 onwardAdd outage questions, then plan questions, raising the share of chats one step at a time. Keep the quality review sample running permanently.
What to measure
Resolution without repeat contact
The share of chats closed with no hand-off and no new contact on the same topic within seven days. This is the number the business case depends on.
Hand-off accuracy
Chats that should have gone to an agent and did not, and chats handed off that the assistant could have answered. The first kind matters more.
p95 time to first token, including outages
Track it separately for outage hours. That is when the endpoint is busiest and when customers are least patient.
Agent re-asks after hand-off
How often an agent has to ask again for something already in the summary or the account lookup. If it rises, fix the summary prompt.
Models to test first
Candidates from our catalog, not a ranking. The project picks one by testing them on its own labeled examples.
Qwen3-30B-A3B-Instruct-2507
The starting choice. A mixture-of-experts model that activates about 3B of its 30B parameters for each token, which keeps each reply fast when hundreds of conversations run at once. Apache 2.0, with a native context of 262,144 tokens, and Qwen describes it as strong at tool calling. It has no thinking mode, so replies start without a reasoning preamble.
Llama-3.3-70B-Instruct
A dense 70B model with a 128K-token context and support for tool calling. About 141 GB of weights in 16-bit precision, so it needs two GPUs with tensor parallelism. Test whether it resolves more chats on the held-out set before paying for the extra GPUs.
gpt-oss-20b
OpenAI’s smaller open-weight model, under Apache 2.0. It was trained for tool use, and its reasoning effort can be set to low, medium or high, so routine replies stay quick and harder billing disputes get more reasoning.
gemma-3-27b-it
Accepts images as well as text, which matters if customers send photos of their router lights or screenshots of a bill. Tell us if you need image input so we can confirm it for your deployment.
Sizes are total parameters as listed in our catalog. Mixture-of-experts models use only part of them for each token. Check each model's license for your use. Several, such as the Llama and Gemma licenses, carry their own terms.
When fine-tuning helps
Keep facts in retrieval and in the tools, not in the weights. Prices, promotions and plan rules change every month, and the help center and billing API pick up a change the next time they are queried, while a fine-tuned model keeps what it was trained on.
Fine-tuning helps with what stays fixed: the provider’s tone, the shape of a good reply, and when to hand off. Resolved chats that agents wrote, and the drafts they sent unchanged during shadow mode, are the training data. If the provider runs separate brands for broadband and mobile, each can have its own adapter on the same endpoint.
When the data has to stay put
Support chats contain names, addresses, account numbers and billing history. Telecom privacy rules often restrict how customer account information is used and who processes it. The endpoint can run inside the provider’s own network or in a specific region, next to the billing system and the chat platform it reads from.
Tell us about your project
Bring us your volumes and a sample of your data. We will tell you which models to test, what the endpoint would need, and what a performance commitment would cover. Self-service accounts can create inference endpoints to try the platform, but not on our top-tier GPU capacity.