Policy and Procedure Assistant at a Regional Bank
An example project: 3,000 operations staff at a regional bank ask questions of 40,000 internal policies, procedures and product manuals and get answers that cite the source paragraph, with every part of the system running inside the bank's network.
This is an example project, not a customer story. The company is made up, and every number on this page is an assumption stated so you can replace it with your own.
About 131,000 staff hours a year go back to the operations team
Build cost assumes engineers at $180,000 a year fully loaded. Time from other staff, such as reviewers labeling examples, is not counted. Payback is counted from go-live. See the arithmetic.
Staff spend minutes on every lookup, or ask someone who knows
A regional bank has about 3,000 operations staff: branch operations, deposit and loan servicing, payments, card disputes and back-office compliance. Their work is governed by about 40,000 internal documents. These include credit and deposit policies, step-by-step procedures, product manuals, fee schedules, and the internal bulletins that announce changes to them. They live in the bank’s document management system and intranet.
Finding the right paragraph is slow. Intranet search matches keywords in titles, several versions of a procedure often turn up, and the answer is usually on page 30 of a 60-page manual. Staff look something up about three times a day. Many skip the search and ask a senior colleague, or email the policy team’s shared mailbox and wait until the next day. A typical lookup takes about ten minutes, counting the colleague’s time when one is asked.
The bank wants staff to type a question and get a short answer that quotes and links the paragraph it came from, from the current version of the document. Policy and regulation rule out sending these documents or questions to a service outside the bank’s network.
Retrieval decides what the model reads
The model writes the answer from passages it is given. Indexing, search and access control run separately, and our catalog has no embedding model, so that part is not on this endpoint.
Index the documents
bank's indexing job and embedding modelA nightly job pulls current, approved documents from the document management system, splits them into passages of about 500 tokens with their heading and section number, and embeds each passage. The embedding model is an open model the bank runs on its own hardware, or the vector search it already has. It is not part of this endpoint.
Search, filtered by access
bank's search serviceThe question is embedded and matched against the index, combined with keyword search for product codes and form numbers. Passages from documents the user is not entitled to see are filtered out here, before the model sees anything. A reranker, if the bank uses one, runs here too and is also separate.
Answer with citations
modelThe model receives the question and the top eight passages, each labeled with its document, version and section. It answers in a few sentences and cites the passage labels it used. If no passage answers the question, it says so and points to the policy team mailbox.
Check the citations
bank's codeCode confirms that every cited label was in the prompt and that each document is still the current version, and turns the labels into links to the exact paragraph. An answer with no valid citation is not shown as an answer.
Log and feedback
bank's codeQuestions, passages, answers and thumbs up or down are logged inside the bank. Questions with no answer go to the policy team every week; they show which procedures are missing or unclear.
One GPU covers the bank's busiest hour
The arithmetic we would start from. Profiling on real traffic replaces these estimates before anything goes into a contract.
| Questions per business day (4,500 lookups, two questions each including follow-ups) | 9,000 |
|---|---|
| Input tokens per question (about 600 of instructions, eight passages at 500 tokens each, and 400 for the question and earlier turns) | 5,000 |
| Output tokens per question (the answer with citations) | 250 |
| Tokens per day | 45M in, 2.25M out |
| Busiest hour, if 15% of the day’s questions arrive in it | 1,350 questions, about 1,875 input and 94 output tokens per second |
| Starting configuration to profile | Granite 4.0 H Small on one RTX PRO 6000 (96 GB) |
| GPUs, if one GPU sustains 4,000 input tokens per second with p95 time to first token under two seconds | 1 |
| Index, separate from the endpoint: 40,000 documents at about 7,500 tokens each | about 300M tokens, about 600,000 passages |
Nearly every token is input, and the model reads the whole prompt before it writes anything, so time to first token grows with the number of passages. At the assumed rate the busiest hour uses a little under half of one GPU’s prefill capacity. That rate is the first number profiling checks; if it is half, the GPU is fully used at the peak and a second worker is needed for the busiest hour.
One worker is enough for an internal tool that can be unavailable for a few minutes during a restart. If the bank wants no gap at all during working hours, set the minimum to two workers.
Granite 4.0 H Small has 32B parameters, so its weights are about 64 GB in 16-bit precision, which leaves about 30 GB on a 96 GB GPU. Only 4 of its 40 layers use attention; the other 36 are Mamba-2 layers whose state does not grow with prompt length. That keeps the memory each long prompt holds small.
The index is a separate, much smaller job. Embedding 300M tokens once, and the few documents that change each night, is work for the bank’s embedding model, not for this endpoint.
How the numbers add up
| Lookups a day today (3,000 staff, about three each) | 9,000 |
|---|---|
| Time per lookup today, including a colleague’s time when one is asked | 10 minutes |
| Time per lookup with the assistant (read the answer, open the cited paragraph) | 3 minutes |
| Share of lookups that go to the assistant (the target; the pilot tells you the real rate) | 50% |
| Staff time saved, over 250 business days | about 131,000 hours a year |
| Value of that time, at $45 an hour fully loaded | about $5.9 million a year |
| One RTX PRO 6000 running all year, at our reference price of $1.25 per GPU-hour | $11,000 a year |
| Embedding model on one L40S, if the bank runs it on a GPU priced the same way ($1.10 per GPU-hour) | $9,600 a year |
| One-time build: two engineers for four months, at $180,000 a year loaded | $120,000 |
| Payback on the build, counting staff time at its loaded cost | under 1 month |
At these assumptions the assistant saves about 131,000 hours of staff time a year, against about $21,000 a year of GPUs for generation and embedding. If the endpoint runs on hardware the bank owns, price that hardware instead. The first year also carries the build, about $120,000 in engineering plus the policy team’s time on the evaluation set.
Usage is the assumption that matters. At 15% of lookups instead of 50%, the saving is about 39,000 hours, worth about $1.8 million. If each lookup saves three minutes instead of seven, at 50% usage, it is about 56,000 hours, worth about $2.5 million. In every case the GPUs are a small fraction of the value.
Saved time is capacity, not cash, unless the bank plans hiring around it. Read the table as the hours that go back to servicing loans and resolving disputes.
The risk that does not show in the table is a wrong answer. A member of staff who follows an outdated procedure creates a compliance error. That is why the citation check and the current-version filter are in the build, and why the evaluation measures citation accuracy, not only whether answers read well. The benefit that does not show is the weekly list of unanswered questions, which tells the policy team which documents to fix.
How the project would run
Build the evaluation set
weeks 1 to 3Collect 600 real questions from the policy team mailbox and from senior staff, across every operations area. The policy team marks the paragraph that answers each one, or marks it unanswerable. Hold back 200 that no prompt or fine-tune ever sees.
Tune retrieval, then compare models
weeks 4 to 7First measure retrieval alone: how often the right paragraph is in the top eight passages. Fix chunking and filters until that is high, because no model can cite a paragraph it was not given. Then run three or four candidate models on the same passages and score answer correctness and citation accuracy.
Pilot with one department
weeks 8 to 11Card disputes or deposit servicing, about 200 people. Every answer shows its citations, and staff rate answers. The policy team reviews a sample of answers each week and every answer rated down.
Roll out by department
weeks 12 to 16Add departments one at a time, each after its documents are checked for duplicate and outdated versions. Old versions in the index are the most common cause of wrong answers.
Run as a service
after rolloutRe-run the held-out set after every change to the model, the prompt, the chunking or the embedding model, and keep the weekly review of unanswered questions.
What to measure
Citation accuracy
The share of answers whose cited paragraph actually supports the answer, scored on the held-out set and on the weekly review sample.
Retrieval hit rate
How often the paragraph the policy team marked is among the passages the model received. When answers get worse, check this first.
Usage
Weekly active staff and questions per person, by department. The business case depends on it, and a department with low usage usually has a document problem.
Questions to the policy team mailbox
Should fall as the assistant is adopted. Questions that still arrive show what the assistant cannot answer.
Models to test first
Candidates from our catalog, not a ranking. The project picks one by testing them on its own labeled examples.
granite-4.0-h-small
The starting choice. A 32B mixture-of-experts model with about 9B parameters active per token, and a hybrid of Mamba-2 and attention layers that keeps memory use low on long prompts. Apache 2.0 with a 128K-token context. IBM lists retrieval-augmented generation and question answering among its intended uses.
Qwen3-235B-A22B-Instruct-2507
A much larger model to test against the starting choice: 22B of its 235B parameters are active per token. Its weights are about 470 GB in 16-bit precision, so it needs several GPUs on one node with tensor parallelism. Only worth it if citation accuracy on the held-out set improves.
Llama-3.3-70B-Instruct
A dense 70B model with a 128K-token context and support for tool calling, so the model could issue its own follow-up searches instead of relying on one retrieval pass. About 141 GB of weights in 16-bit precision, so it needs two 96 GB GPUs with tensor parallelism.
Mistral-Nemo-Instruct-2407
A 12B model with a 128K-token context, built by Mistral AI with NVIDIA. A smaller option to test, since many answers mostly restate the passage they cite.
The embedding model that builds the index, and any reranker, are separate models the bank runs itself. They are not in our catalog and not part of this endpoint. The endpoint exposes an OpenAI-compatible API, so the bank’s retrieval code calls it like any other model API.
Sizes are total parameters as listed in our catalog. Mixture-of-experts models use only part of them for each token. Check each model's license for your use. Several, such as the Llama and Gemma licenses, carry their own terms.
When fine-tuning helps
Start without fine-tuning. Most wrong answers in a project like this trace back to retrieval or to outdated documents in the index, and a fine-tuned model still cannot cite a paragraph it was not given.
Once retrieval is good, fine-tuning can teach the bank’s citation format, its internal product codes and acronyms, and when to decline and point to the policy team. The policy team’s reviewed answers from the pilot are the training data. Measure the effect on citation accuracy with the held-out set before switching.
When the data has to stay put
Every prompt contains passages from internal policies and procedures, and questions often include customer details. Banking regulators and the bank’s own policies require that this data stays under its control. Every part of this system can run inside the bank’s network: the index, the embedding model, the search service and the endpoint. The endpoint can also run air-gapped if the bank requires it.
Tell us about your project
Bring us your volumes and a sample of your data. We will tell you which models to test, what the endpoint would need, and what a performance commitment would cover. Self-service accounts can create inference endpoints to try the platform, but not on our top-tier GPU capacity.