A Self-Hosted Coding Assistant at a Payments Software Company
An example project: a payments software company with 400 engineers runs code completion, IDE chat and a pull-request review bot on open models inside its own network, on two small endpoints that cost about $52,000 a year in GPUs.
This is an example project, not a customer story. The company is made up, and every number on this page is an assumption stated so you can replace it with your own.
The assistant costs about $142,000 a year to run and pays for itself at about a minute saved per engineer a day
Build cost assumes engineers at $180,000 a year fully loaded. Time from other staff, such as reviewers labeling examples, is not counted. Payback is counted from go-live. See the arithmetic.
Four hundred engineers, and the code may not leave the network
A company of about 400 engineers builds payment processing software that banks and card processors run. Its security policy, and several of its contracts with bank customers, say that source code may not be sent to a third-party service. Engineers are not allowed to use hosted coding assistants, and some use them anyway on personal machines, which the security team considers worse than an approved internal tool.
The engineering leads want three things inside the network: code completion in the editor, a chat panel in the editor for questions about the code, and a bot that reviews every pull request and leaves comments before a person reviews it. About 150 pull requests are opened on a working day, and most wait several hours for a first human review.
Completion is the hard part. A suggestion that arrives after the engineer has typed the next few characters is useless, so it has to be fast at the busiest hour of the day, not only on average.
The model answers three kinds of request; the company's code decides what it sees
The model writes completions, answers and review comments. Authentication, context gathering and posting to the code host are ordinary code the platform team owns.
Editor extension
existing open-source extensionAn open-source editor extension that accepts an OpenAI-compatible endpoint, configured to point at the company’s gateway. It collects the code before and after the cursor for completion, and the open files for chat. No editor plugin is written from scratch.
Gateway
company's codeA small service in front of both endpoints. It checks the engineer’s single sign-on token, applies per-user rate limits, routes completion and chat to their endpoints, and logs request sizes and latency. Prompts are not stored by default, because they contain source code.
Code completion
model, completion endpointThe model receives the code before and after the cursor and fills in the middle. Suggestions are capped at a few lines, which keeps output short and fast.
Chat about the code
model, chat endpoint, with existing code searchThe gateway adds results from the code search tool the company already runs, such as definitions of the functions in the open file, before calling the model. The search itself is not part of the endpoint.
Pull-request review bot
company's CI job, then modelA CI job collects the diff, the full text of the changed files and the team’s review guidelines, and calls the model. It posts the model’s comments on the pull request, marked as automated. The bot never approves or blocks a merge. A person still reviews every change.
Peak-hour completion latency sets the size, not token volume
The arithmetic we would start from. Profiling on real traffic replaces these estimates before anything goes into a contract.
| Engineers using it on a working day | 300 of 400 |
|---|---|
| Completions, if each engineer triggers 300 a day (1,500 tokens in, 40 out each) | 90,000 a day |
| Chat requests, at 25 per engineer a day (6,000 tokens in, 500 out each) | 7,500 a day |
| Pull-request reviews, 150 pull requests reviewed twice each (25,000 tokens in, 1,500 out each) | 300 a day |
| Tokens per working day | About 190M in, 7.8M out |
| Busiest hour, if 15% of the day’s completions and chats arrive in it | 3.75 completions a second, about 7,500 input and 310 output tokens per second |
| Starting configuration to profile | Qwen3-Coder-30B-A3B on H100 (80 GB) for completion, and on RTX PRO 6000 (96 GB) for chat and review |
| Completion GPUs, if one H100 sustains 8 completions a second within the latency target | 1 at peak, 2 workers in working hours |
| Chat and review GPUs | 1 at peak, 2 workers in working hours |
The token volume is modest. What sets the size is that completions must start within 300 ms while hundreds of engineers type at once. Completion and chat run on separate endpoints so that a burst of long chat requests or a batch of reviews never slows down completions. The completion endpoint is set to favor latency; the chat and review endpoint can favor throughput, and reviews wait in a queue.
The model’s weights are about 61 GB in 16-bit precision. On an 80 GB H100 that leaves about 19 GB for cache, which is enough for completion requests of a few thousand tokens. The H100 is chosen for completion because its higher memory bandwidth gives faster output per request; profile the RTX PRO 6000 for completion too, since it costs less per hour. Chat and review prompts are longer, so they go on the 96 GB GPU with more room for cache.
Bursts inside the busiest hour can reach twice the average, about 7.5 completions a second, which is still under the assumed 8 per GPU. That rate is the first thing to check in profiling. The second worker on each endpoint covers restarts and bursts during working hours; each endpoint autoscales between one and two workers.
How the numbers add up
| Completion endpoint: one H100 all year plus a second for 12 hours on 250 working days (11,760 GPU-hours at our reference price of $3.15) | about $37,000 a year |
|---|---|
| Chat and review endpoint: the same pattern on RTX PRO 6000 at $1.25 per GPU-hour | $14,700 a year |
| Half of one platform engineer to run the gateway, roll out the extension and maintain evaluations, at $180,000 a year loaded | $90,000 a year |
| Total running cost | about $142,000 a year |
| Engineer time saved needed to break even, at $100 an hour loaded and 300 engineers on 250 working days | about 1.1 minutes per engineer a day |
| Value if each engineer saves 20 minutes a day (the pilot measures the real figure) | about $2.5M a year |
| Yearly tokens, to compare with a per-token API at your own prices | about 47B input, 2B output |
| One-time build: two platform engineers for three months | $90,000 |
| Payback on the build, at 20 minutes saved a day (at 5 minutes, about 2 months) | under 1 month |
This project is not justified by cost savings against a hosted service, because the company cannot send its code to one. The choice is between a self-hosted assistant and no assistant. The case rests on engineer time, and the break-even point is low: about 1.1 minutes a day per engineer covers the GPUs and the half engineer who runs it. The GPUs alone are covered by about 25 seconds a day.
The 20-minute figure is an assumption, and self-reported time savings from coding tools are often higher than measured ones. Even at 5 minutes a day the value is about $625,000 a year against $142,000 of cost. The pilot should measure suggestion acceptance and review turnaround rather than rely on surveys.
If your code could go to a per-token API, multiply the yearly token figures by its prices. Most of the input is completion context that changes a little with each keystroke, so the result depends heavily on how the provider prices cached input. At low volume a per-token API can cost less than a dedicated endpoint, because the dedicated endpoint is sized for peak latency rather than for total volume.
The benefit that does not show in the table is the review bot. Pull requests get a first set of comments within minutes, so authors fix obvious problems before a human reviewer spends time on them.
How the project would run
Build the evaluation set
weeks 1 to 2From the company’s own repositories, take 2,000 recent merged commits, hide a span of each changed function, and check whether the model’s completion matches or passes the existing tests. For review, take 300 past pull requests where a human reviewer asked for a change, and check whether the model raises the same issue. Keep a held-out part that no prompt is tuned on.
Compare models and profile latency
weeks 3 to 4Run the candidate models on a trial endpoint against the evaluation set, and replay a recorded hour of completion traffic from the pilot team’s editors to measure first-token latency at the target load. Pick the smallest model that meets the latency target and the completion quality bar.
Pilot with one team
weeks 5 to 8Thirty engineers on the payments ledger team use completion and chat. The review bot runs on their repositories and posts comments, but only the pull-request author sees them at first. Track suggestion acceptance, latency, and which bot comments led to a code change.
Roll out to all engineering
weeks 9 to 12Open the extension to all engineers, a few teams a week, while watching peak-hour latency. Turn on visible review comments for each repository once its team agrees the comments are worth reading.
Tune and fine-tune
after rolloutUse the logged acceptance data to decide whether a fine-tuned adapter on internal code is worth training, and compare it against the base model on the held-out set.
What to measure
Suggestion acceptance rate
The share of shown completions the engineer keeps. Track it by language and by repository, since internal frameworks are where a general model is weakest.
Completion latency at the 95th percentile
Measured at the gateway during the busiest hour. If it rises above the target, engineers stop waiting for suggestions and acceptance falls.
Review comments that led to a change
The share of bot comments followed by a code change on the same lines. A low rate means the comments are noise and people will learn to ignore them.
Time to first human review
Compared with the months before the bot. If authors fix more problems before review, reviewers should approve sooner.
Models to test first
Candidates from our catalog, not a ranking. The project picks one by testing them on its own labeled examples.
Qwen3-Coder-30B-A3B-Instruct
The starting choice. Alibaba’s coding model, with 3.3B of its 30.5B parameters active per token, which keeps output fast enough for completion on one GPU. The Qwen3-Coder project states that every Qwen3-Coder version supports fill-in-the-middle, and the native context of 262,144 tokens is enough for long review prompts. Apache 2.0. One model for both endpoints keeps operations simple.
gpt-oss-120b
A candidate for chat and review, not for completion. Apache 2.0, trained for tool use, with reasoning effort you can set per request, so the review bot can use more reasoning than chat. OpenAI sized it to run on a single 80GB GPU.
GLM-4.6
A larger model to test for the review bot only, where requests are queued and latency matters less. Z.ai lists coding among its improvements over GLM-4.5, with a 200K-token context. Its 355B parameters need several GPUs on one node with tensor parallelism. Measure whether its comments lead to more code changes before paying for them. MIT license.
Sizes are total parameters as listed in our catalog. Mixture-of-experts models use only part of them for each token. Check each model's license for your use. Several, such as the Llama and Gemma licenses, carry their own terms.
When fine-tuning helps
Payments code uses internal frameworks, ledger conventions and service names that no public model has seen. A LoRA adapter trained on the company’s merged code can teach the completion model those patterns. Measure it on the held-out completions from the company’s own repositories: if acceptance on internal-framework code rises, the adapter is worth keeping.
For the review bot, past review comments that led to a change are the training data, and an adapter can learn the team’s review standards. Code changes daily, so current repository contents belong in the prompt through code search, not in the weights. Adapters for completion and for review can run on the same endpoints.
When the data has to stay put
The company’s bank customers audit how it handles source code, and its contracts say code stays in its environment. Prompts to a coding assistant are mostly source code, and chat prompts can also contain log lines and test data. Both endpoints run inside the company’s network, in the same environment as its code host and CI, so code, prompts and suggestions stay there.
Tell us about your project
Bring us your volumes and a sample of your data. We will tell you which models to test, what the endpoint would need, and what a performance commitment would cover. Self-service accounts can create inference endpoints to try the platform, but not on our top-tier GPU capacity.