Measured performance, and the method behind every figure
Any vendor can quote tokens per second. Whether that number means anything depends on what was running, how many requests were in flight and whether the cache was warm. Below are our figures, the method that produced them, and the runs we threw away.
Capacity figures for local AI are quoted carelessly across this industry. A throughput number taken from a single request on a warm cache, with a short prompt and a short answer, can be five to ten times better than the same system doing real work for several people at once. It is not a lie exactly; it is a measurement of something nobody will ever experience.
So the figures below come with the method attached. They were measured on one compute node on a stated date, with the runs we discarded listed alongside the ones we kept. Figures for a specific workload — your document lengths, your user count — are available on request, because those depend on what you are actually doing.
The figures, measured on one node
Run of 17 September 2026. One RAPTOR compute node: NVIDIA GB10, 121.7 GiB unified memory, Ubuntu 24.04 LTS.
Figures are for a single node. A two or three node unit runs more capabilities at once; it does not make one model faster.
Local language model, chat workload
Open-weight model served by vLLM
| Concurrent users | Total output | Per user | Time to first token (p50) |
|---|---|---|---|
| Concurrent users:1 | Total output:50 tok/s | Per user:51 tok/s | Time to first token (p50):110 ms |
| Concurrent users:2 | Total output:91 tok/s | Per user:47 tok/s | Time to first token (p50):171 ms |
| Concurrent users:4 | Total output:147 tok/s | Per user:38 tok/s | Time to first token (p50):288 ms |
| Concurrent users:8 | Total output:243 tok/s | Per user:32 tok/s | Time to first token (p50):469 ms |
| Concurrent users:16 | Total output:367 tok/s | Per user:24 tok/s | Time to first token (p50):594 ms |
| Concurrent users:32 | Total output:526 tok/s | Per user:17 tok/s | Time to first token (p50):727 ms |
| Concurrent users:64 | Total output:721 tok/s | Per user:12 tok/s | Time to first token (p50):839 ms |
Responsive capacity
32 concurrent users
p95 time to first token under 2 s, at least 10 tok/s each, no failed requests
Fluid capacity
16 concurrent users
p95 time to first token under 1 s, at least 20 tok/s each, no failed requests
Other capabilities
54 tok/s per user, 325 ms to first token
Vision-language model answering questions about an image, single user.
36 tok/s per user, 315 ms to first token
Unified chat and vision endpoint, single user. 16 users still hold 20 tok/s each.
26 pages per minute
Scanned multi-page PDF to structured Markdown, measured per page rather than per file.
19.7 images per minute
1024px diffusion output, roughly 3 s per image, with caching disabled.
3.8× realtime
One stream of 192 s audio per request. Four concurrent streams fall to roughly realtime.
Speech recognition accuracy
Google FLEURS test split, 50 utterances per language, identical normalisation for both models
| Model | Language | Word error rate | Character error rate | Speed |
|---|---|---|---|---|
| Model:Whisper large-v3-turbo | Language:English | Word error rate:4.2% | Character error rate:1.7% | Speed:34.5× realtime |
| Model:Whisper large-v3-turbo | Language:German | Word error rate:4.2% | Character error rate:1.3% | Speed:68.3× realtime |
| Model:omniASR 7B | Language:English | Word error rate:5.3% | Character error rate:2.5% | Speed:2.8× realtime |
| Model:omniASR 7B | Language:German | Word error rate:5.7% | Character error rate:1.5% | Speed:2.9× realtime |
Whisper is both more accurate and an order of magnitude faster on these two languages, so it is the default. omniASR stays in the catalogue for its far wider language coverage, which is the reason to accept the cost.
Runs we threw away
Four result sets from this run were discarded before anything was published. They are listed because a benchmark page that shows only its good runs is asking you to trust the ones it kept.
- 01
A warm cache made latency look ten times better
The first language model run reused the same seed at every concurrency level, so the serving cache answered repeated prompts. Time to first token read 74 ms where the honest figure for the same 4,096-token prompt was 726 ms. The whole run was discarded and the harness fixed to use a unique prompt per request.
- 02
Transcription recorded successes it never had
Every request in the first transcription run failed, because that container could not decode MP3 — and the harness still wrote completion records. Discarded, and the failure path fixed so an error can no longer be counted as a result.
- 03
OCR counted a nineteen-page document as one page
Pages per minute was computed per file rather than per page, and a second part of the sweep never ran at all because two jobs shared an output directory. Both fixed; the published figure is per page.
- 04
Image generation returned cached images
An identical prompt and seed let the pipeline return a cached image in 0.11 s. With caching defeated, every generation took 2.86 to 2.95 s, which is the figure published above.
Six numbers, and what each one hides
Each of these can be made to look better than it is. The note under each says how, so you can ask anyone else the same question.
TTFT, milliseconds
How long the operator waits before anything appears.
Measured per request under a defined concurrency, with prefix caching explicitly disabled and a unique prompt per request. A cached prefix can make this figure roughly ten times better than it will be in use, which is why it is the single easiest number to flatter.
tokens per second, per request and aggregate
How fast an answer arrives once it starts, and how much total work the node is doing.
Reported both per request and aggregated across concurrent requests, because the two diverge sharply as load rises. A single-stream figure quoted as system capacity is meaningless.
simultaneous requests
How many people the node supports before it degrades.
Swept from one request upwards until time to first token crosses an agreed threshold. The useful answer is a curve, not a maximum, and the threshold belongs to your workload rather than ours.
tokens per second, prefill
The cost of long inputs: documents, transcripts and retrieved context.
Measured separately from generation, because document-heavy work is dominated by prefill. Systems that look fast on chat can be slow here, and this is the number that decides whether document analysis is practical on a given unit.
word error rate
Whether the transcript can be trusted.
Word and character error rate on a public multilingual benchmark, scored with the same normalisation across every model compared, so the comparison is like for like rather than a quoted figure from each vendor.
pages per second
How long a stack of scanned paper takes.
Per page, on real multi-page documents rather than a single clean image. Counting a nineteen-page PDF as one item is an easy way to produce an impressive number that does not survive contact with a scanner.
The rules we hold ourselves to
These exist because each of them has, at some point, caught a number that was wrong.
- Measured on the hardware that ships
- Figures come from the platform running on the same class of hardware you would deploy, not from a datacentre GPU with the results scaled down.
- Cold caches unless stated
- Prefix caching is disabled for latency measurement. Where a figure benefits from a warm cache, it is labelled as such, because in production the cache is warm only for repeated prompts.
- Failures count
- A run that errors is not quietly dropped from the average. We have discarded whole result sets during development after finding that a pipeline had written completion records for work it never did.
- Western open-weight models only
- The deployed catalogue is restricted to Western-developed open-weight models, and each is compared on accuracy as well as speed. A faster model that is wrong more often is not a better model.
Ask for the figures
The figures above are for a general workload on one node. Tell us yours — how many people, what they are doing, how long the documents are — and we will measure a matching configuration and send you the numbers with the method attached.
Request the figuresKeep reading
The capabilities these measurements are taken against, and how an operator deploys them on the node.
Why the measurements are taken on the unit in the first place, and what has to be true for AI to run offline.
The engineering position behind the method, and how an engagement runs.
Ready to deploy?
Talk to the team about configurations, deployment timelines and getting a unit in front of your operators.