Latency & cold starts
These are measured numbers from the deployed service, not targets. They will move as hardware and engine versions change; when they do, this page changes with them.
| Metric | Measured | Conditions |
|---|---|---|
| Cold start, first token | ≈ 60 s | First request after the model has been idle for 30 s. |
| Warm time to first token | ≈ 0.5 s | Short prompt, single request, model already running. |
| Single-stream decode | ≈ 130 tok/s (0.6B), ≈ 95 tok/s (1.7B) | One request alone on a worker. |
| Warm aggregate throughput, 32 concurrent | ≈ 2,200 tok/s | 32 concurrent requests on one worker (RTX A5000), 0.6B model. |
| Warm time to first token, 32 concurrent | p50 2.3 s, p95 2.5 s | Same 32-way run: per-request first token while the worker batches. |
| Time to first token, 26k-token prompt | 2–5 s | Prefill of a long document, warm worker. |
Cold starts
Models scale to zero when idle. The first request after roughly 30 seconds without traffic starts a fresh worker: the engine loads the weights, builds its compiled graphs and allocates its cache before it can answer. Today that takes about 60 seconds to the first token. Requests that arrive while a worker is starting wait for it; once it is up, everything is warm until the next idle gap. Bursts that exceed one worker's capacity start a second worker with the same delay while the first keeps serving.
Client timeouts
Set your HTTP client's timeout to at least 120 seconds for the first request and use streaming, so a cold start shows up as a delayed first chunk rather than an error. The OpenAI SDKs default to 10 minutes; many hand-written clients default to 30 seconds, which is too short.
Warm latency
With a worker running, a short prompt returns its first token in about half a second and decodes at roughly 130 tokens per second for the 0.6B model. Under concurrent load a worker batches requests: with 32 concurrent requests on one RTX A5000 worker, aggregate throughput reaches about 2,200 tokens per second while each request's first token moves out to 2.3 seconds at the median and 2.5 seconds at p95. Your per-token price is the same either way.
Long context
The advertised context of 32,768 tokens is the engine's limit for prompt and completion together. We verify recall before advertising it: a needle placed at 10%, 50% and 90% depth of a ~26,500-token prompt was recovered at every depth by Qwen3-0.6B, with 2–5 seconds to the first token. Leave room for the answer — and for the model's reasoning — when you size a prompt near the limit.
How we measure
Numbers on this page come from requests sent through the public API exactly as a customer would send them, timed at the client. Cold start is measured after an idle gap longer than the scale-down timeout. Throughput is measured with concurrent 256-token completions of the same prompt. The scripts live in the service repository and are run before any number here changes.