Serving Cost/Latency Simulator
Estimate inference cost and latency for a model-plus-GPU combination before committing infrastructure, grounded in real Azure GPU pricing and published memory-bandwidth specs from the serving-the-token field note.
Deterministic estimate + AI narrative — illustrative, not a benchmark
The numeric estimate (tokens/second, latency, cost) is computed by a fixed formula in this page's backend, not by an AI model, using real published Azure GPU retail pricing (East US, queried 2026-09-01) and real NVIDIA memory-bandwidth spec-sheet figures. Only the short written explanation underneath is AI-generated, and it is instructed to interpret the numbers, not invent new ones. This is a simplified, illustrative model, not a substitute for a real load test.
How it works
This calculator uses the same memory-bandwidth-bound approximation of autoregressive decoding discussed in the serving-the-token field note ↗: each decode step reads the model's weights from GPU memory once and serves every sequence in the batch during that same read, so latency per token is constant regardless of batch size while cost per 1,000 tokens falls as batch size grows — the core economics of batched serving.
Pick a model size, GPU, batch size, and output length below. If the model's weights don't reasonably fit the GPU's memory, the tool says so instead of returning a misleading number.
Try it
Loading options…
Loading configuration options…
Why this exists
This product extends the Cloud Capacity project record's theme of matching real infrastructure decisions to real demand: before committing a GPU SKU and fleet size to a serving workload, this gives a grounded, order-of-magnitude read on the cost and latency tradeoff, using the exact batching economics reasoning documented in the serving-the-token field note.
Production checklist
- Inputs are constrained to fixed, allow-listed options — no arbitrary numeric abuse.
- GPU pricing and bandwidth figures are real, cited, published data, not fabricated numbers.
- A VRAM capacity check flags combinations that would not physically fit on a single GPU.
- The AI layer only narrates already-computed numbers; it cannot alter or recompute them.
- Automated backend tests cover both the fits and does-not-fit cases.