Single-User Inference Card
SUIC
This is a write-up about a piece of hardware that does not exist yet. But it will, because we need it badly.
I will call it a Single-User Inference Card, or SUIC for short. Everyone likes a good acronym.
The idea for its usage is:
for AI inference (not for AI training and not for playing video games)
for a single user only (not serving 1,000 enterprise chatbot users from a datacenter)
The restrictions are:
no fantasy super-chips, we can use ONLY combinations of technology available today
it should not cost you much more than the PC you use right now
This is the thing you would plug into your PC, and you could have smarts comparable to SOTA models, but locally. Full privacy, full availability, at the cost of the electricity used for computation.
Building the card
For the compute base, I will pick 3 popular consumer NVIDIA cards, all released in 2025. Their launch prices ranged from $550 to $2,000.
For memory, I will pick something commercially available since 2024 called HBM3E.
It is mass-produced by SK Hynix, Micron, and Samsung. A six-pack of 36 GB stacks costs about $2,500.
What we do now is attach them to the cards.
In truth, that would require a modified GPU die with a new memory interface and different packaging. BUT none of the technology is new.
The GPU and HBM stacks need an interposer/package comparable to CoWoS. That currently costs about $750. Add another $500 for the special PCB, voltage regulators, and extra cooling.
Now that we’ve got our 3 SUIC cards, I want to benchmark them. But against what?
Choosing the model to test on
We could pick any of 100 different usage scenarios, but my gut feeling tells me software devs would be among the first customers. So agentic coding it is.
What I wanted to show was calculations on a dense model. That would be beautifully simple and would look super strong, but I promised you frontier smarts.
Claude Opus-category models like Kimi K3, GLM-5.2, DeepSeek-V4-Flash-0731, MiniMax M2.7 are all MoEs.
Best I found was Mistral Medium 3.5 128B, but that is just not in the same league. It’s more like Sonnet 4.5/4.6. In quality, it’s not much better than the 28.6 GB Qwen3.6-27B-Q8_0 that you can already run on RTX 5090.
I will proceed with the DeepSeek-V4-Flash-0731.
My reasoning is that it has comparable benchmarks to the recently popular Opus 4.8 (no comparisons to Opus 5 published yet), and I found testing results that give me the exact measurements I need on a real MI300X card (with HBM3 memory) to estimate the performance of our SUIC 5070, SUIC 5080, and SUIC 5090 cards.
The reported model weights are 156.67 GiB (about 168 GB), which leaves us about 48 GB for KV cache and the other memory spenders.
MI300X numbers:
67.3 t/s native decode
152.6 t/s with DSpark-7 speculative decoding (a later run)
11,690 t/s uncached single-request prefill
These are the benchmarks I mentioned:
The ugly number here is NL2Repo. This measures performance on long-running tasks (”long-horizon”). You tell the agent what to build and check it in the morning. Opus 4.8 would do better. I guess Opus 5 or Fable would do even better than that. But... there will be a model in 2027 that beats those, and another one a year later. These numbers are a signal, not a true measure of what your experience will be.
For now, let’s just assume that you could do very good work with it, if you had it running locally as much as you wanted and if the performance was good enough.
Performance gains
OK, now on to the best part: fantasy numbers for fantasy hardware.
About the memory, and why does this all even work
Consumer cards RTX 5070, RTX 5080, RTX 5090 use GDDR7 memory.
Datacenter cards like MI300X, H100, H200, B300 use HBM3 or HBM3E memory.
Fun fact: GDDR7 is much faster per pin
We are talking like 30 vs 9 Gb/s. More than 3x!
The elephant in the room with today’s inference hardware is the bus width.
Math is super simple
Bandwidth = (pin speed x bus width in bits) / 8
HBM3E stacks are manufactured with a lot of pins. 1024 per each.
So our 6-pack has 6144-bit bus width.
Let’s do some multiplications (source):
9.375 Gb/s x 6144 bit / 8 = 7200 GB/s
All three SUICs get the same 216 GB memory capacity.
As for memory bandwidth. What we know from NVIDIA H100, H200, AMD’s MI355X and others is that the die can pull about 30-36 GB/s per SM. NVIDIA reports that B300 can pull 50 GB/s (8TB/s divided by 160 SM).
My guess is that is safe to assume that with our new design we can get 42.5 GB/s per SM so our 5090 can exactly pull the complete 7200 GB/s.
In general, especially with dense models, at low concurrency, decode will be memory-bound, while prefill is compute heavy. On the other hand MoE model that I chose is launch-bound as the author states or I would rather name that latency-bound.
Before we do the calculation, I just want to say a word about “at low concurrency” part. That was our proposition at the start - single user. Perhaps it’s not truly a “single request at a time”, but overall concurrency is low, under 10.
For high concurrency scenarios we already have inference cards - like NVIDIA’s H100, H200, B200, B300 or AMD’s MI300X. That territory is covered much better than the one we are testing for.
Calculations
In the mentioned testing results native single-stream decode is 67.3 tok/s.
Dense
If this was a dense model, which it is not, the bandwidth ceiling would be 32 tok/s (31.17 ms to stream 168 GB with 5.3 TB/s) on the MI300X and 43 tok/s (23.33 ms to stream 168 GB with 7.2 TB/s) on the SUIC 5090.
You would then ask, do the SUIC cards have enough compute?
The model card says DeepSeek-V4-Flash-0731 has 304B params.
We would use general compute estimate transformers
FLOP/token = 2 x active param count
And we would see that for 304B we need 608 GFLOP per token.
608 GFLOP/tok x 43 tok/s = 26 TFLOP/s
From the NVIDIA specs:
And we would conclude that even the RTX 5070 has enough compute for decoding a dense 304B model at that speed.
MoE
But that is not what’s going on unfortunately. What we have is a MoE with 13B active params. So by above formula it needs just 2 x 13 B = 26 GFLOP / token. Meaning even less compute is needed - 23 times less.
And being memory bound? Those 13B mixed precision weights take about 8 GB. At 7,200 GB/s we feed those weights in 1.1 ms. If that was our roof, we would do 900+ tok/s easy.
Compute is not the issue and memory bandwidth is not the issue. Not for our SUICs, nor for the tested MI300X.
All of them would struggle with latency that is caused by the MoE architecture.
This specific model has 43 layers and each layer has to execute multiple kernels.
A kernel is just standard lingo for a small GPU program launched to perform one operation inside the model.
For one transformer layer, the inference runtime will launch separate GPU kernels for operations such as:
RMSNorm,
attention projection
attention itself
MoE router/top-k selection
gathering tokens for selected experts
expert matrix multiplication
activation function
combining expert outputs
residual add
Launching each GPU kernel has a fixed cost. The runtime has to prepare the launch, put it into the GPU command stream, schedule it, maybe synchronize with previous work, set up arguments and so on. The GPU then has to start executing the kernel - this is “the actual work”.
In cases where you have high concurrency kernel will have to do a lot of work, so you will perhaps have 15µs for kernel launch, then kernel will do 5ms of computation. Overall it means instead of 5ms you got 5.015ms. Who cares?
But, when you have batch-1 work (single user, single task), you very often have tiny pieces of work to do.
It can look like 5µs for kernel launch, then kernel will do 20µs of computation and your overhead is now huge.
The profiler here measured 1970 GPU kernel launches for a single-stream native decode step.
Most of the launches are about 6µs. There is a detailed deconstruction, but the important part is that per token about 5.8 ms is fixed, just overhead. No amount of compute or extra memory bandwidth can change that.
Ryan Zhou gave measurements that don’t show native decoding but speculative decoding (attached module guesses several tokens ahead and the main model verifies them together in one pass using spare compute). This is why the single user gets 149.2 tok/s instead of 67.3 tok/s mentioned in the beginning.
As concurrency increases so does the aggregate speed - this is because efficiency rises, same fixed cost for kernel launches, but more work done with each. I added the TTFT and per user generation speed just to show that concurrency has also a cost to per user performance - it’s not a magic solution. TTFT will rise because you wait for everyone else’s prefill. This was only a 400 word example. For agentic coding contexts get a lot larger, the same measurements show that 52K token prompt with MI300X took 7.3 s.
It is easy to imagine that your request lands inside a batch with someone’s large 100k request. You wait for their prefill too. This is what you pay for when using APIs where someone else hosts the model for you.
Let’s now finally see how this MoE would perform on our SUIC cards.
MoE numbers, MI300X first
MI300X has 192 GB of HBM3 memory at 5.3 TB/s.
It can do 2,600 TFLOP/s at FP8
Our model needs 26 GFLOP per token and has about 8 GB of weights. The weights are FP4 and FP8.
It can feed those weights in 1.5 ms, meaning that the memory limits it to about 660 tok/s.
Compute needed for the measurements is about 1.75 TFLOP/s, and we have 2,600 TFLOP/s (yes more than 1500x, not a typo).
And still, at native decode, we got 67 tok/s.
That is about 15 ms per token, and only 1.5 ms was pulling weights from memory.
At peak 2,600 TFLOP/s the compute is:
26 GFLOP (need per token) / 2,600 TFLOPs = 0.01 ms (need per token)
15 ms per token = 1.5 ms memory + 0.01ms compute + 13.49 ms overhead
Overhead is 90%.
Note: there is a difference in numbers I want to explain: 5.8 ms is the irreducible launch floor; the rest to the overall 13.49 ms is sparse attention, skinny BF16 calls, and kernels too small to fill the chip.
MoE numbers, NVIDIA
MI300X uses ROCm. It is less polished than CUDA, and we are using NVIDIA GPUs for the base.
It would be nice to get CUDA based measurements, but the only ones I found online were using multiple cards.
The one that best describes what we are trying to estimate is done using two RTX PRO 6000 cards and it was using native weights. Measurements show 105 tok/s for a concurrency-1 case. RTX PRO 6000 uses GDDR7 like an RTX 5090. They are not identical but are very similar cards. Big difference is that 6000 has 3x the memory. Architecture is the same, memory bandwidth also. Compute inside 10-20% margin.
So two RTX PRO 6000 have theoretical bandwidth of 2 x 1800 = 3.6 TB/s. Theoretical because RTX PRO 6000 doesn’t even support NVLink. So any sync goes over 5gen PCIe - that’s 64 GB/s each direction.
At 105 tok/s work being done needs about 2.73 TFLOP/s.
Tested card has 1007.6 TFLOP/s (FP8 dense). Compute is again tiny. Out of the complete second it spends about 3ms on compute. We can just ignore it, it is a rounding error. Memory fetch would be 2.2 ms per token if the 3.6 TB/s was on a single card. 2.2 ms per token translates to 454 tok/s and we got 105. About 77% of overhead even with the “2 cards PCIe sync tax”. Impressively better than ROCm’s 90% without multiple cards sync.
Only claim that I found online with a number of a single card measurement is some random comment on HN. It says B300 runs it over 400 tok/s for single stream (we called that “concurrency-1 case” here). A single B300 has 288 GB of HBM3e, 8 TB/s of bandwidth, 15 PFLOPS dense FP4, and 5 PFLOPS dense FP8.
Memory would cap it to 1ms per token, so about 1000 tok/s. The 400+ tok/s claim would put the overhead time to less than 1.5 ms per token in other words overhead is 60% or less. Underwhelming, but I will take it. I expected the sync of cards over the PCIe to have a larger influence.
MoE numbers, SUIC
For our 3 SUIC cards I want to proceed with the something close to B300 numbers. They have the same Blackwell architecture, and don’t have the in between card sync. I think its fine to go with 2 ms per token.
SUIC 5070 2,000 GB/s -> 4ms to load weights + 2 ms overhead -> 166 tok/s
SUIC 5080 3,600 GB/s -> 2.2ms to load weights + 2 ms overhead -> 238 tok/s
SUIC 5090 7,200 GB/s -> 1.1ms to load weights + 2 ms overhead -> 322 tok/s
How about the TTFT?
That depends on context size and for this model the FP8 calculations.
MI300X is faster than our cards, it has 2614 TFLOP/s FP8 and reported “11.69K tok/s steady prefill”.
So multiply our GPU FP8 values with this factor:
11690/2614 = 4.47
Note: these are cold context prefills. This gets KV cached, you don’t wait for it every time you make a request.
Summary
For context most providers get you somewhere around 60 tok/s by default.
With our cards you could have Opus-category better concurrency-1 performance locally than with Opus itself:
Just one dense example
Models aren’t designed in a vacuum. Their architectures are in part defined in dependence of what exists in the hardware market. I’m sure that if we had SUICs we would soon see models that would better use the higher bandwidth. In the recent few months Qwen 3.6 27B was very popular in the software development community. It could be run on a 32GB card and gave solid results. A few days ago Qwen 3.8 was released. I found some casual measurements on a single RTX PRO 6000 on a full BF16 and FP8 quantizations.
If you read until this point, you already see the connections, 2x the size half the decode speed - this is memory bound. Lets do some super-quick math. RTX PRO 6000 just as RTX 5090 has the 1.792 TB/s GDDR7 bandwidth.
Official releases:
Now this is dense, not a MoE, so for BF16 we pull 55.6 GB of weights per token (this is so nuts!), so at 1.792 TB/s we can do that around 32.23 times per second. Decode was 28.64 tok/s.
For the smaller FP8 30.9 GB -> 57.99 times per second. Decode was 50.08 tok/s.
Efficiency here is awesome: 88% and 86%. We can be conservative and assume only 85% for our SUICs.
And we can now just calculate the decode for SUICs:
(bandwidth / weight size in GB) x 0.85
So, SUIC 5070 with BF16 would be slow-ish, FP8 would have the speed like working with Claude. SUIC 5080 would be like that for BF16. All else would be fast, real fast!
Note:
27B params mean 54B operations. To hit each of this speeds we need from 1.6 TFLOP/s to 10.7 TFLOP/s. So we have more than 100x of compute necessary.
Why don’t we have SUICs already
This all sounds so crazy that you might wonder what’s going on with those NVIDIA engineers. Why do they build hardware that is so extremely unbalanced?
They aren’t nuts and the reason is super simple - when you buy a RTX 5090 you buy a graphics card. It’s made for games! Ray tracing, low latency. The load for that is completely different than for inference. For inference, the card needs to read the entire model out of its own memory for every single token (obviously the active experts in MoE case).
For games the GPU does a lot of math on a small amount of data. Textures, geometry, the frame buffer... hundreds or thousands of operations per byte. For inference it’s 2 operations per weight, one add and one multiply. And the CUDA cores mostly drink pina coladas and soak the sun, or whatever cores do when on vacation.
The other reason might be that there is no incentive to do so. NVIDIA uses GDDR7 in their consumer cards, AMD uses GDDR6 which is a lot slower. The memory is sold out for the year 2026. There is no competition in this space. And large AI labs are buying the GPUs in truckloads.
But I have no doubt that time will come for the SUICs, just not this year.