DeepSeek V4-Flash for a 32-person team: faster than H100 and B200, at half the cost per token

TL;DR: We put DeepSeek V4-Flash (284 billion parameters) on one machine with eight NVIDIA RTX PRO 6000 GPUs and had our load-testing team throw 32 simultaneous chat users at it for 90 minutes. From their report:

• 68 tokens/s per person on a typical reply, and 57.5 tokens/s for the slowest 10% of replies. Lambda's published H100 and B200 runs average 41-49.

• 0.53 s to the first word on a typical reply, and 2.3 s for the unluckiest 1%. Lambda's unluckiest 1% waited 10-32 s.

• 0% of reply time lost to freezes, and all 11,413 completed replies within the service level.

• 1,320 tokens/s for the whole machine, more than the 1,222 published for the model's 8x B200 reference, at about half the cost per token on Google Cloud.

Every per-person number in the report landed in its "Good" band, and other traffic was sharing the server during the run, so these numbers are on the conservative side. The same model also runs on eight 24 GB NVIDIA L4s. As far as we can tell, nobody had tried that before.

Why we did this

If you want to run DeepSeek V4-Flash yourself, every published deployment points you at a data-center server: eight H100s, H200s or B200s. Those cost tens of dollars an hour, often come with a waitlist, and almost always live in someone else's cloud. For a bank or a hospital, that last part usually ends the discussion before price even comes up.

So we asked a narrow question. Can a team of 32 people share one frontier model, all at once, on GPUs that are easier to get and cheaper to rent, inside their own cloud account, and still have it feel fast?

It can. GraphX runs V4-Flash on the NVIDIA RTX PRO 6000, a Blackwell workstation card with 96 GB of memory, and installs it into the customer's own Google Cloud project with one command. Most of the numbers below come from our load-testing team's 90-minute, 32-person run and the report their tool produced. For comparison we use Lambda's published V4-Flash benchmark on H100 and B200, which is the most complete public benchmark for this model we know of.

What's in this post:

1. Speed per person, against Lambda's H100 and B200 runs.

2. The full 90 minutes, minute by minute.

3. The machine, the memory and the money.

4. Why the same model shows up as 284 GB, 146 GB and 167 GB.

5. Answer quality and exactness checks.

6. V4-Flash on eight NVIDIA L4s, and why that was the hardest of all.

The test

The load test was built to look like a busy working day for a 32-person team using a chat assistant:

The report grades each run against a service level. "Good" means a reply starts within 2 seconds, streams faster than 15 tokens/s, loses less than 5% of its time to freezes, and nothing fails. Here is where graphx landed on the per-person measures:

Two things in the report work against us, which makes these numbers conservative. The server wasn't ours alone: its health endpoint showed more requests in flight than the test was sending, so other traffic was competing. And the model's default reasoning setting was on. Any thinking text it produced cost time but isn't counted in the speeds.

Result 1: Faster per person than H100 and B200

Lambda's benchmark sends 32 concurrent requests, each with 8,192 input and 1,024 output tokens, to 8x H100 and to B200, using SGLang and vLLM. It reports the mean and p99 time between words and the mean and p99 wait for the first word. Side by side with our load test:

The number we'd point to is 57.5. That's our slowest 10% of replies, and it's still faster than the average reply in every Lambda configuration. A typical GraphX reply is 39-66% faster than Lambda's averages.

The unlucky users

People don't complain about the average. They complain about the reply that took forever, so we look hard at the slowest 1%:

• Time per word. 20 ms for graphx, against 28.5-54.9 ms for Lambda. That still works out to 50 tokens/s for our worst replies.

• First word. 2.3 s for graphx, against 10.1-32.2 s for Lambda. Our unluckiest user waited about as long as Lambda's average user did.

• Freezes. None. 0% of reply time over the whole 90 minutes.

One reply, start to finish

A typical reply: the first word shows up after about half a second, and the full 591-token answer has streamed about 9 seconds later. Reading it at a normal 5 tokens per second takes about two minutes. So the text arrives roughly 13 times faster than you can read it, and that's true for all 32 people at once.

The first turn of each conversation carries the 4,000-token document, and its first word arrived in 1.06 s. Later turns, which only add the new message, started in 0.44 s.

A fair reading of the comparison

The two tests aren't identical, and here's why the comparison still favors graphx:

• Lambda's prompts are longer. 8,192 tokens, with 32 requests in flight at all times. Ours use ~4,000-token documents and natural pauses, so 20 requests are in flight on average and 30 at peak. That's closer to how people actually chat, since they read and type between messages.

• Our run was harder in other ways. A shared server, reasoning switched on, and 90 minutes of continuous load rather than a short benchmark window.

• We also ran the harder shape. In release testing we started all 32 users at the same instant on 35,000-52,000-token contexts, four to six times Lambda's prompt length. Median time per word was 14.5 ms, still below every Lambda average. All 38 replies in that test were word-for-word identical to what the model writes when it serves that user alone.

Result 2: Ninety minutes without drift

Charts from the load-test report: total output, per-reply speed, first-word wait and time between words over the full 90 minutes.

Short benchmarks can hide slow problems, like memory filling up or queues growing until the second hour falls apart. The 90-minute charts don't show any of that. After the first minute of ramp-up:

• Total output stayed between about 1,150 and 1,550 tokens/s and averaged 1,320.

• Speed per reply held at about 68 tokens/s from minute 2 to minute 90.

• The typical first-word wait moved between 0.45 and 0.75 s.

• Time between words sat at about 15 ms throughout.

• 115-135 replies finished every minute, 127 on average.

The machine was also reading 8,557 prompt tokens per second during the run, so people kept pasting documents without slowing everyone else down. Over the run as a whole, 0% of reply time was lost to freezes.

The error column

The report lists 29 requests (0.25%) that ended in a client-side connection error rather than a reply. They trace to a keep-alive timing problem between the test client and the server, which has been addressed in later releases. All 11,413 replies that completed met the service level.

The full scorecard

Every percentile in the report:

Speed including the wait is 62.1 tokens/s, only 9% below the streaming speed. The first word comes fast enough that it barely changes how long a whole reply takes. And 79% of replies finished on their own before the cap, so these speeds describe complete answers.

More traffic than a real team would send

The simulated users are much busier than real people. Over 90 minutes the 32 of them got 11,413 replies, about 357 each, which is one reply every 15 seconds or so per person. Someone who actually reads a 591-token answer at 5 tokens per second needs about two minutes before sending the next message. By that yardstick each simulated user sends the traffic of several real ones, and the test behaves like a much larger team.

That's also why the machine never got a break. On average 20.3 requests were in flight at any moment, and at the peak 30 of the 32 users were waiting on it at once.

Scaled to a working day

If you take the measured rates and run them for an 8-hour day:

• about 61,000 replies (127 a minute);

• about 38 million answer tokens (1,320 a second), roughly 28 million English words;

• about 259 million prompt tokens read (9,002 a second);

• no time lost to freezes, assuming the day looks like the 90 minutes we measured.

Speed at each stage of a reply

• Before it starts: about half a second to the first word, 1.6 s for the unlucky 10%.

• While it streams: 68 tokens/s typically, 57.5 for the slowest 10%, and 50 tokens/s even at the p99 gap between words.

• The whole reply, wait included: 62.1 tokens/s.

• The whole machine: 1,320 tokens/s, flat from minute 2 to minute 90.

Result 3: A smaller machine that delivers more

DeepSeek V4-Flash's published guidance names eight NVIDIA B200s as the reference configuration. The load-test report sets our machine next to it:

Our machine is smaller on every line of that table: half the memory, under a quarter of the bandwidth, a fifth of the compute and no NVLink. It still put out 8% more tokens per second than the reference figure, at 56% of the monthly price.

Cost per token

The report also works out what a million output tokens costs at the service level, running all month:

On Google Cloud that's 48% less per token than the reference. On GPU marketplaces, where RTX PRO 6000s list from about $1 per GPU-hour, it's up to 9x cheaper, though marketplace stock comes and goes. The Google Cloud figures are standard on-demand list prices for machines that were in stock.

How many people one machine can serve

The report also sizes the machine for heavy users, defined as someone using a coding assistant all day: 60 messages, 800-token replies, 8 active hours. That person uses about 1.4 million output tokens a month. At the measured capacity, one graphx machine covers about 2,400 of them, against about 2,230 for the B200 reference. The 32-person test tells you how fast it feels; the capacity tells you how many people it can carry over a month.

Result 4: Same model, three file sizes

Lambda lists V4-Flash at about 284 GB for its H100 run and about 146 GB for its B200 run, and our storage bucket holds about 170 GB. It's a fair question whether these are three different models. They aren't. All three are the same 284-billion-parameter DeepSeek V4-Flash. What differs is the number format the weights are stored in, and which formats each GPU can do math in.

How DeepSeek ships it. The released checkpoint mixes two formats. The expert layers, which hold most of the parameters, use FP4 (four bits per number). Everything else uses FP8 (eight bits per number). FP4 is why the native checkpoint is so compact.

Why B200 uses ~146 GB. B200 is a Blackwell GPU, and Blackwell does FP4 math directly. It runs DeepSeek's checkpoint as released.

Why H100 needs ~284 GB. H100 is a Hopper GPU. It does FP8 math but not FP4. To run on H100, the FP4 expert weights get rewritten in FP8, which takes twice the bits for the same numbers, so the H100 checkpoint is all FP8. FP8 is one byte per number, and 284 billion parameters at one byte each come to about 284 GB. That's why the size on H100 matches the parameter count. The conversion changes how the numbers are stored, not what the model learned. It does cost speed, though: every word means reading the active weights from memory, and on H100 those weights are twice the size.

Why ours is 167 GB (about 170 GB in the bucket). The RTX PRO 6000 is also Blackwell, so like B200 it does FP4 math and runs the FP4 + FP8 weights exactly as DeepSeek released them. We deploy DeepSeek's DSpark edition, which is the same weights plus a ~7 GB draft module that DeepSeek publishes alongside the model to speed up generation. The model files come to 166.9 GB. The bucket shows about 170 GB because it also holds ~3 GB of download cache. Counted the same way, the native checkpoint itself is about 160 GB in decimal gigabytes, the same files Lambda rounds to ~146 GB.

Result 5: Answer quality

Fast answers are only useful if they're the right answers, so every release is checked for that as well as for speed.

Exactness. In exact mode, each reply is compared against what the model writes when it serves that user alone, and they have to match word for word. Across our release tests, 81 of 81 checked replies matched, including 32 users at once and a ~190,000-token document.

Quality battery. Release 1.0.7 was graded against the exact reference build, with 32 requests running in parallel:

On a replay of the same load-test traffic, release 1.0.7 measured 73.4 tokens/s per person, up from 66.5 (+10%), with quality scores equal to the exact reference.

Release gates. Every release is installed on a fresh machine and has to pass its gates before it ships. Release 1.0.2 passed 7 of 7:

A fresh one-click deploy of release 1.0.7 was serving in 9 minutes.

Result 6: V4-Flash on eight NVIDIA L4s

The NVIDIA L4 is a 24 GB, 72-watt card built for video, recommendation and small-model inference. Every major cloud offers it, and it's easy to get in regions where H100s aren't. Eight of them give you 192 GB in total, for a 284-billion-parameter model. Public sizing tools say a 24 GB card is too small for V4-Flash, NVIDIA's supported-hardware list for the model starts at the A100, and we couldn't find anyone who had even attempted it on L4s.

graphx runs it there, for many users at once, and it has served a live chat-and-agents product. It's the hardest thing in this post, and we're comfortable calling it an engineering marvel.

Why it's the hardest deployment of all

Every resource that matters for serving a big model is at its tightest on the L4. Eight L4s next to the machines this model normally runs on:

• The model barely fits. Even in DeepSeek's compact format, the model fills most of each card. What's left, about 4 GB per card, has to hold the working memory and every user's conversation, some of them 200,000 tokens long. An eight-H100 server still has more than half its memory free after loading the bigger FP8 copy.

• No FP4 math, and no room for the H100 workaround. The L4 is an Ada-generation GPU. Like H100, it can't do FP4 math. Unlike H100, it can't fall back to the FP8 copy either: at 284 GB, that copy is bigger than the L4 machine's entire 192 GB.

• Less memory bandwidth than a single H100. Each word means streaming the active weights out of memory. All eight L4s together read at 2.4 TB/s. One H100 reads at 3.35 TB/s, and an eight-H100 server at more than ten times our total.

• About an eighth of an H100 server's compute for reading documents and serving many users at once.

• No NVLink. All eight cards share PCIe, the slowest link of any machine in this post.

• About a tenth of the power. The eight cards draw 576 W together, less than one H100 (700 W).

Any one of these would make a 284-billion-parameter model hard to serve. The L4 has all of them at the same time.

The model takes 83% of every GPU's memory. The other 17% serves 32 users with long conversations.

What it adds up to

A machine with less memory bandwidth than one H100, about a tenth of an H100 server's power, no FP4 math and about 4 GB of free memory per card does all of this:

• serves the full 284-billion-parameter model to 32 users at once, at about 10 tokens/s each;

• gives a single user about 17 tokens/s, more than three times reading speed;

• reads an 8,000-token document in about 5 seconds;

• holds 150 simultaneous sessions with flat ~79 ms word spacing and zero errors;

• recalls facts from 200,000 tokens of context at every depth we tested.

Every one of those is above human reading speed and comes from the same certified production profile, with tool use, streaming, coding agents and an OpenAI-compatible API all working.

It also widens who can run a frontier model. The L4 is the most widely available inference GPU in the cloud, offered in regions and quotas where data-center GPUs aren't. A model that normally needs an eight-H100 or eight-B200 server runs on cards a team can rent today.

Summary table

What this means for a team

• One machine for the whole team. 32 people chatting at once at 68 tokens/s each, with replies starting in about half a second.

• GPUs you can actually get. RTX PRO 6000 machines are generally much easier to find than H100, H200 or B200 capacity, and L4s are everywhere.

• About half the cost per token of the B200 reference on Google Cloud, and far less on GPU marketplaces.

• Your cloud, your keys, your data. The stack installs into your own Google Cloud project. There's no shared endpoint, and prompts and answers stay in your account.

• Verified releases. Every update passes speed, exactness and quality gates on a fresh machine before it reaches you.

Who it's for

• Regulated industries. Banks, insurers, healthcare providers and public-sector teams that can't send documents or conversations to a third-party API.

• Company-wide assistants. Rolling a chat assistant out to hundreds or thousands of employees, with one machine covering about 2,400 heavy users a month.

• Platform teams watching cost. Anyone paying per token today who'd rather have a predictable monthly bill at about half the reference cost per token.

• Teams stuck on a GPU waitlist. RTX PRO 6000 and L4 capacity is available now.

• Products where the first word matters. Customer-facing chat, support and search, where even the unluckiest user sees text in about two seconds.

Why GraphX

1. Three firsts on the hardest targets.

   • DeepSeek V4-Flash on RTX PRO 6000, faster per person than Lambda's published H100 and B200 runs.

   • DeepSeek V4-Flash on eight 24 GB NVIDIA L4s, which as far as we can tell nobody had attempted before, on a machine with less memory bandwidth than a single H100.

   • GLM-5.2 on RTX PRO 6000 at 110+ tokens/s for one user at 128K context, ahead of every published setup on the same GPUs (see our companion post).

2. Workstation and inference GPUs get second-class support elsewhere. Today's open-source stacks are built for data-center GPUs first, and public issue trackers still show key kernels for the RTX PRO 6000's chip family failing or falling back to slow paths.

3. Long runs, not short benchmarks. The headline numbers here come from 90 minutes of continuous load and a full report.

4. The lead compounds. Each model we bring up starts from what the last one taught us, and every speed claim is tied to a quality check. Our estimate is that a well-funded team would need 12-24 months to get where we are now.

What the next POC will target

These are proof-of-concept results, and there's more speed available on the same hardware. Next up:

• higher per-person speed on long answers, building on release 1.0.7's +10%;

• faster first words when many large documents arrive at the same moment;

• more users per machine at the same per-person speed;

• more models on the same GPUs, with GLM-5.2 already running.

FAQ

Is it the full model? Yes. The RTX PRO 6000 results in this post use the complete DeepSeek V4-Flash, all 284 billion parameters, in exactly the FP4 + FP8 format DeepSeek released, plus DeepSeek's own draft module. Nothing is removed or distilled, and answer quality is checked on every release.

Who ran the test? Our load-testing team, using a dedicated load-testing tool with scripts that model real chat traffic. We quote the numbers as the report gives them.

Why does Lambda's H100 run use a 284 GB model? H100 can't do FP4 math, so the FP4 expert weights are stored in FP8 instead, which roughly doubles their size. FP8 is one byte per parameter, so 284 billion parameters come to about 284 GB, which is why the two numbers match. It's the same model in a wider format.

Why is your copy 170 GB and not 146 GB? We deploy DeepSeek's DSpark edition, which adds a ~7 GB draft module to the native weights (166.9 GB of model files), and the bucket also holds ~3 GB of download cache. Lambda's ~146 GB is the native checkpoint without the draft module, rounded.

What hardware do I need? One machine with 8x NVIDIA RTX PRO 6000. No H100, H200 or B200. If availability matters most, the model also runs on 8x NVIDIA L4.

Can it run in my cloud? Yes. One command installs it into your Google Cloud project, with your keys and your data, and no shared endpoint.

How many people can use one machine? The test ran 32 people at once at 68 tokens/s each. Over a month, the measured capacity covers about 2,400 heavy users.

Which clients work with it? Anything that speaks the OpenAI API, including streaming and tool calls.

How do updates work? Each release is a verified container with checksummed weights that passes the release gates on a fresh machine before it ships. You update with the same one-click command.

What about data privacy? The model, the conversations and the API key all live in your own cloud project. graphx doesn't run a shared endpoint.

Can we try it on our own workload? Yes. A proof of concept runs on your traffic (your documents, your users, your concurrency) and reports the same metrics as this post.

Will it get faster? Yes. The next POC targets higher per-person speed, faster first words under bursts and more users per machine.

Glossary

• Token. A piece of text, about three-quarters of an English word on average.

• Tokens per second (tok/s). How fast text streams to one person once it starts.

• ms per word (time per output token). The gap between consecutive tokens. 15 ms per word is about 67 tokens/s.

• First word (time to first token). How long someone waits before text starts to appear.

• Freeze. A pause in the middle of a streaming reply that's long enough to notice.

• p50, p90, p99. The value that 50%, 90% or 99% of replies beat. p99 describes the unluckiest 1%.

• Service level. The report's minimum bar for a reply: at least 10 tokens/s and a first word within 10 s.

• FP4, FP8. Number formats that store each weight in 4 or 8 bits.

• Release gate. A check a release has to pass before it ships.

Methodology

Load test. Our load-testing team's report, "Load test report: GraphX DeepSeek V4 Flash (8xRTX-6K)", run on 2026-10-26 at 21:00 UTC: 32 people, 90 minutes, 4-turn conversations opening with a ~4,000-token document, 4-7 s pauses, 30-60 s ramp, natural replies capped at 1,024 tokens. Reported: 11,413 replies; output 1,320 tok/s; decode speed per reply p50 68.1 and p10 57.5 tok/s; end-to-end 62.1 tok/s; first token p50 534 ms, p90 1.58 s, p99 2.32 s (turn 1: 1.06 s, later turns: 437 ms); time per output token p50 15 ms, p99 20 ms; 0% of time stalled; 0.3% errors (27 client fetch errors, 2 connection errors); prompt reading 8,557 tok/s; 20.3 requests in flight on average, 30 at peak. The report notes that the server carried other traffic during the run and that the default reasoning level was "medium". Working-day figures are the measured rates multiplied by 8 hours.

Lambda. Lambda's DeepSeek V4-Flash benchmark: 32 concurrent requests, 8,192 input and 1,024 output tokens. H100: 8 GPUs, FP8 checkpoint, SGLang + EAGLE. B200: native checkpoint, tensor parallel 4, SGLang + EAGLE and vLLM. Speed per person is 1 divided by mean time per output token. Lambda reports means; our typical values are medians from the load-test report.

Hardware and cost. From the load-test report. Reference configuration: 8x B200 192GB, per DeepSeek V4-Flash's published guidance. Google Cloud on-demand, us-central1, October 2026: g4-standard-384 (8x RTX PRO 6000) at $36.00/hour ($26,280/month) and a4-highgpu-8g (8x B200) at $64.44/hour ($47,041/month); Google Cloud offers B200 only as this 8-GPU machine. Neocloud: lowest marketplace listings (RTX PRO 6000 at $1.00 per GPU-hour, 8x B200 at $70/hour); stock varies. Monthly capacity is output tokens/s times 730 hours. The reference capacity is Lambda's published 1,222 output tok/s measured on its B200 run. Heavy user: 60 messages a day, 800-token replies, 8 active hours.

Model sizes. H100 and B200 sizes as listed by Lambda. graphx deploys deepseek-ai/DeepSeek-V4-Flash-DSpark: 166.9 GB of model files (170.3 GB in the bucket, including ~3.4 GB of download cache). The native checkpoint's files total 159.6 GB (148.6 GiB).

In-house measurements. Exact mode: 32 users released at once, a shared 17K-token preamble plus a 35K-token code context each, every reply compared word for word with a solo run. Release gates: release 1.0.2 on a fresh one-click deploy, 7/7. Quality battery and the 73.4 tok/s figure: release 1.0.7 against the exact reference, on a replay of the load-test traffic shape.

L4. Certified 8x L4 production profile. Hardware comparison uses vendor specifications summed over 8 GPUs (L4: 24 GB, 300 GB/s, 121 TFLOPS dense FP16, 72 W; H100 SXM: 80 GB, 3.35 TB/s, ~989 TFLOPS dense FP16, 700 W; B200: 192 GB, 8 TB/s, 2.25 PFLOPS dense FP16, 1,000 W). The ~4 GB per card is what's left of each L4's 24 GB after the model's 83%. "Nobody had attempted it" reflects our search of public reports, benchmarks and issue trackers.

Sources. Supported hardware: NVIDIA NIM V4-Flash. RTX PRO 6000 (sm_120) issues: vLLM #47818, SGLang #39868.