A 90 second stall, fixed
I serve DeepSeek-V4.1-Flash across two DGX Sparks with the
MiaAI-Lab recipe
on the TensorFold engine. This morning a chat froze for two minutes the moment a second request
arrived with a very long prompt. I wrote it up as an issue, then wrote the fix. The lab merged it into
main the same day.
The bug
With --parallel 4, a stream that has already finished its prompt pass is supposed to keep decoding
while another request prefills. It didn't. Admitting a request runs its whole prompt pass before the engine
takes another decode round, so every live stream waited out the entire pass. A 201,336-token cold prompt took
119 seconds; the stream in front of it drew one token and then nothing.
Cancels had the same shape. The server only checks for disconnects at the top of its loop, so a client that left mid-pass was not noticed until the pass returned (logged 100 ms after it ended). The GPU was not stuck: once the pass finished, the cold request decoded 386 tokens in about five seconds. The engine was busy, not hung, and it was busy in one uninterruptible block.
The fix
Patch 0005-prefill-interleave: the engine already runs a prompt pass in 2,048-row chunks. Between two
chunks, rank 0 now runs decode rounds for the other live streams for up to N ms, delivers their
tokens, and takes their cancels. The pass itself is left alone, and a cancel of the request being filled is
seen at the next chunk boundary. The patch ships off by default; the recipe turns it on with
PREFILL_INTERLEAVE_MS=250, passed to rank 0 as TF_DS_PREFILL_INTERLEAVE_MS.
Prompts with images still run whole. I added a host test with it: a fake backend, a six-chunk pass, and a
decoding stream that must get tokens and cancel before the pass ends.
The numbers
These are the lab's measurements on their pair, not mine:
| Test | Before | After |
|---|---|---|
| Longest gap in a decoding stream during a ~120K-token cold prompt | 90.1 s | 1.3 s |
| Cancel noticed mid-pass | after the pass ends | 0.74 s |
tools/exact.py, output vs previous image | — | 31 / 31 identical |
The 31-case match is the one I care about most. Interleaving reorders work; it must not change tokens.
What I don't claim
I reproduced the stall on my pair (the log lines are in the issue). The before/after table above comes from the maintainers' own run, and they rewrote the README and changelog for the merged state. I have not re-run it on their hardware or mine since the merge.
Sources:
issue #3 ·
PR #5 (closed after the
content was merged as e65785c) ·
credit in the repo's CREDITS.md and CHANGELOG.