Hyun-woo

A useful way to remember which one, if you would rather not re-derive it: the model reads left to right and writes from the right-hand end. Whatever is at the right-hand end is what it continues from, so that had better not be filler.

Also worth knowing that this whole class of problem disappears if your serving stack does continuous batching rather than static batching — it manages sequences independently rather than padding them into a rectangle, so there is no padding side to get wrong.

That is a good reason to use a real inference server rather than hand-rolling batched generation, once you are past experimenting. You inherit correct handling of this and several similar traps.

The mechanism is simpler than people expect, and once you see it the failure is obvious.

There is no built-in notion of "finished". The model is a next-token predictor in a loop: given everything so far, produce a distribution over the vocabulary, sample one token, append it, repeat. Nothing in that loop knows about answers or turns.

Stopping is imposed from outside, by exactly three things:

1. The token budget — stop after N tokens. Blunt, always present.
2. A stop sequence — the runtime watches the output text for a string you specify and cuts generation when it appears.
3. The end-of-sequence token — a normal token in the vocabulary that the model was trained to emit when a sequence is complete. The loop checks each sampled token against a configured id and halts on a match.

So the model does not stop because it decided to. It emits a particular token, and your runtime recognises that token and stops the loop. Both halves have to work.

Now the failure you have, which is almost always one of these two:

The id mismatch. Instruction-tuned models generally have two special tokens: a classic end-of-sequence, and a separate end-of-turn token used by the chat format. The model is trained to end its reply with the end-of-turn token. But the tokenizer configuration frequently still lists the old end-of-sequence as the stop id, so the model correctly signals it is done, the runtime does not recognise that particular token as a stop, and generation continues straight into a new turn. Which is precisely your symptom: a complete answer, then a hallucinated user, then another answer.

Fix: find the model's end-of-turn token id and add it to the stop ids. Most runtimes accept a list, so add both. Check the model's own config and template rather than assuming — this varies per model family and it is exactly the thing that gets copied wrongly between projects.

The wrong prompt format. The end-of-turn behaviour is trained together with a specific chat template — particular delimiters around the system, user and assistant turns. Feed the model a plain string, or someone else's template, and it is not in the state where it learned to emit that token. It just continues the text, which is what a language model does by default.

Fix: use the model's own chat template. Every serious runtime can apply the template that shipped with the weights; do not hand-roll the delimiters.

One more possibility worth ruling out: it may not be an instruct model at all. A base model has no notion of turns and will never stop naturally — continuing forever is correct behaviour for it. If you downloaded the base rather than the instruct or chat variant, no amount of stop-token configuration will help and the fix is a different download.

Belt and braces once the above is right: set a stop sequence on the string that begins a user turn in your template. It catches the case where the model produces the delimiter as ordinary text rather than as the special token, which does happen, particularly with heavily quantised models.