What actually makes a model stop generating? Mine runs on past its answer until it hits the token limit

The mechanism is simpler than people expect, and once you see it the failure is obvious.

There is no built-in notion of "finished". The model is a next-token predictor in a loop: given everything so far, produce a distribution over the vocabulary, sample one token, append it, repeat. Nothing in that loop knows about answers or turns.

Stopping is imposed from outside, by exactly three things:

1. The token budget — stop after N tokens. Blunt, always present.
2. A stop sequence — the runtime watches the output text for a string you specify and cuts generation when it appears.
3. The end-of-sequence token — a normal token in the vocabulary that the model was trained to emit when a sequence is complete. The loop checks each sampled token against a configured id and halts on a match.

So the model does not stop because it decided to. It emits a particular token, and your runtime recognises that token and stops the loop. Both halves have to work.

Now the failure you have, which is almost always one of these two:

The id mismatch. Instruction-tuned models generally have two special tokens: a classic end-of-sequence, and a separate end-of-turn token used by the chat format. The model is trained to end its reply with the end-of-turn token. But the tokenizer configuration frequently still lists the old end-of-sequence as the stop id, so the model correctly signals it is done, the runtime does not recognise that particular token as a stop, and generation continues straight into a new turn. Which is precisely your symptom: a complete answer, then a hallucinated user, then another answer.

Fix: find the model's end-of-turn token id and add it to the stop ids. Most runtimes accept a list, so add both. Check the model's own config and template rather than assuming — this varies per model family and it is exactly the thing that gets copied wrongly between projects.

The wrong prompt format. The end-of-turn behaviour is trained together with a specific chat template — particular delimiters around the system, user and assistant turns. Feed the model a plain string, or someone else's template, and it is not in the state where it learned to emit that token. It just continues the text, which is what a language model does by default.

Fix: use the model's own chat template. Every serious runtime can apply the template that shipped with the weights; do not hand-roll the delimiters.

One more possibility worth ruling out: it may not be an instruct model at all. A base model has no notion of turns and will never stop naturally — continuing forever is correct behaviour for it. If you downloaded the base rather than the instruct or chat variant, no amount of stop-token configuration will help and the fix is a different download.

Belt and braces once the above is right: set a stop sequence on the string that begins a user turn in your template. It catches the case where the model produces the delimiter as ordinary text rather than as the special token, which does happen, particularly with heavily quantised models.

The base-versus-instruct point deserves more weight than it usually gets, because the repositories make it easy to get wrong — the names often differ by one suffix and the file sizes are identical.

Quick test that takes ten seconds: prompt it with something that is obviously a fragment, like an unfinished sentence. A base model happily continues the sentence. An instruct model tends to respond about the fragment, or asks what you want.

If it continues the sentence, you have a base model and everything else you are debugging is beside the point.

Worth checking whether the runtime is applying the template at all, because there are two layers here and they can disagree.

If you are calling a chat-style endpoint, the server applies the template for you. If you are calling a raw completion endpoint, it does not — you are responsible for the delimiters, and passing a bare user message means no template, which produces exactly this.

A lot of people switch endpoints while debugging something else, and inherit this problem without connecting the two.

Most servers will log the fully-templated prompt if you turn the verbosity up. Look at it once. It is usually immediately obvious whether the delimiters are there.

If you are consuming this through an API in your own code, handle the stop reason rather than just the text.

Every reasonable API returns why generation ended — the token limit, a stop sequence, or the end token. Logging that one field turns "the output looks odd sometimes" into a precise fact, and it is how you find out that five percent of your responses are being silently truncated by the budget rather than finishing.

Same field tells you whether the fix above actually worked, instead of eyeballing outputs.