Why does batched generation produce garbage unless I set the padding side to left?

Symptom worth adding to the list, because it is the quiet version of the same bug: subtly worse quality rather than obvious garbage.

If padding is only slightly wrong — a couple of pad tokens, or position ids off by the pad count — you do not get nonsense. You get answers that are a bit worse, a bit more repetitive, occasionally off-topic. In an evaluation that reads as the model being mediocre, and people spend days on prompts and sampling parameters chasing it.

Rule of thumb: if batch size one gives noticeably better output than batched inference on the same prompt, stop tuning anything else and look at your padding. Same prompt, same seed, both ways — that comparison finds it in a minute.

The attention mask does exactly what you think — the model does not attend to pad positions. The problem is not attention. It is where generation continues from.

A decoder-only model generates by taking the last position of the sequence and predicting what comes next. That is the whole mechanism. The next token is a function of the hidden state at the final position.

With right padding, a short prompt in the batch looks like:

[tok tok tok PAD PAD PAD]

The final position is a pad token. So the model is asked to predict what follows a pad — a position it has no meaningful representation for, because it was masked out of attention and never trained as a continuation point. You get whatever falls out of an undefined state, which is your repeated tokens and unrelated text. The longest sequence in the batch has no padding, which is why it looks fine and the short ones do not, and why batch size one never shows the problem.

With left padding:

[PAD PAD PAD tok tok tok]

every sequence's final position is a real token — the actual last token of its prompt. Generation continues from the right place for every row. The padding is at the front where the attention mask cleanly excludes it.

So the rule is not arbitrary: left padding puts the generation point where generation expects it.

Now why training is the opposite, which is the part that makes the rule look contradictory.

Training a causal model is not a loop. Every position predicts the next token in one forward pass, and the loss is computed across all positions, with pad positions masked out of the loss. There is no single "continuation point", so where the padding sits does not affect correctness — right padding is the convention because it keeps sequences aligned from index zero, which makes label alignment and slicing straightforward.

So:

• Generation → left, because only the last position matters and it must be real.
• Training → right, because all positions matter equally and alignment is easier.

They are not contradictory, they are answers about two different computations.

Two related things that will bite you next:

Position ids. With left padding the real tokens no longer start at position zero. Modern implementations derive positions from the attention mask and handle this correctly, but if you construct position ids yourself, or use an older or custom implementation, you will produce a subtle degradation rather than obvious garbage — which is far worse to find. Let the library compute them from the mask.

A missing pad token. Many causal models ship without one, and the usual workaround is to set the pad token to the end-of-sequence token. That is fine, but be aware you have now made the pad token identical to the stop token, so anything that scans output for the stop token needs the mask to disambiguate. Set it explicitly rather than letting something guess.

Strip the padding from the output. With left padding the generated continuation starts after the input length, and if you slice by a fixed offset across the batch you will cut into real text for some rows. Slice per row using each sequence's own input length.

A useful way to remember which one, if you would rather not re-derive it: the model reads left to right and writes from the right-hand end. Whatever is at the right-hand end is what it continues from, so that had better not be filler.

Also worth knowing that this whole class of problem disappears if your serving stack does continuous batching rather than static batching — it manages sequences independently rather than padding them into a rectangle, so there is no padding side to get wrong.

That is a good reason to use a real inference server rather than hand-rolling batched generation, once you are past experimenting. You inherit correct handling of this and several similar traps.

Add it to your tests. One case: run a batch containing prompts of clearly different lengths, and assert the output for each row matches the output you get running that same prompt alone.

It is a cheap test and it catches this whole family — padding side, position ids, per-row slicing — the moment someone changes the batching code. Without it the failure is invisible in aggregate metrics and only shows up as users saying the thing feels worse.