The attention mask does exactly what you think — the model does not attend to pad positions. The problem is not attention. It is where generation continues from.
A decoder-only model generates by taking the last position of the sequence and predicting what comes next. That is the whole mechanism. The next token is a function of the hidden state at the final position.
With right padding, a short prompt in the batch looks like:
[tok tok tok PAD PAD PAD]
The final position is a pad token. So the model is asked to predict what follows a pad — a position it has no meaningful representation for, because it was masked out of attention and never trained as a continuation point. You get whatever falls out of an undefined state, which is your repeated tokens and unrelated text. The longest sequence in the batch has no padding, which is why it looks fine and the short ones do not, and why batch size one never shows the problem.
With left padding:
[PAD PAD PAD tok tok tok]
every sequence's final position is a real token — the actual last token of its prompt. Generation continues from the right place for every row. The padding is at the front where the attention mask cleanly excludes it.
So the rule is not arbitrary: left padding puts the generation point where generation expects it.
Now why training is the opposite, which is the part that makes the rule look contradictory.
Training a causal model is not a loop. Every position predicts the next token in one forward pass, and the loss is computed across all positions, with pad positions masked out of the loss. There is no single "continuation point", so where the padding sits does not affect correctness — right padding is the convention because it keeps sequences aligned from index zero, which makes label alignment and slicing straightforward.
So:
- Generation → left, because only the last position matters and it must be real.
- Training → right, because all positions matter equally and alignment is easier.
They are not contradictory, they are answers about two different computations.
Two related things that will bite you next:
Position ids. With left padding the real tokens no longer start at position zero. Modern implementations derive positions from the attention mask and handle this correctly, but if you construct position ids yourself, or use an older or custom implementation, you will produce a subtle degradation rather than obvious garbage — which is far worse to find. Let the library compute them from the mask.
A missing pad token. Many causal models ship without one, and the usual workaround is to set the pad token to the end-of-sequence token. That is fine, but be aware you have now made the pad token identical to the stop token, so anything that scans output for the stop token needs the mask to disambiguate. Set it explicitly rather than letting something guess.
Strip the padding from the output. With left padding the generated continuation starts after the input length, and if you slice by a fixed offset across the batch you will cut into real text for some rows. Slice per row using each sequence's own input length.