27
Why does batched generation produce garbage unless I set the padding side to left?
I batch several prompts together for generation. With a batch size of one everything is fine. As soon as I batch prompts of different lengths, the shorter ones come back as nonsense — repeated tokens, unrelated text,…