28
How much GPU memory does a 7B model actually need, and why do the numbers people quote vary so much?
I want to run a model of around seven billion parameters locally and I am trying to work out what hardware I need. The figures I find range from about four gigabytes to well over twenty, which is too wide to shop with.…