$700 total. Motherboard has one PCIe x16 slot and one x4, 750W PSU, and I would rather not replace either.
Two 3060 12GB cards gets me 24GB for around $500 with room left over. One used 3090 is 24GB for roughly $650-700 and leaves nothing. Same VRAM on paper.
Goal is a 70B at Q4 for code review on my own repos. Which way do people go, and does the x4 slot ruin the two-card plan?
Two 3060s do work fine in llama.cpp with a tensor split, and if you value capacity per dollar over speed they are defensible - I ran that pair for a year. Numbers from mine: 30B class at Q4, roughly 11 tok/s split across both, versus about 26 on a single 3090 doing the same thing. Half the price, less than half the speed, twice the fan noise.
Reframe the goal. At $700 the question is not "how do I run a 70B", it is "what is the best code review I can get from 24GB".
A 32B coding model at Q4_K_M on one 3090 runs at 28 tok/s with 8k of context and is genuinely useful on real diffs. A 70B at 3 tok/s is a thing you demo once and never open again, because reviewing a 400-line diff takes eleven minutes and you will just read it yourself.
Speed is a quality feature for interactive tools. Nobody iterates against something that slow.
Before committing either way, rent an hour on a cloud box with enough VRAM and run the 70B against ten of your own diffs. Ten dollars answers the question of whether the bigger model is even better for your repos, which is not a given. I did this and the 32B was better on my code because the 70B kept inventing framework APIs.
Also budget for the case and airflow. Two cards in consumer slots means the top one is breathing the bottom one's exhaust, and thermal throttling turns your bandwidth advantage into a slideshow after twenty minutes of sustained generation.
$700 for hardware that will be embarrassing in 18 months to review code that takes you 4 minutes to read yourself. I say this as someone with two of them under the desk.
@orm_tamsin · 6mo ago · 3 replies
Single 3090, and I want to be honest that it does not get you what you asked for either.
Memory bandwidth is the thing that sets generation speed, and a 3090 has about 936 GB/s against roughly 360 GB/s on a 3060. Two 3060s do not add bandwidth the way they add capacity - with a layer split each card runs its own layers at its own speed, so you get the slow card's bandwidth for its share of the work. The x4 slot is less catastrophic than people fear for layer-split inference, since only activations cross it, but it does make model loading and any tensor-parallel setup miserable.
The part nobody says out loud: a 70B at Q4_K_M is roughly 40GB of weights. Neither of these configurations runs it fully offloaded. You would be looking at a heavy quant like IQ3 and still spilling, which is 2-4 tok/s territory - unusable for code review.
Reply
Report
@hornworm_hunt · 6mo ago
This is the answer I did not want and clearly needed. I had done the VRAM arithmetic on parameter count and forgotten the cache and the buffers entirely.
Reply
Report
@teardown_tues · 6mo ago
One more thing on used 3090s specifically: check transient power draw against that 750W. Mine trips a perfectly adequate PSU on spikes unless I power-limit it to 300W, which costs about 6% of performance and solves it completely.
Reply
Report