The two numbers people forget until they matter: cost and latency on your actual inputs.
Both are easy to measure while you are running the comparison and both are frequently the deciding factor:
Cost per real request, not per token in the abstract. A model that is cheaper per token and more verbose can cost more. Measure with your prompts and your typical inputs.
Latency at the percentile your users feel, not the average. If a request sits in front of somebody waiting, the slow tail is what they experience.
Write both into the same sheet as the quality scores. Then the decision is three numbers rather than a feeling, and it is often clear immediately - a small quality gain that doubles your latency is usually not worth it for an interactive product, and is fine for a background job.
Two operational notes since you run this in production:
Pin the version you tested. Testing one thing and running another because an alias moved underneath you is the most common way this whole exercise gets wasted.
Keep the old one available for a while after switching. The regressions you did not catch surface within days from real traffic, and being able to switch back in one config change is the difference between an inconvenience and a bad week.