Exhaust the cheaper options first, in this order, because each is hours rather than a fortnight.
Improve the prompt properly. Not marginally, actually iterate against a held-out set of your examples and measure. Most prompts in production have never been through more than two revisions.
Choose examples deliberately. Few-shot examples chosen to cover your awkward cases beat more examples chosen at random. Three good ones often beat ten arbitrary ones, and that also cuts your cost.
Retrieve examples rather than fixing them. Pick the most similar few from your two thousand at request time. This gets you a lot of what fine-tuning promises with none of the training, and it improves automatically as your set grows.
Constrain the output. If the problem is format, structured output or a grammar enforces it absolutely rather than probabilistically, which is better than training for it.
Try a smaller model with the improved prompt. Your cost motivation might be satisfied without any training at all.
If after all that there is still a gap, you now have something valuable: a measured baseline and a set of failures. That is exactly what you need to tell whether a fine-tune helped, and starting without it is why people finish a training run unable to say whether it worked.