I have a dataset and I am not sure fine-tuning is the answer - when is it actually worth it?

Exhaust the cheaper options first, in this order, because each is hours rather than a fortnight.

Improve the prompt properly. Not marginally, actually iterate against a held-out set of your examples and measure. Most prompts in production have never been through more than two revisions.

Choose examples deliberately. Few-shot examples chosen to cover your awkward cases beat more examples chosen at random. Three good ones often beat ten arbitrary ones, and that also cuts your cost.

Retrieve examples rather than fixing them. Pick the most similar few from your two thousand at request time. This gets you a lot of what fine-tuning promises with none of the training, and it improves automatically as your set grows.

Constrain the output. If the problem is format, structured output or a grammar enforces it absolutely rather than probabilistically, which is better than training for it.

Try a smaller model with the improved prompt. Your cost motivation might be satisfied without any training at all.

If after all that there is still a gap, you now have something valuable: a measured baseline and a set of failures. That is exactly what you need to tell whether a fine-tune helped, and starting without it is why people finish a training run unable to say whether it worked.

The distinction that decides it: fine-tuning changes how a model responds. It is a poor way to change what it knows.

So it works well for:

Format and structure. Always producing a particular shape, in a particular style, without being asked each time.

Tone and voice. Consistently sounding like your product rather than like a general assistant.

A narrow task done repeatedly, where you want a small fast model to match a large one on that one thing.

Reducing prompt size, which is your cost problem - behaviour baked in is behaviour you stop sending.

It works badly for:

Teaching facts. Training on documents does not reliably make a model able to recall them, and it will confidently produce plausible variations. Retrieval is the tool for knowledge, not training.

Anything that changes often. Retraining is not an edit.

Improving general reasoning. A small model fine-tuned on your data is still a small model.

So the question to ask about your two thousand examples: are they teaching a shape or teaching content? If the model already can do the task and is inconsistent about how, fine-tuning is the right tool. If it is getting the substance wrong, it is not.

The cost that gets left out of the comparison: a fine-tuned model is a thing you now own and maintain.

What that means in practice:

You are pinned to a base model. When a better one arrives you do not get it for free - you retrain, re-evaluate and redeploy. People with a fine-tune from eighteen months ago are frequently behind what a good prompt on a current model does.

Every change to the task means retraining, and retraining means re-evaluating.

You need an evaluation set anyway, which is most of the work people imagine fine-tuning avoids.

You are running inference yourself, with the operational load that implies.

None of that is a reason not to do it. It is a reason to be honest that the fortnight is the beginning rather than the end.

The cases where I would do it without hesitation: a narrow high-volume task where a small local model matching a large hosted one saves real money, or a hard requirement that the data never leaves your machines.

The case where I would not: hoping it makes the model generally better. That is where the fortnight disappears and the result is a model that is worse at everything and slightly better at formatting, which is exactly the outcome you were worried about, and your instinct is right.