There are real, explainable drivers, and there is story on top. Roughly in order of how much of the price they account for:
Picking standard. How much of each pluck is bud and young leaf versus mature leaf and stem. This is labour, it is measurable, and it is most of the difference at the lower and middle of the range. A bud-heavy pick takes far more hands to produce the same weight.
Season. First flush of the year is limited and in demand; later pickings are more plentiful. Same field, same maker, several times the price.
Processing skill and scale. Hand-finished small batches cost more than machine processing, and for oolong in particular the roasting and rolling stages are where a skilled maker earns the money. This is real and it is the hardest thing to see on a label.
Origin scarcity. A specific small area with a reputation. Real, and this is where the multiplier gets steep and where the story starts doing the work.
Story. Old trees, wild, single tree, a named picker. Sometimes true, unverifiable in most cases, and priced as though always true.
So the honest answer: the first three are worth paying for and you can taste them. The fourth is real but the price curve leaves the quality curve behind. The fifth is the part you are right to be sceptical about.