Your instinct is right, your worry is also right, and both are resolvable - but measure before you refactor a dozen routes, because the answer differs a lot by runtime.
What to measure. Timestamp at the very top of the module, timestamp at the start of the handler, and log the difference on a cold invocation. That gap is your init cost, and it is the number this whole exercise is about. Without it you are guessing at which imports matter.
What you will usually find: a small number of dependencies account for nearly all of it. Anything with native bindings, anything that builds large lookup tables at import, anything that reads or parses files at module scope. The PDF library is a classic. Nine small imports are usually irrelevant.
So refactor the two or three that dominate, not all twelve routes.
On your worry: yes, a lazy import moves the cost to the first request that needs it. Whether that is worse depends entirely on the ratio. If one request in fifty makes a PDF, forty-nine requests get faster and one gets slower: clearly good. If half your traffic hits that route, you have moved the cost around for nothing.
The rule: lazy-load what is rare and expensive. Eager-load what is common, however expensive.