Before moving imports, look for work happening at module scope, because that is often bigger than the imports themselves and much easier to fix.
Things that run when the module loads, whether or not anyone asked:
Client construction. Database clients, SDK clients, HTTP clients created at the top of a file. Some are cheap; some open connections or read configuration.
Configuration parsing and validation. A schema validating your whole environment at startup is doing real work on every cold start.
Anything awaited at module scope. This is the worst case, because the module cannot finish loading until it completes - a network call there is directly in the cold path of every request.
Large constants built by code, rather than declared. A lookup table computed from a list at import time.
The fix for most of these is the same shape as your lazy import: build it on first use and keep it, rather than building it at load. A small helper that creates the thing once and returns the cached instance afterwards covers nearly all of it, and it is a much smaller change than restructuring imports.
That pattern also solves your worry more neatly, because the cost is paid once per instance rather than once per request - the second request to that route is fast again.