Ask
158
@row_level_sam ·

internal tool nobody touches at the weekend takes 4s on the monday hit but 900ms cold on a weekday

node function on a serverless platform, managed postgres with scale to zero, about 200 requests a day and all of them in office hours. a normal weekday cold start is around 900ms and consistent. the first request after a quiet weekend is reliably three to four seconds and i cannot work out which part is slower. is there a longer idle penalty than the usual cold start or is something else waking up?

9 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @gramgrader_gus · 2w ago · 3 replies

    You have two cold starts stacked and you are measuring their sum. The runtime cold start is your 900ms and it does not care whether it idled for ten minutes or three days. The extra seconds are almost certainly the database resuming, because scale to zero providers suspend after a short idle window and the resume is measured in seconds, not milliseconds. Instrument the connect separately from the query and from the handler and the picture resolves immediately.

    241
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @four_percent_fi · 2w ago

      exactly the shape i saw. weekday requests keep the db awake by accident and the weekend removes that.

      68
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @row_level_sam · 2w ago

      that would explain why it is the same 900ms on a weekday, since the db is never actually idle long enough during the week.

      52
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @kernel_ring_buf · 2w ago

    if you do want it fixed cheaply, one scheduled job at eight on monday morning that opens a connection and runs select 1. costs nothing, runs once a week, and the first human of the week gets a warm path.

    111
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @tessa_ondrak · 2w ago

    i timed this on ours by wrapping four phases with timestamps: process init, module imports, first db connect, first query. on a monday the connect was 2.8s of a 3.9s request and everything else was unchanged from a weekday. thirty minutes of instrumenting saved a week of guessing at bundle size.

    187
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @pinned_versions · 2w ago

    for an internal tool with 200 requests a day, consider not fixing it. a loading state that says waking up costs you an hour and the alternative is paying for something to stay running all weekend for one slow request on monday.

    128
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @rollback_rae · 2w ago · 2 replies

    worth ruling out the other candidate before you settle on the database. after a long idle your deployment artifact can fall out of whatever regional cache it was in, so the platform fetches and unpacks it again rather than reusing a warm copy nearby. that also shows up as a first hit that is worse than a normal cold start and it looks identical from the outside if you only measure total duration.

    149
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @visa_run_val · 2w ago

      true, and the way to separate them is a hello world function on the same platform with no database. if that is also slow on a monday it is the platform, if it is not, it is your db.

      91
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @borrowck_ben · 2w ago

    i solved this the wrong way first. added a cron to ping the function every few minutes, which kept the function warm all weekend and made no difference at all on monday morning, because the ping did not touch the database. the warmer has to exercise the slow dependency or it is just burning invocations.

    163
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report