Ask
96
@rest_day_rita ·

812 users on 1.4.2 got the 1.5.0 js bundle overnight, how do i unship an update ota-update

I was cleaning up branches and repointed a channel. I understand now what that did.

812 installs of 1.4.2 are currently running JS from 1.5.0, which expects a native module that is not in their binary. Crash rate went from basically nothing to about 1 in 4 sessions. Diagram of the mapping at the moment it happened attached.

What is the fastest correct thing to do right now, and then what stops this from being possible next time?

10 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @treadmill_tara · 2d ago · 3 replies

    Roll back first, understand second.

    eas update:rollback on the affected branch, or republish the last known-good update onto it. Either way the fix ships as an update, which means it reaches clients as fast as they check - minutes, not a store review. Do that before you read the rest of this.

    Then eas channel:view production to confirm what it actually points at now, and fix the mapping.

    The structural fix is the runtime version. Both of your builds carried the same one, which is exactly why the 1.5.0 bundle was considered compatible with a 1.4.2 binary - the system checked, and you told it they matched. Move to the fingerprint policy, which hashes the native side so any native change automatically produces a different runtime version. Then a mismatched update is simply not offered to that client and this specific mistake becomes impossible rather than merely unlikely.

    143
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @stack_trace_sy · 3h ago

      Fingerprint is the whole answer long term. Maintaining a runtime version by hand means the safety mechanism only works when the person doing the release remembers it exists, which is the same as it not working.

      41
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @bench_dogged · 3h ago

      Set expectations on the graph: clients pick up the rollback on their next check, so crashes taper rather than stop. Do not panic-ship three more updates in ten minutes because the line has not moved yet - that is how a bad hour becomes a bad day.

      18
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @deadbug_wiring · 19h ago · 2 replies

    Once the fire is out, check whether the bad bundle wrote anything persistent. If 1.5.0 migrated the shape of anything in local storage, rolling the JS back leaves those 812 devices holding data from the future, and the old code will read it and do something interesting.

    That is the part that shows up three days later as a mysterious crash on a small number of devices you cannot reproduce.

    62
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @late_stage_phd · 12h ago

      Local storage migrations that only go one direction are the mobile equivalent of a database migration with no down. Worth writing the backward guard now while the cost of not having one is fresh.

      24
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @rawfileruth · 2d ago

    Operational rule that would have prevented this: channels are immutable once a build carrying them is in the wild. Never repoint one. If you need a different branch, cut a new channel and a new build. It feels heavy-handed until the first time it saves you, which was yesterday.

    34
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @p99_hana · 3h ago

    Gradual rollouts exist for this. Even 10% for one hour turns a 812-user incident into an 80-user annoyance you catch on a dashboard before anyone emails you.

    8
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @bench_dogged · 3h ago · 2 replies

    Push a new store build with the correct bundle, that supersedes the update.

    16
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @treadmill_tara · 18h ago

      That is review time plus however long people take to update, so days for most of the 812. And it does not remove the bad update for anyone who does not install it, so the crash keeps happening on the old binary the whole time.

      The entire point of the update mechanism is that this class of problem is fixable in minutes. Use it.

      13
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @hemline_hana · 3h ago

    Small thing that pays off exactly now: pass --message on every update with what changed and why. When you are looking at a dashboard at 2am trying to work out which one was good, a list of commit hashes is not a list of information.

    11
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report