176
@trace_tilly ·

Build the manual version by hand or spend two weekends automating it first Cut the Scope

My idea is a weekly digest of new listings in a niche marketplace. Doing it by hand takes me about 90 minutes a week: search, copy, write short notes, send. I have six subscribers, all people I know. Writing the scraper and the sending pipeline would take me two full weekends, maybe more given the site changes its layout regularly. Part of me thinks automating first is obviously right and part of me thinks that is the trap.

8 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @ask_first_ari · 9mo ago · 3 replies

    Do it by hand until it hurts, and be specific about what hurting means so you know when you have arrived.

    Right now 90 minutes a week for six people is completely sustainable. Automating that saves you 90 minutes and costs you two weekends, so it pays back in about ten weeks if nothing breaks, and something will break because you already said the site changes layout.

    The number I would use as a trigger is subscribers. At six, do it by hand. At fifty, the manual version starts costing real time and the digest matters enough to be worth building. Between those, automate the single most annoying step rather than the whole pipeline. For most people doing this, that is the copying, not the sending.

    There is also a thing you get from the manual version that you cannot buy later: you will notice which listings are actually interesting, and that judgement is the only part of your digest that a scraper cannot reproduce.

    268
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @scopecreep_sam · 9mo ago

      The trigger number is the useful part. Without one, do it manually becomes forever, and people burn out doing 90 minutes a week for two years while telling themselves they are validating.

      96
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @margin_notes_mo · 9mo ago

      Second the point about judgement. The digests I actually read are the ones where a human decided something was worth including. A complete automated feed of everything new is a thing I already have and ignore.

      42
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @runway_math · 9mo ago · 2 replies

    Split the difference. Do not build the scraper, build the boring middle. A spreadsheet with the columns you fill in and a template that turns rows into a formatted email is maybe two hours of work and cuts your 90 minutes to perhaps 50. Scraping a site that changes layout is the expensive fragile part, and the formatting and sending is the cheap durable part.

    Automate durable things first, fragile things last, ideally never.

    142
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @null_pointer_ok · 9mo ago

      Durable versus fragile is a good filter generally. Anything that depends on somebody else's HTML is fragile by definition, and anything that only touches your own data will still work in three years.

      51
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @boundaries_bo · 9mo ago

    Track where the 90 minutes actually goes for two weeks before you build anything. Everybody assumes it is evenly spread and it never is. Mine turned out to be 60 minutes of searching and reading and 30 of everything else, which meant the pipeline I was about to build would have saved me half an hour.

    108
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @capsule_curious · 9mo ago

    Six subscribers who are all people you know is also worth thinking about. Before spending two weekends, send it to twenty strangers and see whether anyone opens it twice. That tells you whether the automation is worth building at all.

    26
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @gradschool_gary · 9mo ago

    One risk of automating first that people underrate: once the machine is producing it, you stop reading it closely, quality drifts, and you do not notice until someone unsubscribes. The manual version keeps you honest for as long as you can stand it.

    57
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report