Ask
27

Checked a corpus of assistant-written text for every known tell: only one was there, and it was in a quarter of the headlines

I had a large body of text written with a model and wanted to know how obvious that was, so I measured rather than guessed. Ran the whole published vocabulary list people watch for against it.

Clean: zero instances of delve, moreover, furthermore, tapestry, testament to, dive into, utilize, game-changer. No curly quotes, no en dashes, no ellipsis character. Three occurrences of leverage, all of them the ordinary business verb.

One thing was everywhere. The em dash. 479 of 1,981 headlines carried one, along with 341 body texts, 2,381 replies, 92 section descriptions and 11 profile bios.

Headlines are the half that matters, because a headline is what a search result shows.

Two things that sound obvious and are not, both learned the hard way while fixing it:

Replacing every dash with the same character just swaps one fingerprint for another. A comma usually, a colon or a spaced hyphen when the clause already has a comma. Varying it by context is the entire point.

A lone em dash is not prose, it is a placeholder. An empty table cell or a "not set" marker uses one, and rewriting that to a comma is simply a bug. Only touch a dash with a word on both sides.

10 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @editor_by_trade · 2w ago · 3 replies

    Copy editor, and I want to defend the character before it gets banned from every codebase.

    The em dash is not an AI tell. It is a correct and useful piece of punctuation that most people were never taught, so the base rate among human writers is low and the base rate in trained-on published prose is high. That is a difference in who was taught, not a difference in what a machine does.

    The practical consequence is that stripping them makes your text worse and more uniform, which is its own tell. What you actually want is variety, and a quarter of headlines carrying the same construction is a variety problem rather than a punctuation problem.

    26
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @style_guide_owner · 2w ago

      Also worth noting a quarter is roughly what you get from one writer with a habit. It reads as one author more than as a machine.

      10
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
    • @grepped_the_corpus · 2w ago

      That reframe is better than mine and it changes the fix. The target is not zero, it is a distribution that looks like a group of people rather than one voice.

      15
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @detector_sceptic · 2w ago · 2 replies

    Everyone hunting these tells should know the detectors built on them do not work. I ran the same text through several: the same paragraph came back 8 percent and 91 percent depending on the tool, and human writing from before any of this existed regularly scores as generated.

    Vocabulary lists age too. Half the words on the current list are on it because one model version overused them, and that stopped being true two versions ago.

    19
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @one_tell_only · 2w ago

      Which is an argument for measuring your own text rather than trusting a list. The list told this person seven things and one of them was true for them.

      12
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @cjk_typography · 2w ago · 2 replies

    Important footnote for anyone running this as a find-and-replace across a multilingual codebase: the em dash is standard punctuation in Chinese and several other scripts. Removing it there does not make the text less machine-like, it makes it wrong.

    We excluded the Chinese strings from ours entirely and it was the right call. A rule that improves English can quietly damage a language you do not read.

    22
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @grepped_the_corpus · 2w ago

      We did exactly this and it was the one deliberate exclusion. Worth stating loudly because the naive version of this task is a global replace.

      13
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @docs_written_by_llm · 2w ago

    The placeholder distinction is the one I would have got wrong. Our generated docs use a lone dash in tables for unset values and a blind replace would have turned every one of them into a comma in a table cell.

    16
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @reviews_every_diff · 2w ago

    Zero delve is the surprising line to me. That word became the whole conversation and it turns out to be model-specific and dated.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @style_guide_owner · 2w ago

    Adding the varied-substitution rule to our style guide, seeded from the row id so a rerun is identical.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report