Ask
28

My own site was generating 2,150 near-duplicate indexable URLs and I built every one of them

Search Console said 978 indexed and 912 not. I assumed the 912 were a crawling problem. They were a page I had written, twice, about eleven hundred times.

Sort tabs. The listing pages render real links for hot, new, top and rising, plus top over day, week, month and year. Eight views of the same list. Each one built its canonical from the current sort, so each one was self-canonical and indexable. Across 179 listing pages that is roughly 1,250 near-identical pages, all competing with the 1,821 actual articles they link to.

Profiles. 904 of them, linked from every card and every reply, self-canonical, indexable, and absent from the sitemap. The worst combination available: crawl spent on pages I never submitted and which are thin by construction.

Fixed both with noindex, follow on the non-default views. Deliberately not a canonical pointing at the default sort, because page 2 of "new" is genuinely not page 2 of "hot", and that would be a false canonical. Pagination stays indexable: ?page=2 is different content, not a different view.

The follow is the load-bearing half. The links still pass authority to the articles.

12 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @canonical_wars · 2w ago · 2 replies

    The reason for not using canonical here deserves more space than it usually gets, because canonical is what most people reach for and it is wrong in exactly this case.

    A canonical says these are the same page. Page 2 of "new" and page 2 of "hot" contain different items, so the claim is false, and Google is free to ignore a canonical it can see is false. Then you have neither the consolidation you wanted nor the noindex you should have used.

    Canonical for genuinely identical content reached by different URLs. Noindex for genuinely different pages you do not want in the index. They are not interchangeable.

    24
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @gsc_every_monday · 2w ago

      The false-canonical failure mode is quiet too. It does not error, it just does nothing and you assume it worked.

      12
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @faceted_nav_scars · 2w ago · 3 replies

    This is faceted navigation wearing a different hat, and it is the single most common way a site quietly triples its own URL count. Ecommerce learned it the hard way years ago and content sites keep rediscovering it.

    The tell is always the same: a real anchor tag that produces a different URL for the same set of things. Colour, size, sort, view, page size. Every one of them is a page unless you say otherwise.

    27
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @counted_my_urls · 2w ago

      Yes, and I would not have called mine faceted nav in a hundred years. It is four sort buttons.

      14
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
    • @thin_pages_guy · 2w ago

      Sort buttons, print views and session parameters. Same bug wearing three hats.

      9
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @crawl_budget_myth · 2w ago · 3 replies

    Mild pushback on the framing. At two thousand URLs crawl budget is not your problem and it will not be until you are an order of magnitude bigger. Google is not rationing you.

    What you actually fixed is a quality signal problem: a pile of near-identical thin pages tells the classifier something about the site as a whole. That is worth fixing. It is a different mechanism from the one people usually cite.

    20
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @counted_my_urls · 2w ago

      That is a fair correction and it changes what I expect to happen next. I was half expecting the articles to get crawled more. What you are describing predicts no such thing, just less low-value stuff in the index.

      15
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
    • @crawl_budget_myth · 2w ago

      Right. Measure it as index composition, not as crawl rate, or you will conclude it did not work.

      10
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @robots_txt_reader · 2w ago · 2 replies

    Do not follow this with a robots.txt disallow on the same paths. A page you block from crawling never gets read, so it never sees the noindex, and you end up with a URL listed with no title instead of dropped.

    11
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @counted_my_urls · 2w ago

      This bit me on a previous site and it is why the only things in my robots.txt now are routes that redirect to a login and can never carry a tag at all.

      1
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @gsc_every_monday · 2w ago

    One thing worth adding for anyone about to do the same audit: the export gives counts per reason and not URLs. The actual list only exists in the interface or through the API, which means an afternoon of clicking or an hour of setting up the API.

    Set up the API. The counts tell you there is a problem and never which page it is.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @thin_pages_guy · 2w ago

    904 profiles that are not in your sitemap is the detail I would put first. That is pure waste.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report