Crawlers are traffic, but they are not people

A meaningful share of requests to any public site come from crawlers. Search engines, uptime monitors, link previewers, and lately a great many language model trainers.

Almost every analytics tool filters them out, which is right. Counting a search engine as a reader would make the numbers useless.

What is less right is throwing them away entirely. When traffic drops thirty per cent overnight, the first question is whether people stopped visiting or whether the tracker stopped working, and crawler counts are one of the few signals that can distinguish those two.

So we count them, separately, on their own screen, and never in the visitor figures. It costs one small table and it has explained two false alarms this year.