How much of a small site's traffic is bots? We counted ours

On Shipfolks, 29% of page requests over a 15-day sample came from software that openly identifies itself as a crawler. On top of that, 3,744 requests were automated scans for WordPress logins and leaked config files, on a site that runs neither. The real bot share is higher than 29%, because some bots dress up as browsers and cannot be told apart from people.

If you run a small product, your traffic probably looks similar, and any counter that does not filter it is telling you a story. Here is exactly what we found in our own logs, how we counted it, and what it changes about which numbers to trust.

How we counted

The source is the nginx access log for shipfolks.com from 27 September to 11 October 2026: every request the server answered, 13,192 of them. We removed our own visits, and requests for static files like images and stylesheets, and then sorted what was left by user agent, the line of text every client sends to say what it is.

A request counts as a declared crawler if that text names a bot: Googlebot, Amazonbot, SemrushBot and so on. Everything else counts as browser-looking. That second group is not the same as "human", and the difference is most of this post.

The numbers

The scans are not counted in the 29%. They are a separate category: automated sweeps of the whole internet looking for a known weak spot. The most common targets in our log were /wp-admin (96 requests), /@fs (80, a path a misconfigured development server exposes), /.git (73) and /.env (40). Shipfolks is not a WordPress site and has none of those files public, so every one of them came away with nothing. They still arrived, and a counter that counts requests counts them.

Who the crawlers were

Page requests from named crawlers over the same 15 days:

Three things stand out.

One crawler did a third of it. Amazonbot made 739 of the 2,151 declared-crawler page requests. A single busy crawler can make a quiet week look like a good one.

The crawler you want most came least. Googlebot fetched 20 pages in 15 days. For a new, small site that is normal, and it is the useful number in this table: if you want to be found in Google, how often Google actually visits matters more than how many pages you publish. Your sitemap and internal links decide what it sees on those visits.

AI crawlers are now a real slice. ClaudeBot, GPTBot, OAI-SearchBot, PerplexityBot and ChatGPT-User together made about 120 page requests. One of them is different from the rest: ChatGPT-User fetches a page because a person asked ChatGPT something, so those 15 visits are closer to a reader than a crawl.

The bots you cannot see

71% browser-looking is not 71% people. Some automated traffic sends a normal browser user agent on purpose, and from a log line alone you cannot prove which.

You can see its shape, though. Shipfolks keeps its own count of project page views, and across the 110 projects listed in July the views are strangely even: the 10th percentile has 44 views and the 90th has 70. Human attention does not spread like that. A few projects should pull far ahead and most should trail. Software that walks every page in turn produces exactly the flat line we have.

The same counter says September was our busiest month for project views. September also had zero clicks on "contact" and zero messages. Those two facts do not fit together if the views are people. They fit perfectly if many of them are not.

What your analytics tool already does

Google Analytics removes known bots automatically. Its documentation says known bot traffic is excluded using Google's own research and the International Spiders and Bots List maintained by the IAB, and that you cannot turn this off or see how much was removed (Google's page on it, checked in October 2026).

That covers the declared crawlers in the table above. It cannot cover a bot that sends a browser user agent, because nothing in the request says it is one. So a JavaScript analytics tool will usually show far fewer visits than your server log, and the truth is somewhere between the two, closer to the lower number.

A counter you wrote yourself, one row per page view in your own database, filters nothing unless you make it. Ours filtered nothing until 11 October.

What to do with this

1. Stop reporting page views as traction. Count things that need a person: sign-ups, replies, purchases, a contact click from someone signed in. Those are smaller and they are real.
2. If you keep your own counter, drop what you can prove is a bot. It is crude, and it is what ours does now: skip any user agent that names a crawler or an HTTP library, and any request without an Accept-Language header, which browsers send on every page load and a script has to be told to send.

$ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
if ($ua === '' || empty($_SERVER['HTTP_ACCEPT_LANGUAGE'])
    || preg_match('/bot\b|crawl|spider|slurp|headless|python|curl|wget/i', $ua)) {
    return; // do not count it
}

A bot that drives a real browser still gets through, because nothing in its request gives it away.

1. Compare two sources. Your server-side count against your analytics tool. The gap between them is roughly your bot traffic, and watching it change tells you when a new crawler has found you.
2. Grep for Googlebot. If it visits a handful of pages a week, publishing more is not your bottleneck. Being linked to, and a sitemap that lists what matters, are.
3. Block the crawlers you do not want in robots.txt. The well-behaved ones publish how. The scanners ignore it, which is fine: they find nothing.

If you are choosing an analytics tool for a small product, the comparison of Plausible, Fathom, Umami and PostHog covers what each one counts. Whatever you pick, the rule from this log holds: the smaller your site, the bigger the share of it that is machines.