Bot and AI crawler traffic

Crawlers and AI bots don’t run JavaScript, so the browser tracker never sees them. This hook runs on your server instead and reports those requests directly.

What it tracks

Every bot hit gets sorted into one of three categories, plus a catch-all for anything unrecognized:

• AI answers: bots that fetch a page live to answer a user’s question, like ChatGPT-User, Claude-User, and Perplexity-User
• Indexing: search engine crawlers building their index, like Googlebot and Bingbot
• Training: bots scraping content to train AI models, like GPTBot and ClaudeBot
• Other: any automated request we can identify as a bot but can’t place in the categories above

Why it needs a server-side hook

The script tag tracks pageviews by running JavaScript in a visitor’s browser. Crawlers request a page and read the HTML, but they don’t execute scripts, so they never fire that tracker. The hook below sits in your server’s request path instead, where it sees every request, bot or human, before any JavaScript is involved.

Install

Pick your stack and add the snippet to your server’s request path.

import { sequence } from '@sveltejs/kit/hooks';
import { createBotTrackingHandle } from '@didtheyclick/sdk/bots';

export const handle = sequence(
	createBotTrackingHandle({ websiteId: 'YOUR_WEBSITE_ID' })
);

import { NextResponse, type NextFetchEvent, type NextRequest } from 'next/server';
import { trackBotRequest } from '@didtheyclick/sdk/bots';

export function proxy(request: NextRequest, event: NextFetchEvent) {
	trackBotRequest(request, { websiteId: 'YOUR_WEBSITE_ID' }, event);
	return NextResponse.next();
}

export const config = {
	matcher: ['/((?!api|_next/static|_next/image|favicon.ico).*)']
};

import { createExpressBotMiddleware } from '@didtheyclick/sdk/bots';

app.use(createExpressBotMiddleware({ websiteId: 'YOUR_WEBSITE_ID' }));

import { trackBotRequest } from '@didtheyclick/sdk/bots';

app.use('*', async (c, next) => {
	await next();
	trackBotRequest(c.req.raw, { websiteId: 'YOUR_WEBSITE_ID' }, c.executionCtx, c.res.status);
});

Options

All five integrations accept the same options object:

• websiteId: your project’s public website ID (required)
• endpoint: override the default https://www.didthey.click/api/bots collect URL, useful if you proxy requests through your own domain
• publicOrigin: the public origin of your site, used to resolve relative request URLs when your server sits behind a reverse proxy
• disableAnswerFetch: stop reporting AI answer bots (ChatGPT-User, Claude-User, Perplexity-User, and similar)
• disableSearchCrawlers: stop reporting search indexing bots (Googlebot, Bingbot, and similar)
• disableTrainingCrawlers: stop reporting AI training bots (GPTBot, ClaudeBot, and similar)
• disableOtherCrawlers: stop reporting any bot that doesn’t fall into the categories above

What gets sent

Each reported hit includes the request path with its query string stripped, the bot’s user agent, status code, and referrer when one is present. The client IP is sent along for verifying the bot is who it claims to be, but it is never stored.

Discovery files are always reported

Static assets (images, fonts, stylesheets) are skipped, but discovery files always get reported even though they’re static: robots.txt, llms.txt, llms-full.txt, sitemap XML files, and .md files. These are exactly the files crawlers check first, so seeing hits on them tells you a bot is actively discovering your site.

Any other stack

No SDK for your framework yet? Call the endpoint directly from your own server-side request handler.

curl -X POST https://www.didthey.click/api/bots \
  -H 'Content-Type: application/json' \
  -d '{"websiteId":"YOUR_WEBSITE_ID","href":"https://example.com/docs","userAgent":"<request User-Agent>","ip":"<client ip>","statusCode":200}'

The request needs no auth header. websiteId is the same public ID used everywhere else, and ip and referrer are optional.

Verifying it works

Load a page on your site with a known bot’s user agent, then check the Crawlers page in your dashboard.

curl -A "ChatGPT-User/1.0" https://your-site.com/

The hit should show up under its category within a few seconds.