# Bot and AI crawler traffic

Crawlers and AI bots don't run JavaScript, so the browser tracker never sees them. This hook runs
on your server instead and reports those requests directly.

## What it tracks

Every bot hit gets sorted into one of three categories, plus a catch-all for anything unrecognized:

- **AI answers**: bots that fetch a page live to answer a user's question, like ChatGPT-User,
  Claude-User, and Perplexity-User
- **Indexing**: search engine crawlers building their index, like Googlebot and Bingbot
- **Training**: bots scraping content to train AI models, like GPTBot and ClaudeBot
- **Other**: any automated request we can identify as a bot but can't place in the categories
  above

## Why it needs a server-side hook

The script tag tracks pageviews by running JavaScript in a visitor's browser. Crawlers request a
page and read the HTML, but they don't execute scripts, so they never fire that tracker. The hook
below sits in your server's request path instead, where it sees every request, bot or human,
before any JavaScript is involved.

## Install

Pick your stack and add the snippet to your server's request path.

**SvelteKit**

```typescript
import { sequence } from '@sveltejs/kit/hooks';
import { createBotTrackingHandle } from '@didtheyclick/sdk/bots';

export const handle = sequence(
	createBotTrackingHandle({ websiteId: 'YOUR_WEBSITE_ID' })
);
```

**Next.js**

```typescript
import { NextResponse, type NextFetchEvent, type NextRequest } from 'next/server';
import { trackBotRequest } from '@didtheyclick/sdk/bots';

export function proxy(request: NextRequest, event: NextFetchEvent) {
	trackBotRequest(request, { websiteId: 'YOUR_WEBSITE_ID' }, event);
	return NextResponse.next();
}

export const config = {
	matcher: ['/((?!api|_next/static|_next/image|favicon.ico).*)']
};
```

**Express**

```typescript
import { createExpressBotMiddleware } from '@didtheyclick/sdk/bots';

app.use(createExpressBotMiddleware({ websiteId: 'YOUR_WEBSITE_ID' }));
```

**Hono**

```typescript
import { trackBotRequest } from '@didtheyclick/sdk/bots';

app.use('*', async (c, next) => {
	await next();
	trackBotRequest(c.req.raw, { websiteId: 'YOUR_WEBSITE_ID' }, c.executionCtx, c.res.status);
});
```

> **Note: Replace YOUR_WEBSITE_ID**
>
> Use the same website ID as your browser tracker, the `data-website-id` value from your project's install snippet.

## Options

All five integrations accept the same options object:

- `websiteId`: your project's public website ID (required)
- `endpoint`: override the default `https://www.didthey.click/api/bots` collect URL, useful if
  you proxy requests through your own domain
- `publicOrigin`: the public origin of your site, used to resolve relative request URLs when
  your server sits behind a reverse proxy
- `disableAnswerFetch`: stop reporting AI answer bots (ChatGPT-User, Claude-User,
  Perplexity-User, and similar)
- `disableSearchCrawlers`: stop reporting search indexing bots (Googlebot, Bingbot, and similar)
- `disableTrainingCrawlers`: stop reporting AI training bots (GPTBot, ClaudeBot, and similar)
- `disableOtherCrawlers`: stop reporting any bot that doesn't fall into the categories above

## What gets sent

Each reported hit includes the request path with its query string stripped, the bot's user agent,
status code, and referrer when one is present. The client IP is sent along for verifying the bot
is who it claims to be, but it is never stored.

## Discovery files are always reported

Static assets (images, fonts, stylesheets) are skipped, but discovery files always get reported
even though they're static: `robots.txt`, `llms.txt`, `llms-full.txt`, sitemap XML files, and
`.md` files. These are exactly the files crawlers check first, so seeing hits on them tells you a
bot is actively discovering your site.

## Any other stack

No SDK for your framework yet? Call the endpoint directly from your own server-side request
handler.

```bash
curl -X POST https://www.didthey.click/api/bots \
  -H 'Content-Type: application/json' \
  -d '{"websiteId":"YOUR_WEBSITE_ID","href":"https://example.com/docs","userAgent":"<request User-Agent>","ip":"<client ip>","statusCode":200}'
```

The request needs no auth header. `websiteId` is the same public ID used everywhere else, and
`ip` and `referrer` are optional.

## Verifying it works

Load a page on your site with a known bot's user agent, then check the Crawlers page in your
dashboard.

```bash
curl -A "ChatGPT-User/1.0" https://your-site.com/
```

The hit should show up under its category within a few seconds.

> **Tip: Bot hits are free**
>
> Bot and crawler requests never count against your monthly events quota.