Citelayer® AI Bot Analytics — Track Which AI Crawlers Visit Your Site
What Is Bot Analytics?
When AI systems like ChatGPT, Claude, or Perplexity want to learn about your site, they send crawlers. These bots read your content, build indexes, and use what they find to answer user questions or generate citations. The problem is you’ve had no visibility into this process. Your server logs exist, but parsing them for AI traffic is a manual, error-prone task most site owners never do.
Citelayer® Bot Analytics solves this. It automatically detects AI crawlers visiting your site, logs their activity to a local database table, and presents the data in a clean dashboard. You see which bots come, how frequently, and which pages attract the most AI attention. No external service, no data leaving your server, no setup beyond activating the feature.
What Gets Logged — and What Doesn’t
Bot Analytics only logs visits from AI-related bots. It ignores standard web crawlers like Googlebot, Bingbot, and other traditional search engine spiders. This is intentional: standard crawlers generate enormous traffic volume, and logging them would bloat your database without adding useful information.
Citelayer® uses two categories to classify AI bots:
confirmed_ai — Bots with a verified, public affiliation with AI products. Examples include GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and similar crawlers that have publicly documented their identity and purpose.
possible_ai — Bots whose user-agent strings suggest AI-related activity but whose affiliation hasn’t been formally confirmed. This category catches emerging crawlers before they become well-known.
The classification is based on the same bot list that powers Citelayer®’s llms.txt protocol — the LLMS_TXT::KNOWN_BOTS registry maintained within the plugin.
Standard search engine crawlers fall into a third category (standard) but are explicitly excluded from logging. They never appear in your analytics data.
The Bot Analytics Dashboard
Navigate to Citelayer® → Bot Analytics in your WordPress admin to access the dashboard. The interface is organized across several sub-tabs: Overview, Top Bots, Crawler Logs, Cache Config, and CSV Export.
Stat Cards
The Overview tab opens with four stat cards showing total bot visits across different time windows:
- Total Visits — all-time visit count in the database
- Last 7 Days — visits in the past week
- Last 30 Days — visits in the past month
- Last 90 Days — visits in the past quarter
These numbers update in near-real-time. The dashboard caches stats for five minutes to avoid repeated database queries on busy sites — if you just enabled logging and check immediately, wait a few minutes for the first cache to populate.
30-Day Trend Chart
Below the stat cards, a SVG trend chart displays daily visit counts for the past 30 days. This makes it easy to spot patterns: is AI crawler traffic growing? Did a spike coincide with a specific publication date? Are you seeing regular weekly patterns?
The chart renders server-side as SVG, so it works without JavaScript and loads fast.
Top Bots Leaderboard
The Top Bots section shows which AI crawlers visit your site most frequently, sorted by total hit count. Each row displays the bot name, its category (confirmed_ai or possible_ai), and the visit count for the selected time period. Use the time period filter — 7d / 30d / 90d / All-time — to narrow the view.
This leaderboard answers the most common question: “Is GPTBot actually crawling my site?”
How the Logging Works
Bot detection runs on WordPress’s template_redirect hook at priority 2 — early enough to capture the request before any significant processing, but after WordPress has parsed the URL.
Before logging a visit, Citelayer® checks several conditions. It skips requests from the WordPress admin area, AJAX requests, WP-Cron jobs, REST API calls, feed requests, and 404 pages. This filtering keeps the log clean and focused on actual content visits.
When a bot is detected, Citelayer® records:
- hit_at — the timestamp of the visit (UTC)
- bot_name — the identified bot (e.g., “GPTBot”)
- bot_category — confirmed_ai or possible_ai
- url_path — the path that was visited (e.g.,
/blog/my-post/) - url_hash — an MD5 hash of the URL path, used for efficient deduplication queries
- ip_address — only recorded if IP logging is enabled (off by default — see Privacy section below)
All data goes into a custom database table: wp_citelayer_bot_visits. The table has database indexes on the timestamp (bot_at), the URL hash (url_hash), the bot category (category), and the visit date (hit_date). These indexes keep queries fast even with large datasets.
Time Period Filter
The Overview, Top Bots, and Crawler Logs tabs all include a time period filter with four options: 7 Days, 30 Days, 90 Days, and All Time. Select a period and the dashboard reloads to show data for that window only.
The filter state persists for the current browser session. If you frequently work with a specific time window, it will stay selected as you navigate between tabs.
Privacy and GDPR Compliance
Bot Analytics was designed with privacy in mind from the start. A few things worth understanding clearly:
It only tracks bots. The logging code fires only when a known AI bot user-agent is detected. Real human visitors are never logged in this table.
IP logging is off by default. The ip_address column exists in the database schema, but Citelayer® does not populate it unless you explicitly enable IP logging via the option citelayer_bot_analytics_log_ip. Bot IP addresses are generally not personal data under GDPR (they belong to data center infrastructure, not individuals), but keeping the default off respects the principle of data minimization.
Everything stays on your server. There is no external service, no telemetry, no analytics platform. The data lives in your WordPress database and nowhere else.
You control data volume. The 90-day automatic cleanup (described below) means data doesn’t accumulate indefinitely.
Automatic Data Cleanup
Bot visit records older than 90 days are automatically deleted by a WP-Cron job that runs daily. This keeps the database table from growing without bound on active sites.
If you need a longer retention window — for example, to track year-over-year trends — you can adjust the cleanup threshold using the citelayer_bot_analytics_cleanup_days filter:
// Keep bot analytics data for 180 days instead of the default 90
add_filter( 'citelayer_bot_analytics_cleanup_days', function( $days ) {
return 180;
} );Set it to 0 to disable automatic cleanup entirely, though this is not recommended on high-traffic sites.
Dashboard Performance and Caching
The stat cards and chart data are cached as WordPress transients for five minutes. On sites where AI bots visit frequently, this prevents the dashboard from running expensive aggregate queries on every page load.
The five-minute cache means the numbers you see may be slightly behind real-time. This is expected behavior. For debugging purposes — for example, if you want to verify that a specific bot visit just logged correctly — use the Crawler Logs tab, which always queries the database directly.
For more on the Crawler Logs tab, see Citelayer® Crawler Logs — Full AI Bot Visit History.
Cache Plugin Compatibility
If your site uses a caching plugin, there’s an important consideration: cached pages are served directly from the cache layer, which means WordPress — and therefore Citelayer®’s bot detection code — never runs. Bot visits to cached pages won’t be logged.
The Cache Config tab in the Bot Analytics dashboard detects which caching plugin you’re running and provides specific instructions for configuring bot-user-agent exceptions. Citelayer® auto-detects WP Rocket, LiteSpeed Cache, W3 Total Cache, WP Super Cache, and WP Optimize.
See Citelayer® Cache Plugin Compatibility — Setup for AI Bot Tracking for the full setup guide.
CSV Export
You can export your bot visit data as a CSV file for use in spreadsheets or external analysis tools. The export option is available in the CSV Export tab of the Bot Analytics dashboard.
See Citelayer® CSV Export — Download Your Bot & Referral Data for details on export options and column definitions.
Finding Pages AI Bots Ignore
Bot Analytics tells you which bots visit your site. The companion AI Blind Spots feature tells you which of your pages no AI bot has visited at all. These two views together give you a complete picture of your AI visibility.