Skip to content
Reference

Citelayer® Cache Plugin Compatibility — Setup for AI Bot Tracking

5 min read

Why Caching Breaks Bot Detection

Citelayer®’s AI bot detection runs as part of WordPress’s PHP execution. On template_redirect, before the page is served, Citelayer® inspects the incoming HTTP User-Agent, identifies whether it belongs to a known AI crawler, and logs the visit if it does.

The problem is that caching plugins exist specifically to skip that process. When WP Rocket, LiteSpeed Cache, or any similar plugin has a cached copy of a page, it serves that copy directly — often from memory, often without invoking PHP at all. WordPress never runs, Citelayer® never executes, and the bot visit is never logged.

The result is that your Bot Analytics dashboard may show dramatically fewer visits than actually occurred, or none at all, depending on how aggressively your caching plugin operates.

The fix is to configure your caching plugin to bypass the cache for known AI bot user-agents. This means AI crawlers get fresh PHP-rendered responses (which are logged), while human visitors and standard search engine bots continue getting the fast cached responses (which don’t need logging).

Checking Your Cache Configuration

Navigate to Citelayer® → Bot Analytics → Cache Config. This tab automatically detects which caching plugin is active on your site and shows you:

  • Which plugin was detected
  • Whether it’s currently configured to pass AI bots through the cache
  • Step-by-step instructions for your specific plugin

Citelayer® detects the following caching plugins automatically:

PluginDetection Method
WP RocketWP_ROCKET_VERSION constant
LiteSpeed CacheLSCWP_V or LSCWP_VERSION constant
W3 Total CacheW3TC constant
WP Super CacheWPCACHEHOME constant
WP OptimizeWPO_Cache_Config class

If you’re running a different caching plugin not on this list, scroll to the manual configuration section at the bottom of this article.

WP Rocket Configuration

  1. Go to Settings → WP Rocket in your WordPress admin.
  2. Click the Advanced Rules tab.
  3. Scroll to the Never Cache for User Agents field.
  4. Paste the AI bot user-agent patterns from Citelayer®’s list (available in the Cache Config tab — click “Copy Bot Patterns”).
  5. Save changes.

WP Rocket will now let requests from these user-agents bypass the cache entirely. The patterns use partial string matching, so a pattern like GPTBot matches any User-Agent containing that string.

LiteSpeed Cache Configuration

  1. Go to LiteSpeed Cache → Cache in your WordPress admin.
  2. Click the Excludes tab.
  3. Find the Do Not Cache User Agents field.
  4. Enter each AI bot pattern on a separate line (copy the list from the Citelayer® Cache Config tab).
  5. Save changes.

LiteSpeed’s exclusion rules apply at the application level, before the LiteSpeed web server’s built-in cache. This configuration works for both the WordPress plugin’s object cache and LiteSpeed’s server-side page cache.

W3 Total Cache Configuration

  1. Go to Performance → Page Cache in your WordPress admin.
  2. Click the Advanced tab within Page Cache settings.
  3. Find the Rejected User Agents textarea.
  4. Enter each AI bot pattern on a new line.
  5. Save settings and clear the W3TC page cache.

W3TC applies user-agent rejection at the WordPress level — requests from matching user-agents are processed by PHP normally rather than served from the disk or memory cache.

WP Super Cache Configuration

  1. Go to Settings → WP Super Cache in your WordPress admin.
  2. Click the Advanced tab.
  3. Find the Rejected User Agents section.
  4. Add each AI bot pattern to the field.
  5. Click Update Status to save.

WP Super Cache bypasses the cache for rejected user-agents, falling through to normal WordPress PHP execution.

Getting the Bot Patterns

Every caching plugin above requires a list of AI bot user-agent patterns. Citelayer® provides this list ready-to-copy in the Cache Config tab of the Bot Analytics dashboard. The patterns cover all bots in the confirmed_ai and possible_ai categories — the same bots that get logged.

The list is updated when Citelayer® updates its bot registry. After a plugin update, it’s worth checking the Cache Config tab to see if new patterns need to be added to your cache configuration.

Nginx-Level Caching

If your server uses Nginx’s FastCGI cache, proxy cache, or similar server-level caching (independent of any WordPress plugin), the caching plugin configuration steps above won’t help. Nginx makes caching decisions before PHP is invoked at all.

Nginx-level cache bypass for bot user-agents needs to be configured in your Nginx server configuration, typically using a map block and a cache bypass condition. This requires server access and is outside the scope of what Citelayer® can configure automatically.

A basic example:

map $http_user_agent $skip_cache {
    default 0;
    ~GPTBot 1;
    ~ClaudeBot 1;
    ~PerplexityBot 1;
    # Add other AI bot patterns here
}

fastcgi_cache_bypass $skip_cache;
fastcgi_no_cache $skip_cache;

Place this in the appropriate server or location block. The Citelayer® Cache Config tab lists all bot patterns; translate them to ~pattern format for the Nginx map block.

If you’re on a managed hosting platform (WP Engine, Kinsta, Flywheel, etc.) that handles Nginx caching at the infrastructure level, contact your hosting provider. Most managed hosts have documented processes for user-agent-based cache bypass.

After Configuration

Once your cache plugin is configured, verify that bot logging is working by temporarily checking your Crawler Logs after a few hours. If you see bot visits appearing — even just one or two per day from major crawlers — the configuration is working. If you see zero visits despite bot traffic being plausible for your site’s content and traffic level, revisit the cache bypass rules.

For more detail on reading crawler log data, see Citelayer® Crawler Logs — Full AI Bot Visit History.