Skip to content
Protocols

llms.txt for WordPress — AI Discovery File Setup

6 min read


Why This Matters

AI crawlers need a map of your site. They don’t browse page by page like humans — they look for structured discovery files that describe what your site offers and where to find key content.

llms.txt is a plain-text discovery file served at /llms.txt — think of it as robots.txt for AI models. First proposed by Andrej Karpathy, it gives language models a concise overview of your site: what it’s about, what content exists, and where to access it programmatically.

Without llms.txt, AI models guess. With it, they know exactly what your site provides. Citelayer® generates this file dynamically from your WordPress data — no manual editing, no static files to maintain.

How It Works

Citelayer® registers a WordPress rewrite rule that intercepts requests to /llms.txt and generates the file dynamically. The output is structured plain text with a consistent format that AI models can parse reliably.

Output Structure

Your generated llms.txt follows this structure:

# Your Site Name

Your site's tagline or description.

## About This Site
- URL: https://yoursite.com
- Language: en-US
- Character Set: UTF-8

## Contact
- Admin Email: admin@yoursite.com

## Content Statistics
- Posts: 142
- Pages: 12

## Semantic API Endpoints
- Schema REST API: https://yoursite.com/wp-json/citelayer/v1/schema
- UCP Discovery: https://yoursite.com/.well-known/ucp.json
- WebMCP Context: https://yoursite.com/wp-json/citelayer/v1/mcp

## AI-Optimized Content
- Markdown Index: https://yoursite.com/wp-json/citelayer/v1/markdown-index

## Recent Posts
- How to Configure Schema Markup: https://yoursite.com/configure-schema-markup/
- Getting Started with Citelayer: https://yoursite.com/getting-started/
- (up to 10 recent published posts)

Generated by Citelayer® (https://citelayer.ai) — AI Visibility Layer for WordPress

The file adapts to your configuration. Sections like Semantic API Endpoints and AI-Optimized Content only appear when the corresponding features are enabled. If you activate UCP Discovery, its endpoint shows up automatically. Same for Markdown and WebMCP.

Content-Type and Caching

Citelayer® serves llms.txt with Content-Type: text/plain; charset=utf-8. Browser caching is disabled (via nocache_headers) so you always see the latest version during development. Behind the scenes, WordPress transient caching keeps generation fast — the file rebuilds from cache rather than querying the database on every request.

Endpoint

The file is available at a single URL:

https://yoursite.com/llms.txt

Citelayer® registers this via a WordPress rewrite rule:

^llms.txt$ → index.php?citelayer_llms_txt=1

The rewrite rule is flushed automatically on plugin activation. If you move or reinstall Citelayer®, the rule re-registers itself.

Configuration

Navigate to Settings → Citelayer → llms.txt in your WordPress admin to configure the feature.

Enable or Disable

You can disable llms.txt generation entirely from the admin settings. When disabled, requests to /llms.txt fall through to WordPress’s normal 404 handling.

SEO Plugin Compatibility

Some SEO plugins (Rank Math, Yoast) generate their own llms.txt. Citelayer® detects these and offers two modes:

  • Enhance mode: If your SEO plugin already generates llms.txt, Citelayer® enhances that output by appending semantic endpoint information, content statistics, and other data the SEO plugin doesn’t include. The SEO plugin stays in control — Citelayer® adds to it.
  • Replace mode: Citelayer® generates its own llms.txt and disables the SEO plugin’s version. Use this when you want full control over the output.

Yoast special case: If Yoast SEO is active but has no physical llms.txt file, Citelayer® takes over automatically. No configuration needed — Citelayer® detects the absence and fills the gap.

Conflict Detection

If Citelayer® detects multiple SEO plugin families trying to control llms.txt (for example, both Rank Math and Yoast active simultaneously), it returns a 503 Service Unavailable response with a Retry-After: 3600 header. This prevents conflicting output. Resolve the conflict by deactivating one of the SEO plugins or choosing a specific mode in Citelayer® settings.

Content Filter

Use the citelayer_llmstxt_content filter to modify the content array before Citelayer® renders the output. This lets you add custom sections, remove default ones, or restructure the file to match your needs.

add_filter( 'citelayer_llmstxt_content', function( $content ) {
    // Add a custom section
    $content['custom_section'] = [
        'heading' => '## Custom Data',
        'lines'   => [
            '- API Documentation: https://yoursite.com/api-docs/',
            '- Developer Portal: https://yoursite.com/developers/',
        ],
    ];
    return $content;
} );

For a full list of available filters, see Hooks & Filters.

Verify Your Setup

After activating Citelayer®, confirm that llms.txt works correctly:

  1. Open your browser and visit https://yoursite.com/llms.txt.
  2. Check the heading. The first line should read # Your Site Name (matching your WordPress site title).
  3. Verify the footer. The last line should say Generated by Citelayer® (https://citelayer.ai) — AI Visibility Layer for WordPress.
  4. Confirm dynamic sections. If you have Schema REST API or Markdown endpoints enabled, their URLs should appear under Semantic API Endpoints or AI-Optimized Content.
  5. If using Enhance mode with Rank Math or Yoast, verify that the SEO plugin’s base output is present with Citelayer®’s additions appended.

You can also verify with curl:

curl -s https://yoursite.com/llms.txt

Technical Details

Caching

Citelayer® stores the generated llms.txt content in a WordPress transient:

  • Transient key: citelayer_llmstxt_content
  • TTL: 1 hour
  • Auto-invalidation events: save_post, delete_post, switch_theme, updates to blogname or blogdescription

When any of these events fire, the transient is deleted. The next request to /llms.txt regenerates the content and stores a fresh transient. This means your llms.txt stays current — publish a new post and it appears in the Recent Posts section within seconds.

Rewrite Rules

The rewrite rule ^llms.txt$ is registered during plugin activation and flushed automatically. If you notice /llms.txt returning a 404, visit Settings → Permalinks in WordPress admin and click Save Changes to flush rewrite rules manually.

How citelayer.ai Uses This

The Citelayer® website at citelayer.ai runs on WordPress with Citelayer® installed. Visit citelayer.ai/llms.txt to see a live example of the generated output.