Crawler Logs: Inspect Matching Requests
Crawler Logs shows the request records collected inside WordPress. It is the detail view behind the bot dashboard, useful for investigating paths and changes.
Filter the records
- Choose a date range that falls within retained data.
- Filter by matched bot name or path.
- Inspect the records around the time an endpoint or content problem appeared.
- Export the relevant slice if you need to compare it outside WordPress.
What is stored
Records contain a timestamp, matched bot name/category and normalized URL path. IP logging is optional and off by default. The logger skips several non-frontend contexts and does not log ordinary traditional crawlers in this AI view.
Understand a bot label
Matching uses known User-Agent substrings. Anyone can claim a User-Agent, and some recognized patterns are classified as possible AI. Verify provider identity separately when that distinction matters.
From a record to a next step
Open the recorded public path and inspect the response format and current content. Compare server/cache/WAF behavior if requests seem missing. A crawler visit is not proof of indexing or citation. Export instructions and dashboard interpretation explain the available scope.
Next step
Continue with the quick start checklist, or check the common questions. For a reproducible problem, send the plugin versions, affected URL, expected result and actual response through Support. Do not include passwords, license keys or customer details.
Reviewed 6 October 2026 against citelayer® Free 1.13.5, WooCommerce add-on 1.2.19 and EDD add-on 1.2.1. Features depend on the active components and settings on your site.