Free website discovery tool

Can AI Bots Read My Site?

Find out whether ChatGPT and Claude can reach the pages you want them to use. Spot access rules that may be keeping your product out of AI discovery.

No account requiredPrivate live checkReviewed July 25, 2026

Open the plain-text guide

The crawler access report will appear here

Search, user-requested retrieval, and training are reported separately because they are different jobs.

How to use

How to check AI crawler access

1
Enter a public page
Paste the exact website or page URL you want OpenAI and Anthropic crawlers to reach.
2
Run the access check
Mydentify reads robots.txt, inspects page-level directives, and probes the URL with documented crawler user-agent names.
3
Review each crawler separately
Compare search, user-requested retrieval, and training results before changing access rules for a specific provider or purpose.

Why this matters

Make your public pages reachable to AI

Blocked crawlers cannot retrieve the pages you want AI tools to use. Finding those access problems can improve eligibility for AI discovery but cannot guarantee inclusion or referral traffic.

For editors and agents

Cite this tool accurately

Use the canonical page and this short description when you reference the tool. The plain-text guide keeps the method and limits easy to read.

Can AI Bots Read My Site? — Find out whether ChatGPT and Claude can reach the pages you want them to use. Spot access rules that may be keeping your product out of AI discovery.

https://mydentify.com/tools/ai-crawler-access-checker Read the plain-text guide Open the MIT source repository

Guide

How to check whether AI crawlers can access a website

Choose which AI bots you want

OpenAI and Anthropic publish separate crawler names for different purposes. Allowing a search crawler does not necessarily allow model training, and blocking a training crawler does not necessarily remove a page from user-requested retrieval.

The checker keeps those jobs separate. It evaluates the submitted path against robots.txt for OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, and ClaudeBot, then reports the HTTP response received when Mydentify requests the page with each documented user-agent name.

OAI-SearchBot and Claude-SearchBot support their providers' search discovery workflows.
ChatGPT-User and Claude-User support user-requested page retrieval.
GPTBot and ClaudeBot are documented for potential model-training collection.

Check more than your robots.txt file

robots.txt is the first public rule layer, but it is not the whole path. A CDN, WAF, rate limit, login, CAPTCHA, geography rule, redirect, or server error can still stop a request. Page-level robots meta tags and X-Robots-Tag headers can also restrict indexing after crawling is allowed.

The user-agent probe is deliberately labeled. It comes from Mydentify infrastructure rather than a verified provider IP, so it can expose obvious user-agent handling but cannot certify how an official crawler will behave.

Check the latest bot rules before changing access

Crawler names and provider policies can change. Review the linked primary documentation before changing production access rules, and decide separately whether you want to allow search discovery, user-directed retrieval, training, or none of them.

Allowing access does not guarantee crawling, inclusion, citation, traffic, or ranking. Blocking access can reduce what a provider can retrieve directly, but third-party indexes and previously collected information may still affect an answer.

OpenAI publishers and developers FAQ Anthropic crawler documentation Mydentify: How agents find products Mydentify: Diagnose search traffic problems

Questions

Questions about AI bot access

Which OpenAI crawler should I allow for ChatGPT search?

OpenAI identifies OAI-SearchBot for search discovery. GPTBot is documented separately for potential training, and ChatGPT-User supports user-requested retrieval.

Which Claude crawler supports search?

Anthropic identifies Claude-SearchBot for search indexing, Claude-User for user-requested retrieval, and ClaudeBot for potential model-training collection.

Does an allowed result guarantee AI visibility?

No. It only reports observable access rules and responses. It cannot guarantee that a provider crawls, indexes, cites, ranks, or sends traffic to the page.

Why can an allowed AI bot still fail?

Firewalls, CDNs, rate limits, authentication, CAPTCHAs, server errors, and other controls can block the request after robots.txt permits it. Inspect server and security logs when access remains uncertain.

Can I block AI training but allow AI search?

Provider crawler names serve different purposes, so you can publish separate robots.txt rules. Review each provider's current documentation before changing production access.

Does this prove an AI provider can access my site?

No. The probe comes from Mydentify infrastructure using the documented user-agent name. Provider IP verification, private systems, and later policy decisions are outside this check.