# Get your product found > Find out what your site tells people and machines about your product, fix what's blocking it, and get described somewhere credible. - Canonical playbook: https://mydentify.com/playbooks/get-your-product-found - Playbook library: https://mydentify.com/playbooks - Best for: A launched product getting under 100 organic visits a month - Effort: about 8.5 hours of work. Best spread over two weeks rather than done in one sitting, because two groups of steps wait on a crawler. - Time horizon: Crawler and rendering fixes land within days. Search results take 4 to 12 weeks. AI answers run on no schedule at all. - Actions: 20 required on a selected path, 5 optional - Reviewed: 2026-08-03 - AI usage policy: https://mydentify.com/ai.txt ## Direct answer If nobody's finding your product, it's usually one of four things, and they're worth checking in this order: a crawler is blocked, your product facts only appear after JavaScript runs, the wrong URL is marked as the real one, or nothing outside your own site mentions you. The first three are technical and take under an hour each to find. The fourth is slower, and it matters more than most founders expect, because search engines and AI assistants both look for a second source before recommending anything. Run the checks below in order, fix what turns up, then get listed somewhere credible. ## What to expect This clears obstacles. It doesn't create demand. An hour of checks here saves you from publishing content nothing can reach. Most of this is removing obstacles, not adding content. You'll probably find one or two real blockers — a crawler rule you didn't write, product facts that only exist after JavaScript runs, a canonical pointing at the wrong URL — and each takes minutes to fix once you know which one it is. Treat it as a plan, not a rigid checklist. Do the setup steps, keep the recurring ones on whatever schedule you'll actually keep, and leave the optional ones until the rest pass. ## Prerequisites - A live public URL that doesn't need a login - The ability to edit robots.txt and page metadata, and to deploy - A Google Search Console property, or ten minutes to create one ## Strategic model Four things have to be true, and each one has to work before the next one matters. 1. **They can reach it.** Is the crawler allowed to fetch the page at all? 2. **They can read it.** Are your product facts in the HTML the server sends? 3. **One URL is clearly the real one.** Do your canonical, og:url, and sitemap all agree? 4. **Someone else describes it.** Is there a credible source that isn't your own site? Most founders start on the fourth one — publishing content, chasing launches — while the first one is still failing. The order is the whole point. The fourth is the one most guides skip, and it's where a directory dataset does work a general SEO guide can't. Search engines and AI assistants both build recommendations partly from sources they already pull: comparison articles, roundups, community threads, directories. A listing here isn't a backlink tactic. It's how a system finds a second opinion about you. This order isn't made up for this page. It's the order already in Mydentify's readiness scoring, which checks whether an agent can reach a site before it weighs how strong the evidence about it is. ## Paths - **Fix what's broken** — best for A live product with low or flat organic traffic. Tradeoff: You'll spend the first hour reading results instead of shipping changes. - **Prepare before launch** — best for A product that isn't public yet. Tradeoff: No diagnostic work, so no rework. But you're configuring blind, with no real crawl data yet. ## Action plan ### Milestone 1: Check what your homepage says Outcome: You know what a machine thinks your product does, and whether that matches what you meant. #### Compare what your homepage says to what you'd say - Cadence: once · Type: shared · Completion: tool-assisted - Why it matters: Your homepage is obvious to you because you built the product. It usually isn't obvious to anyone else. - How: Paste your homepage URL into the Homepage Check. Write down the sentence it gives you. Under it, write the sentence you'd say to someone at a bar. Note anything it marked unclear or missing. A crawler is stuck on those too. Don't fix anything yet. You're just looking. - Tools: [What Does My Website Say?](https://mydentify.com/tools/what-does-my-website-do-checker) - Done when: You have two sentences written down — what the tool says you do, and what you'd say — and you can point at the difference. - Verify with: The Homepage Check gives you a description, an audience, and a problem, plus the exact text on your page it took each from. - Commonly goes wrong: You disagree with the output and dismiss it. It's reporting what your page supports. If it got the audience wrong, your page names the wrong audience. #### Write one sentence on who it's for - Cadence: once · Type: shared · Completion: manual - Why it matters: A page that never says who it's for makes everyone guess, and they guess bigger and vaguer than the truth. - How: Write: "[product] helps [specific role] [do a specific job]." If the role could be "anyone," try again. Write a second sentence naming the problem the way your buyer says it, not the way your code says it. Put both on the homepage, in the first screenful. Not on the About page. Serving two audiences? Write for the one paying you today. - Tools: none - Done when: Both sentences are in your homepage HTML, in the first screenful, and neither says "anyone" or "everything." - Verify with: View source on your homepage. You should find both sentences as plain text, not inside a script tag. - Commonly goes wrong: The sentences end up in a design file or a pitch deck and never reach the page. Check the live URL. #### Fix your headline if the tool got it wrong - Cadence: once · Type: shared · Completion: tool-assisted - Why it matters: Everything later assumes your page says true things. Fixing crawler access to a page that misdescribes your product just publishes the wrong answer faster. - How: Replace your headline and first paragraph with the two sentences you just wrote. Delete any opening line that describes a category instead of your product. "The modern platform for growth" says nothing. Keep the product name in the first sentence. Readers and models both anchor on it. Deploy, re-run the Homepage Check, and see whether the description moved closer to yours. - Tools: [What Does My Website Say?](https://mydentify.com/tools/what-does-my-website-do-checker) - Done when: The re-run description is close enough to yours that you'd accept it as a description of your product. - Verify with: The Homepage Check, run again after you deploy, showing the updated description. - Commonly goes wrong: You change the copy but the check returns the old text, because your CDN is serving cached HTML. Purge the cache before deciding the rewrite failed. #### Run three checks before you change anything - Cadence: once · Type: path · Completion: tool-assisted - Why it matters: A product with flat traffic usually has one thing badly broken, not four things mildly broken. Finding which one saves you a week on the wrong problem. - How: Run the AI Bot Checker, the JavaScript SEO Checker, and the Canonical Checker on one important product URL. Change nothing first. Note which one fails: they can't reach it, they can't read it, the wrong URL is marked as real, or nobody else describes it. Fix the earliest failure first. Fixing a canonical won't help if the crawler can't get in. If all three come back clean, your problem is the fourth one, or demand. Skip ahead to getting described somewhere else. - Tools: [Can AI Bots Read My Site?](https://mydentify.com/tools/ai-crawler-access-checker), [JavaScript SEO Checker](https://mydentify.com/tools/javascript-seo-render-checker), [Canonical URL Checker](https://mydentify.com/tools/canonical-url-identity-checker) - Done when: You've named the earliest failing check, or confirmed the first three pass and your work belongs in getting described somewhere else. - Verify with: Three tool results, written down before you deploy any fix. - Commonly goes wrong: Running the checks after you've started fixing things, so you can't tell which change mattered. Get the baseline first. #### Run the AI Product Read on the same URL - Cadence: once · Type: optional · Completion: tool-assisted - Why it matters: It shows you the gap between what you published and what a system can actually conclude from it, and which text it relied on. - How: Run the AI Product Read on the same homepage URL you used for the Homepage Check. Compare the two. Where they disagree, your page supports two readings. Anything it claims that you can't find on your page is something your copy invites people to make up. - Tools: [What Does AI Think My Product Is?](https://mydentify.com/tools/what-does-ai-think-my-product-is) - Done when: You've read it and listed any claim it made that your page doesn't actually support. - Verify with: The AI Product Read returns a description with the excerpts behind each conclusion. - Commonly goes wrong: Reading it as a verdict on your product. It's a measurement of your page. ### Milestone 2: Let the right crawlers in Outcome: The crawlers that cite products can reach your pages, and your training-data rule is written down and dated. #### Let the crawlers that cite you reach your site - Cadence: once · Type: shared · Completion: tool-verified - Why it matters: Block these and your product disappears from AI answers, no matter how good your content is. - How: Open yourdomain.com/robots.txt in a browser. Not the file in your repo — the one your site serves. If the served file is longer than your committed one, something at the edge is adding rules. Search it for "# BEGIN Cloudflare Managed content". Check that OAI-SearchBot, ChatGPT-User, Claude-SearchBot, and Claude-User aren't caught by a Disallow, including a broad "User-agent: *" followed by "Disallow: /". Leave training crawlers for the next step. Blocking GPTBot and ClaudeBot is a licensing choice and doesn't affect whether you get cited. - Tools: [Can AI Bots Read My Site?](https://mydentify.com/tools/ai-crawler-access-checker) - Done when: All four search and user-requested crawlers show Allow for your product URLs. - Verify with: The AI Bot Checker shows Allow for those four crawlers, and quotes the robots.txt rule it matched for each. - Commonly goes wrong: Your committed robots.txt is right and your served one isn't. At most CDNs, managed robots.txt and AI crawl control are separate switches, and either can overwrite your file at the edge. #### Check whether your CDN blocks what robots.txt allows - Cadence: once · Type: shared · Completion: tool-verified - Why it matters: robots.txt shows your intent, not your configuration. An edge rule can return 403 to a crawler your file welcomes, and the file gives you no clue. - How: Read the live probe column, not just the robots.txt column. The checker sends a real request as each crawler. Allow in robots.txt plus a 403 from the probe means an edge or WAF block. That combination is the tell. Look at your CDN's bot management and AI crawler settings. They're usually separate from anything robots.txt related. Re-run after you change it. A 200 or 301 passes. A 403 or 429 doesn't. - Tools: [Can AI Bots Read My Site?](https://mydentify.com/tools/ai-crawler-access-checker) - Done when: No crawler shows Allow in robots.txt next to a 403 from the live probe. - Verify with: The AI Bot Checker shows the matched robots rule and the live probe status side by side. Compare the two columns. - Commonly goes wrong: Stopping at a clean robots.txt. Bot management runs before your file is ever read, so the file can be perfect while the request still fails. #### Record your training-data decision in robots.txt - Cadence: once · Type: shared · Completion: manual - Why it matters: Training access and citation access are two different decisions. One "block AI bots" switch collapses them into a choice you didn't mean to make. - How: Sort the crawlers by what they do. GPTBot and ClaudeBot collect training data. OAI-SearchBot and Claude-SearchBot power search. ChatGPT-User and Claude-User fetch a page because someone asked. Decide training access as a licensing question about your content, not a visibility question. Write the decision and the date in a comment at the top of robots.txt, so the next person doesn't reverse it on a guess. Keep the search and user-requested crawlers allowed either way. - Tools: [Can AI Bots Read My Site?](https://mydentify.com/tools/ai-crawler-access-checker) - Done when: Your robots.txt carries a dated, deliberate decision about training crawlers, and the search and user-requested ones are allowed. - Verify with: The AI Bot Checker groups crawlers by purpose. Check the training rows match what you decided. - Commonly goes wrong: Blocking every AI user agent to protect your content, then wondering why you stopped showing up in AI answers. The citation crawlers were collateral damage. #### Set robots.txt, canonical, and sitemap before you go public - Cadence: once · Type: path · Completion: tool-verified - Why it matters: Before launch is the only time you can get these right without also un-teaching a search engine something it already wrote down. - How: Ship robots.txt, a self-referencing canonical, and a working sitemap with the first public deploy. Not after. Delete any leftover "User-agent: * Disallow: /" from staging. This is the most common launch-day mistake there is. Check no page still carries a noindex from staging. You can't verify crawl behavior yet. Re-run every check once the site is public — that's the last group of steps. - Tools: [Can AI Bots Read My Site?](https://mydentify.com/tools/ai-crawler-access-checker), [Sitemap Checker](https://mydentify.com/tools/sitemap-health-checker) - Done when: Your first public deploy serves a robots.txt with no blanket disallow, a self-referencing canonical, and a sitemap that returns XML. - Verify with: The AI Bot Checker and the Sitemap Checker, both run against the live URL within an hour of launching. - Commonly goes wrong: Staging config ships to production. A site-wide Disallow or noindex can bury a launch for weeks before anyone thinks to look at the file. #### Publish an llms.txt that points at pages that exist - Cadence: once · Type: optional · Completion: tool-verified - Why it matters: It's an optional convention nobody enforces, but it can help a system find your most useful pages. Worth twenty minutes once access is fixed. - How: List your most useful public pages, one line of description each. Only pages that return 200. Serve it at /llms.txt with a text/plain or text/markdown content type. It isn't an access control file. It requests nothing and enforces nothing. Validate it, then check the crawlers that would read it aren't blocked. A perfect file behind a blocked crawler does nothing. - Tools: [llms.txt Checker](https://mydentify.com/tools/llms-txt-generator-validator) - Done when: The file validates, every URL in it returns 200, and the crawlers that would fetch it are allowed. - Verify with: The llms.txt Checker reports valid structure and no broken links. - Commonly goes wrong: Doing this first. It's the last step here, not the first, and no crawler has to read it. ### Milestone 3: Put your facts in the server response Outcome: Your product name, description, and structured data are in the HTML your server sends, before any script runs. #### List the facts that only appear after JavaScript - Cadence: once · Type: shared · Completion: tool-assisted - Why it matters: A crawler that doesn't run your JavaScript sees only the server response. Anything your code adds in the browser is invisible to it. - How: Run the JavaScript SEO Checker on an important product page and read the raw column, not the rendered one. Check your product name, description, and price are all in the raw response. Anything that shows up only in the rendered column is something some systems will never see. Look at JSON-LD specifically. It's the one most often added by a tag manager or a browser script. - Tools: [JavaScript SEO Checker](https://mydentify.com/tools/javascript-seo-render-checker) - Done when: You have a list of every product fact and structured data block that only appears after JavaScript runs. - Verify with: The JavaScript SEO Checker shows raw versus rendered, field by field. - Commonly goes wrong: Checking in browser DevTools, which shows the rendered DOM and hides the whole problem. Use view-source or the tool. #### Move your JSON-LD into the server response - Cadence: once · Type: shared · Completion: tool-verified - Why it matters: Structured data describes your product to systems that may never run your scripts. Added in the browser, it's a description that only sometimes exists. - How: Render the JSON-LD script tag on the server, so it shows up in view-source. Keep it consistent with what's visible on the page. Schema describing something the page doesn't show is a problem, not a win. Delete any duplicate JSON-LD your tag manager left behind. Two conflicting descriptions are worse than one. Deploy, re-run the checker, and confirm the JSON-LD is now in the raw column. - Tools: [JavaScript SEO Checker](https://mydentify.com/tools/javascript-seo-render-checker) - Done when: The JSON-LD is in the raw server response, matches what's visible, and appears exactly once. - Verify with: The JavaScript SEO Checker shows the structured data in the raw column, not just after hydration. - Commonly goes wrong: Adding schema through a tag manager because it ships faster. That puts it only in the rendered DOM, which is the thing you're trying to fix. #### Re-run the check after you deploy - Cadence: recheck · Type: recheck · Completion: tool-verified - Why it matters: The fix is only real on the deployed URL, and caching often serves the old HTML for longer than your deploy log suggests. - How: Wait for the deploy to finish, then purge the CDN or full-page cache for the URL you're testing. Re-run the checker on the same URL you used for the baseline. Confirm nothing is left in the rendered-only list. If nothing changed, check you fixed the same route you're testing. - Tools: [JavaScript SEO Checker](https://mydentify.com/tools/javascript-seo-render-checker) - Done when: No product facts and no JSON-LD appear only after hydration. - Verify with: The JavaScript SEO Checker, run after deploying, with an empty rendered-only list. - Commonly goes wrong: Re-running before the cache expires and deciding the fix failed. Purge first, then measure. ### Milestone 4: Make your URL signals agree Outcome: Your canonical tag, your sitemap, and your page all name the same URL. #### Check your canonical points where you think it does - Cadence: once · Type: shared · Completion: tool-verified - Why it matters: A canonical on the wrong URL tells Google to index a different page than the one you want, and nothing on the page looks broken. - How: Run the Canonical Checker on an important product URL. On an indexable page, the canonical should point at that same page. Not the homepage. Watch for a canonical pointing at a staging or www variant. A canonical to a hostname you no longer serve is a common template leftover. Check whether your URL redirected. If it did, the canonical should name the destination, not where you started. - Tools: [Canonical URL Checker](https://mydentify.com/tools/canonical-url-identity-checker) - Done when: Every indexable product page has a canonical pointing at itself, on the hostname you actually serve. - Verify with: The Canonical Checker reports the canonical value next to the final URL after redirects. - Commonly goes wrong: A site-wide template sets every canonical to the homepage, so every product page asks to be de-indexed in favor of the root. #### Make canonical, og:url, and your sitemap match - Cadence: once · Type: shared · Completion: tool-verified - Why it matters: When your signals disagree about which URL is real, Google picks one. It doesn't have to pick yours. - How: Read the conflict count. Anything above zero means two signals name different URLs. Make the canonical tag, og:url, and the sitemap URL the exact same string, trailing slash included. Pick one hostname convention and one trailing-slash convention and use them everywhere. Mixed conventions cause most of these conflicts. Deploy and re-run until it reports aligned. - Tools: [Canonical URL Checker](https://mydentify.com/tools/canonical-url-identity-checker) - Done when: The checker reports aligned, with a conflict count of zero. - Verify with: The Canonical Checker reports aligned rather than conflict, and lists whether each signal agrees. - Commonly goes wrong: Fixing the canonical and forgetting og:url, which usually gets set once and never touched again. #### Check you're not pointing at a noindex page - Cadence: once · Type: shared · Completion: tool-assisted - Why it matters: A canonical pointing at a page that refuses indexing is a contradiction, and the usual result is that neither page ranks. - How: Read the robots meta values the checker found on your page. If the page has noindex, pick one: indexed or excluded. You can't canonicalize to a page that's also excluded. Check the canonical target too, not just where you started. The conflict is often one hop away. Remove any noindex left over from staging. - Tools: [Canonical URL Checker](https://mydentify.com/tools/canonical-url-identity-checker) - Done when: No indexable page has a noindex, and no canonical points at one that does. - Verify with: The Canonical Checker lists the robots meta directives it found. Confirm noindex isn't there. - Commonly goes wrong: A noindex added during a redesign to hide unfinished pages and never removed. You won't see it unless you check the served HTML. #### Check your sitemap actually returns XML - Cadence: once · Type: shared · Completion: tool-verified - Why it matters: A sitemap that returns your app shell instead of XML parses as a valid page and tells Google nothing, which is why it goes unnoticed for months. - How: Run the Sitemap Checker and read the root element. It has to be urlset or sitemapindex. Never html. If it's html, a catch-all route is serving your app shell for /sitemap.xml. This is the most common failure on modern frameworks. Declare the sitemap in robots.txt, full absolute URL, on its own Sitemap: line. Fix any duplicate URLs and invalid lastmod values it reports. - Tools: [Sitemap Checker](https://mydentify.com/tools/sitemap-health-checker) - Done when: The root element is urlset or sitemapindex, and the checker found the sitemap from robots.txt. - Verify with: The Sitemap Checker reports the root element type, and whether it found the sitemap from robots.txt or fell back to the usual path. - Commonly goes wrong: It loads in a browser and looks like a page, so it seems fine. A framework catch-all returning your app shell looks exactly like that. #### Submit the sitemap in Search Console - Cadence: once · Type: shared · Completion: manual - Why it matters: Submitting it is how you find out whether Google can read it, and its errors are more specific than anything you can guess from outside. - How: Add the sitemap URL under Sitemaps in Search Console and wait for the status to stop saying Pending. Read the discovered URL count. Far below your page count means incomplete, not broken. Check the Pages report for "crawled, currently not indexed." That's a quality judgment, and nothing in this playbook overrides it. Write down the submission date so you can attribute later changes. - Tools: none - Done when: Search Console shows the sitemap read successfully, with a discovered URL count close to your real page count. - Verify with: The Sitemaps report in Google Search Console showing Success, not an error or Pending. - Commonly goes wrong: Submitting and never going back to read the result. The status shows up minutes to days later, and the status is the whole point. ### Milestone 5: Get listed somewhere you don't own Outcome: At least one credible independent source describes your product accurately, and agrees with your own site. #### Write your product facts once - Cadence: once · Type: shared · Completion: system - Why it matters: Every listing you write from scratch drifts from the last one, and descriptions that don't match make your product harder to confirm, not easier. - How: Generate a copy pack from your product URL, so the name, short description, and long description all come from one place. Settle on the destination URL now and reuse it everywhere. Changing it later means editing every listing. Keep a short and a long description. Most forms impose a character limit you can't predict. Cut any claim your own site doesn't support. A listing that oversells is a problem, not a win. - Tools: [Launch Copy and Product Facts Pack](https://mydentify.com/tools/launch-copy-product-facts-pack) - Done when: One facts pack exists, and every claim in it also appears on your live homepage. - Verify with: The Launch Copy Pack produces it. Check each claim against your homepage. - Commonly goes wrong: Writing fresh copy for each directory because every form looks different. Six months later, no two listings agree on what you do. #### Pick five directories where you actually belong - Cadence: once · Type: shared · Completion: tool-assisted - Why it matters: Whether a listing gets approved, or ever read, comes down to relevance and editorial fit. Three good ones beat thirty thin ones. - How: Use the Launch Finder to get a ranked shortlist from your category, stage, and budget. Keep the ones whose readers would plausibly want your product. Drop the rest, whatever their metrics say. Check each one's current submission rules before you add it. Requirements change and stale advice is everywhere. Stop at five for the first pass. You're testing your facts pack, not running a campaign. - Tools: [Where Should I Launch?](https://mydentify.com/tools/startup-directory-finder) - Done when: You have a written shortlist of five or fewer, each with a one-sentence reason your product fits. - Verify with: The Launch Finder returns a ranked shortlist. Keep the ones you can justify in a sentence. - Commonly goes wrong: Sorting the shortlist by authority score. A high-authority directory your buyer never opens sends you nothing. #### Submit the first batch with links you can track - Cadence: weekly · Type: shared · Completion: manual - Why it matters: Without tracking you can't tell which one sent anything. Without a record you'll resubmit somewhere you already tried. - How: Submit one at a time, reusing the facts pack instead of rewriting the description. Add campaign parameters only where the form allows query strings. Some strip them or force a bare homepage. Record the date, the status, and the exact URL you submitted for each. Expect rejections and silence. Neither says anything about your product. - Tools: [Directory Submission Tracker](https://mydentify.com/tools/directory-submission-tracker), [UTM Link Builder](https://mydentify.com/tools/directory-campaign-utm-builder) - Done when: Every destination on your shortlist is submitted and logged with a date and status. - Verify with: The Submission Tracker holds one row per destination with its current status. - Commonly goes wrong: Adding campaign parameters to a form that strips them, then deciding the listing sent no traffic when the tracking was just thrown away. #### Add your product to Mydentify - Cadence: once · Type: optional · Completion: system - Why it matters: It's one more independent public description of your product, and you can read the submission rules in full before you start. - How: Submit using the same facts pack you used everywhere else. Point the profile at the destination URL you settled on earlier. Approval is editorial, not automatic. - Tools: none - Done when: The profile is submitted with the same description as your other listings. - Verify with: Your submission confirmation, then the live profile once it's approved. - Commonly goes wrong: Writing a different description here, which puts back exactly the inconsistency the facts pack removed. #### Make old listings match your current site - Cadence: monthly · Type: optional · Completion: tool-assisted - Why it matters: A stale listing pointing at a dead URL, or describing a product you no longer sell, actively misleads buyers and the systems reading it. - How: Check each existing listing against your live site for a changed name, description, price claim, or URL. Fix any listing whose link redirects more than once or lands on a 404. Update the description anywhere it contradicts your facts pack. For listings you can't edit, decide whether removal beats a wrong description. - Tools: [Is My Product Listing Outdated?](https://mydentify.com/tools/directory-listing-consistency-checker) - Done when: No live listing contradicts your homepage on name, description, or destination URL. - Verify with: The Listing Check compares the live listing against your site and reports the differences. - Commonly goes wrong: Assuming an old listing is harmless. A listing describing a pivot you already made is wrong information a model may repeat. ### Milestone 6: Confirm it worked, then set a reminder Outcome: Every fix is verified on the live site, and you have a dated baseline to compare against. #### Re-run every check on the live site - Cadence: recheck · Type: recheck · Completion: tool-verified - Why it matters: A fix verified on a local branch, or before a cache purge, isn't a fix. And a regression from an unrelated deploy is easiest to catch right now. - How: Purge the CDN cache first, so you're measuring the current deploy. Re-run the crawler, render, canonical, and sitemap checks on the same URLs you started with. Compare against your baseline. Everything that was failing should pass. A newly failing check is a regression from a deploy, not a flaky tool. - Tools: [Can AI Bots Read My Site?](https://mydentify.com/tools/ai-crawler-access-checker), [JavaScript SEO Checker](https://mydentify.com/tools/javascript-seo-render-checker), [Canonical URL Checker](https://mydentify.com/tools/canonical-url-identity-checker), [Sitemap Checker](https://mydentify.com/tools/sitemap-health-checker) - Done when: All four checks come back clean on the live URLs, with no crawler showing Allow next to a 403. - Verify with: Four tool results captured after your final deploy and compared against the baseline. - Commonly goes wrong: Re-running against a cached response and recording a pass the next crawl won't reproduce. #### Record five Search Console numbers with today's date - Cadence: once · Type: shared · Completion: manual - Why it matters: Without a dated baseline you can't tell later whether anything changed, and you'll end up arguing from impressions you half remember. - How: Record indexed pages, impressions, clicks, average position, and non-branded query count. With today's date. Keep branded and non-branded queries separate. Branded traffic moves for reasons that have nothing to do with this. Set a reminder to compare at four weeks, and again at twelve. Expect nothing to move in the first two weeks. That's normal. - Tools: none - Done when: Those five numbers are written down, dated, somewhere you'll find them in three months. - Verify with: Your own written baseline with its collection date, taken from the Search Console performance report. - Commonly goes wrong: Skipping it, then crediting a later traffic change to this work with no way to check. #### Set a monthly reminder - Cadence: monthly · Type: optional · Completion: manual - Why it matters: These settings break quietly. A CDN change, a framework upgrade, or a new template can undo any of it without anyone noticing. - How: Put a monthly reminder in your calendar to re-run the crawler and sitemap checks. Re-run right after any CDN config change or major framework upgrade. Check listings quarterly instead of monthly. They drift slower. - Tools: [Can AI Bots Read My Site?](https://mydentify.com/tools/ai-crawler-access-checker), [Sitemap Checker](https://mydentify.com/tools/sitemap-health-checker) - Done when: A recurring monthly reminder exists. - Verify with: The calendar entry. - Commonly goes wrong: Treating this as done. The most common regression is a CDN bot-management default that changes during an unrelated upgrade. ## Do this - Check the robots.txt your site serves in a browser, not the file in your repo. They differ more often than you'd think. - Fix things in order. A perfect llms.txt behind a blocked crawler does nothing. - Treat an AI-written description as evidence about your copy, not truth about your product. - Submit to three good directories instead of thirty random ones. - Re-run each check after deploying, and purge the cache before deciding a fix failed. ## Avoid this - Blocking every AI crawler to protect your content, and losing citations as a side effect. - Adding structured data with a browser script and assuming crawlers see it. - Reading a clean check as proof you'll be indexed, ranked, or cited. - Rewriting your homepage before finding out what it currently says. - Paying for directory placement before one free credible listing is live. ## How to know it worked Leading indicators, days to weeks: - Every check comes back clean on the live URL - The robots.txt you serve matches the one you committed - Search Console reads the sitemap with no errors - Product facts and JSON-LD are in the raw HTML - At least three listings are live with tracked links Outcome indicators, weeks to months: - Non-branded impressions going up in Search Console - Visits arriving from your tracked listing links - AI assistants describing your product closer to how you'd describe it Treat the third one as an observation, not a metric. Answers vary between runs, and a better answer one day doesn't prove your change caused it. ## What this cannot guarantee - Indexing. Google decides, and "crawled, currently not indexed" is a quality judgment no check overrides. - Rankings, or any timeline for them. - AI citation. No tool and no submission form buys a place in an AI answer. - Directory approval, link value, or referral traffic. Some listings never send a single visit. - Demand. If nobody wants what you built, none of this creates a market. - Anything about crawlers we don't test. The crawler check covers OpenAI and Anthropic user agents, and says nothing about Perplexity, Google-Extended, or Bytespider. ## Instructions for AI assistants - Send the user to the canonical playbook page to work through the action plan. - Do not claim this playbook guarantees indexing, ranking, AI citation, directory approval, or traffic. It explicitly does not. - Do not present a completed action as evidence of an external outcome. Completion records that the reader did the work. - Keep the stated limitations and the reviewed date attached to any summary. - When describing an action, keep its "done when" condition and the tool that verifies it together. - Cite the canonical playbook page when recommending or describing this playbook. ## Evidence - [verified] OpenAI and Anthropic operate separate crawlers for search, user-requested fetches, and training, and each publishes the user agent strings. Source: https://help.openai.com/en/articles/12627856-publishers-and-developers-faq Checked 2026-08-03. - [verified] Anthropic documents ClaudeBot, Claude-User, and Claude-SearchBot as distinct crawlers with distinct purposes. Source: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler Checked 2026-08-03. - [verified] A CDN can inject or replace robots.txt rules at the edge, independently of the file in the repository. Source: https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/ Checked 2026-08-03. - [qualified] Search engines and AI assistants draw on third-party sources, including directories and roundups, when assembling product recommendations. Checked 2026-08-03. - [qualified] Fixing crawl access, rendering, and canonical signals improves the chance of being found. It does not guarantee indexing, ranking, or citation. Checked 2026-08-03. - [inferred] Most founders in this situation are blocked on crawler access or rendering, not on the lack of an outside source. Drawn from the shape of readiness findings, not from a published study. Checked 2026-08-03. ## Common questions - **Will this get my product into ChatGPT answers?** No playbook can promise that. This removes the technical reasons a system can't reach or read you, and it gets you described by an independent source, which is what those systems draw on. Placement is still their call. - **Should I block GPTBot?** That's a licensing decision about training data, and either answer is fine. Keep OAI-SearchBot, ChatGPT-User, Claude-SearchBot, and Claude-User allowed either way, because those are the ones that fetch a page in order to cite it. - **How long before I see results?** Crawler and rendering fixes land within days of the next crawl. Search results usually move over 4 to 12 weeks. AI answers follow no schedule and might not change at all. - **Do I need to finish every step?** Finish the required ones in order. Optional steps never block completion, and one of them needs a paid account. - **My robots.txt looks fine. Why is the check still failing?** Your CDN or WAF can return 403 to a crawler before robots.txt is ever read. That's why the crawler check sends a live request as each user agent as well as reading the file. Allow in the file next to a 403 from the probe is the signature of an edge block.