Get your product found
Find out what your site tells people and machines about your product, fix what's blocking it, and get described somewhere credible.
By Mydentify · Updated August 2026
- Best for
- A launched product getting under 100 organic visits a month
- Effort
- About 8.5 hours · 20 steps
- Works best with
- Homepage Check, AI Bot Checker, JavaScript SEO Checker
What to expect: This clears obstacles. It doesn't create demand. An hour of checks here saves you from publishing content nothing can reach.
This is the long version. If 8.5 hours of detail is more than you want right now, the 6-step plan covers the same ground and links back here per step.
What to check before launch
If nobody's finding your product, it's usually one of four things, and they're worth checking in this order: a crawler is blocked, your product facts only appear after JavaScript runs, the wrong URL is marked as the real one, or nothing outside your own site mentions you. The first three are technical and take under an hour each to find. The fourth is slower, and it matters more than most founders expect, because search engines and AI assistants both look for a second source before recommending anything. Run the checks below in order, fix what turns up, then get listed somewhere credible.
- They can reach it. Is the crawler allowed to fetch the page at all?
- They can read it. Are your product facts in the HTML the server sends?
- One URL is clearly the real one. Do your canonical, og:url, and sitemap all agree?
- Someone else describes it. Is there a credible source that isn't your own site?
Complete the required checks first. Open a step only when you need the instructions.
Before you start · 3 things
- A live public URL that doesn't need a login
- The ability to edit robots.txt and page metadata, and to deploy
- A Google Search Console property, or ten minutes to create one
Action plan
Ticking steps needs JavaScript. Every step is readable without it.
Check what your homepage says
You know what a machine thinks your product does, and whether that matches what you meant.
Step 1Compare what your homepage says to what you'd say
How to do it- Paste your homepage URL into the Homepage Check.
- Write down the sentence it gives you. Under it, write the sentence you'd say to someone at a bar.
- Note anything it marked unclear or missing. A crawler is stuck on those too.
- Don't fix anything yet. You're just looking.
Pass when: You have two sentences written down — what the tool says you do, and what you'd say — and you can point at the difference.
Avoid the common failure
Your homepage is obvious to you because you built the product. It usually isn't obvious to anyone else.
If it goes wrong: You disagree with the output and dismiss it. It's reporting what your page supports. If it got the audience wrong, your page names the wrong audience.
Step 2Write one sentence on who it's for
How to do it- Write: "[product] helps [specific role] [do a specific job]." If the role could be "anyone," try again.
- Write a second sentence naming the problem the way your buyer says it, not the way your code says it.
- Put both on the homepage, in the first screenful. Not on the About page.
- Serving two audiences? Write for the one paying you today.
Pass when: Both sentences are in your homepage HTML, in the first screenful, and neither says "anyone" or "everything."
Avoid the common failure
A page that never says who it's for makes everyone guess, and they guess bigger and vaguer than the truth.
If it goes wrong: The sentences end up in a design file or a pitch deck and never reach the page. Check the live URL.
Step 3Fix your headline if the tool got it wrong
How to do it- Replace your headline and first paragraph with the two sentences you just wrote.
- Delete any opening line that describes a category instead of your product. "The modern platform for growth" says nothing.
- Keep the product name in the first sentence. Readers and models both anchor on it.
- Deploy, re-run the Homepage Check, and see whether the description moved closer to yours.
Pass when: The re-run description is close enough to yours that you'd accept it as a description of your product.
Avoid the common failure
Everything later assumes your page says true things. Fixing crawler access to a page that misdescribes your product just publishes the wrong answer faster.
If it goes wrong: You change the copy but the check returns the old text, because your CDN is serving cached HTML. Purge the cache before deciding the rewrite failed.
Step 4Run three checks before you change anything
How to do it- Run the AI Bot Checker, the JavaScript SEO Checker, and the Canonical Checker on one important product URL. Change nothing first.
- Note which one fails: they can't reach it, they can't read it, the wrong URL is marked as real, or nobody else describes it.
- Fix the earliest failure first. Fixing a canonical won't help if the crawler can't get in.
- If all three come back clean, your problem is the fourth one, or demand. Skip ahead to getting described somewhere else.
Pass when: You've named the earliest failing check, or confirmed the first three pass and your work belongs in getting described somewhere else.
Avoid the common failure
A product with flat traffic usually has one thing badly broken, not four things mildly broken. Finding which one saves you a week on the wrong problem.
If it goes wrong: Running the checks after you've started fixing things, so you can't tell which change mattered. Get the baseline first.
Step 5 · optionalRun the AI Product Read on the same URL
How to do it- Run the AI Product Read on the same homepage URL you used for the Homepage Check.
- Compare the two. Where they disagree, your page supports two readings.
- Anything it claims that you can't find on your page is something your copy invites people to make up.
Pass when: You've read it and listed any claim it made that your page doesn't actually support.
Avoid the common failure
It shows you the gap between what you published and what a system can actually conclude from it, and which text it relied on.
If it goes wrong: Reading it as a verdict on your product. It's a measurement of your page.
Let the right crawlers in
The crawlers that cite products can reach your pages, and your training-data rule is written down and dated.
Step 6Let the crawlers that cite you reach your site
How to do it- Open yourdomain.com/robots.txt in a browser. Not the file in your repo — the one your site serves.
- If the served file is longer than your committed one, something at the edge is adding rules. Search it for "# BEGIN Cloudflare Managed content".
- Check that OAI-SearchBot, ChatGPT-User, Claude-SearchBot, and Claude-User aren't caught by a Disallow, including a broad "User-agent: *" followed by "Disallow: /".
- Leave training crawlers for the next step. Blocking GPTBot and ClaudeBot is a licensing choice and doesn't affect whether you get cited.
Pass when: All four search and user-requested crawlers show Allow for your product URLs.
Avoid the common failure
Block these and your product disappears from AI answers, no matter how good your content is.
If it goes wrong: Your committed robots.txt is right and your served one isn't. At most CDNs, managed robots.txt and AI crawl control are separate switches, and either can overwrite your file at the edge.
Step 7Check whether your CDN blocks what robots.txt allows
How to do it- Read the live probe column, not just the robots.txt column. The checker sends a real request as each crawler.
- Allow in robots.txt plus a 403 from the probe means an edge or WAF block. That combination is the tell.
- Look at your CDN's bot management and AI crawler settings. They're usually separate from anything robots.txt related.
- Re-run after you change it. A 200 or 301 passes. A 403 or 429 doesn't.
Pass when: No crawler shows Allow in robots.txt next to a 403 from the live probe.
Avoid the common failure
robots.txt shows your intent, not your configuration. An edge rule can return 403 to a crawler your file welcomes, and the file gives you no clue.
If it goes wrong: Stopping at a clean robots.txt. Bot management runs before your file is ever read, so the file can be perfect while the request still fails.
Step 8Record your training-data decision in robots.txt
How to do it- Sort the crawlers by what they do. GPTBot and ClaudeBot collect training data. OAI-SearchBot and Claude-SearchBot power search. ChatGPT-User and Claude-User fetch a page because someone asked.
- Decide training access as a licensing question about your content, not a visibility question.
- Write the decision and the date in a comment at the top of robots.txt, so the next person doesn't reverse it on a guess.
- Keep the search and user-requested crawlers allowed either way.
Pass when: Your robots.txt carries a dated, deliberate decision about training crawlers, and the search and user-requested ones are allowed.
Avoid the common failure
Training access and citation access are two different decisions. One "block AI bots" switch collapses them into a choice you didn't mean to make.
If it goes wrong: Blocking every AI user agent to protect your content, then wondering why you stopped showing up in AI answers. The citation crawlers were collateral damage.
Step 9 · optionalPublish an llms.txt that points at pages that exist
How to do it- List your most useful public pages, one line of description each. Only pages that return 200.
- Serve it at /llms.txt with a text/plain or text/markdown content type.
- It isn't an access control file. It requests nothing and enforces nothing.
- Validate it, then check the crawlers that would read it aren't blocked. A perfect file behind a blocked crawler does nothing.
Pass when: The file validates, every URL in it returns 200, and the crawlers that would fetch it are allowed.
Avoid the common failure
It's an optional convention nobody enforces, but it can help a system find your most useful pages. Worth twenty minutes once access is fixed.
If it goes wrong: Doing this first. It's the last step here, not the first, and no crawler has to read it.
Put your facts in the server response
Your product name, description, and structured data are in the HTML your server sends, before any script runs.
Step 10List the facts that only appear after JavaScript
How to do it- Run the JavaScript SEO Checker on an important product page and read the raw column, not the rendered one.
- Check your product name, description, and price are all in the raw response.
- Anything that shows up only in the rendered column is something some systems will never see.
- Look at JSON-LD specifically. It's the one most often added by a tag manager or a browser script.
Pass when: You have a list of every product fact and structured data block that only appears after JavaScript runs.
Avoid the common failure
A crawler that doesn't run your JavaScript sees only the server response. Anything your code adds in the browser is invisible to it.
If it goes wrong: Checking in browser DevTools, which shows the rendered DOM and hides the whole problem. Use view-source or the tool.
Step 11Move your JSON-LD into the server response
How to do it- Render the JSON-LD script tag on the server, so it shows up in view-source.
- Keep it consistent with what's visible on the page. Schema describing something the page doesn't show is a problem, not a win.
- Delete any duplicate JSON-LD your tag manager left behind. Two conflicting descriptions are worse than one.
- Deploy, re-run the checker, and confirm the JSON-LD is now in the raw column.
Pass when: The JSON-LD is in the raw server response, matches what's visible, and appears exactly once.
Avoid the common failure
Structured data describes your product to systems that may never run your scripts. Added in the browser, it's a description that only sometimes exists.
If it goes wrong: Adding schema through a tag manager because it ships faster. That puts it only in the rendered DOM, which is the thing you're trying to fix.
Step 12 · recheckRe-run the check after you deploy
How to do it- Wait for the deploy to finish, then purge the CDN or full-page cache for the URL you're testing.
- Re-run the checker on the same URL you used for the baseline.
- Confirm nothing is left in the rendered-only list.
- If nothing changed, check you fixed the same route you're testing.
Pass when: No product facts and no JSON-LD appear only after hydration.
Avoid the common failure
The fix is only real on the deployed URL, and caching often serves the old HTML for longer than your deploy log suggests.
If it goes wrong: Re-running before the cache expires and deciding the fix failed. Purge first, then measure.
Make your URL signals agree
Your canonical tag, your sitemap, and your page all name the same URL.
Step 13Check your canonical points where you think it does
How to do it- Run the Canonical Checker on an important product URL.
- On an indexable page, the canonical should point at that same page. Not the homepage.
- Watch for a canonical pointing at a staging or www variant. A canonical to a hostname you no longer serve is a common template leftover.
- Check whether your URL redirected. If it did, the canonical should name the destination, not where you started.
Pass when: Every indexable product page has a canonical pointing at itself, on the hostname you actually serve.
Avoid the common failure
A canonical on the wrong URL tells Google to index a different page than the one you want, and nothing on the page looks broken.
If it goes wrong: A site-wide template sets every canonical to the homepage, so every product page asks to be de-indexed in favor of the root.
Step 14Make canonical, og:url, and your sitemap match
How to do it- Read the conflict count. Anything above zero means two signals name different URLs.
- Make the canonical tag, og:url, and the sitemap URL the exact same string, trailing slash included.
- Pick one hostname convention and one trailing-slash convention and use them everywhere. Mixed conventions cause most of these conflicts.
- Deploy and re-run until it reports aligned.
Pass when: The checker reports aligned, with a conflict count of zero.
Avoid the common failure
When your signals disagree about which URL is real, Google picks one. It doesn't have to pick yours.
If it goes wrong: Fixing the canonical and forgetting og:url, which usually gets set once and never touched again.
Step 15Check you're not pointing at a noindex page
How to do it- Read the robots meta values the checker found on your page.
- If the page has noindex, pick one: indexed or excluded. You can't canonicalize to a page that's also excluded.
- Check the canonical target too, not just where you started. The conflict is often one hop away.
- Remove any noindex left over from staging.
Pass when: No indexable page has a noindex, and no canonical points at one that does.
Avoid the common failure
A canonical pointing at a page that refuses indexing is a contradiction, and the usual result is that neither page ranks.
If it goes wrong: A noindex added during a redesign to hide unfinished pages and never removed. You won't see it unless you check the served HTML.
Step 16Check your sitemap actually returns XML
How to do it- Run the Sitemap Checker and read the root element. It has to be urlset or sitemapindex. Never html.
- If it's html, a catch-all route is serving your app shell for /sitemap.xml. This is the most common failure on modern frameworks.
- Declare the sitemap in robots.txt, full absolute URL, on its own Sitemap: line.
- Fix any duplicate URLs and invalid lastmod values it reports.
Pass when: The root element is urlset or sitemapindex, and the checker found the sitemap from robots.txt.
Avoid the common failure
A sitemap that returns your app shell instead of XML parses as a valid page and tells Google nothing, which is why it goes unnoticed for months.
If it goes wrong: It loads in a browser and looks like a page, so it seems fine. A framework catch-all returning your app shell looks exactly like that.
Step 17Submit the sitemap in Search Console
How to do it- Add the sitemap URL under Sitemaps in Search Console and wait for the status to stop saying Pending.
- Read the discovered URL count. Far below your page count means incomplete, not broken.
- Check the Pages report for "crawled, currently not indexed." That's a quality judgment, and nothing in this playbook overrides it.
- Write down the submission date so you can attribute later changes.
Pass when: Search Console shows the sitemap read successfully, with a discovered URL count close to your real page count.
Avoid the common failure
Submitting it is how you find out whether Google can read it, and its errors are more specific than anything you can guess from outside.
If it goes wrong: Submitting and never going back to read the result. The status shows up minutes to days later, and the status is the whole point.
Get listed somewhere you don't own
At least one credible independent source describes your product accurately, and agrees with your own site.
Step 18Write your product facts once
How to do it- Generate a copy pack from your product URL, so the name, short description, and long description all come from one place.
- Settle on the destination URL now and reuse it everywhere. Changing it later means editing every listing.
- Keep a short and a long description. Most forms impose a character limit you can't predict.
- Cut any claim your own site doesn't support. A listing that oversells is a problem, not a win.
Pass when: One facts pack exists, and every claim in it also appears on your live homepage.
Avoid the common failure
Every listing you write from scratch drifts from the last one, and descriptions that don't match make your product harder to confirm, not easier.
If it goes wrong: Writing fresh copy for each directory because every form looks different. Six months later, no two listings agree on what you do.
Step 19Pick five directories where you actually belong
How to do it- Use the Launch Finder to get a ranked shortlist from your category, stage, and budget.
- Keep the ones whose readers would plausibly want your product. Drop the rest, whatever their metrics say.
- Check each one's current submission rules before you add it. Requirements change and stale advice is everywhere.
- Stop at five for the first pass. You're testing your facts pack, not running a campaign.
Pass when: You have a written shortlist of five or fewer, each with a one-sentence reason your product fits.
Avoid the common failure
Whether a listing gets approved, or ever read, comes down to relevance and editorial fit. Three good ones beat thirty thin ones.
If it goes wrong: Sorting the shortlist by authority score. A high-authority directory your buyer never opens sends you nothing.
Step 20 · weeklySubmit the first batch with links you can track
How to do it- Submit one at a time, reusing the facts pack instead of rewriting the description.
- Add campaign parameters only where the form allows query strings. Some strip them or force a bare homepage.
- Record the date, the status, and the exact URL you submitted for each.
- Expect rejections and silence. Neither says anything about your product.
Pass when: Every destination on your shortlist is submitted and logged with a date and status.
Avoid the common failure
Without tracking you can't tell which one sent anything. Without a record you'll resubmit somewhere you already tried.
If it goes wrong: Adding campaign parameters to a form that strips them, then deciding the listing sent no traffic when the tracking was just thrown away.
Step 21 · optionalAdd your product to Mydentify
How to do it- Submit using the same facts pack you used everywhere else.
- Point the profile at the destination URL you settled on earlier.
- Approval is editorial, not automatic.
Pass when: The profile is submitted with the same description as your other listings.
Avoid the common failure
It's one more independent public description of your product, and you can read the submission rules in full before you start.
If it goes wrong: Writing a different description here, which puts back exactly the inconsistency the facts pack removed.
Step 22 · monthly · optionalMake old listings match your current site
How to do it- Check each existing listing against your live site for a changed name, description, price claim, or URL.
- Fix any listing whose link redirects more than once or lands on a 404.
- Update the description anywhere it contradicts your facts pack.
- For listings you can't edit, decide whether removal beats a wrong description.
Pass when: No live listing contradicts your homepage on name, description, or destination URL.
Avoid the common failure
A stale listing pointing at a dead URL, or describing a product you no longer sell, actively misleads buyers and the systems reading it.
If it goes wrong: Assuming an old listing is harmless. A listing describing a pivot you already made is wrong information a model may repeat.
Confirm it worked, then set a reminder
Every fix is verified on the live site, and you have a dated baseline to compare against.
Step 23 · recheckRe-run every check on the live site
How to do it- Purge the CDN cache first, so you're measuring the current deploy.
- Re-run the crawler, render, canonical, and sitemap checks on the same URLs you started with.
- Compare against your baseline. Everything that was failing should pass.
- A newly failing check is a regression from a deploy, not a flaky tool.
Pass when: All four checks come back clean on the live URLs, with no crawler showing Allow next to a 403.
Avoid the common failure
A fix verified on a local branch, or before a cache purge, isn't a fix. And a regression from an unrelated deploy is easiest to catch right now.
If it goes wrong: Re-running against a cached response and recording a pass the next crawl won't reproduce.
Step 24Record five Search Console numbers with today's date
How to do it- Record indexed pages, impressions, clicks, average position, and non-branded query count. With today's date.
- Keep branded and non-branded queries separate. Branded traffic moves for reasons that have nothing to do with this.
- Set a reminder to compare at four weeks, and again at twelve.
- Expect nothing to move in the first two weeks. That's normal.
Pass when: Those five numbers are written down, dated, somewhere you'll find them in three months.
Avoid the common failure
Without a dated baseline you can't tell later whether anything changed, and you'll end up arguing from impressions you half remember.
If it goes wrong: Skipping it, then crediting a later traffic change to this work with no way to check.
Step 25 · monthly · optionalSet a monthly reminder
How to do it- Put a monthly reminder in your calendar to re-run the crawler and sitemap checks.
- Re-run right after any CDN config change or major framework upgrade.
- Check listings quarterly instead of monthly. They drift slower.
Pass when: A recurring monthly reminder exists.
Avoid the common failure
These settings break quietly. A CDN change, a framework upgrade, or a new template can undo any of it without anyone noticing.
If it goes wrong: Treating this as done. The most common regression is a CDN bot-management default that changes during an unrelated upgrade.
Tips and traps
- Check the robots.txt your site serves in a browser, not the file in your repo. They differ more often than you'd think.
- Fix things in order. A perfect llms.txt behind a blocked crawler does nothing.
- Treat an AI-written description as evidence about your copy, not truth about your product.
- Submit to three good directories instead of thirty random ones.
- Re-run each check after deploying, and purge the cache before deciding a fix failed.
- Blocking every AI crawler to protect your content, and losing citations as a side effect.
- Adding structured data with a browser script and assuming crawlers see it.
- Reading a clean check as proof you'll be indexed, ranked, or cited.
- Rewriting your homepage before finding out what it currently says.
- Paying for directory placement before one free credible listing is live.
How to know it worked
- Every check comes back clean on the live URL
- The robots.txt you serve matches the one you committed
- Search Console reads the sitemap with no errors
- Product facts and JSON-LD are in the raw HTML
- At least three listings are live with tracked links
- Non-branded impressions going up in Search Console
- Visits arriving from your tracked listing links
- AI assistants describing your product closer to how you'd describe it
What this cannot guarantee
- Indexing. Google decides, and "crawled, currently not indexed" is a quality judgment no check overrides.
- Rankings, or any timeline for them.
- AI citation. No tool and no submission form buys a place in an AI answer.
- Directory approval, link value, or referral traffic. Some listings never send a single visit.
- Demand. If nobody wants what you built, none of this creates a market.
- Anything about crawlers we don't test. The crawler check covers OpenAI and Anthropic user agents, and says nothing about Perplexity, Google-Extended, or Bytespider.
Treat the third one as an observation, not a metric. Answers vary between runs, and a better answer one day doesn't prove your change caused it.
Sources · 6 claims checked
Verified means a named primary source states it. Qualified means it is well supported but conditional. Inferred means it is our reading, not a published finding.
- verifiedOpenAI and Anthropic operate separate crawlers for search, user-requested fetches, and training, and each publishes the user agent strings. Source
- verifiedAnthropic documents ClaudeBot, Claude-User, and Claude-SearchBot as distinct crawlers with distinct purposes. Source
- verifiedA CDN can inject or replace robots.txt rules at the edge, independently of the file in the repository. Source
- qualifiedSearch engines and AI assistants draw on third-party sources, including directories and roundups, when assembling product recommendations.
- qualifiedFixing crawl access, rendering, and canonical signals improves the chance of being found. It does not guarantee indexing, ranking, or citation.
- inferredMost founders in this situation are blocked on crawler access or rendering, not on the lack of an outside source. Drawn from the shape of readiness findings, not from a published study.
Questions
Will this get my product into ChatGPT answers?
No playbook can promise that. This removes the technical reasons a system can't reach or read you, and it gets you described by an independent source, which is what those systems draw on. Placement is still their call.
Should I block GPTBot?
That's a licensing decision about training data, and either answer is fine. Keep OAI-SearchBot, ChatGPT-User, Claude-SearchBot, and Claude-User allowed either way, because those are the ones that fetch a page in order to cite it.
How long before I see results?
Crawler and rendering fixes land within days of the next crawl. Search results usually move over 4 to 12 weeks. AI answers follow no schedule and might not change at all.
Do I need to finish every step?
Finish the required ones in order. Optional steps never block completion, and one of them needs a paid account.
My robots.txt looks fine. Why is the check still failing?
Your CDN or WAF can return 403 to a crawler before robots.txt is ever read. That's why the crawler check sends a live request as each user agent as well as reading the file. Allow in the file next to a 403 from the probe is the signature of an edge block.