What Robots.txt means
Robots.txt is a public file that tells crawlers which parts of your site they may or may not fetch. It is a request crawlers can choose to honor, not a security lock.
A founder trying to hide an unfinished section of their site by disallowing it in robots.txt might be surprised to learn the page is still publicly reachable by anyone with the direct link.
Robots.txt does not protect private data, and a rule blocking one path does not remove that path from the internet. For real privacy, use authentication, not a robots.txt entry.
Check your robots.txt file directly at yourdomain.com/robots.txt and confirm it is not accidentally blocking your product pages or listing pages from being crawled.
Why this matters for builders
A blocked path can hide product pages from search and AI crawlers.
Mistake builders make often
Some builders use robots.txt as if it were a security measure to hide sensitive pages, when it is only a request to crawlers and does not restrict actual access.
What to do now
Check your file with the Can AI Bots Read My Site? tool, and pair it with the Sitemap Checker to confirm the pages you want found are reachable.