What is robots.txt?

Every website has a small text file named exactly robots.txt. To see the client's: in your browser's address bar, type the site's domain name followed by /robots.txt and press Enter. For example, if the client's website is joespizza.com, type joespizza.com/robots.txt.

Think of it as a "who's allowed in" sign for automated visitors. Search engines like Google have always read it. Now AI assistants — ChatGPT, Gemini, Perplexity, and others — send their own automated readers too. If the site's robots.txt says "stay away," those AI assistants can never read the site. The client is invisible to them.

Why this is the #1 finding

Everything else in the report only matters if AI can read the site in the first place. Unblocking it is the single highest-impact fix.

The fastest fix: forward it to the client's web person

Many clients have a web person or host who edits this file — and that's fine. The SERAVERRA report has a "Copy message for your web person" card. Click its copy icon. It copies a ready-to-send message containing: what the problem is, the exact text to add, and exactly where to put it. Paste it into an email or text message to whoever manages the site and send it.

That's it. They'll know what to do.

Doing it yourself

If the client pays for web hosting directly (a company like Bluehost, GoDaddy, HostGator, or SiteGround) and you have access to that hosting account, you can do this in about two minutes:

  1. See the current file

    In your browser, go to yourdomain.com/robots.txt — using the client's real domain name. For example: joespizza.com/robots.txt. You'll see what's there now.

  2. Open the hosting file manager

    Log in to the client's hosting account (the company the client pays for web hosting), or ask the client for access. Look for "File Manager" — in cPanel it's an icon labeled exactly that. Open it, then open the website's main folder (often called public_html). Find the file named exactly robots.txt and click it to edit.

  3. Don't delete what's there

    The site's robots.txt may already have rules for Google and other search engines. Leave every existing line alone. You're only adding new lines.

  4. Paste the new lines at the very top

    In the SERAVERRA report, click Copy on the text box under this finding. Go back to the robots.txt editor. Click at the very beginning of the file — before the first character — and paste. The new lines must sit above any line that begins with the word Disallow.

  5. Save and verify

    Click Save (or Save Changes). Go back to yourdomain.com/robots.txt in your browser and refresh the page — you should see the new lines at the top.

Why the top?

When two rules conflict, the most specific one wins. The new "Allow" lines override a general "Disallow" for the AI readers listed — but only if they're placed above it.

What if the site is on Wix, Squarespace, or WordPress.com?

Here's the honest answer: on most website-builder platforms, nobody can open robots.txt directly. The platform controls that file. Here's exactly what to do on each:

  • Wix: Log in to the client's account at wix.com (or ask the client to add you to the site) and open the site's dashboard (not the editor — the dashboard). Click Settings in the left menu, then SEO, then look for a Robots.txt section. If you see it, you can add the lines from the report there. If you don't see it, the client's plan doesn't include it — use the "Copy message for your web person" card in the report or contact Wix support at wix.com/contact.
  • Squarespace: Squarespace does not let you edit robots.txt on any plan. Go to squarespace.com/help, click Contact Us, and ask: "Please allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and others) in this site's robots.txt." Or forward the issue to the client's web person using the button in the report.
  • WordPress.com (hosted): On the Free, Personal, and Premium plans, you cannot edit robots.txt. On the Business plan or higher (which allows plugins): install the free Yoast SEO plugin, then in the site's WordPress dashboard go to Yoast SEO → Tools → File Editor and add the lines there. On a lower plan, your options are upgrading or contacting WordPress.com support.
  • WordPress.org (self-hosted): You can edit it. In the site's WordPress dashboard, install the free Yoast SEO plugin (Plugins → Add New, search "Yoast SEO", Install, Activate). Then go to Yoast SEO → Tools → File Editor, paste the lines from the report at the top, and click Save changes to robots.txt. Or edit the file directly through your hosting file manager as described above.
  • Shopify: Shopify generates robots.txt automatically and doesn't allow direct editing on any plan. Contact Shopify support or forward it to the client's web person.

Bottom line: if the platform won't let you touch the file, use "Copy message for your web person" in the report. They deal with platform limitations every day.

After the fix

In the report, click the Rescan icon (or New scan in the top bar) to scan the same website again. The crawler check reads the site's robots.txt fresh each time — once the AI readers are allowed, this finding disappears and the score goes up.