A Disallow: / left over from staging quietly removes a whole site from search. This skill finds that and ten other robots.txt and sitemap problems, and tells you how to fix each one.

Your agent runs a scanner on robots.txt and every sitemap XML file in the project or build output. It also checks each sitemap URL against robots.txt, using Google's longest-match rule. It returns BLOCK, REVIEW, or NOT FLAGGED, with the file, line, reason, and a safer approach.

What it checks

BLOCK (fix before shipping)

• R01: Disallow: / (or /*) for all crawlers or Googlebot

• R02: localhost, private IP or staging URLs in the sitemap

• R03: a sitemap that is not valid sitemap XML

REVIEW (check before release)

• R11: no Sitemap line in robots.txt

• R12: a relative Sitemap line

• R13: CSS, JS or asset folders blocked

• R14: misspelled or unsupported rules such as Noindex

• R15: rules before any User-agent line

• R16: mixed http/https or hosts in a sitemap

• R17: bad or future lastmod dates

• R18: sitemap URLs that robots.txt blocks

What's included

• The scanner (Python 3.9+, standard library only)

• A rules guide with the reason and safer approach for each rule

• Risky and safe example files, and the real report the scanner produces on the risky ones

Limits

It reads files as written and does not fetch the live site, unzip .xml.gz sitemaps, or follow a sitemap index to its child sitemaps. NOT FLAGGED is not a guarantee of rankings.

Contents

One ZIP file with 12 files: SKILL.md, the scanner in scripts/ (2 Python files), references/rules.md (every rule with its reason and safer approach), and examples/ (7 example files plus report.md, the real scanner output on the risky example).

How to use: This is an agent skill: a folder with a SKILL.md file and Python scripts that an AI coding agent such as Claude Code, Cursor, or Codex CLI can run. Unzip it into your agent's skills folder (for Claude Code: ~/.claude/skills/), then ask your agent to check your project. The scripts also run on their own with Python 3.9 or newer, with nothing to install. They read your files only: no network access, and nothing is sent anywhere.