--- name: site-proof description: Proof a site before doing content or launch work — find missing pages, broken or wrongly-sized images, thin pages, missing 404/robots/sitemap/llms.txt, noindex state, and heading and metadata problems. Use at the START of any site build, migration, redesign or launch, when someone says a site is "ready", or before a cutover, so the gaps are known before work is redone. --- # Proof the site first Run this **before** doing content work, not after. The point is to find what is missing while it is still cheap to fix — a page nobody mentioned, an image that only exists at 400px, a stub that was never written. Finding those after the copy is signed off means doing the work twice. **Report. Do not fix.** The output of this skill is a list handed to a human, plus the two or three questions only they can answer. Fixing comes after they have decided what is in scope. ## 1. Ask first Three things cannot be inferred from the files. Ask them before running anything, in one message: 1. **What is the full page inventory?** Get it from whoever owns the content, not from the prototype. Prototypes are missing pages far more often than they have extra ones, and a page nobody built is invisible to every automated check below. 2. **Does this site need an `llms.txt`?** Cheap now, annoying once copy is frozen. If they do not know what it is: a plain-text file at the root that tells AI crawlers what the site is and what matters on it. 3. **When does it launch, and should it be indexable before then?** This decides whether missing `noindex` is a defect or correct. Ask, then run the checks while you wait. Do not block on the answers. ## 2. Run the checks Write the script at the bottom of this file to `/tmp/proof.py` and run it against either a build directory or a live URL: ```bash python3 /tmp/proof.py ./dist # local build python3 /tmp/proof.py https://staging.example.com ``` Stdlib only — no `pip install`, works on any machine. It reports and never writes to the site. Local-directory mode checks what is on disk. URL mode additionally checks real HTTP status codes, which is the only way to catch a **soft 404** (unknown paths returning 200 with the homepage). Run URL mode against staging when there is one. What it covers: | Area | Checks | |---|---| | Pages | every internal link resolves; nothing 404s | | Content | thin pages (< 150 words); leftover lorem/placeholder/TODO | | Images | missing files, hotlinks to the old site, no `alt`, upscaled sources, sources far larger than their slot, files over 500 KB, missing width/height | | Headings | an `h1` on every page, exactly one, at least one `h2`, no skipped levels | | Metadata | title and description present, canonical present, no duplicates across pages or languages | | Infrastructure | `robots.txt`, sitemap (via `robots.txt`, then the usual paths), `llms.txt`, a real 404, `noindex` consistency | Severities: **BLOCK** is broken and must be fixed. **WARN** needs a human to look. **ASK** is a question for the client. **INFO** is worth knowing. ## 3. Report it as a decision list Do not paste the JSON. Turn it into something the person can act on: - **Lead with what is missing, not what is wrong.** Pages that do not exist and images that were never supplied are the things that change the plan. A missing `alt` does not. - **One line per distinct problem.** The script already collapses per-page repeats — a footer logo with no `alt` is one fix, not fourteen. Keep it that way in your summary. - **Separate "I can fix this" from "you must decide this."** Alt text you can write. Whether `/careers` ships in v1 is not yours to choose. - **Say what you did not check.** The script reads static HTML. A JS-rendered site will look thin to it and its images may not appear at all — say so rather than reporting a false all-clear. Then stop and wait. Do not start fixing. ## Judgement calls the script cannot make - **Thin pages are a signal, not a verdict.** A contact page is legitimately 60 words. Check the flagged pages by eye before repeating the number. - **`noindex` has two failure modes** and the script cannot tell which one you are in: a staging site indexed by Google outranks the real one, and a launch that forgets to lift `noindex` is worse than both. Whichever state the site is in, confirm there is a ticket for the transition. - **Image sizing needs the rendered slot**, which is a layout fact, not a file fact. The script compares against the `width` attribute where one exists. Where none does, measure the real slot in a browser before trusting it. - **Links to downloads are not pages.** The script skips `.md`, `.pdf`, `.xml` and friends when crawling — it still reports them if they 404, but it will not complain that a PDF has no `h1`. If you add a new download type, that list is `NOT_A_PAGE`. - **Duplicate titles across languages are usually real.** `/` and `/en/` sharing a title means the translated page never got its own — it is not a false positive just because the pages are different languages. ## The script Write this to `/tmp/proof.py` verbatim. ```python #!/usr/bin/env python3 """Pre-launch proof pass. Stdlib only — no pip install, works on any machine. python3 proof.py python3 proof.py Reports. Never fixes. """ import sys, os, re, json, struct, urllib.request, urllib.parse, urllib.error from html.parser import HTMLParser THIN_WORDS = 150 OVERSIZE_RATIO = 2.0 # source pixels vs declared width HEAVY_BYTES = 500_000 # ---------------------------------------------------------------- image dims def dims(b): """(w,h) from raw bytes for png/jpg/gif/webp. None if unreadable.""" try: if b[:8] == b'\x89PNG\r\n\x1a\n': return struct.unpack('>II', b[16:24]) if b[:3] == b'\xff\xd8\xff': i = 2 while i < len(b) - 9: if b[i] != 0xFF: i += 1; continue m = b[i+1] if m in (0xD8, 0x01) or 0xD0 <= m <= 0xD7: i += 2; continue ln = struct.unpack('>H', b[i+2:i+4])[0] if 0xC0 <= m <= 0xCF and m not in (0xC4, 0xC8, 0xCC): h, w = struct.unpack('>HH', b[i+5:i+9]); return (w, h) i += 2 + ln if b[:6] in (b'GIF87a', b'GIF89a'): return struct.unpack('> 14) & 0x3FFF) + 1) if c == b'VP8X': w = b[24] | b[25] << 8 | b[26] << 16 h = b[27] | b[28] << 8 | b[29] << 16 return (w + 1, h + 1) except Exception: pass return None # ---------------------------------------------------------------- html parse class Page(HTMLParser): def __init__(self): super().__init__(convert_charrefs=True) self.title = None; self.desc = None; self.canonical = None self.robots = ''; self.headings = []; self.imgs = []; self.links = [] self._in_title = False; self._skip = 0; self._text = [] self._h = None def handle_starttag(self, tag, attrs): a = dict(attrs) if tag == 'title': self._in_title = True elif tag in ('script', 'style', 'nav', 'header', 'footer'): self._skip += 1 elif tag == 'meta': n = (a.get('name') or '').lower() if n == 'description': self.desc = a.get('content', '') if n == 'robots': self.robots = (a.get('content') or '').lower() elif tag == 'link' and 'canonical' in (a.get('rel') or ''): self.canonical = a.get('href') elif tag in ('h1','h2','h3','h4','h5','h6'): self._h = (int(tag[1]), []); self.headings.append(self._h) elif tag == 'img': self.imgs.append({'src': a.get('src'), 'alt': a.get('alt'), 'w': a.get('width'), 'srcset': a.get('srcset'), 'loading': a.get('loading')}) elif tag == 'a' and a.get('href'): self.links.append(a['href']) def handle_endtag(self, tag): if tag == 'title': self._in_title = False elif tag in ('script','style','nav','header','footer'): self._skip = max(0, self._skip-1) elif tag in ('h1','h2','h3','h4','h5','h6'): self._h = None def handle_data(self, d): if self._in_title: self.title = (self.title or '') + d.strip() if self._h is not None: self._h[1].append(d.strip()) if not self._skip: self._text.append(d) @property def words(self): return len(re.findall(r"[A-Za-zÀ-ÿ0-9']+", ' '.join(self._text))) # ---------------------------------------------------------------- sources class Src: """Uniform read interface over a local dir or a live base URL.""" def __init__(self, target): self.url = target.startswith(('http://', 'https://')) self.base = target.rstrip('/') if self.url else os.path.abspath(target) def get(self, path): """-> (status, bytes) ; status None when unreachable.""" if self.url: u = self.base + '/' + path.lstrip('/') req = urllib.request.Request(u, headers={'User-Agent': 'preflight-proof/1'}) try: with urllib.request.urlopen(req, timeout=20) as r: return r.status, r.read() except urllib.error.HTTPError as e: return e.code, b'' except Exception: return None, b'' p = os.path.join(self.base, path.lstrip('/')) if os.path.isdir(p): p = os.path.join(p, 'index.html') if not os.path.exists(p): return 404, b'' return 200, open(p, 'rb').read() def pages(self): if not self.url: out = [] for root, _, files in os.walk(self.base): for f in files: if f.endswith('.html'): rel = os.path.relpath(os.path.join(root, f), self.base) out.append('/' + rel.replace(os.sep, '/')) return sorted(out) out = [] for sm in self.sitemaps(): st, b = self.get(sm) if st != 200 or not b: continue locs = [l.decode() for l in re.findall(rb'\s*([^<]+?)\s*', b)] # a sitemap index points at sitemaps, not pages — follow one level if b'\s*([^<]+?)\s*', b2)] locs = [l for l in locs if '.xml' not in l] out += [urllib.parse.urlparse(l).path or '/' for l in locs if is_page(l)] return sorted(set(out)) if out else ['/'] def sitemaps(self): """Sitemap URLs, robots.txt first — that is where the real one is declared. Only /sitemap.xml is a convention, not a rule; a site can publish /sitemap-index.xml and be perfectly correct.""" found = [] st, b = self.get('/robots.txt') if st == 200: for m in re.findall(r'(?im)^\s*sitemap:\s*(\S+)', b.decode('utf-8', 'replace')): found.append(urllib.parse.urlparse(m).path or m) for c in ('/sitemap.xml', '/sitemap-index.xml', '/sitemap_index.xml'): if c not in found: found.append(c) return found # Extensions that are downloads, not pages. Following them produces nonsense # findings — a .md file has no and no h1 because it is not a page. NOT_A_PAGE = ('.md','.txt','.xml','.json','.pdf','.zip','.csv','.svg','.png', '.jpg','.jpeg','.webp','.gif','.ico','.mp4','.webm','.css','.js') def is_page(p): ext = os.path.splitext(urllib.parse.urlparse(p).path)[1].lower() return ext in ('', '.html', '.htm') def norm(p): p = p.split('#')[0].split('?')[0] if p.endswith('/index.html'): p = p[:-10] return p or '/' def main(): if len(sys.argv) < 2: print(__doc__); sys.exit(2) src = Src(sys.argv[1]) findings = [] def add(sev, area, msg, where=''): findings.append({'sev': sev, 'area': area, 'msg': msg, 'where': where}) paths = src.pages() if src.url: # crawl one level from the seeds so unsitemapped pages still surface seen, queue = set(), list(paths) while queue and len(seen) < 60: p = norm(queue.pop(0)) if p in seen: continue seen.add(p) st, b = src.get(p) if st != 200: continue pg = Page(); pg.feed(b.decode('utf-8', 'replace')) for h in pg.links: if h.startswith(('http', 'mailto:', 'tel:', '#', 'javascript:')): continue q = norm(urllib.parse.urljoin(p if p.endswith('/') else p + '/', h)) if q.startswith('/cdn-cgi/') or not is_page(q): continue if q not in seen: queue.append(q) paths = sorted(seen) pages, titles, descs = {}, {}, {} for p in paths: st, b = src.get(p) if st != 200: add('BLOCK', 'pages', f'{p} does not resolve (HTTP {st})'); continue pg = Page(); pg.feed(b.decode('utf-8', 'replace')) pages[norm(p)] = pg # --- content if pg.words < THIN_WORDS: add('WARN', 'content', f'thin page — {pg.words} words of body text', norm(p)) if re.search(r'lorem ipsum|coming soon|placeholder text|TODO|FIXME', ' '.join(pg._text), re.I): add('BLOCK', 'content', 'placeholder text still in the page', norm(p)) # --- headings h1 = [h for h in pg.headings if h[0] == 1] if not h1: add('BLOCK', 'headings', 'no h1', norm(p)) elif len(h1) > 1: add('WARN', 'headings', f'{len(h1)} h1 elements', norm(p)) if not [h for h in pg.headings if h[0] == 2]: add('WARN', 'headings', 'no h2 — page has no visible structure', norm(p)) lvls = [h[0] for h in pg.headings] for a, b_ in zip(lvls, lvls[1:]): if b_ - a > 1: add('WARN', 'headings', f'h{a} followed by h{b_} — level skipped', norm(p)); break # --- metadata if not pg.title: add('BLOCK', 'meta', 'no <title>', norm(p)) else: titles.setdefault(pg.title, []).append(norm(p)) if not pg.desc: add('WARN', 'meta', 'no meta description', norm(p)) else: descs.setdefault(pg.desc, []).append(norm(p)) if not pg.canonical: add('WARN', 'meta', 'no canonical URL', norm(p)) # --- images for im in pg.imgs: s = im['src'] if not s: add('BLOCK', 'images', '<img> with no src', norm(p)); continue if im['alt'] is None: add('BLOCK', 'images', f'no alt attribute: {s}', norm(p)) if s.startswith('http'): add('WARN', 'images', f'hotlinked off-site: {s}', norm(p)); continue ip = s if s.startswith('/') else urllib.parse.urljoin( norm(p) if norm(p).endswith('/') else norm(p) + '/', s) ist, ib = src.get(ip) if ist != 200: add('BLOCK', 'images', f'missing image (HTTP {ist}): {s}', norm(p)); continue if len(ib) > HEAVY_BYTES: add('WARN', 'images', f'{len(ib)//1024} KB: {s}', norm(p)) d = dims(ib) if d and im['w'] and str(im['w']).isdigit(): want = int(im['w']) if d[0] < want: add('BLOCK', 'images', f'upscaled — {d[0]}px source in a {want}px slot: {s}', norm(p)) elif not im['srcset'] and d[0] > want * OVERSIZE_RATIO: add('WARN', 'images', f'{d[0]}px source for a {want}px slot, no srcset: {s}', norm(p)) elif d and not im['w']: add('INFO', 'images', f'no width/height attribute ({d[0]}×{d[1]}) — layout will shift: {s}', norm(p)) # --- cross-page duplicates for t, ps in titles.items(): if len(ps) > 1: add('BLOCK', 'meta', f'duplicate title across {len(ps)} pages: "{t[:60]}"', ', '.join(ps)) for d, ps in descs.items(): if len(ps) > 1: add('WARN', 'meta', f'duplicate description across {len(ps)} pages', ', '.join(ps)) # --- broken internal links known = set(pages) for p, pg in pages.items(): for h in pg.links: if h.startswith(('http', 'mailto:', 'tel:', '#', 'javascript:')): continue q = norm(urllib.parse.urljoin(p if p.endswith('/') else p + '/', h)) # /cdn-cgi/ is injected by Cloudflare (email obfuscation) and is not # part of the site. It 404s to a crawler and is not a defect. if q in known or q.startswith('/cdn-cgi/'): continue # a download that 404s is still a defect, but it is not a missing page if not is_page(q): if src.get(q)[0] != 200: add('BLOCK', 'pages', f'link to a missing file: {h}', p) continue st, _ = src.get(q) if st != 200: add('BLOCK', 'pages', f'link to a page that does not exist: {h}', p) # --- site-level files st, _ = src.get('/robots.txt') if st != 200: add('BLOCK', 'infra', f'no robots.txt (HTTP {st})') sm_ok = [sm for sm in (src.sitemaps() if src.url else ['/sitemap.xml']) if src.get(sm)[0] == 200] if not sm_ok: add('WARN', 'infra', 'no sitemap found (tried robots.txt and the usual paths)') else: missing = sorted(set(pages) - set(norm(x) for x in src.pages())) if missing: add('WARN', 'infra', f'{len(missing)} page(s) reachable but not in {sm_ok[0]}', ', '.join(missing[:6])) st, _ = src.get('/llms.txt') if st != 200: add('ASK', 'infra', 'no llms.txt — ask whether this site wants one') if src.url: st, _ = src.get('/definitely-not-a-real-page-9f3a') if st != 404: add('BLOCK', 'infra', f'unknown URLs return HTTP {st}, not 404 — soft 404') st, _ = src.get('/404.html') if st != 200 and not src.url: add('BLOCK', 'infra', 'no 404.html in the build') noindexed = [p for p, pg in pages.items() if 'noindex' in pg.robots] if not noindexed: add('ASK', 'infra', 'nothing is noindexed — confirm this site is meant to be ' 'indexable already, or add noindex until launch') elif len(noindexed) < len(pages): add('BLOCK', 'infra', f'noindex on only {len(noindexed)} of {len(pages)} pages — inconsistent', ', '.join(sorted(set(pages) - set(noindexed))[:5])) else: add('ASK', 'infra', f'all {len(pages)} pages are noindex — there must be a ticket to lift this at launch') # One finding per distinct problem, not per page. A footer image with no alt # is one thing to fix, not fourteen, and a report that lists it fourteen # times buries the four things that are actually distinct. merged, order = {}, [] for f in findings: k = (f['sev'], f['area'], f['msg']) if k not in merged: merged[k] = {'sev': f['sev'], 'area': f['area'], 'msg': f['msg'], 'where': []} order.append(k) if f['where']: merged[k]['where'].append(f['where']) out = [] for k in order: m = merged[k] w = m['where'] m['pages'] = len(w) m['where'] = ('site-wide (%d pages)' % len(w)) if len(w) > 3 else ', '.join(w) out.append(m) rank = {'BLOCK': 0, 'ASK': 1, 'WARN': 2, 'INFO': 3} out.sort(key=lambda m: (rank.get(m['sev'], 9), m['area'])) print(json.dumps({'target': sys.argv[1], 'pages': len(pages), 'paths': sorted(pages), 'summary': {s_: sum(1 for m in out if m['sev'] == s_) for s_ in ('BLOCK', 'ASK', 'WARN', 'INFO')}, 'findings': out}, indent=1)) if __name__ == '__main__': main() ```