Live crawl · —
How machines read
your website.
— AI user-agents were sent at the live origin, robots.txt was parsed against RFC 9309 matching rules, and — pages were fetched and stripped to raw text exactly the way a retrieval pipeline does it — no JavaScript, no rendering, no mercy.
The one-line verdict
—
02AI user-agent access matrix
Each agent below was resolved against robots.txt using longest-match precedence, then actually sent to the origin to catch WAF, Cloudflare or plugin-level blocks that robots.txt never reveals.
| User-agent | Operator | Purpose | Named in robots | robots.txt verdict | Live origin | TTFB |
|---|
03Robots.txt lab
A working robots.txt evaluator running the live file. Pick an agent, type any path, and see which directive wins and why.
04LLM scraper simulator
Replays the exact pipeline an answer engine runs: fetch raw HTML → discard script/style/svg → walk every text node → drop chrome → chunk → embed. What survives on the right is the entire universe the model gets to reason about.
Run a crawl to populate the extracted text stream.
05Structured data coverage
JSON-LD types found per page. Green cells are types answer engines actively use to resolve entities; grey is navigation filler that adds no semantic value.
06Page-level audit
Click any column header to sort. Extractable words counts only body copy — navigation, menus and footers are excluded, mirroring readability extraction.
| Page | Extractable words | Text ratio | Chrome | H1 | Schema | Alt gaps | HTML | TTFB | Signal |
|---|
07Prioritised fix plan
Ordered by impact on machine readability. Each item states the finding, why a retrieval system cares, and the concrete change.
Take it with you
Get this audit as a branded PDF
Every finding above, with the evidence, the reasoning and the exact fix, formatted as a report you can hand to a developer or a client. Nothing above is hidden — the PDF is just the portable version.