Methodology
Every number CatalogReady produces is deterministic: same store, same HTML, same score. This page documents the whole method.
How we crawl
CatalogReady identifies itself honestly with this user agent:
Mozilla/5.0 (compatible; CatalogReadyBot/1.2; +https://yourdomain/methodology)
- We fetch raw HTML only and never execute JavaScript. This matches how GPTBot, PerplexityBot, ClaudeBot and similar AI crawlers read pages — if your product data only exists after scripts run, they don't see it, and neither do we.
- We respect a hard budget per scan: at most 25 HTTP requests, 10 product pages, 4 concurrent connections, 10-second timeouts and a 45-second total limit. We read robots.txt and report what it allows, and we stop at 2 MB per page.
- Product pages are discovered via Shopify's products.json, sitemaps, or links from the homepage — in that order.
The checks and their point values
The score is site (35) + average across crawled product pages (65), out of 100. If robots.txt blocks OpenAI, Perplexity and Google-Extended all at once, the total is capped at 40 — nothing else matters until crawlers can read the store. Grades: 85–100 “Agent-ready”, 65–84 “Visible, with gaps”, 40–64 “At risk”, 0–39 “Invisible”.
Site level — 35 points
AI crawler access starts at 25 points; each blocked bot group subtracts its penalty (a firewall that blocks unknown bots subtracts 8). Technical foundation adds up to 10.
| access.openai | OpenAI crawlers allowed (GPTBot, OAI-SearchBot, ChatGPT-User) | −6 if blocked |
| access.perplexity | Perplexity crawlers allowed (PerplexityBot, Perplexity-User) | −6 if blocked |
| access.google_extended | Google-Extended allowed (Gemini / AI Mode) | −6 if blocked |
| access.claude | ClaudeBot allowed (Anthropic) | −4 if blocked |
| access.bing | Bingbot allowed (Copilot) | −3 if blocked |
| access.waf | No firewall blocking unknown bots | −8 if blocked |
| tech.https | HTTPS resolves cleanly | 2 |
| tech.sitemap | Sitemap reachable | 2 |
| tech.rendering | Product content present in raw HTML | 4 |
| tech.canonical | Canonical tags on crawled pages | 2 |
llms.txt presence is noted as informational only — no points either way.
Page level — 65 points per page, averaged
Product schema — 25 points
| schema.product | JSON-LD Product markup | 10 |
| schema.price | Offer with price and currency | 5 |
| schema.availability | Availability in schema | 3 |
| schema.image | Image in schema | 3 |
| schema.description | Description in schema | 4 |
Identifiers — 10 points
| ids.gtin | GTIN or MPN present | 5 |
| ids.sku | SKU present | 2 |
| ids.brand | Brand present | 3 |
Descriptions & attributes — 15 points
| content.length | Description at least 300 characters | 5 |
| content.attributes | 3+ concrete attributes (material, size, ...) | 5 |
| content.title_h1 | Unique title and one H1 naming the product | 3 |
| content.placeholder | No placeholder text | 2 |
Reviews — 10 points
| reviews.aggregate | AggregateRating with a rating value | 6 |
| reviews.count | Review count of 1 or more | 2 |
| reviews.review | At least one Review item in markup | 2 |
Images — 5 points
| images.present | At least one product image | 1 |
| images.alt | Alt text on 80%+ of images | 3 |
| images.alt_quality | Descriptive, non-generic alt text | 1 |
Sources
- OpenAI crawler documentation (GPTBot, OAI-SearchBot, ChatGPT-User) — platform.openai.com/docs/bots
- Perplexity crawler documentation (PerplexityBot, Perplexity-User) — docs.perplexity.ai
- Anthropic crawler documentation (ClaudeBot) — support.anthropic.com
- Google crawler documentation (Google-Extended, Bingbot equivalents) — developers.google.com/search/docs/crawling-indexing
- schema.org Product, Offer, AggregateRating and Review definitions — schema.org/Product
- robots.txt evaluation follows RFC 9309 (longest-match, allow wins ties)
What the score is not
We never query ChatGPT, Perplexity or any assistant, and we don't measure rankings. Scores reflect machine readability measured against published AI crawler and schema.org standards. Assistant behavior varies; no ranking or recommendation is guaranteed.