Page discovery

Retrieval works on your own pages, so the first job is knowing which pages you have. Discovery crawls the project's domain and builds the list that scoring then works through.

What the crawl records

Per page, beyond the URL and the title tag:

• Word count. Thin pages are visible as thin rather than as a bad score.
• Inlinks. How many of your own pages link to it. A page with none is invisible to a crawler following links, including a model's.
• Indexable. Whether the page allows indexing at all.
• Status code. A redirect or an error, which explains a surprising absence.
• Page type. Home, location, service, landing, blog, product, tool, listicle or other, inferred from the URL.
• Folder. The section it belongs to, which is what the rollups group by.

Sections

Pages are grouped into sections by the first part of their path, so /blog/... and /features/... become separate rows in the rollup. This is the unit most of the retrieval screen works in, because a site's problems are rarely spread evenly: one section is usually written for people who already know the product, and that section scores badly across the board.

A section can be tracked or not tracked. Untracked sections are not scored and do not count in the rollup, which is how you keep a thousand-page archive out of a report about twenty commercial pages.

When a page is missing

Four causes, in the order worth checking.

1. Nothing links to it and the sitemap omits it. Fix the links. This is also why assistants are not finding it.
2. It is blocked. Robots rules or an authentication wall. Expected on a private page, a problem on a public one.
3. The content is JavaScript only. The HTML comes back as a shell and the words arrive afterwards.
4. It returns an error or redirects. The status code says which.

The JavaScript case

A page whose text exists only after client-side rendering can be discovered and still come back with no content. That shows as an unreadable page rather than a score of zero, because zero would imply it was read and found wanting.

The fix is rendering the content that matters on the server. It is also the fix for several assistants, which read raw HTML and nothing more.

Re-crawling

Discovery is repeatable. Analyze a section after a site change, a new set of pages or a sitemap fix, and those pages join the scoring queue. Tracked pages are also re-scored weekly, so the picture stays current between crawls.