Serve a Markdown version of each content page to agents that ask for it
An agent that asks for Markdown gets the page without menus, scripts or popups, so it reads what you wrote rather than your layout. The same words as the HTML, never extra ones: a version only machines can see is cloaking.
Step 1.3.3.6
Time: 1h
Done when
Request three content pages with the header Accept: text/markdown. For each, give the Content-Type returned, the Vary header, and the canonical URL in its Link header, and say whether the headings match the HTML page. Anything other than text/markdown with Vary: Accept and a canonical pointing at the HTML page fails. (Proof: a list of the items.)
If this is a change to your product
- What must become true
- A content page answers Accept: text/markdown (and the same path plus .md) with a Markdown copy of the same text, sent as text/markdown, with Vary: Accept and an HTTP Link header naming the HTML page as canonical.
- Where it usually lives
- Middleware or a route handler in front of the content pages, rendering from the same source the HTML uses, or the CDN’s own Markdown conversion if it has one.
- The trap
- Leaving out Vary: Accept. A cache then stores the Markdown under the page URL and serves it to browsers and to Googlebot. And a Markdown file cannot hold a <link rel="canonical"> tag, so the canonical has to go in the HTTP Link header.