Technical SEO: the foundation everything stands on

Technical SEO is the work of making sure search engines can reach, fetch, render and store your pages. It earns no applause when it works, but nothing else works without it. The best article on the internet ranks nowhere if a crawler cannot get to it, and in 2026 the same gate applies to AI answers: Google's guidance on AI features requires content to be indexed, crawlable and snippet eligible before it can appear anywhere. This page covers the layers in the order a crawler meets them.

Crawl access: robots.txt and status codes

The first file a crawler checks is robots.txt, which tells it which paths it may fetch. Two failure modes recur constantly: blocking a directory that contains content you want found, and blocking the CSS or JavaScript files a page needs to render. Review the file whenever the site structure changes, not once at launch.

Status codes are the second gate. A page you want indexed should return 200. Moved content should return a 301 to its new address, not a chain of hops. Removed content should return 404 or 410 rather than a soft page that returns 200 with an error message, because soft errors waste crawl attention and confuse deduplication. Repeated 5xx responses tell Google the server is struggling, and crawling slows down in response.

XML sitemaps

A sitemap is a machine-readable list of the URLs you consider canonical, optionally with a last-modified date. It does not force indexing and it does not override quality judgments; it makes discovery reliable, which matters most on large sites, on new sites with few inbound links, and on sites where pages change often. Keep it current automatically. A sitemap full of dead or redirected URLs trains crawlers to trust it less.

IndexNow, accurately described

IndexNow is a protocol for pinging participating search engines the moment a URL is added, updated or deleted, instead of waiting for a recrawl. Bing and Yandex participate. Google does not use IndexNow, so a ping does nothing for Google Search; for Google, sitemaps and internal links remain the discovery mechanism. IndexNow is cheap to implement and worth having for the engines that listen, as long as you are honest with yourself about which engines those are.

Rendering and JavaScript

Google can execute JavaScript, but rendering happens as a separate, deferred step after crawling. Content that only exists after client-side scripts run is indexed later and more fragilely than content present in the initial HTML, and it breaks entirely if the required scripts are blocked or fail. The safe pattern is simple: put the content that must rank in the server-rendered HTML, and let JavaScript enhance rather than construct the page.

Speed and Core Web Vitals

Core Web Vitals measure loading, interactivity and visual stability as a user experiences them. Google's page experience documentation says these signals are used by its ranking systems, while being explicit that relevance comes first: a slow page with the best answer can still outrank a fast page with a worse one. Treat speed as an experience signal that breaks ties and protects conversions, not as a substitute for substance.

Canonicals and duplicate handling

Most sites serve the same content at several addresses: with and without trailing slashes, through URL parameters, across staging copies. Google picks one canonical version and consolidates signals onto it. Your job is to make that choice easy: one internal linking convention, rel canonical tags that point where you intend, redirects that agree with the canonicals, and a sitemap that lists only canonical URLs. When these disagree, Google decides for you, and not always the way you wanted.

What breaks AI visibility too

Every failure above also removes you from AI answers. Because Google grounds its AI features through the same index and ranking systems as classic search, a page that is blocked, noindexed, unrenderable or canonicalized away cannot be retrieved, and a page that cannot be retrieved cannot be cited. Not crawlable means not citable. There is no separate technical channel for AI: Google Search ignores llms.txt, and no special file substitutes for a fetchable page. Get the plumbing right, then focus on on-page structure and the substance that earns citations.