Chunking: how retrieval systems split a page

Two statements about chunking are both true, and they are easy to mistake for a contradiction. Google tells publishers there is no requirement to break content into small pieces for AI. Retrieval systems nevertheless split documents into passages and match queries at passage level. Both hold, because the splitting is the engine's job rather than yours. What reaches you is a consequence, not an instruction: a passage that cannot be understood on its own is a passage that retrieves badly and quotes badly.

How retrieval-augmented systems split documents

A retrieval system cannot load a whole website into a model's context window, so it indexes pieces. The pipeline is broadly the same across implementations:

Splitting. The document is cut into chunks. The simplest method is a fixed window of a set number of tokens, usually with an overlap, so that a sentence cut in half at one boundary survives whole inside the neighboring chunk. Structural splitting cuts on headings, paragraphs or list items instead. Semantic splitting looks for topic shifts and cuts there.
Embedding. Each chunk is converted into a vector representing its meaning and stored with a pointer back to the source page.
Retrieval. The query is embedded the same way, and the system finds the chunks whose vectors sit closest to it. The score is computed per chunk, not per page.
Assembly. The surviving chunks, sometimes together with their immediate neighbors, are placed in the model's context and the answer is generated from them.

Which of the retrieved chunks actually reach the answer is decided by a further scoring stage, covered in rerankers and the step that decides which passages get cited. The wider selection process sits in how answer engines choose their sources.

What is published, and by whom

Exact chunk sizes and splitting strategies are implementation details, and no major answer engine publishes its own. Two primary sources publish figures for their own systems. OpenAI's file search documentation states that by default max_chunk_size_tokens is set to 800 and chunk_overlap_tokens is set to 400. Anthropic's write-up on contextual retrieval describes chunks in this pattern as usually no more than a few hundred tokens. Both describe developer retrieval products. Neither describes Google Search, ChatGPT's web retrieval or any consumer answer engine. Read them as evidence that the working unit is roughly a paragraph or two, and as nothing more than that.

The chunk is the unit of citation, not the page

Google's ranking systems guide describes passage ranking as an AI system used to identify individual sections, or passages, of a web page in order to judge how relevant that page is to a search. The generative layer inherits the same granularity. A page can rank well overall and contribute nothing to an answer, because the thing being scored, retrieved and quoted is a passage.

Why passages that depend on earlier context retrieve badly

Splitting has one destructive side effect. Every reference backwards across a chunk boundary breaks. A passage that opens with "it", "this approach" or "as mentioned above" arrives at the retrieval index without its antecedent, because the antecedent is sitting in a different chunk.

Anthropic illustrates the failure with a chunk stating that revenue grew by 3 percent over the previous quarter, which names neither the company nor the quarter. On its own it matches almost no useful query and supports almost no answer. Anthropic reports that prepending a short generated context sentence to each chunk before embedding reduced failed retrievals by 49 percent in its own tests, and by 67 percent when combined with reranking. Those numbers describe Anthropic's experimental setup, not a live search engine. The transferable part is the mechanism: context that lives outside the passage is context the retrieval step cannot see.

Reconciling the two claims

Set the claims next to each other.

Claim one. Google's AI features guide states that there is no requirement to break your content into tiny pieces for AI to better understand it, and that you do not need to write in a specific way just for generative AI search. The full document is walked through in what Google actually says about AI search optimization.

Claim two. Retrieval operates on passages, and a passage that depends on earlier context performs worse.

They are compatible for three reasons. First, the splitting already happens without you. Google is answering publishers who are considering fragmenting a page into machine-targeted blocks, and that work is redundant, because the engine cuts the page anyway. Second, no special format is involved. Naming the subject of a paragraph inside that paragraph is ordinary editing, and it was good practice long before retrieval systems existed. Third, the two claims target different things. Google rules out a tactic, formatting for machines. The passage property rules in a habit, writing prose that stands on its own. Nothing in Google's guidance says context-independence is irrelevant. It says do not restructure for AI, and self-contained writing is not a restructuring.

The practical pattern

Name the subject instead of referring back. Repeat the product, company or method name at the start of each section rather than carrying it forward with a pronoun.
Answer in the first sentence. Use a heading that matches the question a reader would ask, then state the answer immediately below it and elaborate afterwards.
Keep one idea per passage. A paragraph covering three claims matches queries for all three weakly instead of one strongly.
Make tables and lists self-labeling. Column headers and list items should carry their own units, timeframes and subjects, because a table can be lifted away from the sentence that introduced it.

Each of these also makes the page better for a reader who arrives mid-scroll. That overlap is the point. What earns citations once a passage is retrieved is covered in how to get cited by AI engines.

What cannot be verified

Any specific word count sold as the optimal passage length is unverifiable. The engines do not publish their splitters, the values that exist in public documentation belong to developer products rather than to answer engines, and a boundary rule cannot be reverse-engineered from citation output. Treat numeric targets for paragraph length as guesses. The property that matters is whether a passage still makes sense when the paragraphs around it are removed, and you can check that by reading.