MarkPrep
All posts
Published · Updated · 9 min

Markdown Is Step One. Chunking Is Where RAG Breaks.

In a RAG (retrieval-augmented generation) pipeline, converting PDFs to Markdown is the easy first step, and chunking is where retrieval usually goes wrong. Tutorials often weigh up vector databases and compare embedding models, then split the documents with a one-liner that chops text every N characters. A bad split gives no error. The damage shows only when a user gets a confident, wrong answer built from the wrong three paragraphs.

This post explains why fixed-size splitting hurts, what a sensible default looks like and why dirty text spoils your embeddings. It also shows why each chunk's metadata matters as much as its text.

Why you still chunk when context windows are huge

Current chat models take about a million tokens. You still need to split documents, for two reasons.

First, embedding models have hard token limits, and they are far smaller. The model that turns your text into vectors typically takes 2,000 to 32,000 tokens; OpenAI's embedding models take 8,192. You can't embed a 90-page document as one vector even if you can prompt with it. Retrieval needs pieces.

Second, a model reads a long input unevenly, even when everything fits. It uses information buried in the middle less well than information at the start or end, an effect known as "lost in the middle". Stuffing the whole corpus into context and hoping the model finds the relevant clause works badly. Retrieving a few focused chunks and handing over only those works better.

Fixed-size splitting is the default, and it's the problem

The most common chunker splits on a raw count: every 1,000 characters, or every 500 tokens, wherever that lands. It is trivial to build, which is why it is everywhere. It also treats your document as a featureless stream of characters, and that is the flaw.

Documents have structure (sections, subsections, tables, definitions), and that structure carries meaning. A fixed-size cut ignores all of it. It will cut a sentence mid-clause, or split a table so the header lands in one chunk and the rows in the next. It can also join the last paragraph of a section on termination to the first paragraph of a section on governing law. The resulting chunk mixes two topics, and its vector represents neither well. When you later search for termination terms, the relevant text is diluted by its neighbour, and retrieval quality suffers.

Header-aware chunking, and a default that works

The better approach uses the structure the document already has. Markdown makes this practical because headings are explicit: #, ## and ### are clear boundaries a chunker can respect. Header-aware chunking breaks at those boundaries first and falls back to size limits only within a section. Its chunks line up with the document's own sections, so each one is about one thing. A fixed-size chunk can mix two topics; a header-aware chunk rarely does.

MarkPrep's defaults are a reasonable start:

  • Chunk size: 800 tokens by default, adjustable from 100 to 2,000. Small enough to stay under embedding limits and keep each vector focused, and large enough to carry a complete thought.
  • Overlap: 100 tokens by default, adjustable from 0 to 400. Overlap repeats a little text between neighbouring chunks, so an idea that crosses a boundary survives whole in at least one chunk. Too little and you cut context that spans a boundary. Too much and you fill your index with repeats and pay to embed the same sentences again.

Treat these as starting points and test them against your own retrieval results.

Approach Boundary logic Typical failure
Fixed-size Every N tokens, blind Split tables, cut sentences, mixed-topic chunks
Header-aware At headings, size as fallback Uneven chunk sizes (usually fine)

Dirty text spoils the vectors before you chunk

Dirty Markdown wrecks chunking before it starts. Raw PDF extraction drags along repeated headers, footers and page numbers. There is the running header repeated on all 90 pages and the footer with the confidentiality notice. There are page numbers such as "Page 12 of 90", and words hyphenated across a line break and never rejoined.

Feed that into a chunker and the problems add up. The repeated header shows up in every chunk. Every vector then shares the same block of boilerplate, which pulls unrelated chunks closer together and blurs the differences retrieval depends on. Headers, footers and page numbers spend tokens, in your prompt and your embedding budget, on text with no meaning. A word split as inter- / national embeds as two fragments instead of "international". Chunk boundaries aligned to headings don't help if the text inside them is full of noise.

So clean the text before you chunk it. MarkPrep's clean-up is on by default. In documents of three or more pages, it drops any line that sits at the top or bottom of 60% or more of the pages. It also removes page numbers, rejoins words hyphenated across line breaks, collapses runs of blank lines and tidies tables. The chunker then gets the content and little else, so the vectors represent what the document says.

Metadata turns a chunk into something you can retrieve

What you attach to a chunk decides how useful it is once retrieved.

Heading path. Each chunk should carry the trail of headings above it, such as Master Services Agreement > Liability > Limitation of Liability. The path tells you where a retrieved chunk came from, so you can cite it. It lets you filter or boost retrieval by section. It also gives the chunk context its own text may lack. "Such liability shall not exceed the fees paid" is ambiguous alone and clear under the heading path that places it.

Page number. For a PDF, keep the source page. When a model cites a chunk, you can send a person to page 47 of the original to check it. In legal, medical and financial work, answers must be checkable, and traceability decides whether the system is usable. MarkPrep stores each PDF chunk's page number, and the slide or chapter number for PowerPoint and EPUB files. For other formats the field is empty.

With this metadata in place, you can ask for more than similarity. For example: find chunks similar to the query, prefer those under "Indemnification", and show their source pages.

Ship it as JSONL

Retrieval pipelines want structured records, one per line. JSONL (one JSON object per line) is the common format for this. You can stream it, load it row by row and push it into an indexing job. Here is one line of MarkPrep's export for a file called msa.pdf:

{"id": "msa-12", "text": "Such liability shall not exceed...", "metadata": {"source": "msa", "title": "Master Services Agreement", "chunk_index": 12, "heading_path": ["Master Services Agreement", "Liability", "Limitation of Liability"], "page": 47, "tokens": 780}}

Each line has an id, the chunk's text and a metadata object. The metadata has the source file name, the document title, the chunk's position and its heading path. It also has the page (null when the format has no pages) and the estimated token count. Your pipeline can read the file as exported, without a script to rebuild the heading paths or pages.

The order that matters

The vector database and the embedding model, the parts tutorials dwell on, are the most forgiving of small mistakes. The clean-up and the chunking come first and need the most care.

To see how your own document splits, open the chunk preview and drop in a file. Move the size and overlap sliders, then export the JSONL. For the wider pipeline, from conversion to indexing, see MarkPrep for RAG pipelines.

Try it on your own file

Convert a document and watch the token counter. It's free, needs no account and uploads nothing.