Turning a Pile of Research Papers Into a RAG Knowledge Base
To turn a folder of research papers into a RAG (retrieval-augmented generation) knowledge base, convert each PDF to Markdown and clean it. Then split it on the paper's own sections, and embed and retrieve the chunks. The folder might hold two hundred papers. The goal is to ask questions across all of them and find the paper that made a particular argument. In effect, you want an assistant that has read your field. Academic PDFs are also among the hardest documents to convert cleanly, and a bad conversion undermines the whole knowledge base.
Why papers fight back
A typical journal PDF throws several problems at a converter at once:
- Two-column layouts. Read without column detection, a two-column page mixes the columns. Lines from the left and right columns can end up side by side or interleaved, so the text is all there but out of order.
- Dense tables. Results tables are the point of many papers, and they are often complex. Flattened or misaligned, they become useless.
- Equations. Mathematical notation rarely survives as clean text. Expect it to be approximate at best.
- Citations, headers and footers. Running heads, page numbers and reference markers clutter the text if they are not removed.
MarkPrep's PDF converter reads each line straight across the page, through both columns, from the top of the page to the bottom. It has no column detection, so on a two-column paper, check the output before you add it to the knowledge base. Single-column papers and preprints convert more cleanly. MarkPrep does remove repeated headers, footers and page numbers, and it rebuilds tables whose columns line up. It doesn't convert equations to LaTeX.
Clean the text before you embed
Skipping clean-up harms retrieval without warning. When you embed a chunk of text, everything in that chunk becomes part of its meaning. That includes the header repeating "J. Neural Methods, Vol 12" on every page, the page numbers and the citation clutter. MarkPrep's clean-up removes repeated headers, footers and page numbers during conversion, and it is on by default.
Chunk on the paper's sections
Once each paper is clean Markdown, its heading structure (Abstract, Introduction, Methods, Results, Discussion) gives you ready-made chunk boundaries. Chunk on those boundaries and each chunk is a coherent passage, which is what you want a retriever to return. Slice every 500 characters instead and you will cut through the middle of a Methods description and strand half a Results table. A query then gets back a fragment that answers nothing.
The chunk preview marks each chunk's start and end, so you can check that every section stays whole. Each chunk also carries its heading path, so a retrieved passage shows where in the paper it came from.
Keep unpublished papers on your machine
Much of what is in that folder is unpublished: your own drafts, papers under review, confidential preprints, industry data. Uploading those to a cloud converter to build your knowledge base is the wrong move. MarkPrep converts in the browser, so your unpublished work stays on your machine while you build the knowledge base. If your lab or employer forbids uploading pre-publication material, a converter that never uploads keeps you within that rule.
Putting it together
The pipeline for a paper-based knowledge base:
- Convert each PDF to Markdown, and check the reading order on two-column papers.
- Clean the text: remove headers, footers and page numbers.
- Chunk on section headings, with the heading path as metadata.
- Embed and retrieve the resulting passages.
Convert a few papers with PDF to Markdown and read the first page of each result, especially two-column ones. Then follow the full route from conversion to JSONL export on MarkPrep for RAG pipelines.
Try it on your own file
Convert a document and watch the token counter. It's free, needs no account and uploads nothing.