MarkPrep
All posts
Published · Updated · 6 min

How Many Tokens Is My Document? (And Why You Should Care)

To find out how many tokens your document is, count them with a tokenizer for the model you plan to use. Word count and page count only hint at the answer, because tokens, words and pages are different units. Paste a long document into an LLM without a count, and you may get an error about a token limit instead of an answer. The number in that error means little if you never measured the document.

What a token is

A token is the unit an LLM reads in. It is smaller than most words and larger than a single character. Common words are often one token. Longer or rarer words are split into several, and punctuation and the space before a word affect the split.

The vendors publish rules of thumb for English text. OpenAI and Google both put a token at about 4 characters. Google adds that 100 tokens is about 60 to 80 English words. For its current Claude models, Anthropic says 1 million tokens hold about 555,000 words or 2.5 million characters. That is about 2.5 characters per token. So the same text is a different number of tokens to different models.

Take a 10,000-word report. Google's figure puts it at about 12,500 to 16,700 tokens. Anthropic's figure for current Claude models puts it at about 18,000. The old shortcut of "word count times 1.3" undercounts Claude today.

Why word count misleads

You see "12 pages" and assume the document is small. Yet the amount of text on a page depends on its layout. Font size, margins, images and white space decide how much text lands on a page, and none of that tells a model anything. Two 10-page PDFs can hold very different amounts of text.

The token count is not even a fixed property of the text. It depends on the model's tokenizer. The same paragraph is a different number of tokens to GPT, Claude and Gemini, because they encode text differently. Code, tables, unusual formatting and non-English text use more tokens than clean English prose, which raises the count without any sign on the page. So "how long is this document?" has no single answer until you tie it to a model.

Why the number matters: the context window

A token count becomes useful when you compare it with your model's context window. That is the maximum number of tokens the model can hold at once, counting your input and its output.

If your document fits, paste away. If it doesn't, the model may reject it, cut it off without saying so, or leave you to split it by hand. Context windows differ from model to model, so compare your model's current limit with this document's token count. A window size you remember may no longer be right.

How do I count the tokens in a document?

You can count the tokens in a document in three ways:

  • Estimate. Divide the character count by the vendor's characters-per-token figure. This is fine for a rough check, but it can be wrong enough to catch you out near a limit.
  • A tokenizer tool. Paste text in and get a count for a given model. You have to extract the text first and repeat it for each model.
  • Count at conversion time. If you are turning a PDF or Word file into Markdown anyway, get the token count in the same step. You then see it per model, without a second tool. MarkPrep's token counter does this. It converts the file in your browser, counts the tokens in the Markdown and shows whether the result fits your model's window.

Every count in MarkPrep is an estimate, labelled "approx". OpenAI counts use the o200k_base encoding. Claude and Gemini models use each vendor's characters-per-token figure, because neither publishes a tokenizer that runs offline in a browser. The models in the list are current ones, such as Claude Sonnet 5.5, GPT-6.1 Sol and Gemini 3.8 Flash.

What to do when it doesn't fit

When the document is too big, you have three options:

  • Chunk it on purpose. Split on heading boundaries instead of cutting every N characters through the middle of a table. The chunk preview shows each chunk before you split, so you can check that each one holds a whole section.
  • Trim the dead weight. Repeated headers, footers and page numbers add tokens and teach the model nothing. Removing them lowers the count and improves the text at once.
  • Pick another model. If a document is close to the limit, a model with a larger window may take it whole.

Next step

Before you give a document to an LLM, drop the file into the token counter and choose your model. If you are converting a PDF anyway, PDF to Markdown shows the estimated count and the context-window check next to the Markdown.

Try it on your own file

Convert a document and watch the token counter. It's free, needs no account and uploads nothing.