Your RAG Pipeline Is Only as Good as the Text You Fed It

When retrieval returns irrelevant chunks, the reflex is to change the embedding model or tune the similarity threshold. In my experience the problem is almost always earlier: the text that got embedded was already mangled.

Raw text extraction from a PDF gives you a wall of words. Headings, tables, lists and reading order are gone. Then a fixed-size chunker cuts that wall every 1000 characters — mid-sentence, mid-table, splitting a heading from the paragraph it introduces. No embedding model recovers from that.

Convert to markdown first

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("policy.pdf")
print(result.text_content)

The output keeps # heading levels, list structure and tables as markdown tables. It handles PDF, Office documents, HTML, CSV and images with text. The point is not the format — it is that the structure survives, and structure is what makes the next step possible.

Then chunk on structure

import re

def chunk_by_heading(md_text, max_chars=1500):
    sections, current, heading = [], [], ""
    for line in md_text.splitlines():
        if re.match(r'^#{1,3} ', line):
            if current:
                sections.append((heading, "\n".join(current)))
            heading, current = line.lstrip('# '), []
        else:
            current.append(line)
    if current:
        sections.append((heading, "\n".join(current)))
    return sections

A chunk is now a section with a heading, which means it is about one thing. That is the property retrieval needs and fixed-size chunking destroys.

Keep the heading path as metadata

{
  "text": section_text,
  "heading_path": "Refund Policy > International Orders",
  "source": "policy.pdf",
  "page": 4,
}

Two payoffs. Prepending the heading path to the embedded text gives short sections context they otherwise lack. And when the assistant answers, you can cite "Refund Policy → International Orders, page 4" — which is what turns an answer people distrust into one they can verify.

The step everyone skips

Read the converted output. Actually open it. Scanned PDFs come out empty because there is no text layer, and you need OCR. Two-column layouts sometimes interleave. Tables occasionally collapse.

Fifteen minutes of reading the conversion output tells you more about why your retrieval is bad than a week of tuning parameters downstream. Garbage in is not a cliché here — it is the single most common root cause I find in RAG systems that are not working.

Resources

• Repo: microsoft/markitdown
• Docs: github.com/microsoft/markitdown
• Video walkthroughs: YouTube: markitdown rag document ingestion
• Related: Qdrant: vectors without the platform tax