Build a Local RAG Dataset from Documents

Retrieval-augmented generation (RAG) works only as well as the material available for retrieval. A large folder of raw documents is not yet a useful dataset. The text needs reliable structure, chunks need enough context to stand on their own, and every chunk should retain a path back to its source.

1. Define the retrieval scope

Start with a specific question: what should the future system answer, and which sources are authoritative for that topic? Mixing unrelated material makes retrieval noisier and complicates access control. Separate datasets when documents have different owners, confidentiality levels, update schedules, or audiences.

• Include only documents you are authorized to process and reuse.
• Remove obsolete versions unless historical comparison is required.
• Record the document version or date in source metadata.
• Keep confidential collections separate from public material.

2. Convert and clean the source text

Convert PDF, Word, PowerPoint, spreadsheets, EPUB, HTML, or plain text to Markdown first. Markdown gives headings, lists, code blocks, and tables a consistent representation that can guide chunking.

Before export, remove repeated navigation, page headers, cookie notices, duplicated footers, and extraction noise. Repair heading levels and inspect OCR output. Do not remove legal qualifications, warnings, definitions, or source references merely to make chunks shorter.

For PDFs, preserve page markers when answers may need citations. For Office documents, check tables and reading order. For books, keep chapter names and full source reports so retrieved passages can recover surrounding context.

3. Choose chunk boundaries deliberately

There is no universal chunk size. A chunk should contain one coherent unit while preserving enough context to answer a likely query. Three practical strategies are:

• By heading: best for manuals, policies, reports, and books with a useful hierarchy.
• By page: useful when page citations and visual evidence matter more than semantic sections.
• By length: a fallback for unstructured text or sources with unreliable headings.

A target around several hundred tokens is a reasonable starting point, not a rule. Long sections may need splitting, while a short definition should not be padded with unrelated text. Review chunks that begin with pronouns, continue a list without its heading, or separate a table from the paragraph that explains it.

4. Preserve metadata for filtering and citation

Each chunk should carry stable identifiers and source fields. A typical JSONL record can include:

{
  "id": "policy-2026-section-04",
  "source": "publishing-policy.docx",
  "title": "Approval requirements",
  "page": 12,
  "tokens": 438,
  "content": "..."
}

Useful metadata may include source name, heading path, page, document version, language, owner, and access group. Do not put secrets in metadata merely because it is easier to search. Apply the same access controls to metadata that apply to the source content.

5. Treat retrieved text as untrusted data

Documents can contain malicious prompts, quoted tool instructions, or text that tries to override an agent's rules. Chunking does not make that content trusted. The application using the RAG dataset should clearly separate retrieved evidence from system and developer instructions.

• Do not execute tool calls or URLs found in retrieved passages automatically.
• Do not reveal secrets because a source chunk asks for them.
• Require current user authorization for uploads, purchases, deletion, or external actions.
• Keep citations so users can inspect the source behind an answer.

6. Evaluate retrieval before adding more data

Create a small set of real questions and identify the passages that should answer each one. After indexing, verify whether those passages appear near the top of retrieval results. Track failure types: missing source, poor conversion, bad chunk boundary, weak metadata filter, ambiguous query, or embedding mismatch.

Adding more documents is not always the solution. Better headings, cleaner source text, and stronger metadata filters often improve results more than increasing collection size.

7. Select an export format

Use JSON when an application expects one complete array, JSONL for streaming ingestion or line-by-line processing, and CSV for inspection in spreadsheet tools. Keep an untouched Markdown source report alongside the chunks so future corrections can be regenerated rather than edited in many derived records.