Papers
arxiv:2609.24220

Document Retrieval-Aware Chunking (D-RAC): Universal Retrieval-Aware Ingestion of Enterprise Documents via PDF Normalization and Multimodal Markdown Conversion

Published on Sep 21
· Submitted by
Uday Allu
on Sep 22
Authors:
,
,

Abstract

Retrieval-Augmented Generation (RAG) systems over enterprise knowledge bases must ingest heterogeneous document formats -- PDFs, Word documents, presentations, and scans -- whose content is locked inside complex visual layouts, multi-column pages, and dense tables. Rule-based extraction and OCR destroy reading order, flatten tables, and lose heading hierarchy, while fully agentic chunking over extracted text incurs high token costs and hallucination risk. We present Document Retrieval-Aware Chunking (D-RAC), an extension of our Web Retrieval-Aware Chunking (W-RAC) framework to arbitrary document formats. D-RAC first normalizes any input document into PDF, exploiting the fact that virtually every format has a faithful, deterministic PDF rendering. A single multimodal LLM pass then converts rendered pages into retrieval-optimized Markdown -- rewriting tables as self-contained prose statements and preserving heading hierarchy -- after which chunking proceeds exactly as in W-RAC: deterministic parsing into ID-addressable units followed by lightweight LLM-based chunk planning over identifiers rather than text. Source text is never regenerated during chunking, preserving W-RAC's cost, determinism, and observability benefits while unlocking every renderable format as a first-class input. On the 236-document, 795-page PDF subset of the RAG-Multi-Corpus benchmark spanning five enterprise domains, D-RAC converts and chunks the entire corpus in 72 minutes with zero errors, producing 1,748 retrieval-ready chunks. Compared to agentic chunking with frontier LLMs, D-RAC reduces chunking-stage output tokens by 95.7%, cutting chunking cost by 77.8% (GPT-4.1 pricing) to 85.6% (Gemini 2.5 Pro pricing) and chunking time by 75%. D-RAC scales linearly to documents of 500+ pages.

Community

Paper submitter

D-RAC: Universal Retrieval-Aware Ingestion of Enterprise Documents

Enterprise RAG has to ingest PDFs, DOCX, PPTX, XLSX and scans — formats where text extraction breaks reading order, flattens tables and loses heading hierarchy. Agentic chunking repairs some of that, but it regenerates the whole document as output tokens, which is where the cost and the hallucination risk live.

D-RAC extends our earlier W-RAC framework to arbitrary formats with two additions. First, every input is deterministically normalized to PDF and rendered to page images, since virtually any document format has a faithful PDF rendering. Second, a single multimodal LLM pass converts those pages into retrieval-optimized Markdown: every table row becomes one self-contained sentence carrying its column headers, disjunctive merges ("16 or 20 years") are forbidden, decorative imagery is dropped, and heading hierarchy is made explicit. Chunking then proceeds exactly as in W-RAC — deterministic parsing into ID-addressable elements, then LLM planning over identifiers, never text. The multimodal model touches content once; source text is never regenerated.

Every agentic chunking pipeline I've run hits the same wall: the PDF parser decides what the retriever ever sees. So the real test here isn't whether retrieval-aware chunking beats fixed-size — it's whether the normalization step survives tables that span pages and headings buried in footers. If the markdown conversion mangles those, you've just moved the failure from chunking to parsing, and the retrieval-aware part is optimizing garbage. I'd want retrieval quality measured against a bad-parser baseline, not just against naive chunking. That's the number that tells me if this is worth wiring into prod.

This is an automated message from the ResearchStudio team.

We created an interactive ResearchStudio Reel for this paper. It includes a visual poster, a video, and a blog, all available for download in editable formats.

Visual poster for this paper

Open the ResearchStudio Reel →

Download all files from Hugging Face

Please give this comment a thumbs up if you find the Reel helpful!

Want to explore or create Reels for more papers? Visit the ResearchStudio demo.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2609.24220
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2609.24220 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2609.24220 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2609.24220 in a Space README.md to link it from this page.

Collections including this paper 2