RAG Pipeline Preprocessing — Capability Boundary Description
This document defines the capability scope, known limitations, and risk thresholds of the RAG Pipeline preprocessing system for reference.
1. File Type Support Matrix
1.1 Directly Supported File Types
| File Type | Extension | Parsing Step | Notes |
|---|---|---|---|
.pdf | PDF_CONTENT_EXTRACTION / TEXTIN / AZURE_DI / ALI_OCR / LLM_CONTENT_EXTRACTION | Basic parsing only applies to text-based PDFs; scanned documents require an OCR step | |
| Word Document | .docx | DOCX_CONTENT_EXTRACTION | Converted to Markdown via pypandoc, supports paragraphs/tables/embedded images |
| Legacy Word | .doc | Must first be converted via FILE_CONVERT_WITH_SPIRE or FILE_CONVERT_WITH_LIBREOFFICE | Cannot be parsed directly; must first be converted to .docx or .pdf |
| Markdown | .md | MD_CONTENT_EXTRACTION | Natively supported |
| Plain Text | .txt | TXT_CONTENT_EXTRACTION | Natively supported |
| Excel | .xlsx .xls | TABULAR_CONTENT_EXTRACTION | Subject to row/column limits, see §2 |
| CSV / TSV | .csv .tsv | TABULAR_CONTENT_EXTRACTION | — |
| PowerPoint | .pptx .ppt | Must first be converted via FILE_CONVERT_WITH_SPIRE or FILE_CONVERT_WITH_LIBREOFFICE | Direct content extraction is not supported; must first be converted to PDF |
| Image | .png .jpg .jpeg .gif .webp .svg | IMAGE_CONTENT_EXTRACTION | Uses a Vision LLM (such as GPT-4o) to describe image content, up to 300 words |
| Video | .mp4 .avi .mkv .mov | VIDEO_CONTENT_EXTRACTION | Extract audio track → transcribe to text |
| Audio | .mp3 .wav .flac .aac | AUDIO_CONTENT_EXTRACTION | Split into 60-second segments and transcribe segment by segment |
1.2 Unsupported / Explicitly Prohibited File Types
| Type | Description |
|---|---|
.exe | Hardcoded blacklist (_UNSUPPORTED_EXTENSIONS) |
.zip / .rar and other archives | No decompression step; the Pipeline does not process them |
.html / .xml | No dedicated parsing step |
.json / .yaml | No dedicated parsing step |
.eml / .msg emails | Not supported |
.dwg / .dxf CAD files | Not supported |
.rtf | No dedicated step (conversion via LibreOffice may be attempted, but not guaranteed) |
| Encrypted / password-protected files | Encrypted files of all formats cannot be processed |
2. File Size and Resource Limits
2.1 Precheck Limits (file_limit_checker)
A lightweight precheck is performed during upload. Files exceeding the limits will be rejected from upload or flagged with a warning:
| Dimension | Default Threshold | Configurable | Description |
|---|---|---|---|
| Character count | 15,000 characters | ✅ Via the FILE_LIMIT_CHECK_CONFIG environment variable | Applies to text-based files such as PDF/DOCX/TXT/MD/CSV/XLSX |
| Page count | 25 pages | ✅ Same as above | PDF by physical pages, Excel by sheet count, PPT by slide count |
⚠️ Bulk import from knowledge base file sources is not subject to this precheck limit, but oversized files still carry an OOM risk.
2.2 Resource Thresholds Within Pipeline Processing
| Resource Dimension | Threshold / Configuration | Risk |
|---|---|---|
| Parallel PDF page conversion | 50 pages per batch (PDFConversionDefaultOptions.CHUNK_SIZE) | Very large PDFs (500+ pages) may consume high memory during pdf2image conversion, potentially causing OOM |
| Table row count | Single sheet ≤ 20,000 rows | Exceeding the limit causes direct failure and parsing rejection |
| Table column count | Single sheet ≤ 200 columns | Exceeding the limit causes direct failure and parsing rejection |
| Table images | Single file ≤ 150 images (_MAX_IMAGES) | Excess images are ignored |
| Vectorization batch | 100 segments per batch (batch_size) | — |
| Segment image density | Single segment ≤ 5~8 images | Exceeding the limit triggers early splitting to avoid token overflow |
| Audio slicing | 60 seconds / segment | Long audio/video files generate a large number of transcription API calls |
| TextIn OCR | API-level maximum of 1,000 pages | Quota exhaustion will cause errors |
2.3 OOM / Resource Overload Risk Scenarios
| Scenario | Risk Level | Cause | Recommendation |
|---|---|---|---|
PDF > 200 pages + pdf2image | 🔴 High | pdf2image renders each page as an in-memory bitmap at 300 DPI; 200 pages ≈ several GB of memory | Use OCR steps (TextIn/Azure DI) instead of basic parsing |
| Excel single sheet > 10,000 rows | 🟡 Medium | Fully loaded into memory, generating a large number of segments | Pre-split the file or increase worker memory |
| Video > 2 hours | 🟡 Medium | ffmpeg audio extraction + 120 transcription requests of 60s each | Limit video duration or pre-split it |
| Single file containing 100+ embedded images | 🟡 Medium | Each image requires a Vision LLM call, resulting in high time and cost | Extract only key images |
| Bulk import of 1,000+ files | 🟡 Medium | Celery worker queue backlog, gevent concurrency limit of 100 | Import in batches and monitor queue depth |
3. File Parsing Capability Levels
3.1 Parsing Engine Comparison
| Engine | Parsing Method | Applicable Scenarios | Inapplicable Scenarios | Dependencies |
|---|---|---|---|---|
| Basic（pypdf / pdfplumber） | Direct extraction from text layer | Native text PDFs (exported from Word, generated by LaTeX, etc.) | Scanned documents, image PDFs, complex layouts | No external dependencies |
| TextIn OCR | Cloud OCR + layout analysis | Scanned documents, receipts, mixed-layout PDFs | — | TEXTIN_APP_ID / TEXTIN_APP_SECRET |
| Azure Document Intelligence | Cloud Layout/Read models | Table structure preservation, multi-column layouts | — | azure_ocr_endpoint / azure_ocr_key |
| Ali OCR | Alibaba Cloud OCR | Chinese scanned-document scenarios | — | Alibaba Cloud API credentials |
| LLM Parsing（PDF LLM） | Page screenshots → Vision LLM recognition | Extremely complex layouts, mixed text-image layouts | Large files (high cost, slow speed) | Vision LLM (GPT-4o, etc.) |
| pypandoc | Pandoc format conversion | DOCX → Markdown | Complex macros, ActiveX controls | Pandoc binary |
| LibreOffice / Spire | Format conversion engine | .doc→.docx, .ppt→.pdf | — | LibreOffice or Spire runtime |
3.2 Detailed Parsing Capabilities by File Type
PDF
| Feature | Basic | TextIn | Azure DI | LLM |
|---|---|---|---|---|
| Plain text extraction | ✅ | ✅ | ✅ | ✅ |
| Scanned / image PDF | ❌ Returns empty | ✅ | ✅ | ✅ |
| Table structure preservation | ❌ Formatting lost | ✅ | ✅（Layout mode） | ✅ |
| Multi-column layout | ❌ Text disorder | ✅ | ✅ | ✅ |
| Hyperlink extraction | ✅（pdfplumber） | ❌ | ❌ | ❌ |
| Embedded image extraction | ✅（converted to PNG） | ✅ | ✅ | ✅ |
| Processing speed | ⚡ Fast | 🐢 Medium | 🐢 Medium | 🐌 Slow |
| Cost | Free | 💰 Charged per page | 💰 Charged per page | 💰💰 Charged per token |
DOCX
| Feature | Support Status |
|---|---|
| Paragraph text | ✅ |
| Tables | ✅（converted to Markdown tables） |
| Embedded images | ✅（extracted as separate files + Markdown markers） |
| TOC / bookmarks | ⚠️ Partial (may be lost during Pandoc conversion) |
| Macros / VBA | ❌ Ignored |
| Revision marks / comments | ❌ Lost |
| Complex nested tables | ⚠️ Pandoc behavior is unstable |
Tabular Files (Excel / CSV)
| Feature | Support Status |
|---|---|
| Multiple sheets | ✅ Processed sheet by sheet |
| Formulas | ⚠️ Only calculated values are read (data_only=True) |
| Charts / pivot tables | ❌ Ignored |
| Embedded images | ✅（up to 150 images / file） |
| Merged cells | ⚠️ May cause data misalignment |
| More than 20,000 rows | ❌ Parsing rejected |
| More than 200 columns | ❌ Parsing rejected |
Audio and Video
| Feature | Support Status |
|---|---|
| Speech-to-text | ✅（60s segmented transcription） |
| Multilingual recognition | Depends on the transcription service model |
| Speaker diarization | ❌ |
| Background music / noise | ⚠️ Affects transcription quality |
| Video visual content recognition | ❌ Audio track only is extracted |
| Subtitle / CC extraction | ❌ |
Images
| Feature | Support Status |
|---|---|
| Image content description | ✅ Vision LLM description（≤ 300 words） |
| OCR text recognition | ⚠️ Depends on Vision LLM capability; not professional OCR |
| Chart / flowchart understanding | ⚠️ Limited (determined by LLM comprehension) |
| SVG vector graphics | ⚠️ Depends on whether the LLM can render them |
4. Segmentation Capabilities and Limitations
| Segmentation Method | Applicable Scenarios | Parameters | Limitations |
|---|---|---|---|
Fixed character count（FIXED_SIZE） | General text | chunk_size=1024 | May split in the middle of a sentence; single segment ≤ 5~8 images |
By page（PAGE） | PDF documents | — | Only valid for PDF; segment quality may vary greatly when page content differs significantly |
By title（TITLE） | Markdown documents | chunk_size=1024 | Only recognizes # / ## / ### headings; without headings, degrades to one segment for the entire document |
Tabular（TABULAR） | Excel / CSV | — | Dedicated to tabular files |
LLM intelligent（LLM） | Scenarios requiring semantic coherence | — | Slow and costly; uneconomical for large files |
Refine（REFINE） | Secondary optimization of coarse segmentation results | — | Must be used together with upstream coarse segmentation steps |
General limitations:
- Protects tables / code blocks from being split during segmentation (
RecursiveCharacterTextSplitterfallback separators) - Overlong segments (> 65% of chunk_size) use spaCy for sentence-level refinement
- Too many images in a single segment (> 5 images) trigger forced splitting
5. Field Extraction Capabilities
| Step | Function | Dependency | Limitation |
|---|---|---|---|
METADATA_EXTRACTION | Document-level metadata (title/author/date, etc.) | LLM | Schema must be predefined |
SEGMENT_METADATA_EXTRACTION | Segment-level metadata | LLM | One LLM call per segment; costly for large files |
KEYWORDS_EXTRACTION | Keyword extraction | LLM | — |
SEGMENT_SUMMARY | Segment summary | LLM | One LLM call per segment |
DOCUMENT_SUMMARY | Full-document summary | LLM | Long documents require truncation or batch aggregation |
TABLE_CAPTIONING | Table description generation | LLM | — |
TABLE_CAPTIONING_ADVANCED | Table-level summary + row-level narrative grouping | LLM | Effectiveness on complex tables depends on LLM comprehension |
IMAGE_CAPTIONING | Image description generation | Vision LLM | One LLM call per image |
6. Postprocessing Capabilities
| Step | Function | Configuration | Limitation |
|---|---|---|---|
EMBEDDING | Text vectorization | batch_size=100 | Depends on the Embedding model; image placeholders are automatically stripped |
TOKENIZER | Full-text indexing tokenization | spaCy language model | PostgreSQL tsvector length limit |
EMBEDDING_STORE | Write vectors into VectorDB | — | Batch writes; depends on VectorDB performance |
TOKENIZER_STORE | Write tokens into PostgreSQL | — | — |
7. Format Conversion Capabilities
| Input Format | Output Format | Conversion Engine | Limitation |
|---|---|---|---|
.doc | .docx / .pdf | LibreOffice / Spire | Requires the corresponding runtime to be installed on the server |
.ppt | .pdf | LibreOffice / Spire | Animations/transitions are lost |
.pptx | .pdf | LibreOffice / Spire | Same as above |
8. External Service Dependencies
| Service | Environment Variable | Purpose | Impact if Not Configured |
|---|---|---|---|
| TextIn OCR | TEXTIN_APP_ID / TEXTIN_APP_SECRET / TEXTIN_APP_ENDPOINT | PDF/image OCR | The corresponding OCR step is unavailable |
| Azure Document Intelligence | azure_ocr_endpoint / azure_ocr_key | PDF layout analysis | The corresponding step is unavailable |
| Ali OCR | Alibaba Cloud credentials | Chinese OCR | The corresponding step is unavailable |
| Embedding model | LLM Gateway configuration | Text vectorization | The EMBEDDING step is unavailable |
| Vision LLM | LLM Gateway configuration | Image description / LLM PDF parsing | IMAGE/LLM steps are unavailable |
| General LLM | LLM Gateway configuration | Summary/keyword/metadata extraction | Field extraction steps are unavailable |
| LibreOffice | Server-side binary | Format conversion | .doc/.ppt cannot be processed |
| Pandoc | Server-side binary | DOCX → Markdown | DOCX parsing is unavailable |
| ffmpeg | Server-side binary | Video audio-track extraction | Video processing is unavailable |
| spaCy | Python package + language model | Tokenization/sentence splitting | TOKENIZER step and refinement features are unavailable |
9. Summary of Known Boundary Cases
| # | Scenario | Behavior | Recommended Handling |
|---|---|---|---|
| 1 | Scanned PDF + Basic parsing | Extracted content is empty | Switch to TextIn / Azure DI / LLM steps |
| 2 | Multi-column PDF + Basic parsing | Text order is disordered, paragraphs overlap | Use Azure DI Layout mode |
| 3 | Complex tables in PDF + Basic parsing | Table structure is lost, data becomes prose | Use TextIn / Azure DI |
| 4 | Encrypted / password-protected PDF | pypdf throws an exception | Remove password protection during preprocessing |
| 5 | Direct upload of .doc (legacy Word) | No native parser | Ensure the Pipeline includes a file conversion step |
| 6 | Excel exceeds 20,000 rows | Parsing rejected, error thrown | Split into multiple files |
| 7 | Excel merged cells | Data may be misaligned | Unmerge before upload |
| 8 | Excel formulas | Only result values are read, not the formulas themselves | Expected behavior |
| 9 | Video extracts audio track only | Visual content cannot be recognized | If visual information is needed, extract key frames and process separately |
| 10 | Image OCR accuracy | Vision LLM is not professional OCR, accuracy is limited | Use TextIn for OCR-demanding scenarios |
| 11 | Large files + LLM parsing | Extremely high cost (charged by page/token) | Use LLM parsing only for key documents |
| 12 | Markdown without heading structure + TITLE segmentation | The entire document becomes one segment | Switch to FIXED_SIZE segmentation |
| 13 | PPT animations/video embedding | Lost after conversion to PDF | No solution |
| 14 | DOCX containing macros/VBA/ActiveX | All macro content is ignored | Expected behavior |
| 15 | DOCX revision marks/comments | Lost | Accept all revisions before upload |
10. Capacity Planning Recommendations
| File Type | Recommended Upper Limit (Single File) | Reason |
|---|---|---|
| PDF (Basic parsing) | ≤ 50 pages | pdf2image memory consumption; OOM risk beyond 50 pages |
| PDF (OCR parsing) | ≤ 100 pages | Limited by OCR service API constraints and cost |
| DOCX | ≤ 20,000 characters | Pandoc conversion memory usage |
| Excel | ≤ 20,000 rows × 200 columns / sheet | Hard limit |
| CSV | ≤ 20 MB | Fully loaded into memory |
| Audio | ≤ 60 minutes | 60 transcription API calls |
| Video | ≤ 30 minutes | ffmpeg extraction + transcription; the longer the duration, the higher the failure probability |
| Image | ≤ 10 MB / image | Vision LLM input limit |
This document is generated based on code analysis. Actual runtime behavior may vary depending on the deployment environment (memory / CPU / network) and the status of external services. It is recommended to conduct targeted stress testing before use in production environments.