Skip to main content

RAG Pipeline

RAG Pipeline is the workflow orchestration center for knowledge base capabilities, used to connect the three types of capabilities—"file processing, knowledge retrieval, and plugin extension"—into a configurable, reusable, and optimizable RAG execution chain.

Its core value is to enable enterprises to move beyond relying on a single fixed process, and instead flexibly design and continuously optimize retrieval-augmented generation (RAG) workflows based on different document types, different Agents, and different business objectives.

💡 Tip: This feature is only supported in V4.2 and later versions.

Core Features​

  • Orchestratable workflows: Combine capabilities such as preprocessing, retrieval, reranking, and variable processing as nodes to build business workflows as needed.
  • Decoupled capabilities: The three parts—preprocessing, retrieval, and plugins—can be configured and iterated independently, reducing the overall adjustment cost.
  • Reusable strategies: Supports the coexistence of built-in default workflows and custom workflows, making it easy to copy templates and deploy them quickly.
  • Observable execution: Supports trial runs, log viewing, and node-level input/output tracing, making optimization and troubleshooting easier.
  • Sustainable extensibility: Integrates private algorithms or third-party services through a plugin mechanism to meet enterprise-level customization needs.

Capability Composition of RAG Pipeline​

RAG Pipeline consists of the following three types of capabilities:

⚠️ Permission Note: Creating, modifying, and publishing RAG Pipeline all require administrator permissions. Regular users can only use Pipelines that have been authorized and published by administrators.

Retrieval Pipeline​

Used to define knowledge recall strategies and execution logic, and is a key part of RAG question-answering performance.

  • Supports both basic orchestration and advanced orchestration modes, covering scenarios from standard retrieval to complex multi-path recall.
  • Can be combined with nodes such as filtering, reranking, tokenization, templates, and plugins to finely control recall quality.
  • Can be configured by Agent dimension and inherit context parameters to implement differentiated retrieval strategies for different agents.

Preprocessing Pipeline​

Used to define the parsing, splitting, enhancement, and vectorization workflow before files are ingested into the knowledge base, serving as the foundational guarantee for knowledge retrievability.

  • Supports matching different preprocessing rules by file type, with fallback to the default workflow when no match is found.
  • Covers capabilities such as text extraction, text chunking, field extraction, post-processing, data processing, and plugin extension.
  • Supports trial runs and result preview to help verify processing effectiveness before formal ingestion.

Plugin Pipeline​

Used to extend the platform's default capabilities by integrating customized processing logic through custom Python code or uploaded plugin packages.

  • Supports both manual creation and local package upload, adapting to different development habits.
  • Can be used in preprocessing or retrieval stages to meet integration needs for private algorithms, external APIs, and third-party services.
  • Provides independent runtime testing capabilities to facilitate plugin debugging and version iteration.

Application Scenarios​

  • Multi-source document governance: Design differentiated preprocessing strategies for different content types such as PDF, Office documents, spreadsheets, and audio/video.
  • Precise multi-Agent Q&A: Configure independent retrieval workflows for different business agents to avoid "one strategy for all scenarios."
  • Continuous optimization operations: Identify bottlenecks through trial runs and log tracing, and continuously optimize recall quality and response effectiveness.
  • Enterprise private extension: Integrate internal enterprise systems, private model capabilities, or industry-specific algorithms through plugins.

Usage Recommendations​

  • For first-time use, it is recommended to prioritize the built-in default Pipeline to first establish a runnable standard workflow.
  • It is recommended to optimize gradually in the order of "preprocessing quality → retrieval strategy → plugin extension" to reduce debugging complexity.
  • After each adjustment, verify the effect through trial runs and logs before gradually publishing to the production environment.