RAG as a Service

Technical reference for the new RAG ingestion pipeline. The flow is bucket-first: extraction outputs are attached to a bucket, then chunking, embedding, and vector-store loading all operate on the same bucket_uid. Evaluation is the exception — it scores RAG outputs from a self-contained dataset or live traces, not from a bucket.

Pipeline modules

• Bucket management: creates reusable containers for extracted documents and defines the shared Storage layout.
• Chunking: splits bucket sources into retrieval-sized chunks and writes chunking.msgpack.
• Embedding: generates dense and/or sparse vectors from chunking output and writes embeddings.msgpack.
• Vector Store: loads embeddings into Milvus/Zilliz, Qdrant, or Pinecone.
• Evaluation: scores RAG outputs with Ragas — one-shot batch over a JSONL dataset, plus continuous tracking of a live RAG. Not bucket-scoped.

Shared results shape

Chunking, Embedding, and Evaluation result endpoints return a wrapper with a result object shaped like this (Evaluation uses eval_uid in place of bucket_uid):

{
  "bucket_uid": "bkt_uuid",
  "success": true,
  "expires_at": "2026-04-29T12:00:00Z",
  "files": {
    "artifact.msgpack": "https://storage.googleapis.com/..."
  },
  "missing_files": [],
  "error": null
}

The signed URLs are short-lived. File bytes are fetched directly from Storage by the client, not streamed through the API pod.