Vector Store

Loads vectors generated by the Embedding module into a vector DB. Supported providers: Milvus/Zilliz, Qdrant, Pinecone — auto-detected from the URI domain. The user never specifies the provider — only the URI changes.

Provider routing

The vector_db_config.uri field determines the provider server-side:

URI Domain	Provider
`.zilliz.com` or `.zillizcloud.com`	`MilvusVectorDB`
`*.qdrant.io`	`QdrantVectorDB`
`*.pinecone.io`	`PineconeVectorDB`

The user passes the credential as vector_db_config.api_key. It is stored encrypted server-side and resolved only when the job initialises the vector DB client — the cleartext key is never exposed to the job broker.

Differences between providers

	Milvus	Qdrant	Pinecone
Schema	Strict (all fields declared)	Vectors only, schemaless payload	dim + metric only, schemaless metadata
Multi-vector	Yes (named fields)	Yes (named vectors)	No (1 dense + 1 sparse)
Primary key / ID	PK in schema (default `item_id` VARCHAR)	Deterministic UUID5 from `item_id`	String from `item_id` (max 512 chars)
`database` param	`db_name` (namespace)	Must be empty	`namespace`
Distance metric	Ignored (COSINE on index)	Per-vector	Per-index
Batch size (upsert)	400	400	200

Idempotency

All providers use upsert (not insert). Same item_id → same record in the DB, overwrite without duplicates. Relaunching an interrupted job is safe: already-inserted records are overwritten, not duplicated.

Endpoints

POST `/api/v1/vector-store/collections` (synchronous)

Creates the collection in the vector DB. Synchronous operation — it runs inline and returns the result directly, with no background job to poll. The body has a nested structure: collection (with name, fields[], optional description, optional options), vector_db_config (uri required; api_key optional, default null; timeout optional, default 300), and an optional top-level database (default "").

The fields use abstract dtypes (text, int, float, bool, json, dense_vector, sparse_vector) — the provider translates internally to the native type. distance is valid only for dense_vector; omit it for scalar and sparse fields.

Provider-specific options live in two non-overlapping buckets, both with strict validation — unknown keys raise 400 ValueError:

Bucket	Where	Provider whitelist
Field-level (`FieldSchema.options`)	Per-field knobs on the schema.	Milvus/Zilliz: `description`, `max_length`, `auto_id`, `enable_analyzer`, `enable_match`. Pinecone: none. Qdrant: none.
Collection-level (`collection.options`)	Index/collection provisioning knobs.	Pinecone: `cloud`, `region` (ServerlessSpec; default `aws` / `us-east-1`). Milvus/Zilliz: none. Qdrant: none.

Cross-provider behavior differences

The typed FieldSchema contract is the same across providers, but a few fields have provider-specific semantics worth knowing before mixing or migrating between backends.

Field	Pinecone	Milvus/Zilliz	Qdrant
`distance` (`cosine` / `dot` / `euclidean`)	Honored at index creation (`cosine` / `dotproduct` / `euclidean`).	Honored at index creation via `AUTOINDEX` (`COSINE` / `IP` / `L2`). Sparse fields are server-restricted to `IP`.	Honored (`Cosine` / `Dot` / `Euclid`).
`is_primary`	No effect — vector IDs are top-level free-form strings derived from the `item_id` field at insert time. Setting `is_primary=True` logs a warning.	Required and honored — at most one field can be primary; if none is declared, an `item_id` VARCHAR primary key is auto-injected for idempotent upserts.	No effect — point IDs are top-level (int or UUID) derived from the `item_id` field at insert time. Setting `is_primary=True` logs a warning.
Scalar `dtype` (`text`/`int`/`float`/`bool`/`json`)	Stored as metadata. Pinecone metadata is schemaless, so the declared dtype is informational only.	Mapped to a real typed schema column (`VARCHAR`, `INT64`, `FLOAT`, `BOOL`, `JSON`) and enforced at insert.	Stored as payload. Qdrant payload is schemaless until you add a payload index, so the declared dtype is informational only.

Request

{
  "collection": {
    "name": "neurolinker_docs",
    "description": "Neurolinker document embeddings",
    "fields": [
      { "name": "item_id",     "dtype": "text",         "is_primary": true },
      { "name": "content",     "dtype": "text",         "options": { "enable_analyzer": true, "enable_match": true } },
      { "name": "source_file", "dtype": "text" },
      { "name": "text_dense",  "dtype": "dense_vector", "dim": 1024, "distance": "cosine" },
      { "name": "text_sparse", "dtype": "sparse_vector" },
      { "name": "image_dense", "dtype": "dense_vector", "dim": 1024, "distance": "cosine" }
    ]
  },
  "vector_db_config": {
    "uri": "https://YOUR-DB-ENDPOINT",
    "api_key": "YOUR-API-KEY"
  }
}

For a Pinecone serverless index, pass cloud / region via collection.options:

{
  "collection": {
    "name": "neurolinker-docs",
    "options": { "cloud": "aws", "region": "eu-central-1" },
    "fields": [
      { "name": "chunk_id",   "dtype": "text",         "is_primary": true },
      { "name": "text_dense", "dtype": "dense_vector", "dim": 1024, "distance": "cosine" }
    ]
  },
  "vector_db_config": {
    "uri": "https://api.pinecone.io",
    "api_key": "YOUR-API-KEY"
  }
}

Response — 200 (CollectionCreationResult)

{
  "success": true,
  "message": "Collection created",
  "collection": "neurolinker_docs",
  "fields_count": 6,
  "already_existed": false,
  "error": null
}

already_existed: true when the collection already existed (idempotent — no error). Error mapping: 503 on a transient vector-DB error, 400 on a ValueError (bad config), 500 otherwise.

POST `/api/v1/vector-store/jobs` (asynchronous)

Loads the embeddings of a bucket into an existing collection. Bucket ownership is validated (404 if not found or not owned); the call returns immediately with a job_uid and status: "pending" for polling.

`source` field naming convention

The transformer builds a flat record with three non-overlapping namespaces:

Namespace	Origin	Examples
`item_*`	Fields of the atomic embedding item	`item_id`, `item_element_type`, `item_content`, `item_token_count`
`chunk_*`	Fields of the parent chunk; `metadata` is flattened	`chunk_id`, `chunk_content`, `chunk_tokens`, `chunk_header_path`
`<vector_name>`	Vectors — free name chosen by the user in the Embedding request	`text_dense_bge`, `text_sparse_splade`

The item_ and chunk_ prefixes are reserved: a vector_name cannot start with them (validated at the Embedding API boundary).

chunk_metadata does not exist: its keys are emitted separately as chunk_chunk_index, chunk_tokens, chunk_header_path, chunk_pages, chunk_has_table, chunk_has_figure, chunk_small_chunk_merged, and (only after a merge) chunk_merged_chunks. Likewise, chunk_request_uid is not emitted by the current chunking pipeline.

The available fields depend on the element type: text items produce item_content and item_token_count; image items can produce item_description, item_url, item_extracted_text, and item_legend; table items can produce item_description, item_data, and item_legend. The current chunker does not populate image base64 data, so it is not offered as a mapping source. Markdown Field Extraction scalars are dynamic top-level chunk fields, mapped as chunk_<schema_field>.

The Vector DB UI exposes user-facing mapping sources: content (chunk_content), citation metadata (chunk_source_file, chunk_pages, chunk_header_path), exact text-item content (item_content), and vectors produced by the completed embedding job (for example, text_dense_bge). Image/table fields (item_element_type, item_description, item_url, item_extracted_text, item_data) appear only when the completed job produced their modality. A new collection starts only with the automatically configured item_id; vector fields are added after Embedding and all other fields are opt-in. A newly added field is blank until the user selects its source, target, and—for dense vectors—metric. Internal diagnostics, merge bookkeeping, and dynamic Markdown Field Extraction scalars remain backend/API details and are intentionally not selectable in the UI.

Request

{
  "bucket_uid": "bkt_uuid",
  "collection_name": "neurolinker_docs",
  "field_mappings": [
    { "name": "item_id",     "source": "item_id"            },
    { "name": "content",     "source": "chunk_content"       },
    { "name": "source_file", "source": "chunk_source_file"   },
    { "name": "token_count", "source": "chunk_tokens"        },
    { "name": "text_dense",  "source": "text_dense_bge"     },
    { "name": "text_sparse", "source": "text_sparse_splade" },
    { "name": "image_dense", "source": "image_dense_voyage" }
  ],
  "vector_db_config": {
    "uri": "https://YOUR-DB-ENDPOINT",
    "api_key": "YOUR-API-KEY"
  },
  "database": ""
}

The body requires:

bucket_uid — the bucket to load (single, not an array).
collection_name — the target collection (must already exist, created via POST /api/v1/vector-store/collections).
field_mappings[] — each entry has only name (field in the target collection) and source (key in the flat record). For vectors, source must be exactly the vector_name used in the Embedding request.
vector_db_config — uri (required), api_key (optional, default null), timeout (optional, default 300). The api_key is stored encrypted server-side, never exposed in cleartext to the job broker.
database — optional, default "". Milvus uses it as db_name, Pinecone as namespace, Qdrant requires it to stay empty.

Embedding → Vector Store coupling: the vector_name set in the Embedding request (e.g. "text_dense_bge") becomes exactly the source to use in field_mappings. If vector names change in Embedding, the field_mappings must be updated accordingly.

Response — 200 (job started)

{
  "success":    true,
  "job_uid":     "550e8400-e29b-41d4-a716-446655440000",
  "status":     "pending",
  "message":    "Load job started",
  "bucket_uid": "bkt_uuid"
}

GET `/api/v1/vector-store/jobs/{bucket_uid}/{job_uid}`

Polls the job status. Verifies ownership via user_uid (bucket ownership check).

Response (completed)

{
  "job_uid":          "550e8400-e29b-41d4-a716-446655440000",
  "status":          "completed",
  "bucket_uid":      "bkt_uuid",
  "collection_name": "neurolinker_docs",
  "message":         "Load completed successfully",
  "total_records":   62,
  "error":           null,
  "started_at":      "2024-01-01T10:00:00Z",
  "completed_at":    "2024-01-01T10:00:08Z"
}

Status values: pending → processing → completed / failed.

On failed, error is an object { kind, message, retry_safe } — kind is one of transient, partial, or permanent; it is null otherwise. Transient errors are retried automatically; a job is marked failed only once retries are exhausted or the error is non-retryable (retry_safe: false).

No /results endpoint — unlike Chunking and Embedding, the Vector Store does not produce downloadable files. Its output is the indexed vectors inside the configured backend (Milvus / Qdrant / Pinecone), queried via the backend's own API (e.g. Milvus' search, Qdrant's /points/search). The pipeline ends at POST /api/v1/vector-store/jobs + the polling loop above.