Chunking
Splits the markdown produced by Extraction into semantic chunks optimised for retrieval. Always operates per bucket: takes a bucket_uid, reads the bucket sources, downloads the markdown plus the per-document extraction JSON for each one, and produces a single aggregated chunking.msgpack for the entire bucket. Picture/Table chunks are enriched with description, extracted_text, and legend from the extraction JSON, matched by id against the markdown — chunking itself performs no LLM calls.
If you ran Field extraction from markdown on a document (that pipeline runs per document, not per bucket), its extracted scalar fields are stamped as top-level keys on every chunk of that document when you chunk a bucket that includes it. The scalars then travel with the chunk into embedding and the vector store, where they become filterable chunk_* fields. The reserved keys id, source_file, content, metadata, and embedding_chunks are never overwritten.
Available algorithms
Three algorithms selectable via chunking.method (discriminated union):
| Method | Description |
|---|---|
section_greedy |
Structure-aware; groups sections respecting the t_min / t_max token budget. Uses HuggingFace Alibaba-NLP/gte-large-en-v1.5 for tokenisation. |
md_header_level |
Chunks at markdown header boundaries, configurable by level (chunk_at_level). |
block_window |
Sliding window over blocks with configurable overlap. overlap_mode: "within_budget" reuses the last blocks within the t_max budget; "extra_budget" prepends them on top of the budget. |
model_name (used by section_greedy) is the Hugging Face Hub repository id (org/model) of the tokenizer used to measure the t_min / t_max token budget. Override it to match the tokenizer of the embedding model you plan to run downstream.
Endpoints
POST /api/v1/chunk/jobs
Submits a chunking job. The client passes a single bucket_uid plus the algorithm configuration. request_uids is not accepted — the bucket is the only entry point. Bucket ownership is validated (404 if not found or not owned); the call returns immediately with a job_uid and status: "pending" for polling.
Request (section_greedy)
{
"bucket_uid": "bkt_uuid",
"chunking": {
"method": "section_greedy",
"t_min": 600,
"t_max": 1000,
"model_name": "Alibaba-NLP/gte-large-en-v1.5",
"parse_figures": true,
"parse_tables": true,
"parse_headers": true,
"parse_footers": false
}
}
Request (md_header_level)
Request (block_window)
{
"bucket_uid": "bkt_uuid",
"chunking": {
"method": "block_window",
"t_max": 1000,
"overlap_blocks": 1,
"overlap_mode": "within_budget"
}
}
overlap_mode: "within_budget" (default) | "extra_budget".
Response — 200
{
"success": true,
"job_uid": "550e8400-e29b-41d4-a716-446655440000",
"status": "pending",
"message": "Chunking job started",
"bucket_uid": "bkt_uuid"
}
The same bucket_uid may have multiple job_uid entries historically (e.g. re-runs with different parameters).
GET /api/v1/chunk/jobs/{bucket_uid}/{job_uid}
Polls the job status (single object, not an array). Verifies ownership via user_uid (bucket ownership check).
Response
{
"job_uid": "550e8400-e29b-41d4-a716-446655440000",
"status": "completed",
"bucket_uid": "bkt_uuid",
"message": "Chunking completed successfully",
"summary": {
"total_context_chunks": 38,
"total_tokens": 14200,
"avg_tokens": 374.0,
"total_documents": 1,
"total_pages": 12
},
"firebase_path": "uploads/{user_uid}/bkt_uuid/chunking.msgpack",
"error": null,
"started_at": "2024-01-01T10:01:00Z",
"completed_at": "2024-01-01T10:01:08Z"
}
Status values: pending → processing → completed / failed.
POST /api/v1/chunk/analyze
Computes token-distribution statistics for an existing chunking.msgpack, writes chunking_statistics.json + chunking_distribution.png under {user_uid}/{bucket_uid}/, and returns short-lived signed URLs to both artefacts. The client fetches each URL via HTTP GET before expires_at. No file bytes transit through this API.
Request
Response — wraps the shared ResultsData shape.
{
"success": true,
"message": "Analysis completed for bucket bkt_uuid",
"result": {
"bucket_uid": "bkt_uuid",
"success": true,
"expires_at": "2026-04-29T12:00:00Z",
"files": {
"chunking_statistics.json": "https://storage.googleapis.com/...?signature=...",
"chunking_distribution.png": "https://storage.googleapis.com/...?signature=..."
},
"missing_files": [],
"error": null
}
}
If chunking.msgpack is missing or no chunks are present, the response has success: false, files: {}, and error populated.
POST /api/v1/chunk/results
Returns a short-lived signed URL for the chunking job's main output (chunking.msgpack). The client fetches the URL via HTTP GET before expires_at. No file bytes transit through this API. Statistics and the distribution plot are produced by /analyze (see above) and not included here — call /analyze to obtain them.
Request
Response — wraps the shared ResultsData shape.
{
"success": true,
"message": "Results for bucket bkt_uuid",
"result": {
"bucket_uid": "bkt_uuid",
"success": true,
"expires_at": "2026-04-29T12:00:00Z",
"files": {
"chunking.msgpack": "https://storage.googleapis.com/...?signature=..."
},
"missing_files": [],
"error": null
}
}
If the file is missing the response has success: false, files: {}, missing_files: ["chunking.msgpack"], and error populated.
chunking.msgpack structure
The file is msgpack-serialised (binary). Once decoded:
{
"metadata": {
"total_files": 1,
"total_context_chunks": 38,
"files_summary": [
{
"file_name": "report.md",
"context_chunks": 38,
"text_chunks": 30,
"image_chunks": 5,
"table_chunks": 3,
"total_tokens": 14200,
"avg_tokens": 374.0
}
]
},
"chunks": [
{
"id": "chunk_uuid",
"source_file": "report.md",
"content": "...",
"metadata": {
"chunk_index": 0,
"tokens": 312,
"header_path": ["Introduction", "Background"],
"pages": [1, 2],
"has_table": false,
"has_figure": true,
"small_chunk_merged": false
// "merged_chunks": [...] ← added only when small_chunk_merged == true
},
"embedding_chunks": {
"text": {
"id": "text_chunk_uuid",
"content": "...",
"token_count": 312
},
"image": [
{
"id": "img_chunk_uuid",
"description": "...",
"url": "https://firebase-url...",
"image_base64": "...",
"extracted_text": "...",
"legend": { "PE": "Polyethylene", "Re": "Reynolds number" }
}
],
"table": [
{
"id": "table_chunk_uuid",
"description": "...",
"data": "| col1 | col2 |\n|---|---|\n| ... | ... |",
"legend": {}
}
]
}
}
]
}
embedding_chunkscontains the specialised sub-chunks by type (text,image,table) — these are what the Embedding module processes to generate vectors.merged_chunksis present only whensmall_chunk_merged == trueand contains the original chunks that were merged.