Evaluation

Scores RAG outputs with Ragas, in two modes.

One-shot (batch): upload a JSONL dataset of pre-computed RAG outputs; the service scores every row with the Ragas metrics and the completed job exposes a signed URL to result.json. Not bucket-scoped — the input is a self-contained dataset uploaded with the request, not a bucket of documents.

Tracking (continuous): the customer's app sends OpenTelemetry traces to the ingest endpoint; each completed RAG query is scored with Ragas and surfaced through the dashboard read endpoints. Tenant isolation is enforced server-side by user_uid + track_uid.

Input JSONL

Each non-empty line is one JSON object with the Ragas-canonical columns.

Column	Required	Type
user_input	yes	string
response	yes	string
retrieved_contexts	no	list of strings
reference	no	string

The dataset is parsed synchronously on upload, so a bad dataset fails fast with 422 (empty dataset, malformed JSONL line, missing required column, wrong column type, or more than 4000 rows) instead of surfacing later as a failed job. The upload body is capped at 50 MiB and rejected with 413 before it is stored — the byte cap bounds transport/memory while the row cap bounds judge cost; the two are independent (a small file can carry many tiny rows, and a few rows can carry megabytes of context).

Optional columns are treated as absent when every row has them missing, null, or empty ("" for reference, [] for retrieved_contexts). The metric planner then drops the metrics that depended on them rather than firing with empty input.

Metric planner

Metric selection is automatic from the columns actually detected in the dataset. The job reports which metrics fired (metrics_computed) and which were skipped and why (metrics_skipped).

Each metric, what it measures, score interpretation, and the input columns it consumes at call time:

Metric	What it measures	Score	Inputs used
response_relevancy	How relevant the response is to the question (cosine similarity, via embeddings, between the original user_input and N candidate questions regenerated from the response).	Higher better, [0, 1]	user_input, response
faithfulness	Whether the response stays grounded in the retrieved contexts (no hallucinations beyond what the contexts support).	Higher better, [0, 1]	user_input, response, retrieved_contexts
context_relevance	How relevant the retrieved contexts are to the question.	Higher better, [0, 1]	user_input, retrieved_contexts
answer_correctness	Combined factual (claim matching) + semantic (embedding) accuracy of the response against the reference ground truth.	Higher better, [0, 1]	user_input, response, reference
semantic_similarity	Cosine similarity between response and reference embeddings. Pure embedding metric (no judge LLM call).	Higher better, [0, 1]	response, reference
factual_correctness	F1 of atomic claims matched between response and reference (both decomposed into claims first).	Higher better, [0, 1]	response, reference
noise_sensitivity	How often the response shows errors when given noisy or irrelevant contexts.	Lower better, [0, 1]	user_input, response, reference, retrieved_contexts
context_precision	Fraction of retrieved contexts that are relevant to the reference (precision @ K).	Higher better, [0, 1]	user_input, reference, retrieved_contexts
context_recall	How much of the reference is covered by the retrieved contexts.	Higher better, [0, 1]	user_input, retrieved_contexts, reference

Inverted score: noise_sensitivity is the only metric where a lower value is better — it counts errors caused by noise, so 0 means perfectly resilient. The other 8 metrics are all higher = better.

A metric fires when all the columns in its "Inputs used" list are detected in the dataset. user_input and response are always required; retrieved_contexts and reference are optional (see Input JSONL).

Resulting coverage by dataset shape:

Columns present in the dataset	Metrics computed	Count
user_input + response (minimum)	response_relevancy	1
↑ + retrieved_contexts	+ faithfulness, context_relevance	3
user_input + response + reference	response_relevancy, answer_correctness, semantic_similarity, factual_correctness	4
Full set (all 4 columns)	all 9 metrics	9

An optional column is considered "detected" when at least one row has it populated with a non-empty value (None / "" / [] count as absent). Within a dataset where the column is detected, a row that individually omits it is handled per-cell: each metric is scored only on the rows that carry every input it consumes. A metric whose inputs are missing on a given row records null for that cell without being invoked — no judge call, no error. Genuine per-row failures (provider exceptions, non-numeric or non-finite values) are also recorded as null. Either way null values are excluded from the summary aggregation so empty cells do not bias the mean.

Scoring

Scoring uses an infra-managed judge LLM and embedding model — no user-supplied credentials are needed or accepted. Token usage for the judge and embeddings is recorded per evaluation. Embeddings are used only by the metrics that need them (semantic_similarity, response_relevancy, answer_correctness).

---

Endpoints

These are the one-shot endpoints. The tracking endpoints — track CRUD, the OTLP ingest, and the dashboard read endpoints — are grouped under Tracking below.

POST /api/v1/eval/oneshot/jobs

Multipart upload of one .jsonl file. Validates the filename ends with .jsonl (422 otherwise), stores the dataset, and enqueues the evaluation in one round-trip — returns an eval_uid and status: "pending" for polling.

Request — multipart/form-data with one file field carrying the JSONL bytes (extension must be .jsonl; Content-Type application/x-ndjson recommended but not enforced).

Each line of the file body is one Ragas-canonical row (see Input JSONL):

{"user_input":"...","response":"...","retrieved_contexts":["..."],"reference":"..."}
{"user_input":"...","response":"...","retrieved_contexts":["..."]}

Response — 200 (OneshotJobCreateResponse)

{
  "eval_uid": "550e8400-e29b-41d4-a716-446655440000",
  "status":   "pending",
  "message":  "One-shot evaluation enqueued"
}

---

GET /api/v1/eval/oneshot/jobs/{eval_uid}

Polls the job status (single object, not an array). 404 if eval_uid is not a valid UUID, the record does not exist, or it belongs to a different user.

Response — 200 (OneshotJobStatusResponse)

{
  "eval_uid":         "550e8400-e29b-41d4-a716-446655440000",
  "status":           "completed",
  "rows_evaluated":   24,
  "columns_detected": ["user_input", "response", "retrieved_contexts", "reference"],
  "metrics_computed": ["faithfulness", "response_relevancy", "context_relevance", "noise_sensitivity", "answer_correctness", "semantic_similarity", "factual_correctness", "context_precision", "context_recall"],
  "metrics_skipped":  {},
  "summary": {
    "faithfulness": { "mean": 0.92, "p10": 0.80, "p50": 0.94, "p90": 0.99, "count": 24 }
  },
  "result_path":  "user-42/evaluations/550e8400-.../result.json",
  "error":        null,
  "started_at":   "2026-05-28T10:00:00Z",
  "completed_at": "2026-05-28T10:01:13Z"
}

rows_evaluated is all-or-nothing: it equals the total row count when status == "completed", and is null for every other state (pending, processing, failed). The other completion-only fields (summary, metrics_computed, result_path, completed_at, ...) are null until the job reaches completed.

Status values: pending → processing → completed / failed.

---

POST /api/v1/eval/oneshot/results

Returns a short-lived signed URL for result.json. The client fetches the file via HTTP GET before expires_at. No file bytes transit through this API.

When the job exists and is owned by the user but result.json is not yet available (job still pending / processing / failed without output), the endpoint returns 200 with success: false and missing_files: ["result.json"] — this is a legitimate state, not an error.

Request

{ "eval_uid": "550e8400-e29b-41d4-a716-446655440000" }

Response — 200 (OneshotResultsResponse, wraps the ResultsData shape with eval_uid instead of bucket_uid)

{
  "success": true,
  "message": "Results for eval 550e8400-...",
  "result": {
    "eval_uid":   "550e8400-e29b-41d4-a716-446655440000",
    "success":    true,
    "expires_at": "2026-05-28T11:01:13Z",
    "files": {
      "result.json": "https://storage.googleapis.com/...?signature=..."
    },
    "missing_files": [],
    "error": null
  }
}

If the result is not yet available:

{
  "success": false,
  "message": "Result not yet available",
  "result": {
    "eval_uid":      "550e8400-...",
    "success":       false,
    "expires_at":    null,
    "files":         {},
    "missing_files": ["result.json"],
    "error":         "result not yet available"
  }
}

Signed URLs are valid for 1 hour. After expiry, call the endpoint again to obtain a fresh URL.

---

Tracking

A tracking project (track) is a long-lived container for continuous evaluation of a production RAG. Tracks are managed through three CRUD endpoints under /api/v1/eval/tracks, all Bearer-authenticated and scoped to the caller's user_uid. Tenant isolation is by user_uid + track_uid, stamped server-side onto every span at ingest and applied as a filter on read.

Method	Path	Purpose
POST	/api/v1/eval/tracks	Create a track. Body {name} → TrackResponse. Generates track_uid.
GET	/api/v1/eval/tracks	List all the caller's tracks (active + disabled); each row carries the active flag.
PATCH	/api/v1/eval/tracks/{track_uid}	Enable/disable a track. Body {active}. While disabled, ingest refuses its traces and the evaluator skips it; the historical records stay readable. Toggle-able any number of times.

Ownership and isolation: every per-track endpoint validates track_uid as a UUID and confirms the record belongs to the caller. A missing, malformed, or foreign track_uid all return 404. (Ownership is checked independently of active, so a disabled track can still be read and re-enabled.)

Ingest — POST /api/v1/eval/ingest/v1/traces

OTLP/HTTP receiver for production traces. Point your OpenTelemetry exporter here (not at a tracing backend directly). Request:

• authorization: Bearer <user_api_key> → resolves the user.
• neurolinker-track-uid: <track_uid> → the track the traces belong to. Validated for ownership; missing header → 400, malformed/foreign → 404, disabled track → 404.
• Body: OTLP/HTTP protobuf (ExportTraceServiceRequest).

For each completed trace (a root span present in the payload) a per-trace evaluation is scheduled automatically. Traces are scored shortly after they land — a short grace delay lets any straggler spans arrive first, then scoring runs — and appear in the dashboard read endpoints once scored. The SDK's instrument() helper wires all of this up for you, so most integrations never call this endpoint directly.

Dashboard — read endpoints

Read-only endpoints that surface a track's evaluated traffic. All are Bearer-authenticated and validate track ownership upfront (404 on missing/foreign; a disabled track is still readable).

Method	Path	Returns
GET	/api/v1/eval/tracks/{track_uid}/queries?limit=	One row per RAG query, most recent first: trace_id, user_input, response, started_at, latency_ms, status, metrics. limit is 1–500 (default 100); out-of-range values → 422.
GET	/api/v1/eval/tracks/{track_uid}/queries/{trace_id}	Drill-down: the curated record — user_input, response, retrieved_contexts, started_at, latency_ms, status, model, tokens, metrics. 404 if not found / not yet evaluated.

Two layers of signal per query: quality (Ragas metric scores) and operational (latency, status, model, tokens). latency_ms and status are always present. Token usage and model are best-effort — present only when the customer's LLM instrumentation captured them; absent fields are null. A query appears once the evaluator has scored it, shortly after it lands.

---

One-shot — result.json structure

Plain JSON (intended to be human-inspectable). A non-finite metric value is encoded as an explicit null instead of a JSON-incompatible NaN / Infinity.

{
  "eval_uid": "550e8400-e29b-41d4-a716-446655440000",
  "rows": [
    {
      "row_id":     0,
      "user_input": "What is the capital of France?",
      "response":   "The capital of France is Paris.",
      "metrics": {
        "faithfulness":         0.92,
        "response_relevancy":   0.88,
        "context_relevance":    0.91,
        "noise_sensitivity":    0.05,
        "answer_correctness":   0.94,
        "semantic_similarity":  0.97,
        "factual_correctness":  0.90,
        "context_precision":    0.85,
        "context_recall":       0.92
      }
    }
    // ... one entry per input row
  ],
  "summary": {
    "faithfulness": { "mean": 0.92, "p10": 0.80, "p50": 0.94, "p90": 0.99, "count": 24 }
    // ... one entry per metric in metrics_computed
  }
}

• rows[] preserves the input order and carries the per-row metric scores keyed by metric name.
• A failed metric on a single row is null for that cell — other metrics on the same row still score normally.
• summary keys match the metrics_computed set; null cells are excluded from each metric's aggregation so failed cells do not bias the mean.