Skip to main content

​
Chat / DeepSeek — baseline, diagnosis and change ledger

Base SHA: e971558961047b13daa43f7a0b7cac30d74b2876 (origin/staging HEAD at the time of the audit; git rev-list --left-right --count HEAD...origin/staging = 0 0). This document records what was actually present before any edit, so a fix is never applied to a defect that was already closed. Every claim below is anchored to a file and a line, not to the historical CSV.

​
0. What the historical CSV actually proves

The audit CSV (100 messages / 30 threads / 50 assistant answers, 2026-08-09 → 2026-08-15) mixes three model identities and at least three prompt/belt generations: run_status: 47 completed, 3 failed. The single most important observation: tool_names is [] and tool_call_count/tool_result_count/tool_error_count are 0 on all 50 assistant rows — while citation_count is non-zero on 27 of them (up to 44), with citation_source_kinds of gtm_artifact, web, topic, collection, project, record. Citations are minted mechanically from tool results only (chat-tools.ts:781-787 → extractCitations / webCitations / webSearchCitations; chat.service.ts:911). A row with 44 gtm_artifact citations and 0 tool calls is arithmetically impossible if both columns were read from the same run. Therefore tools=[] is a defect of the ad-hoc export query, not evidence that tools failed to persist. Corroborating: feedback_vote is null in every row, which is consistent with a real, un-rated dataset rather than a bug.

​
1. Export diagnosis (§9) — root cause

The CSV column set (thread_ref, message_no, seconds_into_thread, citation_source_kinds, run_status, duration_ms, tokens_in, tokens_out, tool_names, tool_call_count, tool_result_count, tool_error_count) is produced by no committed artifact in this repository. The only committed chat export is scripts/harness/export-chat-feedback.mjs, which selects a completely different projection (thread_id, message_id, answer, reason, question, model, cited_refs) and filters to feedback->>'vote' = 'down'. It emits none of the tool columns. So the CSV came from an uncommitted Supabase SQL snippet (“Supabase Snippet Untitled query”), which is unreviewable and unreproducible. Persistence itself is sound at base SHA:
  • chat.service.ts:934 writes trace: runTrace on the completed path, and :897 writes the partial trace on the failed/stopped path.
  • mastra-runtime.ts:257-276 emits tool_call and tool_result trace events with detail.name.
  • chat.service.ts:612 persists run_id on every executed assistant message.
agent_runs.trace is select: false (agent-run.entity.ts:84), which affects TypeORM reads only — raw SQL is unaffected. That is a plausible trap for a hand-written ORM-based export, but not for the SQL snippet. Conclusion: the fix is to replace the unreviewable query with a committed, unit-tested projection that can never collapse “trace absent” into “zero tools”.

​
2. Defect-by-defect baseline

​
3. Observability baseline (§7)

Working at base SHA:
  • initTelemetry('driftless-api') is called — apps/api/src/main.ts:46.
  • Latitude is configured on both Render services — render.yaml:56-58 and :118-120 (LATITUDE_API_KEY, LATITUDE_PROJECT_SLUG, sync:false).
  • Exactly one agent.run root span per executed turn — chat.service.ts:954-984, pinned by chat-observability.spec.ts:157-161.
  • Opt-in and non-blocking: without the key the global tracer stays the OTEL no-op; export is batched fire-and-forget (libs/telemetry/src/index.ts:40-59).
Missing, against the §7 target list: The regression is documented by the Driftless topic agent-observability-latitude (reviewed/authoritative), which describes the span tree as agent.run → chat {model} → tool {name}. Those two child spans were emitted by apps/api/src/agent-runs/agentic-loop.ts, which no longer exists — it was deleted when Mastra became the runtime. MastraRuntime opens no spans at all, so the chat trace is now a single flat root. The correct seam for the tool span is cognitive/tool-observability.ts: it already wraps every chat tool call with duration + error classification and emits the sanitized ToolExecutionEvent. Adding the child span there reuses the existing event (no second bus, §7), nests automatically under agent.run via OTEL active context, and keeps chat/ at exactly one withSpan so the existing candado stays green.

​
4. TTFT / duration baseline (§8)

  • agent_runs.duration_ms — computed and stored (agent-runs.service.ts:172).
  • model_usage.ttft_ms / duration_ms — columns exist and are documented as “NULL = not measured, which is NOT 0” (model-usage.entity.ts:110-116).
  • ModelUsageService.recordTurn accepts ttftMs / durationMs (model-usage.service.ts:36-39, written at :95-96).
  • ModelUsageService.recordSession never passes them (:109-130), because TurnUsageEvent (managed-session.ts:39-45) has no timing field and GatewayModelContext.onTurnUsage (mastra-model.adapter.ts:70) does not report one.
So both columns are NULL for every row — by omission, not by a wrong value. A real measurement point exists: DriftlessGatewayModel.doStream receives an onContent(delta) callback from the gateway (mastra-model.adapter.ts:342-346). The first invocation of that callback is a genuine observation of the provider’s first content token, and the gateway.route(...) call is an exact per-attempt duration boundary. Semantics that must be documented rather than faked:
  • doGenerate (non-streaming) has no first-token observation → ttft_ms stays NULL; duration_ms is still exact.
  • A tool-call turn may emit no text at all (the provider returns only tool calls). There is no first content token, so ttft_ms stays NULL for that turn. TTFT is time to first content delta, never “time to the final answer” and never derived from total duration.

​
5. Live baseline (§4) — NOT EXECUTED

https://api-staging.driftless.icu/health responds 200 (after a ~51 s cold start), so the deployment is reachable from this environment. No DRIFTLESS_API_KEY is present in the environment, and §4 forbids hardcoding or inventing credentials. The 20-case live DeepSeek baseline was therefore not executed, and no live before/after table is claimed. Consequences, stated plainly:
  • Defects D and E are recorded above as fixed from code reading only. They are protected by new evals rather than re-fixed, per §11.
  • Defect A’s model half (does DeepSeek still emit English in a Spanish thread?) is unverified. Per §5.7 the language repair/validator is therefore deliberately not built — the instruction is to add it only if the current DeepSeek still produces English after prompt+copy are fixed, and that cannot be established without a live run.
  • The runner is delivered and bounded so the team can produce the evidence: pnpm --filter @driftless/api harness:chat-live-evals with DRIFTLESS_API_URL / DRIFTLESS_API_KEY from the environment. It reports SKIP (never PASS) when credentials are absent.

​
6. What changed, and why

​
7. What was deliberately NOT built

  • No language validator/repair. §5.7 permits one only if the CURRENT DeepSeek still emits English after prompt + copy are fixed. That cannot be established without the live baseline, and §10 forbids adding repair pre-emptively.
  • No regex post-processor over the answer. §5.3 is explicit: fix the prompt and the descriptions, never blind-replace the final text.
  • No budget increase. §5.5 says hold the current budgets until evidence says otherwise.
  • No automatic market→web fallback. §10 requires evidence first.
  • No new router, manager, second agent, event store, eval table or observability vendor.
  • No UI change. The dashboard already localizes activity labels and hides internal tool names; nothing in the baseline showed a live UI defect.